Point cloud quality calculation method based on point structured information network

Abstract

The present invention discloses a point cloud quality calculation method based on a point structured information network. The position vector features, distance features, brightness features and brightness difference features of a point cloud block are jointly input into a point structured information network model to extract the structured information features of the point cloud block; the point structured information features are input into a distortion perception flow network to obtain distortion classification features; the structured information features are input into a basic quality perception flow network to obtain basic quality features of the point cloud block. The basic quality features of the point cloud block are fused with the distortion classification features and input into two layers of the third fully connected layer to obtain a predicted quality score; the predicted quality scores of multiple point cloud blocks belonging to the same overall point cloud are averaged to obtain the final score of the overall point cloud. The present invention takes into account the influence of point cloud brightness, distance, relative position and other information on point cloud quality, and introduces brightness difference features and structured information features to comprehensively evaluate the subjective quality of the point cloud.

Description

Technical Field

[0001] The present invention relates to a point cloud quality calculation method based on a point structured information network, and belongs to the technical field of 3D point cloud reference-free quality evaluation. Background Art

[0002] A point cloud is defined as a set of three-dimensional points, where each point is represented by a three-dimensional coordinate and specific attributes (such as color). With the advancement of 3D information capture technology, point clouds have found widespread application in applications such as virtual reality, immersive telepresence, mobile mapping, and 3D information printing. A typical use of point clouds is to represent human holographic images in virtual reality and immersive telepresence. However, to realistically present visual information, a model may consist of millions or even hundreds of millions of points. During transmission, lossy compression schemes are often used. Compared to lossless compression schemes, lossy compression schemes can significantly save transmission resources and increase transmission speeds, but the negative impact is the generation of compressed perception distortion. Furthermore, interference may occur during the acquisition and transmission process, resulting in downsampling perceptual distortion and Gaussian filtering distortion, which can lead to a decrease in human perception. To better manage and control the subjective quality of point clouds, it is crucial to develop a high-performance color point cloud quality assessment that conforms to human perception.

[0003] To quantify this visual perception mechanism, research often focuses on subjective and objective quality assessment. Subjective quality assessment relies on human subjective scoring to provide realistic visual perception scores for varying degrees of distortion. While accurate, these methods are time- and labor-intensive. Objective quality metrics, on the other hand, employ models to assess the visual quality of point clouds. Existing objective quality assessment models can be broadly categorized into three types: full-reference point cloud quality assessment, semi-reference point cloud quality assessment, and no-reference point cloud quality assessment. However, due to the difficulty in obtaining raw point cloud information in most scenarios and the large amount of data required for storage and transmission, full-reference point cloud quality assessment approaches are difficult to implement in real-world scenarios. Therefore, no-reference point cloud quality assessment approaches have gradually become a research focus. Existing no-reference point cloud quality assessment approaches can be primarily categorized into two approaches: handcrafted feature-based metrics and deep learning-based metrics. Handcrafted feature-based methods include projecting 3D point clouds into geometric and color feature domains, extracting quality-perception features using 3D natural scene statistics and entropy, and regressing these features into visual quality scores using support vector machines (3D-NSS). However, handcrafted features are highly dependent on domain knowledge, often limiting the performance of feature representation. Deep learning-based approaches include: obtaining three video sequences by rotating a camera around a point cloud along three specific trajectories, using ResNet3D as a feature extraction model to learn the correlation between captured videos and corresponding subjective quality scores (VS-ResNET); extracting hierarchical features from 3D point clouds, considering both geometric and texture information, using a sparse tensor representation, and feeding the tensor into a CNN to predict quality scores (ResSCNN); inferring point cloud quality using unsupervised adversarial adaptation by treating natural images as the source domain and point clouds as the target domain (IT-PCQA); and predicting the final score through a multi-view joint feature extraction and fusion module, a distortion type identification module, and a quality vector prediction module (PQA-NET). However, the unstructured nature of point clouds in these deep learning-based approaches hinders the direct application of convolutional operations, and therefore projects the 3D point clouds into images or videos. These methods only consider color information and ignore geometric information, which is highly relevant to human perception. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a point cloud quality calculation method based on a point structured information network.

[0005] To achieve the above objectives, the present invention provides a point cloud quality calculation method based on a point structured information network, comprising:

[0006] The position vector features, distance features, brightness features and brightness difference features of the point cloud blocks obtained in advance are jointly input into the point structured information network model to extract the structured information features of the point cloud blocks;

[0007] Input the point structured information features into the distortion-aware flow network that has been iteratively trained to obtain distortion classification features;

[0008] The structured information features are input into the basic quality-aware flow network to obtain the basic quality features of the point cloud blocks.

[0009] Prioritize, the basic quality features of the point cloud block are fused with the distortion classification features and input into two third fully connected layers to obtain the predicted quality score;

[0010] The predicted quality scores of multiple point cloud blocks belonging to the same overall point cloud are averaged to obtain the final score of the overall point cloud.

[0011] Prioritize the pre-acquired position vector features, distance features, brightness features, and brightness difference features, and jointly input them into the point structured information network model to extract the structured information features of the point cloud blocks. This is achieved by the following steps:

[0012] The point structured information network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are connected in sequence;

[0013] The position vector features are input into the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer for processing to obtain structured feature weights;

[0014] The position vector feature, distance feature, brightness feature and brightness difference feature are weighted with the structured feature weight to obtain the structured information feature of the point cloud block.

[0015] Prioritize the point structured information features and input them into the iteratively trained distortion-aware flow network to obtain the distortion classification features. This is achieved by the following steps:

[0016] The distortion-aware flow network before iterative training includes the fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer. The fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer are connected in sequence;

[0017] Freeze the entire distortion-aware stream network after iterative training and remove the linear regression layer;

[0018] The point structured information features are input into the distortion-aware flow network that has been iteratively trained without the linear regression layer to obtain the distortion classification features.

[0019] Preferably, the basic quality-aware flow network includes multiple layers of a seventh convolutional layer, a second maximum pooling layer, a second global average pooling layer, and a second fully connected layer, and the multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer are connected in sequence.

[0020] Preferably, the position vector feature, the distance feature, the brightness feature and the brightness difference feature are obtained in advance, which is achieved by the following steps: sampling the original point cloud according to the principle of the FPS farthest point sampling algorithm to obtain sampling points;

[0021] Using the KNN nearest neighbor algorithm, the 1024 closest distance points of each sampling point are selected to form a point cloud block;

[0022] Calculate the position vector features and distance features of each point cloud block;

[0023] The brightness features and brightness difference features of each point cloud block are calculated.

[0024] First, the position vector features and distance features of each point cloud block are calculated, which is achieved by the following steps:

[0025] Calculate the position vector feature {Δx j ,Δy j ,Δz j}:

[0026] {Δx j ,Δy j ,Δz j}={x j -x0,y j -y0,z j -z0},

[0027] Where p j ={x j ,y j ,z j} represents the three-dimensional coordinates of each sampling point, j = 1, 2, ..., K, p0 = {x0, y0, z0}, p0 is the three-dimensional coordinates of the centroid;

[0028] Calculate the distance feature of each sampling point in the point cloud block:

[0029]

[0030] First, the brightness feature and brightness difference feature of the point cloud block are calculated, which is achieved by the following steps:

[0031] Calculate the brightness feature l of each sampling point in the point cloud block j :

[0032] l j =r j ×0.229+g j ×0.587+b j ×0.114,

[0033] Where c j ={r j ,g j ,b j} indicates that the three-dimensional coordinate is p j The color of the sampling point, r j Indicates the RGB value of red, g j Indicates the RGB value of green, b j Represents the RGB value of blue;

[0034] Calculate the brightness difference feature Δl of each sampling point in the point cloud block j :

[0035] Δl j =l0-l j , where l0 represents the brightness value of the centroid point.

[0036] The beneficial effects achieved by the present invention are:

[0037] The present invention uses the KNN nearest neighbor algorithm to obtain the K nearest neighbor points of each sampling point to form a point cloud block, and calculates four features: position vector feature, distance feature, brightness feature and brightness difference feature based on these nearest neighbor points;

[0038] The present invention inputs the position vector feature, distance feature, brightness feature and brightness difference feature into the point structured information network model, maps the position vectors of adjacent points to weights, and then performs matrix multiplication of the weights and the four features to obtain point cloud structured information;

[0039] The present invention inputs the extracted structural information features into the distortion perception flow network and pre-trains to obtain distortion classification features;

[0040] The present invention inputs the structured information features into the basic quality perception flow network to obtain the basic quality features of the point cloud block;

[0041] The present invention fuses the basic quality features with the distortion classification features and inputs them into two third fully connected layers to obtain the predicted quality score;

[0042] The present invention combines multidimensional features and point structured information network model, and introduces position vector features, distance features, brightness features and brightness difference features into the dual-stream quality evaluation network;

[0043] The present invention not only considers the influence of point cloud brightness, distance, relative position and other information on point cloud quality, but also introduces brightness difference features and structured information features to comprehensively evaluate the subjective quality of point clouds. In terms of model construction, a dual-stream branch network is constructed, and a pre-trained distortion classification model is used to assist the quality evaluation model to more accurately regress the point cloud quality score, which is of great significance to improving the accuracy of point cloud quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is an architectural diagram of the point structured information network model of the present invention;

[0045] Figure 2 It is a framework diagram of the present invention. DETAILED DESCRIPTION

[0046] The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0047] Example 1

[0048] The present invention provides a point cloud quality calculation method based on a point structured information network, comprising:

[0049] In the model application stage, the position vector features, distance features, brightness features and brightness difference features of the point cloud blocks obtained in advance are jointly input into the point structured information network model to extract the structured information features of the point cloud blocks;

[0050] Input the point structured information features into the distortion-aware flow network that has been iteratively trained to obtain distortion classification features;

[0051] The structured information features are input into the basic quality-aware flow network to obtain the basic quality features of the point cloud blocks.

[0052] Furthermore, in the model application stage of this embodiment, the basic quality features of the point cloud block are fused with the distortion classification features and input into the two third fully connected layers to obtain the predicted quality score;

[0053] The predicted quality scores of multiple point cloud blocks belonging to the same overall point cloud are averaged to obtain the final score of the overall point cloud.

[0054] Furthermore, in this embodiment, the pre-acquired position vector features, distance features, brightness features, and brightness difference features are jointly input into the point structured information network model to extract the structured information features of the point cloud block, which is achieved by the following steps: the point structured information network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are connected in sequence;

[0055] The position vector features are input into the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer for processing to obtain structured feature weights;

[0056] The position vector feature, distance feature, brightness feature and brightness difference feature are weighted with the structured feature weight to obtain the structured information feature of the point cloud block.

[0057] Furthermore, in this embodiment, the point structured information features are input into the distortion-aware flow network that has been iteratively trained to obtain distortion classification features, which is achieved through the following steps:

[0058] The distortion-aware flow network before iterative training includes the fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer. The fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer are connected in sequence;

[0059] Freeze the entire distortion-aware stream network after iterative training and remove the linear regression layer;

[0060] The point structured information features are input into the distortion-aware flow network that has been iteratively trained without the linear regression layer to obtain the distortion classification features.

[0061] Furthermore, in this embodiment, the basic quality-aware flow network includes multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer, and the multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer are connected in sequence.

[0062] Furthermore, in this embodiment, the position vector feature, distance feature, brightness feature and brightness difference feature are pre-acquired, which is achieved by the following steps:

[0063] Sample the original point cloud according to the FPS farthest point sampling algorithm principle to obtain sampling points;

[0064] Using the KNN nearest neighbor algorithm, the 1024 closest distance points of each sampling point are selected to form a point cloud block;

[0065] Calculate the position vector features and distance features of each point cloud block;

[0066] The brightness features and brightness difference features of each point cloud block are calculated.

[0067] Furthermore, in this embodiment, the position vector feature and distance feature of each point cloud block are calculated by the following steps: Calculate the position vector feature {Δx j ,Δy j ,Δz j}:

[0068] {Δx j ,Δy j ,Δz j}={x j -x0,y j -y0,z j -z0},

[0069] Where p j ={x j ,y j ,z j} represents the three-dimensional coordinates of each sampling point, j = 1, 2, ..., K, p0 = {x0, y0, z0}, p0 is the three-dimensional coordinates of the centroid;

[0070] Calculate the distance feature of each sampling point in the point cloud block:

[0071]

[0072] Furthermore, in this embodiment, the brightness feature and brightness difference feature of the point cloud block are calculated and achieved by the following steps:

[0073] Calculate the brightness feature l of each sampling point in the point cloud block j :

[0074] l j =r j ×0.229+g j ×0.587+b j ×0.114,

[0075] Where c j ={r j ,g j ,b j} indicates that the three-dimensional coordinate is p j The color of the sampling point, r j Indicates the RGB value of red, g j Indicates the RGB value of green, b j Represents the RGB value of blue;

[0076] Calculate the brightness difference feature Δl of each sampling point in the point cloud blockj :

[0077] Δl j =l0-l j ,

[0078] Where l0 represents the brightness value of the centroid point.

[0079] Example 2

[0080] The technical solution of the present invention includes the following parts:

[0081] 1) Local sampling to form point cloud blocks

[0082] On the one hand, considering the huge number of points in the point cloud samples (usually tens of thousands to millions of points), it is difficult for existing computing resources (such as memory, video memory) to analyze and process the entire point cloud at the same time; on the other hand, the number of samples in the existing point cloud quality evaluation database is relatively small, which cannot meet the requirements of machine learning methods for training samples in terms of large scale and diversity. Therefore, the present invention draws on the idea of ​​local sampling in image quality evaluation, that is, the idea that the local represents the global. Specifically, the farthest point sampling method is used to select N points of each distorted point cloud, and the K nearest neighbor points of each sampling point are selected to form a point cloud block. In the model training stage, the quality score of the overall point cloud is expressed as the true score of the training point cloud block; when the model is applied to quality assessment, the final quality score of the overall point cloud is the average of the scores of all local point cloud blocks.

[0083] The sampling method of local point cloud blocks is as follows: (1) First, the farthest point sampling method is used to select sampling points, and the farthest point from the existing sampling point set is selected iteratively to cover the entire point cloud as much as possible; (2) Then, the K-nearest neighbor (KNN) search method is used to construct local point cloud blocks. By calculating the distance from the sampling point to the neighboring points, these distance calculations are sorted from small to large, and the K samples closest to the sampling point are selected to form a point cloud block.

[0084] 2) Point Structured Information Extraction Module (PSI)

[0085] Previous image quality assessment methods have shown that the human visual system is highly adapted to extract structural information from the observation field, such as gradient, contrast, etc. However, for point clouds, structural information refers to the changes in local 3D blocks, including changes in color and geometry. However, the irregularity of point clouds poses a challenge to the effective extraction of structural information in three-dimensional space. In the present invention, the point structured information extraction module (PSI) module can simultaneously extract geometric and color structural information from point clouds by fitting feature weights as nonlinear functions to three-dimensional relative coordinates. The framework of the PSI module is as follows: Figure 2 shown.

[0086] Feature preprocessing

[0087] like Figure 2 As shown in Figure 2, a sampling block with adjacent points is input into the PSI module. In order to better extract the structural information of irregular points, the input block needs to be preprocessed. Specifically, we represent the patch as P = {p0, p1, p2, ..., p K}. Among them, p0 = {x0, y0, z0} is the sampling point (or centroid point), p j ={x j ,y j ,z j}, j=1,2,…,K are the nearest neighbors of p0. Vector c j ={r j ,g j ,b j} means p j color.

[0088] Since structural information refers to the changes in color and geometric information of local 3D patches, in this study, it is calculated as the changes in geometry and color from neighboring points to the center point, such as position vector, distance, and brightness difference.

[0089] Calculate each point p j The position vector feature, p j ={x j ,y j ,z j}, j = 1, 2, ..., K, the corresponding position vector features are calculated as the centroid points p0 to p j Vectors between:

[0090] {Δx j ,Δy j ,Δz j}={x j -x0,y j -y0,z j -z0}(1)

[0091] Calculate the corresponding distance feature:

[0092]

[0093] Among them, d j Indicates p0 to p j distance.

[0094] Since the human visual system is very sensitive to the relative change of brightness, for each p j ={x j ,y j ,z j}, j = 1, 2, ..., K, its color is converted into brightness value, and the brightness feature of each point cloud block is calculated:

[0095] l j =r j ×0.229+g j ×0.587+b j ×0.114,(3)

[0096] where l j Indicates p j Brightness difference characteristics. j The brightness difference with p0 is calculated as follows,

[0097] Δl j =l0-l j . (4)

[0098] Therefore, for each adjacent point p j ={x j ,y j ,z j}, j = 1, 2, ..., K, we can get a preprocessed input feature, i j ={Δx j ,Δy j ,Δz j ,d j ,l j ,Δl j} T That is, the input of the PSI module is I={i1,i2,…,i K} T ,Right now Weighted based on position vector

[0099] Neighboring points have different relative positions to the same center point, so the local changes between them have different weights on the calculation of the structural information of the 3D block. CNN has shown strong capabilities in learning representations of image features, where the convolution weights are considered as discrete functions of relative positions. On 3D point clouds, the convolution weights are considered as nonlinear continuous functions of the position vector, i.e.

[0100] W=f(Δx,Δy,Δz), (5)

[0101] where Δx={Δx1,Δx2,…,Δx K} T , Δy={Δy1,Δy2,…,Δy K} T and Δz={Δz1,Δz2,…,Δz K} T ,like Figure 2As shown, the nonlinear function f(·) is implemented as 4 convolutional layers. Then, the structural information features of the 3D block are calculated by matrix multiplication with the non-quantized weight W and the input I, i.e.

[0102]

[0103] in The structural information feature representing the 3D point cloud block is also called the ψ feature.

[0104] 3) Point-structured information network structure

[0105] like Figure 1 As shown, in the point structured information network, the structural information feature F output by the PSI module ψ It is fed into two calculation streams, DPS and EQPS.

[0106] First, DPS is pre-trained based on the point cloud distortion classification task. Specifically, F ψ Reshaped into After the network is resized, it is fed into the Conv1 module which has 2 convolutional layers and a max-pooling layer.

[0107] To prevent overfitting, the output of the Conv1 module passes through a global average pooling (GAP) layer of size 2×2, and then maps it to the distortion-related features through two fully connected (FC) layers, which are expressed as Secondly, compared to distortion classification, quality regression requires more complex features. Therefore, EQPS uses two more convolutional modules than DPS: Conv2 and Conv3. Specifically, Conv2 and Conv3 modules each consist of three convolutional layers and one max pooling layer, with 256 and 512 kernels in each convolutional layer, respectively. This is because more convolutional layers and kernels can expand the receptive field and extract richer feature information.

[0108] In general, the structural information features from the PSI module pass through the Conv1, Conv2 and Conv3 modules, a GAP layer of size 1×1, and two FC layers in sequence to obtain a feature, which is expressed as Finally, these two features, namely f dps and f eqps , fused by dot product. Then, based on the fused features, two FC layers are used to regress the predicted quality score of the 3D patch.

[0109] 3) Point Structured Information Network Training: Since the DPS is pre-trained using the distortion classification task, the proposed point structured information network is trained in two stages. The cross-entropy loss between the predicted distortion type and the true distortion type is used to pre-train the DPS. Note that the true distortion type of each 3D point cloud patch is the same as the distortion type of the entire point cloud to which it belongs. The entire network is then trained by freezing the weights of the DPS.

[0110] The mean squared error (MSE) loss between the predicted quality scores and the actual quality scores is used to supervise the training of the entire network, i.e.

[0111]

[0112] Where M is the size of the minimum batch. Symbol Q i and They are expressed as predicted scores and true scores respectively.

[0113] Likewise, the true quality score of each patch is equal to the quality score of the entire point cloud.

[0114] 4) Application of Point Structured Information Network to Point Cloud Quality Assessment

[0115] The predicted scores of multiple point cloud blocks belonging to the same overall point cloud are averaged and the resulting score is used as the final score of the overall point cloud. The specific formula is:

[0116]

[0117] The iterative training phase of the present invention comprises the following steps:

[0118] Step 1: During iterative training, the original point cloud is sampled according to the principle of the FPS farthest point sampling algorithm to obtain sampling points. The sampling points constitute the training data set. The training data set is obtained. The historical sampling points include historical 3D coordinates and historical specific attributes (red, green, and blue colors);

[0119] Using the KNN nearest neighbor algorithm, select the 1024 closest distance points of each sampling point to form a point cloud block, and execute step 2; Step 2: Calculate the historical position vector feature and historical distance feature of each point cloud block according to formula (1) and formula (2), and execute step 3;

[0120] Step 3: Calculate the historical brightness feature and historical brightness difference feature of each point cloud block according to formula (3) and formula (4), and execute step 4.

[0121] Step 4: The pre-acquired position vector features, distance features, brightness features, and brightness difference features are jointly input into the point structured information network model PSI Module to extract the structured information features of the historical point cloud block and proceed to step 5;

[0122] The point structured information network model PSI Module includes the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer. The first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer are connected in sequence. The first convolutional layer, the second convolutional layer and the third convolutional layer are three convolutional layers with a convolution kernel of 3 and an output channel of 8. The fourth convolutional layer is a convolutional layer with a convolution kernel of 3 and an output channel of 16.

[0123] Specifically, the historical position vector features (K×3 dimensions) are first input into the point structured information network model, and processed through the first, second, third, and fourth convolutional layers to obtain the historical structured feature weights (K×16 dimensions). The four convolutional layers are regarded as nonlinear functions, and the position vector features are fitted as feature weights.

[0124] The historical position vector features, historical distance features, historical brightness features, and historical brightness difference features (K×6 dimensions) are weighted with the historical structured feature weights. This is done by matrix multiplication of the two to obtain the structured information features of the historical point cloud blocks. The specific formula is: W = f(Δx, Δy, Δz), where f is the nonlinear function formed by the four convolutional layers (the first, second, third, and fourth convolutional layers).

[0125] Step 5: Input the historical point structured information features into the distortion-aware flow network, and pre-train the distortion-aware flow network using the cross-entropy loss formula to obtain the trained distortion-aware flow network;

[0126] Then freeze the entire distortion-aware stream network after training, remove the linear regression layer to obtain the distortion classification features, and execute step 6;

[0127] The distortion-aware flow network includes the fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer and the linear regression layer. The fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer and the linear regression layer are connected in sequence; the fifth and sixth convolutional layers are two convolutional layers with 3 convolution kernels and 128-dimensional output channels. The first maximum pooling layer has a stride of 2 and a sliding window size of 2.

[0128] The historical point structured information features are regressed into distortion types (4 types in total) through the linear regression layer in the distortion-aware flow network. Step 6: Input the historical structured information features into the basic quality-aware flow network to obtain the basic quality features of the historical point cloud blocks;

[0129] The basic quality-aware flow network includes multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer. The multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer are connected in sequence. Step 7: The basic quality features are fused with the distortion classification features and input into the two third fully connected layers to obtain the predicted quality score.

[0130] Step 8: Iteratively train the point structured network by minimizing the sum of the mean squared error between the predicted quality scores and the true quality scores; the input of the point structured network is the vector features, distance features, brightness features, and brightness difference features of the points in the K point clouds of M batches, and the output of the point structured network is the quality scores of M batches.

[0131] The historical position vector features, historical distance features, historical brightness features and historical brightness difference features are input into the point structured information network model, the distortion perception flow network and the basic quality perception flow network for processing. The point structured information network model, the distortion perception flow network and the basic quality perception flow network are iteratively trained using the mean square error. The expression of the mean square error is:

[0132]

[0133] Where M is the total number of historical sampling points in the training dataset,

[0134] The cross entropy loss function is used to iteratively update the distortion perception flow network before training. The expression of the cross entropy loss function is:

[0135]

[0136] Among them, LOSS is the loss value, M is the number of all sampling points in the training data set, Class is the number of distortion classifications, and y is the true label of the sampling point. Output the predicted labels of the sample points for the distortion-aware flow network.

[0137] Figure 2In the example, the PSI Module is a point structured information network model. The reshape operation is used to transform the output features into a shape suitable for convolutional neural network learning, transforming the K×96 features into features with √k×√k×96 dimensions. Pool-2 is a maximum pooling layer with a stride of 2 and a sliding window of 2, which is used to reduce dimensionality, reduce the number of parameters, and remove redundant information. GAP is a global average pooling layer, which is used to reduce dimensionality and prevent overfitting. Flatten means flattening, which flattens the input tensor so that the fully connected layer can process it. FC is a fully connected layer, which is used to combine the features extracted previously. The symbol means dot product, which means fusing the features of the two branches. Quality Score is the predicted quality score of the network output.

[0138] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A point cloud quality calculation method based on a point structured information network, characterized in that: include: The position vector features, distance features, brightness features and brightness difference features of the point cloud blocks obtained in advance are jointly input into the point structured information network model to extract the structured information features of the point cloud blocks; The position vector features and distance features of each point cloud block are calculated and implemented through the following steps: Calculate the position vector feature {Δx j ,Δy j ,Δz j }: {Δx j ,Δy j ,Δz j }={x j −x0,y j −y0,z j -z0}, Where p j ={x j ,y j ,z j } represents the three-dimensional coordinates of each sampling point, j = 1, 2, ..., K, p0 = {x0, y0, z0}, p0 is the three-dimensional coordinates of the centroid; Calculate the distance feature of each sampling point in the point cloud block: The brightness features and brightness difference features of the point cloud blocks are calculated by the following steps: Calculate the brightness feature l of each sampling point in the point cloud block j : l j =r j ×0.229+g j ×0.587+b j ×0.114, Where c j ={r j ,g j ,b j } indicates that the three-dimensional coordinate is p j The color of the sampling point, r j Indicates the RGB value of red, g j Indicates the RGB value of green, b j Represents the RGB value of blue; Calculate the brightness difference feature Δl of each sampling point in the point cloud block j : Δl j =l0-l j , Where l0 represents the brightness value of the centroid; Input the point structured information features into the distortion-aware flow network that has been iteratively trained to obtain distortion classification features; The structured information features are input into the basic quality-aware flow network to obtain the basic quality features of the point cloud blocks.

2. The point cloud quality calculation method based on point structured information network according to claim 1, characterized in that: The basic quality features of the point cloud block are fused with the distortion classification features and input into the two third fully connected layers to obtain the predicted quality score; The predicted quality scores of multiple point cloud blocks belonging to the same overall point cloud are averaged to obtain the final score of the overall point cloud.

3. The point cloud quality calculation method based on point structured information network according to claim 1, characterized in that: The pre-acquired position vector features, distance features, brightness features, and brightness difference features are jointly input into the point structured information network model to extract the structured information features of the point cloud block. This is achieved through the following steps: The point structured information network model includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer, and the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are connected in sequence; The position vector features are input into the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer for processing to obtain structured feature weights; The position vector feature, distance feature, brightness feature and brightness difference feature are weighted with the structured feature weight to obtain the structured information feature of the point cloud block.

4. The point cloud quality calculation method based on point structured information network according to claim 1, characterized in that: Input the point structured information features into the iteratively trained distortion-aware flow network to obtain the distortion classification features. This is achieved through the following steps: The distortion-aware flow network before iterative training includes the fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer. The fifth convolutional layer, the sixth convolutional layer, the first maximum pooling layer, the first global average pooling layer, the first fully connected layer, and the linear regression layer are connected in sequence; Freeze the entire distortion-aware stream network after iterative training and remove the linear regression layer; The point structured information features are input into the distortion-aware flow network that has been iteratively trained without the linear regression layer to obtain the distortion classification features.

5. The point cloud quality calculation method based on point structured information network according to claim 4, characterized in that: The basic quality-aware flow network includes multiple layers of a seventh convolutional layer, a second maximum pooling layer, a second global average pooling layer, and a second fully connected layer, and the multiple layers of the seventh convolutional layer, the second maximum pooling layer, the second global average pooling layer, and the second fully connected layer are connected in sequence.

6. The point cloud quality calculation method based on point structured information network according to claim 1, characterized in that: Pre-acquire position vector features, distance features, brightness features, and brightness difference features by following these steps: Sample the original point cloud according to the FPS farthest point sampling algorithm principle to obtain sampling points; Using the KNN nearest neighbor algorithm, the 1024 closest distance points of each sampling point are selected to form a point cloud block; Calculate the position vector features and distance features of each point cloud block; The brightness features and brightness difference features of each point cloud block are calculated.

Images