Point cloud quality evaluation method based on projection and multi-scale features and related device

Abstract

The application discloses a point cloud quality evaluation method and related device based on projection and multi-scale features, the method comprises the following steps: determining a preset number of image groups based on a reference point cloud and a distorted point cloud; calculating a texture gradient weight set and a geometric gradient weight set in units of image blocks; calculating a texture quality score based on each image group and the texture gradient weight set of each image group, and calculating a geometric quality score based on each image group and the geometric gradient weight set of each image group; and calculating a quality score of the distorted point cloud based on the texture quality score and the geometric quality score. The application determines image groups through patch projection, then obtains block gradient weight coefficients of the image groups of different scales, and determines a quality score based on the block gradient weight coefficients of different scales and multi-scale features, so that the multi-scale visual characteristics can be explored on the one hand, and the visual importance can be effectively reflected on the other hand, thereby improving the performance of point cloud quality evaluation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a point cloud quality evaluation method and related apparatus based on projection and multi-scale features. Background Technology

[0002] Compared to classic 2D images or videos, 3D visual representation can provide an immersive experience and is an important direction for future interaction between virtual content and the real world. Based on the advantages of 3D visual representation, immersive 3D media has been applied in many fields, such as virtual teleconferencing, immersive sports, and education. Among these, point cloud media has become an important 3D visual representation format due to the continuous development of point cloud media acquisition devices (e.g., distance sensors and multi-camera arrays), encoding methods, and rendering technologies.

[0003] Point clouds can perfectly represent 3D models, but they also generate a large amount of information, making point cloud compression inevitable to reduce storage and transmission data requirements. However, point cloud compression results in noise loss, and lossy compression can lead to visual degradation, affecting the perceptual quality of the model and consequently the user experience. Therefore, evaluating point cloud quality to determine an encoding method that strikes a good balance between visual quality and data size is crucial. However, existing point cloud quality evaluation methods show low correlation with subjective evaluation results when evaluating compressed and distorted point clouds, failing to accurately reflect visual perception and thus affecting the selection of encoding methods.

[0004] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a point cloud quality evaluation method and related apparatus based on projection and multi-scale features, addressing the shortcomings of existing technologies.

[0006] To address the aforementioned technical problems, the first aspect of this application provides a point cloud quality evaluation method based on projection and multi-scale features, the method comprising:

[0007] A preset number of image groups are determined based on the reference point cloud and the distorted point cloud. Each image group includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different.

[0008] Calculate the texture gradient weight set and geometric gradient weight set for each image group on an image patch basis;

[0009] The texture quality score is calculated based on each image group and the texture gradient weight set of each image group, and the geometric quality score is calculated based on each image group and the geometric gradient weight set of each image group.

[0010] Based on the texture quality score and the geometric quality score, calculate the quality score corresponding to the distorted point cloud.

[0011] The point cloud quality evaluation method based on projection and multi-scale features, wherein determining a preset number of image groups based on the reference point cloud and the distorted point cloud specifically includes:

[0012] The projection generates a texture reference map and a geometric reference map corresponding to the reference point cloud, as well as a texture distortion map and a geometric distortion map corresponding to the distorted point cloud, to obtain the first image group;

[0013] The images in the first image group are downsampled to obtain the second image group;

[0014] The images in the second image group are downsampled to obtain the third image group;

[0015] This process continues until a preset number of image groups are obtained.

[0016] The point cloud quality evaluation method based on projection and multi-scale features, wherein the downsampling operation is a downsampling operation performed through a Gaussian pyramid.

[0017] The point cloud quality evaluation method based on projection and multi-scale features, wherein calculating the texture gradient weight set for each image group on an image patch basis specifically includes:

[0018] For each image group, convert the texture reference map in that image group into a grayscale texture reference map;

[0019] The grayscale texture reference image is divided into several texture reference image blocks according to a preset division method, and the pixel gradient value of each pixel in each texture reference image block is calculated.

[0020] The gradient values ​​of each texture reference image block are calculated based on the pixel gradient values ​​of the pixels in each texture reference image block to obtain the texture gradient weight set.

[0021] The point cloud quality evaluation method based on projection and multi-scale features, wherein the calculation of texture quality scores based on each image group and the texture gradient weight set corresponding to each image group specifically includes:

[0022] For each image group, the texture reference image and the texture distortion image are divided into several texture reference image blocks and several texture distortion image blocks according to a preset division method. Based on each texture reference image block, the texture gradient weight set corresponding to the image group, and each texture distortion image block, the texture distortion degree of the image group is calculated. The texture reference image block corresponds one-to-one with the texture distortion image block and the texture image block gradient weight in the texture gradient weight set.

[0023] The texture quality score is calculated based on the texture distortion of each image group.

[0024] The point cloud quality evaluation method based on projection and multi-scale features, wherein the formula for calculating the texture quality score is:

[0025]

[0026]

[0027] Among them, Q T The texture quality score is represented by K, the number of image groups is represented by β. k D represents the multiplicative coefficient of the k-th image group. Tk This represents the texture distortion of the k-th image group, where i represents the i-th texture reference image patch. This represents the texture distortion of the i-th texture reference image block in the k-th image group.

[0028] The point cloud quality evaluation method based on projection and multi-scale features, wherein calculating the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score specifically involves:

[0029] The texture quality score and the geometric quality score are weighted to obtain the quality score corresponding to the distorted point cloud.

[0030] A second aspect of this application provides a point cloud quality evaluation system based on projection and multi-scale features, the system comprising:

[0031] The projection module is used to determine a preset number of image groups based on the reference point cloud and the distorted point cloud. Each image group includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different.

[0032] The weight determination module is used to calculate the texture gradient weight set and geometric gradient weight set of each image group on a per-image-patch basis.

[0033] The first calculation module is used to calculate the texture quality score based on each image group and the texture gradient weight set of each image group, and to calculate the geometric quality score based on each image group and the geometric gradient weight set of each image group.

[0034] The second calculation module is used to calculate the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score.

[0035] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the point cloud quality evaluation method based on projection and multi-scale features as described above.

[0036] A fourth aspect of this application provides a terminal device, which includes: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0037] The communication bus enables communication between the processor and the memory;

[0038] When the processor executes the computer-readable program, it implements the steps in any of the point cloud quality assessment methods based on projection and multi-scale features as described above.

[0039] Beneficial Effects: Compared with existing technologies, this application provides a point cloud quality evaluation method and related apparatus based on projection and multi-scale features. The method includes determining a preset number of image groups based on a reference point cloud and a distorted point cloud; calculating the texture gradient weight set and geometric gradient weight set for each image group on a per-image-patch basis; calculating a texture quality score based on each image group and its texture gradient weight set, and calculating a geometric quality score based on each image group and its geometric gradient weight set; and calculating the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score. This application determines image groups through patch projection, then obtains the block gradient weight coefficients of image groups at different scales, and determines the quality score based on the block gradient weight coefficients at different scales and multi-scale features. This approach can, on the one hand, uncover multi-scale visual characteristics, and on the other hand, effectively reflect visual importance, thereby improving the performance of point cloud quality evaluation and making the point cloud quality evaluation method more suitable for point cloud encoding. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart of the point cloud quality evaluation method based on projection and multi-scale features provided in this application.

[0042] Figure 2 The original flowchart of the point cloud quality evaluation method based on projection and multi-scale features provided in this application.

[0043] Figure 3 A gradient map is constructed from the gradient values ​​of a texture reference image patch for a group of images.

[0044] Figure 4 A gradient map is constructed from the gradient values ​​of a texture reference image patch for a group of images.

[0045] Figure 5 A gradient map is constructed from the gradient values ​​of a texture reference image patch for a group of images.

[0046] Figure 6 A gradient map is constructed from the gradient values ​​of a geometric reference image block of an image group.

[0047] Figure 7 A gradient map is constructed from the gradient values ​​of a geometric reference image block of an image group.

[0048] Figure 8 A gradient map is constructed from the gradient values ​​of a geometric reference image block of an image group.

[0049] Figure 9 The structural principle diagram of the point cloud quality evaluation system based on projection and multi-scale features provided in this application.

[0050] Figure 10 A schematic diagram of the terminal device provided in this application. Detailed Implementation

[0051] This application provides a point cloud quality evaluation method and related apparatus based on projection and multi-scale features. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0054] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0055] Research has shown that compared to classic 2D images or videos, 3D visual representation can provide an immersive experience and represents an important direction for future interaction between virtual content and the real world. The superiority of 3D visual representation has spurred the application of immersive 3D media in many fields, such as virtual teleconferencing, immersive sports, and education. Among these, point cloud media has emerged as an important 3D visual representation format due to continuous advancements in point cloud media acquisition devices (e.g., distance sensors and multi-camera arrays), encoding methods, and rendering technologies.

[0056] Point clouds can perfectly represent 3D models, but they also generate a large amount of information, making point cloud compression inevitable to reduce storage and transmission data requirements. However, point cloud compression results in noise loss, and lossy compression can lead to visual degradation, affecting the perceptual quality of the model and consequently the user experience. Therefore, evaluating point cloud quality to determine an encoding method that strikes a good balance between visual quality and data size is crucial.

[0057] To address this issue, Point Cloud Quality Assessment (PCQA) bridges the gap between subjective and objective quality scores, becoming a research hotspot in point cloud processing and compression. Meynet et al. proposed a Point Cloud Quality Metric (PCQM) that weights and combines geometric features (curvature) and color-based features (luminance, chroma, and hue). Alexiou et al. proposed using structural similarity and local topological degree to reflect the quality of point cloud attributes. Yang et al. proposed a graph-based PCQA called GraphSIM, which generates graphs from resampled point cloud keypoints and then compares the graph similarity between a reference point cloud and a deformed point cloud. Wu et al. proposed a patch projection-based evaluation method called SIAT_PCQA, which projects a 3D point cloud onto multiple 2D patches based on normal vectors and then measures it using existing 2D evaluation methods. L et al. combined region-based geometric distortion and projection-based color features to jointly predict perceived quality, resulting in BQE-CVP.

[0058] Although there are existing point cloud quality assessment methods, when evaluating compressed and distorted point clouds, these methods have several drawbacks. First, they show low correlation with subjective evaluation results, failing to adequately reflect visual perception. Second, when evaluating the quality of projected point cloud geometry and texture maps, the evaluation is performed directly on the projected planar image, whereas the subjective tester actually sees a three-dimensional point cloud image, leading to inconsistencies between subjective and objective evaluations.

[0059] To address the aforementioned issues, in this embodiment, a preset number of image groups are determined based on a reference point cloud and a distorted point cloud. Texture gradient weight sets and geometric gradient weight sets are calculated for each image group, using image blocks as units. Texture quality scores are calculated based on each image group and its texture gradient weight set, and geometric quality scores are calculated based on each image group and its geometric gradient weight set. Finally, the quality score corresponding to the distorted point cloud is calculated based on the texture quality score and the geometric quality score. This application determines image groups through patch projection, then obtains the block gradient weight coefficients of image groups at different scales, and determines the quality score based on the block gradient weight coefficients at different scales and multi-scale features. This approach can, on the one hand, uncover multi-scale visual characteristics, and on the other hand, effectively reflect visual importance, thereby improving the performance of point cloud quality evaluation and making the point cloud quality evaluation method more suitable for point cloud encoding.

[0060] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0061] This embodiment provides a point cloud quality evaluation method based on projection and multi-scale features, such as... Figure 1As shown, the method includes:

[0062] S10. Determine a preset number of image groups based on the reference point cloud and the distorted point cloud.

[0063] Specifically, each of the preset number of image groups includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different. Here, "different image scales corresponding to each image group" means that in any two image groups, the image scales of the texture distortion maps, geometric distortion maps, texture reference maps, and geometric reference maps in the two image groups are different. Furthermore, the image scales of the texture reference map, geometric reference map, texture distortion map, and geometric distortion map in each image group are the same.

[0064] In one implementation, determining the preset number of image groups based on the reference point cloud and the distorted point cloud specifically includes:

[0065] The projection generates a texture reference map and a geometric reference map corresponding to the reference point cloud, as well as a texture distortion map and a geometric distortion map corresponding to the distorted point cloud, to obtain the first image group;

[0066] The images in the first image group are downsampled to obtain the second image group;

[0067] The images in the second image group are downsampled to obtain the third image group;

[0068] This process continues until a preset number of image groups are obtained.

[0069] Specifically, the projection generation can employ the patch projection method SIAT_PCQA, which determines the texture reference map, geometric reference map, texture distortion map, and geometric distortion map through patch projection. The patch projection process can be as follows: first, the distorted point cloud is matched with corresponding points relative to the reference point cloud; then, the reference point cloud and the matched distorted point cloud are patched and packaged to form the texture reference map, geometric reference map, texture distortion map, and geometric distortion map. All three images—texture reference map, geometric reference map, texture distortion map, and geometric distortion map—are two-dimensional images, and they have the same image scale. For example, as shown... Figure 2 As shown, reference point cloud PC ref Distorted point cloud PC dist A texture reference map T is generated using a patch projection method. ref Geometric reference diagram G ref Texture distortion map T dist and geometric distortion map G dist .

[0070] Furthermore, after obtaining the first image group through projection, the texture reference map, geometric reference map, texture distortion map, and geometric distortion map in the first image group are downsampled to obtain the second image group, wherein the image scale of each image in the second image group is smaller than the image scale of each image in the first image group. Similarly, after obtaining the second image group, the texture reference map, geometric reference map, texture distortion map, and geometric distortion map in the second image group are downsampled to obtain the third image group, and so on, to obtain a preset number of image groups.

[0071] In one implementation, when downsampling the first image group, the second image group, ..., the preset number-1 image group, a Gaussian pyramid can be used for downsampling. This involves inputting the texture reference image, geometric reference image, texture distortion image, and geometric distortion image from the first image group into a Gaussian pyramid, obtaining the output items of each layer of the pyramid, and thus obtaining the second image group, the third image group, ..., the preset number of image groups. The number of layers in the Gaussian pyramid is equal to the preset number-1. Essentially, each layer of the Gaussian pyramid reduces the resolution of the output item to half of the input item. For example, let k represent different scales: k=1 represents the image scale corresponding to the first image group generated by projection; k=2 represents downsampling the first image group once, resulting in the second image group having an image scale half that of the first image group, and so on, to obtain the preset number of image groups.

[0072] Of course, in practical applications, after generating the first image group through projection, the second, third, ..., and a predetermined number of image groups can be obtained in other ways. For example, the second image group can be obtained by downsampling the first image group by a factor of 2, the third image group can be obtained by downsampling the first image group by a factor of 4, and so on, to obtain the predetermined number of image groups. Furthermore, the correspondence between the image scales of each image group can also be different. For example, the image scale corresponding to the first image group can be 3 times that of the second image group, and the image scale corresponding to the second image group can be 3 times that of the third image group, and so on.

[0073] S20. Calculate the texture gradient weight set and geometric gradient weight set for each image group, using image blocks as the unit.

[0074] Specifically, calculating the texture gradient weight set for each image group on an image patch basis refers to dividing the texture reference map into several texture reference image patches, and then calculating the texture image patch gradient weight for each texture reference image patch to obtain the texture gradient weight set. It can be understood that the texture gradient weight set includes several texture image patch gradient weights, each texture image patch gradient weight corresponds to an image patch in the texture reference map, and the image patches corresponding to each texture image patch gradient weight are different from each other.

[0075] In one implementation, calculating the texture gradient weight set for each image group on a per-image-patch basis specifically includes:

[0076] For each image group, convert the texture reference map in that image group into a grayscale texture reference map;

[0077] The grayscale texture reference image is divided into several texture reference image blocks according to a preset division method, and the pixel gradient value of each pixel in each texture reference image block is calculated.

[0078] The gradient values ​​of each texture reference image block are calculated based on the pixel gradient values ​​of the pixels in each texture reference image block to obtain the texture gradient weight set.

[0079] Specifically, the texture reference image is a three-channel RGB image, and the grayscale texture reference image is a single-channel grayscale image. In other words, the texture reference image is converted from a three-channel RGB space to a single-channel grayscale space. The preset partitioning method is pre-set and used to divide the grayscale texture reference image into several texture reference image blocks. The preset partitioning method can be based on a preset number of blocks, or on a preset size, etc. In one implementation, the preset partitioning method is based on a preset number of blocks, that is, the texture reference image is equally divided into a preset number of texture reference image blocks.

[0080] After dividing the texture reference image into blocks, the pixel gradient value of each pixel in the texture reference image block is calculated. The pixel gradient value includes the horizontal gradient and the vertical gradient. The formulas for calculating the horizontal gradient and the vertical gradient are as follows:

[0081]

[0082]

[0083] Among them, F H I represents the gradient operator in the horizontal direction. i Let represent the i-th texture reference image block, with a size of M×N, and j represent the j-th pixel in the texture reference image block.

[0084] Furthermore, after obtaining the pixel gradient values ​​of each pixel, the gradient value of the texture reference image block is calculated based on the pixel gradient values ​​of each pixel in the texture reference image block. The formula for calculating the candidate gradient values ​​of the texture reference image block can be:

[0085]

[0086] in, This represents the candidate gradient value.

[0087] After obtaining the candidate gradient values ​​for each texture reference image block, a normalization operation is performed on each candidate gradient value to obtain the gradient value of each texture reference image block. The gradient value of the texture reference image block is used to reflect the importance of the texture reference image block relative to all texture reference image blocks.

[0088] The formula for calculating the gradient value is as follows:

[0089]

[0090] Among them, W i This represents the gradient value of the texture reference image patch. Normal(·) represents the normalization operation, and its specific calculation method is as follows:

[0091]

[0092] The C(·) operation constrains the calculated maximum and minimum values ​​to the range [0,1], i.e., values ​​less than 0 are set to 0, and values ​​greater than 1 are set to 1. x x represents i The mean, σ x x represents i The standard deviation.

[0093] Furthermore, after obtaining the texture gradient weight set corresponding to the texture reference map in each image group, the texture gradient weight set of each image group can convert the texture reference map in each image group into a texture gradient map, that is, multiple texture gradient maps of different scales can be obtained, so as to facilitate the subsequent calculation of the texture distortion degree corresponding to the texture distortion map at different scales, and thus determine the texture quality score. For example, a preset number of image groups include image group a, image group b, and image group 9. The texture gradient map corresponding to the texture reference map of image group a is as follows: Figure 3 As shown, the texture gradient map corresponding to the texture reference map of image group b is as follows: Figure 4 As shown, the texture gradient map corresponding to the texture reference map of image group c is as follows: Figure 5 As shown.

[0094] Furthermore, the calculation process for the geometric gradient weight set is the same as that for the texture gradient weight set. The difference lies in that the geometric reference image is a grayscale image, thus eliminating the need for spatial transformation. Instead, the geometric reference image is directly divided into geometric reference image blocks according to a preset partitioning method, and then the gradient value of each geometric reference image block is calculated. Therefore, each image group will have two gradient weight sets: a texture gradient weight set and a geometric gradient weight set, denoted as W. i T and W i G Through W i T and W i G This indicates the importance of image patch i relative to all image patches.

[0095] Similarly, after obtaining the geometric gradient weight set corresponding to the geometric reference map in each image group, the geometric gradient weight set of each image group can convert the geometric reference map in each image group into a geometric gradient map, that is, multiple geometric gradient maps at different scales can be obtained, so as to facilitate the subsequent calculation of the geometric distortion degree corresponding to the geometric distortion map at different scales, and thus determine the geometric quality score. For example, a preset number of image groups include image group a, image group b, and image group c. The texture gradient map corresponding to the geometric reference map of image group a is as follows: Figure 6 As shown, the geometric gradient map corresponding to the geometric reference map of image group b is as follows: Figure 7 As shown, the geometric gradient map corresponding to the geometric reference map of image group c is as follows: Figure 8 As shown.

[0096] S30. Calculate the texture quality score based on each image group and the texture gradient weight set of each image group, and calculate the geometric quality score based on each image group and the geometric gradient weight set of each image group.

[0097] Specifically, the texture quality score is determined based on the texture gradient weight set of the image group, the distortion of the texture reference map, and the texture distortion map, where the distortion reflects the degree of distortion in the texture distortion map. In one implementation, calculating the texture quality score based on each image group and the corresponding texture gradient weight set specifically includes:

[0098] For each image group, the texture reference image and the texture distortion image are divided into several texture reference image blocks and several texture distortion image blocks according to a preset division method. Based on each texture reference image block, the texture gradient weight set corresponding to the image group, and each texture distortion image block, the texture distortion degree of the image group is calculated.

[0099] The texture quality score is calculated based on the texture distortion of each image group.

[0100] Specifically, the preset partitioning method is the same as the preset partitioning method used to determine the texture gradient weight set, so that several texture reference image blocks and several texture distortion image blocks determined based on the preset partitioning method correspond one-to-one, and several texture reference image blocks correspond one-to-one with the texture image block gradient weights in the texture gradient weight set.

[0101] In one implementation, texture distortion is measured using SSIM distortion. Other distortion metrics, such as MSE, can also be used in other implementations. This embodiment uses SSIM distortion to improve the accuracy of block-by-block image distortion calculation, thereby improving the accuracy of the texture quality score.

[0102] Based on this, the formula for calculating the distortion of texture-distorted image patches can be:

[0103]

[0104] in, This represents the texture patch distortion of the i-th texture-distorted image patch in the k-th image group. This represents the texture image patch gradient weight of the i-th texture reference image patch in the k-th image group. This represents the SSIM distortion metric of the i-th texture distortion image patch in the k-th image group relative to the i-th texture reference image patch.

[0105] Furthermore, after obtaining the distortion degree of each texture distortion image patch, the distortion degrees of each texture distortion image patch are summed to obtain the distortion degree of the texture distortion map of each image group. Then, the texture distortion degree is determined based on the distortion degree of the texture distortion map of each image group. Finally, the texture quality score is determined based on the texture distortion degree. The formula for calculating the texture quality score is as follows:

[0106]

[0107]

[0108] Among them, Q T The texture quality score is represented by K, the number of image groups is represented by β. k D represents the multiplicative coefficient of the k-th image group. Tk This represents the texture distortion of the k-th image group, where i represents the i-th texture reference image patch. This represents the texture distortion of the i-th texture reference image patch in the k-th image group. Furthermore, β... k This represents the multiplicative coefficient of the k-th image group. It can be preset, and the sum of the multiplicative coefficients of each image group is 1. For example, the preset number of image groups is 3, and the multiplicative coefficients of the three image groups are 0.1, 0.3, and 0.6, respectively.

[0109] Furthermore, the calculation process for the geometric quality score is the same as that for the texture quality score, and will not be repeated here. For details, please refer to the calculation process for the texture quality score, i.e., the formula for calculating the geometric quality score is given below:

[0110]

[0111]

[0112]

[0113] in, This represents the geometric image patch distortion of the i-th geometrically distorted image patch in the k-th image group. This represents the geometric image patch gradient weight of the i-th geometric reference image patch in the k-th image group. Q represents the SSIM distortion metric of the i-th geometrically distorted image patch in the k-th image group relative to the i-th geometrically referenced image patch. G The geometric quality score is represented by K, which represents the number of image groups, and β is the geometric quality score. k D represents the multiplicative coefficient of the k-th image group. Gk This represents the geometric distortion of the k-th image group.

[0114] S40. Calculate the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score.

[0115] Specifically, after obtaining the texture quality score and the geometric quality score, the average value of the texture quality score and the geometric quality score can be used as the quality score, one of the texture quality score and the geometric quality score can be used as the quality score, or the quality score can be determined by weighting.

[0116] In one implementation, calculating the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score specifically involves:

[0117] The texture quality score and the geometric quality score are weighted to obtain the quality score corresponding to the distorted point cloud.

[0118] The quality scores of the texture map and geometry map are fused to obtain a visual perception model. The impact of texture and geometry on the quality of the point cloud is evaluated to obtain the final quality score.

[0119] S final = T · T + G · G

[0120] Among them, a T a G These are weighted parameters, for example, set to a respectively. T =0.87, a G =0.13, Q T It is the texture quality score, Q. G It is the geometric mass fraction, S final It is the quality score.

[0121] In summary, this embodiment provides a point cloud quality assessment method based on projection and multi-scale features. The method includes determining a preset number of image groups based on a reference point cloud and a distorted point cloud; calculating the texture gradient weight set of each texture reference map and the geometric gradient weight set of the geometric reference map, using image patches as units; calculating texture quality scores based on each image group and its texture gradient weight set; calculating geometric quality scores based on each image group and its geometric gradient weight set; and calculating the quality score of the distorted point cloud based on the texture and geometric quality scores. This application determines image groups through patch projection, then obtains the block gradient weight coefficients of image groups at different scales, and determines the quality score based on the block gradient weight coefficients at different scales and multi-scale features. This approach can, on the one hand, uncover multi-scale visual characteristics, and on the other hand, effectively reflect visual importance, thereby improving the performance of point cloud quality assessment and making the point cloud quality assessment method more suitable for point cloud encoding.

[0122] Furthermore, to illustrate the effectiveness of the point cloud quality assessment method based on projection and multi-scale features provided in this embodiment, the method was tested on a raster point cloud quality assessment database. Pearson correlation coefficient (PLCC), Spearman rank correlation coefficient (SRCC), root mean square error (RMSE), and Kendall rank correlation coefficient (KRCC) were used as evaluation indicators. Specifically, when calculating PLCC and RMSE, a nonlinear regression operation was first performed on the predicted scores using a five-fold cross function. The optimal PLCC and SRCC values ​​for this embodiment on the IRPC, SJTU-PCQA, and SIAT-PCQD databases were 0.9665 and 0.9429, and 0.8790 and 0.8511, and 0.8319 and 0.8121, respectively.

[0123] Furthermore, the performance of the method provided in this embodiment is compared with other mainstream point cloud quality assessment algorithms. These other mainstream algorithms include four quality assessment methods recommended by the Point Cloud Standardization Working Group: D1-MSE, D1-Hausdorff, D2-MSE, and D2-Hausdorff; two model-based full-reference point cloud quality assessment algorithms, GraphSIM and PCQM; one full-reference point cloud projection quality assessment method, SIAT_PCQA; and one deep learning (DL) based point cloud quality assessment method, BQE-CVP. The comparison results are shown in Tables 1, 2, and 3, where the best result in each column is bolded, and the second-best result is underlined. As can be seen from Tables 1, 2, and 3, the evaluation method provided in this embodiment can effectively predict the degradation of point cloud perceived quality compared to other methods, and it shows a high degree of consistency with human subjective ratings.

[0124] Table 1. Performance comparison of the proposed method and six other point cloud quality assessment methods on the IRPC database.

[0125]

[0126] Table 2. Performance comparison of the proposed method with six other point cloud quality assessment methods on the SJTU-PCQA database.

[0127]

[0128] Table 3. Performance comparison of the proposed method with seven other point cloud quality assessment methods on the SIAT-PCQD database.

[0129]

[0130]

[0131] Based on the above-mentioned point cloud quality evaluation method based on projection and multi-scale features, this embodiment provides a point cloud quality evaluation system based on projection and multi-scale features, such as... Figure 9 As shown, the system includes:

[0132] The projection module 100 is used to determine a preset number of image groups based on the reference point cloud and the distorted point cloud. Each image group includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different.

[0133] The weight determination module 200 is used to calculate the texture gradient weight set and geometric gradient weight set of each image group on a per-image-block basis.

[0134] The first calculation module 300 is used to calculate the texture quality score based on each image group and the texture gradient weight set of each image group, and to calculate the geometric quality score based on each image group and the geometric gradient weight set of each image group.

[0135] The second calculation module 400 is used to calculate the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score.

[0136] Based on the above-described point cloud quality evaluation method based on projection and multi-scale features, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the point cloud quality evaluation method based on projection and multi-scale features as described in the above embodiment.

[0137] Based on the aforementioned point cloud quality evaluation method based on projection and multi-scale features, this application also provides a terminal device, such as... Figure 10 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0138] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0139] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0140] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0141] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A point cloud quality evaluation method based on projection and multi-scale features, characterized in that, The method includes: A preset number of image groups are determined based on the reference point cloud and the distorted point cloud. Each image group includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different. Calculate the texture gradient weight set and geometric gradient weight set for each image group on an image patch basis; The texture quality score is calculated based on each image group and the texture gradient weight set of each image group, and the geometric quality score is calculated based on each image group and the geometric gradient weight set of each image group. The calculation of texture quality score based on each image group and the corresponding texture gradient weight set of each image group specifically includes: For each image group, the texture reference image and the texture distortion image are divided into several texture reference image blocks and several texture distortion image blocks according to a preset division method. Based on each texture reference image block, the texture gradient weight set corresponding to the image group, and each texture distortion image block, the texture distortion degree of the image group is calculated. The texture reference image block corresponds one-to-one with the texture distortion image block and the texture image block gradient weight in the texture gradient weight set. The formula for calculating the texture distortion is: in, This represents the texture patch distortion of the i-th texture-distorted image patch in the k-th image group. This represents the texture image patch gradient weight of the i-th texture reference image patch in the k-th image group. This represents the SSIM distortion metric of the i-th texture distortion image patch in the k-th image group relative to the i-th texture reference image patch; Calculate the texture quality score based on the texture distortion of each image group; The formula for calculating the texture quality score is: Among them, Q T This represents the texture quality score, where K represents the number of image groups. Represents the multiplicative coefficients of the k-th image group. This represents the texture distortion of the k-th image group, where i represents the i-th texture reference image patch. This represents the texture distortion of the i-th texture reference image patch in the k-th image group; The calculation process for the geometric quality score is the same as the calculation process for the texture quality score; Based on the texture quality score and the geometric quality score, calculate the quality score corresponding to the distorted point cloud.

2. The point cloud quality evaluation method based on projection and multi-scale features according to claim 1, characterized in that, The determination of a preset number of image groups based on the reference point cloud and the distorted point cloud specifically includes: The projection generates a texture reference map and a geometric reference map corresponding to the reference point cloud, as well as a texture distortion map and a geometric distortion map corresponding to the distorted point cloud, to obtain the first image group; The images in the first image group are downsampled to obtain the second image group; The images in the second image group are downsampled to obtain the third image group; This process continues until a preset number of image groups are obtained.

3. The point cloud quality evaluation method based on projection and multi-scale features according to claim 2, characterized in that, The downsampling operation is a downsampling operation performed using a Gaussian pyramid.

4. The point cloud quality evaluation method based on projection and multi-scale features according to claim 1, characterized in that, The calculation of the texture gradient weight set for each image group on a patch basis specifically includes: For each image group, convert the texture reference map in that image group into a grayscale texture reference map; The grayscale texture reference image is divided into several texture reference image blocks according to a preset division method, and the pixel gradient value of each pixel in each texture reference image block is calculated. The gradient values ​​of each texture reference image block are calculated based on the pixel gradient values ​​of the pixels in each texture reference image block to obtain the texture gradient weight set.

5. The point cloud quality evaluation method based on projection and multi-scale features according to claim 1, characterized in that, The specific steps for calculating the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score are as follows: The texture quality score and the geometric quality score are weighted to obtain the quality score corresponding to the distorted point cloud.

6. A point cloud quality evaluation system based on projection and multi-scale features, characterized in that, The system includes: The projection module is used to determine a preset number of image groups based on the reference point cloud and the distorted point cloud. Each image group includes a texture reference map, a geometric reference map, a texture distortion map, and a geometric distortion map, and the image scales corresponding to each image group are different. The weight determination module is used to calculate the texture gradient weight set and geometric gradient weight set of each image group on a per-image-patch basis. The first calculation module is used to calculate the texture quality score based on each image group and the texture gradient weight set of each image group, and to calculate the geometric quality score based on each image group and the geometric gradient weight set of each image group. The calculation of texture quality score based on each image group and the corresponding texture gradient weight set of each image group specifically includes: For each image group, the texture reference image and the texture distortion image are divided into several texture reference image blocks and several texture distortion image blocks according to a preset division method. Based on each texture reference image block, the texture gradient weight set corresponding to the image group, and each texture distortion image block, the texture distortion degree of the image group is calculated. The texture reference image block corresponds one-to-one with the texture distortion image block and the texture image block gradient weight in the texture gradient weight set. The formula for calculating the texture distortion is: in, This represents the texture patch distortion of the i-th texture-distorted image patch in the k-th image group. This represents the texture image patch gradient weight of the i-th texture reference image patch in the k-th image group. This represents the SSIM distortion metric of the i-th texture distortion image patch in the k-th image group relative to the i-th texture reference image patch; Calculate the texture quality score based on the texture distortion of each image group; The formula for calculating the texture quality score is: Among them, Q T This represents the texture quality score, and K represents the number of image groups. Represents the multiplicative coefficients of the k-th image group. This represents the texture distortion of the k-th image group, where i represents the i-th texture reference image patch. This represents the texture distortion of the i-th texture reference image patch in the i-th image group; The calculation process for the geometric quality score is the same as the calculation process for the texture quality score; The second calculation module is used to calculate the quality score corresponding to the distorted point cloud based on the texture quality score and the geometric quality score.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the point cloud quality evaluation method based on projection and multi-scale features as described in any one of claims 1-5.

8. A terminal device, characterized in that, include: Processor, memory, and communication bus; The memory stores a computer-readable program that can be executed by the processor; The communication bus enables communication between the processor and the memory; When the processor executes the computer-readable program, it implements the steps in the point cloud quality evaluation method based on projection and multi-scale features as described in any one of claims 1-5.

Images