Projection-based point cloud quality evaluation method, device, equipment and storage medium

Abstract

The present application relates to the field of image processing technology and proposes a projection-based point cloud quality assessment method, apparatus, terminal device, and storage medium. The method comprises: obtaining an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained after performing a preset process on the original point cloud; projecting the original point cloud and the target point cloud onto the respective faces of the bounding box of the target point cloud to obtain a plurality of projection images; performing image quality assessment processing on each of the projection images to obtain an image quality score for each of the projection images; and determining a quality assessment result for the target point cloud based on the image quality score of each of the projection images. Compared with traditional point cloud quality assessment methods based on three-dimensional points, the present application can improve the accuracy of quality prediction for point clouds.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a projection-based point cloud quality evaluation method, apparatus, terminal device, and storage medium. Background Art

[0002] A point cloud is a collection of numerous three-dimensional points in space, each of which can contain information such as geometric and texture properties. Processing such as compression, transmission, and rendering can result in various types of point cloud distortion, which can affect human perception. Therefore, it is often necessary to evaluate the quality of processed point clouds and establish objective point cloud quality evaluation models to determine whether the processed point clouds meet quality requirements.

[0003] Traditional point cloud quality assessment methods based on 3D points typically search for matching points in a point cloud pair, calculate point-to-point or point-to-plane distances, and then evaluate the quality of the processed point cloud based on these distances. However, this approach only measures distances in Euclidean space and fails to consider the visual characteristics of the human eye, resulting in low accuracy in point cloud quality prediction. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a projection-based point cloud quality evaluation method, apparatus, terminal device and storage medium, which can improve the accuracy of quality prediction of point clouds.

[0005] A first aspect of an embodiment of the present application provides a projection-based point cloud quality assessment method, comprising:

[0006] Acquire an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained after performing a preset processing on the original point cloud;

[0007] Projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images;

[0008] Performing image quality assessment processing on each of the projection images to obtain an image quality score for each of the projection images;

[0009] A quality evaluation result of the target point cloud is determined according to the image quality scores of the projection images.

[0010] The embodiment of the present application projects the unprocessed original point cloud and the processed target point cloud onto each surface of the point cloud bounding box to obtain multiple projection images; then, image quality assessment processing is performed on each projection image to obtain an image quality score for each projection image; finally, the quality assessment result of the target point cloud is determined based on the image quality scores of each projection image, thereby completing the quality assessment and prediction of the point cloud. In the above process, the point cloud is projected onto each surface of the bounding box to obtain multiple projection images, and then the image quality scores of each projection image are fused to determine the quality of the point cloud. The overall image characteristics of the point cloud are taken into consideration. Compared with the traditional point cloud quality assessment method based on three-dimensional points, the accuracy of the quality prediction of the point cloud can be improved.

[0011] In one embodiment of the present application, projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images may include:

[0012] Projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of texture images, wherein the texture images are two-dimensional images obtained by converting texture attribute data of the point cloud;

[0013] and / or

[0014] The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of geometric images, where the geometric images are two-dimensional images obtained by converting geometric attribute data of the point cloud.

[0015] During projection, the texture attribute data of the point cloud (such as color) can be converted and projected onto the plane of the point cloud bounding box to obtain a two-dimensional projection image, called a texture image. The texture image is used to represent the color and other texture information of the point cloud. The geometric attribute data of the point cloud (such as size and depth) can also be converted and projected onto the plane of the point cloud bounding box to obtain a two-dimensional projection image, called a geometric image. The geometric image is used to represent the size, depth and other geometric information of the point cloud.

[0016] In one embodiment of the present application, determining the quality evaluation result of the target point cloud according to the image quality scores of the projection images may include:

[0017] Determining, according to the image quality scores of the projection images, the quality scores of the respective faces of the bounding box of the target point cloud;

[0018] A weighted sum operation is performed on the quality scores of the respective faces to obtain a quality evaluation parameter of the target point cloud, wherein a corresponding weight when the weighted sum operation is performed on the quality scores of each face is proportional to the area of the face.

[0019] By using multi-view projection and performing view fusion based on area ratio, different views can be given different importance to reflect the saliency characteristics of the human eye.

[0020] In one embodiment of the present application, projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images may include:

[0021] performing clustering processing on each three-dimensional point contained in the original point cloud to obtain a classification result for each three-dimensional point;

[0022] constructing a plurality of connected regions according to the classification results of each of the three-dimensional points;

[0023] Projecting the multiple connected regions onto respective faces of the bounding box of the target point cloud to obtain multiple reference patches, where the reference patches are two-dimensional regions formed when the connected regions are projected onto the bounding box of the target point cloud;

[0024] For each reference patch, a distorted patch having the same shape as the reference patch is constructed, thereby obtaining a plurality of distorted patches, wherein the distorted patch is a two-dimensional area obtained by performing a preset process on the reference patch;

[0025] Filling the plurality of reference patches and the plurality of distorted patches into a preset two-dimensional blank image to obtain a target image;

[0026] The plurality of projection images are generated according to the target image.

[0027] The 3D points of the point cloud are clustered and segmented based on their normal vectors to form sheet-like connected regions. The point cloud is then projected onto a plane, generating multiple sheet-like projections, called patches. The patches generated by projecting the original and target point clouds are then inserted into the same blank image to form texture and geometric images. Image quality metrics are then used to calculate the quality scores of the geometric and texture images to characterize the visual quality of the point cloud.

[0028] Furthermore, performing clustering processing on each 3D point included in the original point cloud to obtain a classification result for each 3D point may include:

[0029] Calculating a normal vector for each three-dimensional point contained in the original point cloud;

[0030] For each of the three-dimensional points, a dot product of the normal vector of the three-dimensional point and the normal vector of each face of the bounding box of the target point cloud is calculated, and the three-dimensional point is classified into a category corresponding to a first target face of the bounding box of the target point cloud, thereby obtaining a classification result for the three-dimensional point, wherein the dot product of the normal vector of the first target face and the normal vector of the three-dimensional point is the largest;

[0031] The constructing of multiple connected regions according to the classification results of the three-dimensional points may include:

[0032] The three-dimensional points having the same classification result and a distance within a set range are classified into the same connected area, thereby constructing multiple connected areas.

[0033] After performing the same operation for all 3D points, all 3D points are classified into the categories corresponding to the faces of the bounding box. This process is also called initial segmentation of the 3D points. When constructing connected regions, all 3D points with the same classification result and close distances are grouped into the same connected region. This setting can be used to divide the original point cloud into multiple different connected regions.

[0034] Furthermore, after classifying the three-dimensional point into the category corresponding to the first target surface of the bounding box of the target point cloud, the method may further include:

[0035] If the classification results of all other 3D points adjacent to the 3D point belong to the category corresponding to the second target surface of the bounding box of the target point cloud, the classification result of the 3D point is modified to the category corresponding to the second target surface.

[0036] After completing the initial classification of a 3D point, obtain the classification results of other 3D points adjacent to the 3D point. If the classification results of these adjacent 3D points all belong to the category corresponding to another surface, then the 3D point is also classified into the category corresponding to the other surface.

[0037] Furthermore, for each reference patch, a distortion patch with the same shape as the reference patch is constructed to obtain multiple distortion patches, which may include:

[0038] For each 3D point in the original point cloud, the nearest neighboring point to the 3D point is searched in the target point cloud, and a corresponding distorted patch is constructed based on the information of the reference patch to which the 3D point belongs, wherein the relative position of the neighboring point in the distorted patch is the same as the relative position of the 3D point in the reference patch to which it belongs.

[0039] For example, for a point p1 in the original point cloud, the nearest neighbor point p2 corresponding to p1 can be found from the target point cloud using methods such as the K-nearest neighbor (K=1) algorithm or the iterative nearest point algorithm; then, the information of the reference patch to which p1 belongs is copied, and a two-dimensional patch (i.e., a distorted patch) with the same shape as the reference patch is constructed based on the information, and point p2 is assigned to the distorted patch, and the relative position of point p2 in the distorted patch is the same as the relative position of point p1 in the reference patch.

[0040] A second aspect of an embodiment of the present application provides a projection-based point cloud quality assessment device, comprising:

[0041] A point cloud acquisition module is used to acquire an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained by performing a preset process on the original point cloud;

[0042] a point cloud projection module, configured to project the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud, to obtain a plurality of projected images;

[0043] An image quality assessment module, configured to perform image quality assessment processing on each of the projection images to obtain an image quality score for each of the projection images;

[0044] The point cloud quality evaluation module is used to determine the quality evaluation result of the target point cloud according to the image quality score of each of the projection images.

[0045] A third aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the projection-based point cloud quality evaluation method provided in the first aspect of the embodiment of the present application are implemented.

[0046] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the projection-based point cloud quality assessment method provided in the first aspect of the embodiment of the present application.

[0047] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of the projection-based point cloud quality assessment method described in the first aspect of the embodiments of the present application.

[0048] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a flowchart of an embodiment of a projection-based point cloud quality assessment method provided in an embodiment of the present application;

[0051] Figure 2 This is a schematic diagram of an implementation principle of using multi-view orthogonal projection and view fusion based on area ratio provided in an embodiment of the present application;

[0052] Figure 3 This is a flowchart of an embodiment of another projection-based point cloud quality assessment method provided in an embodiment of the present application;

[0053] Figure 4 Schematic diagram of the transformation relationship between three-dimensional points and two-dimensional pixels provided in an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of an implementation principle of point cloud quality evaluation using patch projection provided in an embodiment of the present application;

[0055] Figure 6 This is a structural diagram of an embodiment of a projection-based point cloud quality assessment device provided in an embodiment of the present application;

[0056] Figure 7 This is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details. In addition, in the description of the present application specification and the appended claims, the terms "first," "second," "third," etc. are only used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0058] This application proposes a projection-based point cloud quality assessment method, apparatus, terminal device, and storage medium that can improve the accuracy of point cloud quality prediction. It should be understood that the execution entities of the various method embodiments of this application are various types of terminal devices or servers, such as mobile phones, tablets, laptops, desktop computers, and wearable devices.

[0059] See also Figure 1 , shows a projection-based point cloud quality assessment method proposed in an embodiment of the present application, including:

[0060] 101. Acquire an original point cloud and a target point cloud, where the target point cloud is a point cloud obtained by performing a preset process on the original point cloud;

[0061] First, obtain the original point cloud and the target point cloud. The original point cloud is the unprocessed point cloud, and the target point cloud is the point cloud obtained after performing a predetermined processing on the original point cloud. This processing generally results in a loss of visual quality and distortion of the point cloud. For example, compressing the original point cloud, adding noise, and other processing can produce a distorted target point cloud. The quality of this target point cloud is what is evaluated in this application.

[0062] 102. Project the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images;

[0063] The original point cloud and the target point cloud are then projected onto the faces of the target point cloud's bounding box, resulting in multiple projected images. A bounding box is the smallest geometric volume enclosing a point cloud, encompassing all 3D points within the point cloud. A bounding box is a geometric volume slightly larger than a point cloud and possesses simpler properties, allowing it to approximate complex geometric objects.

[0064] During processing, the original point cloud is projected onto each face of the bounding box. Projecting onto each face produces one projected image. Therefore, if the bounding box has N faces, projecting the original point cloud will produce N projected images. Similarly, projecting the target point cloud onto the N faces of the bounding box will also produce N projected images, for a total of 2N projected images. In addition, different projection methods such as orthogonal projection and perspective projection can be used during projection, and this application does not limit this.

[0065] In one implementation of the present application, projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images may include:

[0066] The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of texture images, where the texture images are two-dimensional images obtained by converting texture attribute data of the point cloud.

[0067] During projection, the texture attribute data of the point cloud (such as color) can be converted and projected into the plane of the point cloud bounding box to obtain a two-dimensional projection image, called a texture image. The texture image is used to represent the color and other texture information of the point cloud.

[0068] In another implementation of the present application, projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images may include:

[0069] The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of geometric images, where the geometric images are two-dimensional images obtained by converting geometric attribute data of the point cloud.

[0070] During projection, the geometric attribute data of the point cloud (such as size, depth, etc.) can also be converted and projected into the plane of the point cloud bounding box to obtain a two-dimensional projection image, called a geometric image. The geometric image is used to represent the geometric information of the point cloud, such as size and depth.

[0071] In addition, when projecting, the present application can simultaneously project texture images and geometric images. If set up in this way, assuming that the point cloud bounding box has N faces, when the original point cloud is projected to the N faces of the bounding box, 2N projection images (N texture images and N geometric images) will be generated. When the target point cloud is projected to the N faces of the bounding box, 2N projection images (N texture images and N geometric images) will also be generated, that is, a total of 4N projection images will be generated.

[0072] Specifically, the bounding box can be a rectangular parallelepiped with 6 faces, a regular dodecahedron with 12 faces, or other different types of geometric bodies. The embodiment of the present application does not limit the shape of the bounding box.

[0073] 103. Perform image quality assessment processing on each of the projection images to obtain an image quality score of each of the projection images;

[0074] After obtaining each projected image, various existing image quality assessment methods can be used to perform image quality assessment on each projected image (texture image or geometric image), thereby obtaining a quality score for each projected image. Existing image quality assessment methods mainly include: GMSD, GSM, IFC, IW-SSIM, MS-SSIM, SSIM, VIF, and VSI. This application does not impose any restrictions on the image quality assessment method used.

[0075] 104. Determine a quality evaluation result of the target point cloud according to the image quality score of each of the projection images.

[0076] After obtaining the quality scores of each projection image, the quality parameters of the target point cloud can be determined based on these quality scores, thereby evaluating the quality of the target point cloud and obtaining the quality evaluation results of the target point cloud. For example, by fusing the quality scores of each projection image, a parameter value that can be used to characterize the quality of the target point cloud can be obtained. In addition, by combining the parameter value to perform point cloud quality evaluation, the quality evaluation results of the target point cloud can be obtained, such as whether the quality of the target point cloud is qualified, whether it meets subsequent processing requirements, and so on. Specifically, when fusing the various quality scores, various fusion methods such as multiplication fusion, linear fusion, and nonlinear fusion can be used. In one implementation of the present application, the quality evaluation results of the target point cloud are determined based on the image quality scores of each of the projection images, which may include:

[0077] (1) determining the quality scores of the respective faces of the bounding box of the target point cloud according to the image quality scores of the respective projection images;

[0078] (2) performing a weighted summation operation on the quality scores of the respective faces to obtain a quality evaluation parameter of the target point cloud, wherein the corresponding weight when performing the weighted summation operation on the quality scores of each face is proportional to the area of the face.

[0079] For the above step (1), the quality score of any face of the bounding box of the target point cloud can be determined based on the quality scores of all projection images obtained by projecting onto the face. Assuming that the point cloud bounding box is a cuboid, it has 6 faces (front view, back view, left view, right view, top view and bottom view), and the quality score of any face (for example, the front view) can be expressed as Q i (P i (pc ref ), P i (pc dist )), P i (pc ref ) represents the quality score of the projection image (texture image and / or geometric image) obtained by projecting the original point cloud to the front view, P i (pc dist ) represents the quality score of the projection image (texture image and / or geometric image) obtained by projecting the target point cloud onto the front view. The pc in the brackets ref Represents the original point cloud, pc dist Represents the target point cloud. i (P i (pcref ), P i (pc dist )) can be based on P i (pc ref ) and P i (pc dist ) are calculated, for example, they can be their average value or weighted sum, etc.

[0080] In step (2), after obtaining the quality scores of each face of the bounding box, a weighted sum operation can be performed on these quality scores to obtain the quality evaluation parameters of the target point cloud. The corresponding weight when performing the weighted sum operation on the quality score of each face can be proportional to the area of the face.

[0081] Specifically, the weight corresponding to a certain face (view) can be calculated using the following formula:

[0082]

[0083] Among them, K represents the number of faces (views), area i represents the area of face (view) i, w i Represents the weight of face (view) i. For example, a point cloud with a bounding box of (200, 300, 400) has a front view area of 60000, a left view area of 120000, and a top view area of 80000. The weight corresponding to the front view is the area of the front view divided by the sum of the areas of all views of the bounding box, and so on. Further, when using the above formula to calculate the weight corresponding to a face (view), area i The area shown can be the effective area of the surface (view) excluding the blank part.

[0084] Specifically, the quality evaluation parameters of the target point cloud can be calculated using the following formula:

[0085]

[0086] Among them, S final Represents the quality evaluation parameter for the target point cloud. A larger value indicates a higher quality target point cloud, which can also be understood as less distortion and closer to the original point cloud. The above process uses multi-view orthogonal projection and area-ratio view fusion to assign different importance to different views, reflecting the saliency characteristics of the human eye.

[0087] The schematic diagram of the implementation principle of using multi-view orthogonal projection and view fusion based on area ratio is as follows: Figure 2 As shown. Figure 2In this method, the original point cloud and the target point cloud are first orthogonally projected to obtain six texture images for each view. Existing image quality assessment methods are then used to assess the quality of each texture image and obtain a quality score for each texture image. Next, the quality score for each of the six views is determined based on the quality scores of the texture images. An area-based multi-view fusion method (i.e., a weighted summation method with the weight of each view determined by its area) is then used to fuse the quality scores of each view, ultimately yielding the quality assessment result for the target point cloud.

[0088] In one embodiment of the present application, after obtaining the quality evaluation result of the target point cloud, the parameters for compression, rendering, and other processing of the original point cloud can be adjusted according to the quality evaluation result, so as to obtain a better point cloud processing effect.

[0089] The embodiment of the present application projects the unprocessed original point cloud and the processed target point cloud onto each surface of the point cloud bounding box to obtain multiple projection images; then, image quality assessment processing is performed on each projection image to obtain an image quality score for each projection image; finally, the quality assessment result of the target point cloud is determined based on the image quality scores of each projection image, thereby completing the quality assessment and prediction of the point cloud. In the above process, the point cloud is projected onto each surface of the bounding box to obtain multiple projection images, and then the image quality scores of each projection image are fused to determine the quality of the point cloud. The overall image characteristics of the point cloud are taken into consideration. Compared with the traditional point cloud quality assessment method based on three-dimensional points, the accuracy of the quality prediction of the point cloud can be improved.

[0090] See also Figure 3 , shows another projection-based point cloud quality assessment method proposed in an embodiment of the present application, including:

[0091] 301. Acquire an original point cloud and a target point cloud, where the target point cloud is a point cloud obtained by performing a preset process on the original point cloud;

[0092] Step 301 is the same as step 101 , and the details can be referred to the relevant description of step 101 .

[0093] 302. Perform clustering processing on each 3D point included in the original point cloud to obtain a classification result for each 3D point;

[0094] A point cloud is a collection of numerous 3D points. In step 302, clustering is performed on each of the 3D points in the original point cloud to obtain classification results for each 3D point. After the 3D points are classified, 3D points of the same category are grouped into the same connected region, thereby constructing multiple different connected regions.

[0095] In one embodiment of the present application, performing clustering processing on each 3D point included in the original point cloud to obtain a classification result for each 3D point may include:

[0096] (1) calculating the normal vector of each three-dimensional point contained in the original point cloud;

[0097] (2) For each of the three-dimensional points, the dot product of the normal vector of the three-dimensional point and the normal vector of each face of the bounding box of the target point cloud is calculated respectively, and the three-dimensional point is classified into the category corresponding to the first target face of the bounding box of the target point cloud, thereby obtaining a classification result of the three-dimensional point, wherein the dot product of the normal vector of the first target face and the normal vector of the three-dimensional point is the largest.

[0098] For step (1), various point normal vector calculation methods in the prior art can be used to determine the normal vector of each 3D point in the point cloud. The normal vector of each 3D point in the point cloud is uniquely determined and can be calculated by combining the attribute parameters of other 3D points near the 3D point.

[0099] For the above step (2), for each 3D point, after calculating the normal vector of the 3D point, the dot product of the normal vector of the 3D point and the normal vector of each face of the bounding box of the target point cloud can be calculated respectively, and then the 3D point can be classified into the category corresponding to the face with the largest dot product. Assuming that the point cloud bounding box is a cuboid with 6 faces (front view, back view, left view, right view, top view and bottom view), the dot product of the normal vector of the 3D point and the normal vector of the front view, the dot product of the normal vector of the 3D point and the normal vector of the back view, and the dot product of the normal vector of the 3D point and the normal vector of the left view are calculated respectively... Then, find the largest dot product. Assuming it is the dot product of the normal vector of the 3D point and the normal vector of the front view, it means that the normal vector of the 3D point is closest to the orientation of the front view, so the 3D point is classified into the category corresponding to the front view. After performing the same operation on all 3D points, the process of classifying all 3D points into the categories corresponding to the faces of the bounding box is completed. This process can also be called the initial segmentation of the 3D points.

[0100] Furthermore, after classifying the three-dimensional point into the category corresponding to the first target surface of the bounding box of the target point cloud, the method may further include:

[0101] If the classification results of all other 3D points adjacent to the 3D point belong to the category corresponding to the second target surface of the bounding box of the target point cloud, the classification result of the 3D point is modified to the category corresponding to the second target surface.

[0102] After the initial segmentation process, further refinement can be performed. Specifically, after completing the initial classification of a 3D point, the classification results of all adjacent 3D points are obtained. If the classification results of these adjacent 3D points all belong to the category corresponding to the other face, the 3D point is also classified into the category corresponding to the other face. Alternatively, a quantity threshold or ratio threshold can be set. If a specified number or ratio of these adjacent 3D points are detected to belong to the category corresponding to the other face, the classification result of the 3D point is modified.

[0103] 303. Construct multiple connected regions according to the classification results of the three-dimensional points;

[0104] After obtaining the classification results of each 3D point in the original point cloud using the above method, multiple connected regions can be constructed based on these classification results. In one implementation of the present application, the construction of multiple connected regions based on the classification results of each 3D point can include:

[0105] The three-dimensional points having the same classification result and a distance within a set range are classified into the same connected area, thereby constructing multiple connected areas.

[0106] When constructing connected regions, all 3D points with the same classification result and close distances are grouped into the same connected region. This allows the original point cloud to be divided into multiple connected regions. The distance range threshold can be appropriately set based on the overall size of the original point cloud and the density of the 3D points.

[0107] 304. Project the multiple connected regions onto respective faces of the bounding box of the target point cloud to obtain multiple reference patches, where the reference patches are two-dimensional regions formed when the connected regions are projected onto the bounding box of the target point cloud.

[0108] After dividing multiple connected regions, these connected regions are projected onto the faces of the bounding box of the target point cloud. Specifically, for each connected region, the projection is performed according to the orientation of the normal vector of the bounding box plane corresponding to the classification results of the 3D points within the connected region. In other words, each connected region is only projected onto the face corresponding to the classification results of the 3D points it contains. When a connected region is projected onto a face of the bounding box, a sheet-like 2D projection area is generated, which is referred to as a reference patch in this application.

[0109] The above steps describe the spatial transformation relationship of projecting 3D points to 2D space. 3D points in a connected region have normal vectors with similar directions. The transformation relationship between 3D points and 2D pixels is as follows: Figure 4As shown in the figure, the vertices of the connected area where the 3D point (x', y', z') is located are (a, b, c). Taking the projection onto the YOZ plane as an example, the projection is recorded as pixels (x, y). The vertices of the 2D patch generated after projection are (u0, υ0), and the length and width of the patch are u1 and υ1 respectively. The geometric relationship between the 3D point (x', y', z') and the projected pixel (x, y) is:

[0110]

[0111] The information of the two-dimensional patch may include u0, υ0, u1, υ1, etc., as well as information of all three-dimensional points contained in the two-dimensional patch.

[0112] 305. For each reference patch, construct a distorted patch with the same shape as the reference patch, thereby obtaining multiple distorted patches, where the distorted patch is a two-dimensional area obtained by performing a preset process on the reference patch;

[0113] Then, for each reference patch, a distorted patch with the same shape is constructed. That is, the number and shape of the reference patches and the distorted patches are the same. The distorted patch is a two-dimensional sheet area constructed from the target point cloud and corresponding to the reference patch.

[0114] In one implementation of the present application, for each reference patch, a distortion patch having the same shape as the reference patch is constructed, thereby obtaining multiple distortion patches, which may include:

[0115] For each 3D point in the original point cloud, the nearest neighboring point to the 3D point is searched in the target point cloud, and a corresponding distorted patch is constructed based on the information of the reference patch to which the 3D point belongs, wherein the relative position of the neighboring point in the distorted patch is the same as the relative position of the 3D point in the reference patch to which it belongs.

[0116] For example, for a point p1 in the original point cloud, the nearest neighbor point p2 corresponding to p1 can be found from the target point cloud using algorithms such as the K-nearest neighbor (K=1) algorithm or the iterative nearest point algorithm; then, the information of the reference patch to which p1 belongs is copied, and a two-dimensional patch (i.e., a distorted patch) with the same shape as the reference patch is constructed based on the information, and point p2 is assigned to the distorted patch, and the relative position of point p2 in the distorted patch is the same as the relative position of point p1 in the reference patch.

[0117] In another implementation of the present application, when establishing a point matching relationship between the original point cloud and the target point cloud, a method of searching for matching points of the original point cloud in the target point cloud can also be used. That is, steps 302-304 can process the target point cloud. In this case, in step 305, a reference patch of the same shape can be constructed based on the distorted patch obtained from the target point cloud.

[0118] 306. Fill the plurality of reference patches and the plurality of distorted patches into a preset two-dimensional blank image to obtain a target image;

[0119] Next, the constructed multiple reference patches and multiple distorted patches are filled into a preset two-dimensional blank image to obtain a target image. The blank image can be an image of a certain size reasonably set according to the size of the original point cloud.

[0120] Furthermore, in order to avoid wasting image space, when filling patches, each patch (including reference patches and distorted patches) can be sorted from large to small, and then inserted into the blank image in order, that is, large patches are inserted first, and then small patches are inserted.

[0121] 307. Generate multiple projection images according to the target image;

[0122] After obtaining the target image, multiple projection images are generated based on the target image. These projection images may include texture images and geometric images. Specifically, based on the color attribute data of each 3D point in each reference patch in the target image, a corresponding texture image can be generated, which can be expressed as T1(pc ref ); Based on the depth value of each 3D point in each reference patch in the target image, the corresponding geometric image can be generated, which can be expressed as G1(pc ref ); Based on the color attribute data of each 3D point in each distortion patch in the target image, a corresponding texture image can be generated, which can be expressed as T2(pc ref , pc dist ); Based on the depth value of each three-dimensional point in each distorted patch in the target image, the corresponding geometric image can be generated, which can be expressed as G2(pc ref , pc dist When generating a texture image, the pixel values can be filled in according to the color attributes of the three-dimensional point (x′, y′, z′) corresponding to the pixel (x, y) in the target image, thereby obtaining a texture image. When generating a geometric image, the pixel brightness values can be filled in according to the depth value x′-a of the three-dimensional point (x′, y′, z′) from the bounding box (taking projection onto the YOZ plane as an example), thereby obtaining a geometric image.

[0123] 308. Perform image quality assessment processing on each of the projected images to obtain an image quality score for each of the projected images;

[0124] After obtaining projection images such as texture maps and geometric maps, image quality assessment processing is performed on each projection image to obtain an image quality score for each projection image. The image quality assessment method used here can also adopt various methods in the existing technology.

[0125] 309. Determine a quality evaluation result of the target point cloud according to the image quality score of each of the projection images.

[0126] Finally, the image quality scores of each projection image are combined to determine the quality evaluation result of the target point cloud. For example, the quality scores of each projection image can be fused to obtain a parameter that can be used to characterize the quality of the target point cloud. The quality of the target point cloud is then evaluated based on this parameter to obtain the final quality evaluation result.

[0127] In one implementation of the present application, determining the quality evaluation result of the target point cloud according to the image quality score of each of the projection images may include:

[0128] The following formula is used to fuse the quality scores of the texture image and the geometric image to obtain the quality evaluation parameters of the target point cloud:

[0129] S final =a·Q(T1(pc ref ), T2(pc ref , pc dist ))+b·Q(G1(pc ref ), G2(pc ref , pc dist ))

[0130] Among them, S final Indicates the quality evaluation parameter of the target point cloud. The larger the quality evaluation parameter, the higher the quality of the target point cloud obtained. It can also be understood as the smaller the distortion of the target point cloud, that is, the closer it is to the original point cloud. ref ) represents the texture image obtained by projecting the original point cloud, T2(pc ref , pc dist ) represents the texture image obtained by projecting the target point cloud; G1(pc ref ) represents the geometric image obtained by projecting the original point cloud, G2(pc ref , pc dist ) represents the geometric image obtained by projecting the target point cloud, Q(*) represents the full reference image quality evaluation to obtain the corresponding image quality score; a and b are preset weight parameters used to adjust the weights of texture features and geometric features, for example, they can be set to a=0.6, b=0.4, or a=0.5, b=0.5, and so on.

[0131] The schematic diagram of the implementation principle of using patch projection to evaluate point cloud quality is as follows: Figure 5 As shown. Figure 5 In this paper, we first perform point clustering on each 3D point of the original point cloud to construct each connected area; then, we project the connected area onto the surface of the point cloud bounding box, and generate the corresponding reference patch by combining the patch information and the 3D point set in the connected area; then, we search for the corresponding adjacent points from the 3D point set of the target point cloud to complete the corresponding point matching, and then, we generate the distorted patch by replacing the corresponding points with the patch information of the reference patch; then, we generate the corresponding texture image T1 (pc) based on the reference patch. ref ) and the geometric image G1(pc ref ), based on the distortion patch, the corresponding texture image T2 (pc ref , pc dist ) and the geometric image G2(pc ref , pc dist ); Finally, the existing image quality evaluation method is used to perform image quality evaluation on each texture image and geometric image, and the quality scores of the texture image and the geometric image are fused to obtain the quality evaluation result of the target point cloud.

[0132] In an embodiment of the present application, the three-dimensional points of the point cloud are clustered and segmented according to the normal vector to form sheet-like connected areas, and then the point cloud is projected onto a plane to obtain multiple sheet-like projections, called patches. Then, the patches obtained by projecting the original point cloud and the target point cloud are inserted into the same blank image to form a texture image and a geometric image, and the quality scores of the geometric image and the texture image are calculated using an image quality index to characterize the visual quality of the point cloud. In addition, in order to solve the problem of patch mismatching, an embodiment of the present application also proposes a matching patch generation method based on the correspondence between the original point cloud and the target point cloud. The shape and position of the distorted patch obtained by this method will be the same as the reference patch, that is, the images formed by the original point cloud and the target point cloud through the patch projection have the same contour shape.

[0133] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0134] The above mainly describes a projection-based point cloud quality evaluation method. The following will describe a projection-based point cloud quality evaluation device.

[0135] See also Figure 6 In one embodiment of the present application, a projection-based point cloud quality assessment device includes:

[0136] The point cloud acquisition module 601 is used to acquire an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained by performing a preset process on the original point cloud;

[0137] A point cloud projection module 602 is configured to project the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projection images.

[0138] An image quality assessment module 603 is configured to perform image quality assessment on each of the projection images to obtain an image quality score for each of the projection images;

[0139] The point cloud quality evaluation module 604 is configured to determine a quality evaluation result of the target point cloud according to the image quality scores of the projection images.

[0140] In one embodiment of the present application, the point cloud projection module may include:

[0141] a texture projection unit, configured to project the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud, to obtain a plurality of texture images, wherein the texture images are two-dimensional images obtained by converting texture attribute data of the point cloud;

[0142] The geometric projection unit is used to project the original point cloud and the target point cloud onto the respective faces of the bounding box of the target point cloud to obtain a plurality of geometric images, which are two-dimensional images obtained by converting the geometric attribute data of the point cloud.

[0143] Furthermore, the point cloud quality evaluation module may include:

[0144] a quality score determining unit, configured to determine a quality score of each face of the bounding box of the target point cloud according to the image quality score of each of the projection images;

[0145] A weighted summation unit is used to perform a weighted summation operation on the quality scores of the respective faces to obtain a quality evaluation parameter of the target point cloud, wherein the corresponding weight when the weighted summation operation is performed on the quality scores of each face is proportional to the area of the face.

[0146] In one embodiment of the present application, the point cloud projection module may include:

[0147] a point clustering unit, configured to perform clustering processing on each 3D point contained in the original point cloud to obtain a classification result for each 3D point;

[0148] A connected region construction unit, configured to construct a plurality of connected regions according to the classification results of the three-dimensional points;

[0149] a patch projection unit, configured to project the plurality of connected regions onto respective faces of a bounding box of the target point cloud, to obtain a plurality of reference patches, wherein the reference patches are two-dimensional regions formed when the connected regions are projected onto the bounding box of the target point cloud;

[0150] a distortion patch construction unit, configured to construct, for each reference patch, a distortion patch of the same shape as the reference patch, thereby obtaining a plurality of distortion patches, wherein the distortion patch is a two-dimensional area obtained by performing a preset process on the reference patch;

[0151] a patch filling unit, configured to fill the plurality of reference patches and the plurality of distorted patches into a preset two-dimensional blank image to obtain a target image;

[0152] A projection image generating unit is configured to generate the plurality of projection images according to the target image.

[0153] Furthermore, the point clustering unit may include:

[0154] a normal vector calculation subunit, configured to calculate a normal vector for each three-dimensional point contained in the original point cloud;

[0155] a dot product calculation subunit, configured to calculate, for each of the three-dimensional points, a dot product of a normal vector of the three-dimensional point and a normal vector of each face of a bounding box of the target point cloud, and classify the three-dimensional point into a category corresponding to a first target face of the bounding box of the target point cloud, thereby obtaining a classification result for the three-dimensional point, wherein the dot product of the normal vector of the first target face and the normal vector of the three-dimensional point is the largest;

[0156] The connected region construction unit may include:

[0157] The three-dimensional point classification subunit is used to classify the three-dimensional points with the same classification result and a distance within a set range into the same connected area, thereby constructing multiple connected areas.

[0158] Furthermore, the point clustering unit may further include:

[0159] The three-dimensional point category modification subunit is used to modify the classification result of the three-dimensional point to the category corresponding to the second target surface if the classification results of each other three-dimensional point adjacent to the three-dimensional point all belong to the category corresponding to the second target surface of the bounding box of the target point cloud.

[0160] Furthermore, the distortion patch construction unit may include:

[0161] A neighbor point search subunit is used to search for the nearest neighbor point to each 3D point in the original point cloud in the target point cloud, and to construct a corresponding distorted patch based on the information of the reference patch to which the 3D point belongs, wherein the relative position of the neighbor point in the distorted patch is the same as the relative position of the 3D point in the reference patch to which it belongs.

[0162] The embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following Figure 1 or Figure 3 The steps of any projection-based point cloud quality assessment method are represented.

[0163] The embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the following Figure 1 or Figure 3 The steps of any projection-based point cloud quality assessment method are represented.

[0164] Figure 7 This is a schematic diagram of a terminal device provided by an embodiment of the present application. Figure 7 As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned embodiments of the projection-based point cloud quality assessment method are implemented, for example Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 6 Functions of modules 601 to 604 are shown.

[0165] The computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the terminal device 7.

[0166] The processor 70 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0167] The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Furthermore, the memory 71 may include both an internal storage unit of the terminal device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the terminal device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0169] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0172] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0174] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. . The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0176] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A point cloud quality evaluation method based on projection, characterized in that: include: Acquire an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained after performing a preset processing on the original point cloud; Projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images; Performing image quality assessment processing on each of the projection images to obtain an image quality score for each of the projection images; Determining a quality evaluation result of the target point cloud according to the image quality scores of each of the projection images; Wherein, determining the quality evaluation result of the target point cloud according to the image quality score of each of the projection images includes: Determining, according to the image quality scores of the projection images, the quality scores of the respective faces of the bounding box of the target point cloud; A weighted sum operation is performed on the quality scores of the respective faces to obtain a quality evaluation parameter of the target point cloud, wherein a corresponding weight when the weighted sum operation is performed on the quality scores of each face is proportional to the area of the face.

2. The method according to claim 1, wherein The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images, including: Projecting the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud to obtain a plurality of texture images, wherein the texture images are two-dimensional images obtained by converting texture attribute data of the point cloud; and / or The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of geometric images, where the geometric images are two-dimensional images obtained by converting geometric attribute data of the point cloud.

3. The method according to claim 1, wherein The original point cloud and the target point cloud are projected onto respective faces of a bounding box of the target point cloud to obtain a plurality of projected images, including: performing clustering processing on each three-dimensional point contained in the original point cloud to obtain a classification result for each three-dimensional point; constructing a plurality of connected regions according to the classification results of each of the three-dimensional points; Projecting the multiple connected regions onto respective faces of the bounding box of the target point cloud to obtain multiple reference patches, where the reference patches are two-dimensional regions formed when the connected regions are projected onto the bounding box of the target point cloud; For each reference patch, a distorted patch having the same shape as the reference patch is constructed, thereby obtaining a plurality of distorted patches, wherein the distorted patch is a two-dimensional area obtained by performing a preset process on the reference patch; Filling the plurality of reference patches and the plurality of distorted patches into a preset two-dimensional blank image to obtain a target image; The plurality of projection images are generated according to the target image.

4. The method according to claim 3, wherein Performing clustering processing on each 3D point contained in the original point cloud to obtain a classification result for each 3D point includes: Calculating a normal vector for each three-dimensional point contained in the original point cloud; For each of the three-dimensional points, a dot product of the normal vector of the three-dimensional point and the normal vector of each face of the bounding box of the target point cloud is calculated, and the three-dimensional point is classified into a category corresponding to a first target face of the bounding box of the target point cloud, thereby obtaining a classification result for the three-dimensional point, wherein the dot product of the normal vector of the first target face and the normal vector of the three-dimensional point is the largest; The step of constructing a plurality of connected regions according to the classification results of the three-dimensional points includes: The three-dimensional points having the same classification result and a distance within a set range are classified into the same connected area, thereby constructing multiple connected areas.

5. The method according to claim 4, wherein After classifying the three-dimensional point into the category corresponding to the first target surface of the bounding box of the target point cloud, the method further includes: If the classification results of all other 3D points adjacent to the 3D point belong to the category corresponding to the second target surface of the bounding box of the target point cloud, the classification result of the 3D point is modified to the category corresponding to the second target surface.

6. The method according to any one of claims 3 to 5, characterized in that For each reference patch, a distortion patch with the same shape as the reference patch is constructed, thereby obtaining multiple distortion patches, including: For each 3D point in the original point cloud, the nearest neighboring point to the 3D point is searched in the target point cloud, and a corresponding distorted patch is constructed based on the information of the reference patch to which the 3D point belongs, wherein the relative position of the neighboring point in the distorted patch is the same as the relative position of the 3D point in the reference patch to which it belongs.

7. A point cloud quality evaluation device based on projection, characterized in that: include: A point cloud acquisition module is used to acquire an original point cloud and a target point cloud, wherein the target point cloud is a point cloud obtained by performing a preset process on the original point cloud; a point cloud projection module, configured to project the original point cloud and the target point cloud onto respective faces of a bounding box of the target point cloud, to obtain a plurality of projected images; An image quality assessment module, configured to perform image quality assessment processing on each of the projection images to obtain an image quality score for each of the projection images; a point cloud quality evaluation module, configured to determine a quality evaluation result of the target point cloud according to the image quality scores of the projection images; Wherein, the point cloud quality evaluation module includes: a quality score determining unit, configured to determine a quality score of each face of the bounding box of the target point cloud according to the image quality score of each of the projection images; A weighted summation unit is used to perform a weighted summation operation on the quality scores of the respective faces to obtain a quality evaluation parameter of the target point cloud, wherein the corresponding weight when the weighted summation operation is performed on the quality scores of each face is proportional to the area of the face.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the projection-based point cloud quality assessment method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the projection-based point cloud quality assessment method according to any one of claims 1 to 6 are implemented.

Images