A point cloud quality assessment method, device and electronic device

Abstract

The present application provides a point cloud quality evaluation method, apparatus and electronic device. The method includes: extracting a plurality of feature points from multi-view images of a target object; calculating an error value between the plurality of feature points and a point cloud to be evaluated of the target object; and performing quality evaluation on the point cloud to be evaluated according to the error value to obtain an evaluation result. In the implementation process of the above solution, the quality evaluation of the point cloud to be evaluated is performed through the error value between the plurality of feature points extracted from the multi-view images of the target object and the point cloud to be evaluated of the target object. Thus, without the need for the real point cloud data of the target object, a quality evaluation result can also be obtained, effectively improving the situation where the quality of the point cloud cannot be evaluated due to the difficulty in obtaining the real point cloud data of the target object, and improving the adaptability of the point cloud quality evaluation method.

Description

Technical Field

[0001] This application relates to the technical field of point cloud data processing. Specifically, it relates to a method, device, and electronic device for evaluating point cloud quality. Background Art

[0002] Currently, most methods for evaluating point cloud quality use the real point cloud data of the target object as a reference gold standard for evaluation. For example: using a laser scanner, stereophotogrammetry instrument, binocular camera, or lidar to scan the target object to obtain the real point cloud data of the target object, and evaluating the quality of the point cloud data to be evaluated by comparing and analyzing the point cloud data to be evaluated with the real point cloud data. However, in some special scenarios, such as scenarios where it is difficult to obtain the real point cloud data of the target object, it is impossible to evaluate the point cloud quality at this time. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a method, device, and electronic device for evaluating point cloud quality, which is used to improve the problem that the real point cloud data of the target object cannot be evaluated due to the difficulty of obtaining it.

[0004] The embodiments of this application provide a method for evaluating point cloud quality, including: extracting a plurality of feature points from multi-view images of the target object; calculating the error values between the plurality of feature points and the point cloud to be evaluated of the target object; and performing quality evaluation on the point cloud to be evaluated according to the error values to obtain an evaluation result. In the implementation process of the above solution, the quality of the point cloud to be evaluated is evaluated through the error values between the plurality of feature points extracted from the multi-view images of the target object and the point cloud to be evaluated of the target object. Therefore, without the real point cloud data of the target object, a quality evaluation result can also be obtained, effectively improving the situation where the real point cloud data of the target object cannot be evaluated due to the difficulty of obtaining it, and improving the adaptability of the point cloud quality evaluation method.

[0005] Optionally, in the embodiments of this application, extracting a plurality of feature points from multi-view images of the target object includes: using a 3D reconstruction tool to process the multi-view images of the target object to obtain a processing result; and extracting a plurality of feature points from the processing result. In the implementation process of the above solution, a plurality of feature points are accurately extracted from the multi-view images of the target object through a 3D reconstruction tool (such as COLMAP), thereby retaining the detailed features in the original multi-view images. Therefore, the accuracy of point cloud quality evaluation is effectively improved.

[0006] Optionally, in the embodiments of the present application, before calculating the error values between multiple feature points and the point cloud to be evaluated of the target object, it further includes: obtaining the deviation value of each feature point among the multiple feature points from the processing result; filtering the multiple feature points according to the deviation value. In the implementation process of the above solution, outliers with large deviations are removed by filtering. These deviation values may be caused by factors such as matching errors, image noise, and occlusion. Then, the multiple feature points are filtered according to the deviation values, thereby improving the adaptability of the point cloud quality evaluation method to different lighting conditions, perspective changes, or image quality differences.

[0007] Optionally, in the embodiments of the present application, the multi-view pictures include: target view pictures; calculating the error values between multiple feature points and the point cloud to be evaluated of the target object includes: for each feature point among the multiple feature points, obtaining the three-dimensional coordinates of the feature point and the target pixel coordinates of the feature point in the target view picture from the processing result; screening out the target three-dimensional point closest to the three-dimensional coordinates of the feature point from the point cloud to be evaluated; using a rendering algorithm to map the target three-dimensional point to the multi-view pictures to obtain the mapped point coordinates; calculating the coordinate error value between the target pixel coordinates and the mapped point coordinates. In the implementation process of the above solution, by mapping the target three-dimensional point closest to the three-dimensional coordinates of the feature point to the mapped point coordinates of the target view picture, the coordinate error value between the mapped point coordinates and the target pixel coordinates of the feature point in the target view picture is calculated, thereby improving the matching accuracy between the point cloud data and the image data.

[0008] Optionally, in the embodiments of the present application, calculating the error values between multiple feature points and the point cloud to be evaluated of the target object includes: converting each feature point among the multiple feature points into three-dimensional coordinates in the world coordinate system; calculating the error values between the three-dimensional coordinates and the world coordinates of the point cloud to be evaluated. In the implementation process of the above solution, by converting the feature points into three-dimensional coordinates in the world coordinate system and calculating the error values between these coordinates and the world coordinates of the point cloud to be evaluated, the accuracy of accurately evaluating the point cloud data can be improved.

[0009] Optionally, in the embodiments of the present application, performing quality evaluation on the point cloud to be evaluated according to the error values includes: counting the number of differences less than the threshold among the multiple error values; determining the accuracy value in the evaluation result according to the number of differences less than the threshold and the number of the multiple error values. In the implementation process of the above solution, by statistically calculating the proportion of the number of error values less than the set threshold in the total error values, the matching quality of the point cloud data can be intuitively quantified, thereby improving the intuitive effect of the point cloud quality evaluation.

[0010] Optionally, in the embodiments of the present application, after obtaining the evaluation result, it further includes: determining the image perspective with the largest number of feature points in the multi-perspective images as the target perspective; screening out the adjacent perspective images with the smallest angle with the target perspective from the multi-perspective images, and determining the image perspective of the adjacent perspective images as the source perspective; visually outputting the evaluation result according to the source perspective and the target perspective. In the implementation process of the above solution, by identifying and selecting the image perspective with the largest number of feature points as the target perspective, and determining the image perspective of the adjacent perspective images as the source perspective, the image data with the richest information and the most representativeness can be preferentially processed, so that the data utilization efficiency can be maximized under limited computing resources, and the processing speed and accuracy can be improved.

[0011] The embodiments of the present application further provide a point cloud quality evaluation device, including: a feature point extraction module, configured to extract a plurality of feature points from multi-perspective images of a target object; an error value calculation module, configured to calculate the error values between the plurality of feature points and the point cloud to be evaluated of the target object; an evaluation result obtaining module, configured to perform quality evaluation on the point cloud to be evaluated according to the error values to obtain an evaluation result.

[0012] Optionally, in the embodiments of the present application, the feature point extraction module includes: an image feature point extraction sub-module, configured to process the multi-perspective images of the target object using a 3D reconstruction tool to obtain a processing result; and extracting a plurality of feature points from the processing result.

[0013] Optionally, in the embodiments of the present application, the point cloud quality evaluation device further includes: a deviation value calculation module, configured to obtain the deviation value of each feature point among the plurality of feature points from the processing result; and a feature point filtering module, configured to filter the plurality of feature points according to the deviation value.

[0014] Optionally, in the embodiments of the present application, the multi-perspective images include: target perspective images; the error value calculation module includes: a feature point coordinate acquisition module, configured to, for each feature point among the plurality of feature points, obtain the three-dimensional coordinates of the feature point and the target pixel coordinates of the feature point in the target perspective image from the processing result; an evaluation point cloud screening module, configured to screen out the target three-dimensional points with the closest distance to the three-dimensional coordinates of the feature point from the point cloud to be evaluated; a mapped point coordinate obtaining sub-module, configured to map the target three-dimensional points to the target perspective image using a rendering algorithm to obtain mapped point coordinates; and a coordinate error value calculation sub-module, configured to calculate the coordinate error value between the target pixel coordinates and the mapped point coordinates.

[0015] Optionally, in the embodiments of the present application, the error value calculation module includes: a three-dimensional coordinate conversion sub-module for converting each feature point among multiple feature points into three-dimensional coordinates in the world coordinate system; a point cloud error calculation sub-module for calculating the error value between the three-dimensional coordinates and the world coordinates of the point cloud to be evaluated.

[0016] Optionally, in the embodiments of the present application, the evaluation result obtaining module includes: a difference quantity statistics sub-module for counting the number of differences less than the threshold from multiple error values; an evaluation result determination sub-module for determining the precision value in the evaluation result according to the number of differences less than the threshold and the number of multiple error values.

[0017] Optionally, in the embodiments of the present application, the point cloud quality evaluation device further includes: a target perspective determination module for determining the picture perspective with the largest number of feature points in the multi-perspective pictures as the target perspective; a source perspective determination module for screening out the adjacent perspective pictures with the smallest included angle with the target perspective from the multi-perspective pictures and determining the picture perspective of the adjacent perspective pictures as the source perspective; a visualization output module for visually outputting the evaluation result according to the source perspective and the target perspective.

[0018] The embodiments of the present application further provide an electronic device, including: a processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are run by the processor, the methods described above are executed.

[0019] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by the processor, the methods described above are executed.

[0020] The embodiments of the present application further provide a computer program product, including: a computer program or computer instructions, and when the computer program or computer instructions are run by the processor, the methods described above are executed. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 A flowchart showing the point cloud quality evaluation method provided by the embodiments of the present application;

[0023] Figure 2 A schematic diagram showing the visualization result provided by the embodiments of the present application;

[0024] Figure 3 Schematic flowchart of the point cloud quality assessment device provided by the embodiment of the present application shown;

[0025] Figure 4 Schematic structural diagram of the electronic device provided by the embodiment of the present application shown. Detailed implementation manners

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the accompanying drawings in the embodiments of the present application are only for the purposes of illustration and description, and are not used to limit the protection scope of the embodiments of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual scale. The flowcharts used in the embodiments of the present application show the operations implemented according to some embodiments of the embodiments of the present application. It should be understood that the operations in the flowchart may not be implemented in sequence, and the steps without logical context relationships may be reversed or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the embodiments of the present application.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the accompanying drawings here may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but merely represents the selected embodiments of the present application.

[0028] It can be understood that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art can understand that the words such as "first" and "second" do not limit the quantity and execution order, and the words such as "first" and "second" do not necessarily limit being different. In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. The term "multiple" refers to two or more (including two). Similarly, "multiple groups" refers to two or more groups (including two groups).

[0029] Before introducing the point cloud quality assessment method provided by the embodiments of the present application, some concepts involved in the embodiments of the present application will be introduced first:

[0030] Point clouds are a typical type of 3D data, which are datasets composed of a large number of discrete data points in 3D space. These points are used to represent the surface or volume information of an object, and the position of each point in space is precisely defined by the three-dimensional coordinates (X, Y, Z) in the Cartesian coordinate system. In addition to the position information, the points in the point cloud can also contain other attributes, such as color information (R, G, B), intensity values (representing the reflection characteristics of the surface material), normals (indicating the surface direction of the point), etc.

[0031] It should be noted that the point cloud quality assessment method provided in the embodiments of this application can be executed by an electronic device. Here, the electronic device refers to a device terminal or a server with the function of executing computer programs. Examples of device terminals include smartphones, personal computers, tablets, personal digital assistants, or mobile Internet devices, etc. A server refers to a device that provides computing services through a network. Examples of servers include x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0032] The following introduces example application scenarios applicable to this point cloud quality assessment method: virtual reality, autonomous driving, medical imaging, 3D reconstruction, and 3D simulation modeling, etc. In application scenarios such as virtual reality, autonomous driving, and medical imaging, point cloud data usually distorts during compression (such as lossy compression), transmission, or reconstruction. At this time, this point cloud quality assessment method can be used to evaluate the quality of the distorted point cloud data, so as to ensure the use of point cloud data with higher quality to improve the user experience. In the application scenarios of 3D reconstruction and 3D simulation modeling, for example, a model application program (APP) for generating a model composed of 3D point clouds. The target users of this application program are mostly ordinary mobile phone users. These ordinary mobile phone users usually do not have professional devices (such as lidar) to obtain real point cloud data (that is, the original point cloud data directly scanned by professional devices), but they need to evaluate the quality of the point cloud data to be evaluated. For example, it is necessary to evaluate the quality of a 3D model composed of indoor decoration point cloud data. In this case, this point cloud quality assessment method can be used to evaluate the quality of this point cloud data based on multi-view images obtained by the mobile phone.

[0033] Please refer to Figure 1 the flowchart showing the point cloud quality assessment method provided in the embodiments of this application; The embodiments of this application provide a point cloud quality assessment method, including:

[0034] Step S110: Extract multiple feature points from the multi-view images of the target object.

[0035] The target object is the target object or target organism whose point cloud needs to be evaluated, such as ancient building sites, ancient forest trees, geological terrains, coral reefs, and so on. The multi-view images of the target object are images obtained by shooting the target object from multiple different angles with the same camera or multiple cameras. Specifically, it can be a set of multi-view images or multiple sets of multi-view images. If it is multiple sets of multi-view images, the above-mentioned point cloud quality evaluation method can be used for quality evaluation for each set of the multiple sets of multi-view images.

[0036] Step S120: Calculate the error values between multiple feature points and the point cloud to be evaluated of the target object.

[0037] It can be understood that there are many ways to calculate the error values between multiple feature points and the point cloud to be evaluated of the target object. It can be the error value between the mapped point coordinates of the point cloud to be evaluated mapped to the multi-view image and the pixel coordinates of the feature points in the multi-view image, or it can be the error value between the three-dimensional coordinates converted from the feature points and the world coordinates of the point cloud to be evaluated.

[0038] Step S130: Perform quality evaluation on the point cloud to be evaluated according to the error values to obtain an evaluation result.

[0039] It can be understood that the calculation process of the above point cloud quality evaluation method does not require manual participation and can directly obtain the evaluation result of the quality evaluation. Compared with the quality evaluation method relying on manual participation, the automation degree is effectively improved.

[0040] In the implementation process of the above solution, the quality of the point cloud to be evaluated is evaluated through the error values between multiple feature points extracted from the multi-view images of the target object and the point cloud to be evaluated of the target object. Thus, without the real point cloud data of the target object, the quality evaluation result can also be obtained, effectively improving the situation where the quality of the point cloud cannot be evaluated due to the difficulty in obtaining the real point cloud data of the target object, and improving the adaptability of the point cloud quality evaluation method.

[0041] As an optional implementation manner of the above step S110, the above implementation manner of extracting multiple feature points from the multi-view images of the target object may include:

[0042] Step S111: Use a three-dimensional reconstruction tool to process the multi-view images of the target object to obtain a processing result;

[0043] Step S112: Extract multiple feature points from the processing result.

[0044] It can be understood that the above three-dimensional reconstruction tool can be a three-dimensional reconstruction tool such as COLMAP (Structure from Motion and Multi-View Stereo). This tool combines Structure from Motion (SFM) and Multi-View Stereo (MVS). This algorithm can extract multiple feature points from multi-view pictures of the target object.

[0045] For example, in the implementation of the above steps S111 to S112: After calculating the multi-view pictures of the target object using the COLMAP algorithm, some intermediate processing results may be obtained. The processing results may include, for example: the pixel coordinates of the feature points in each multi-view picture (such as the target pixel coordinates of the feature points in the target view picture), the three-dimensional coordinates of the feature points, and the pose information corresponding to each picture, etc.

[0046] In the implementation process of the above solution, multiple feature points are accurately extracted from the multi-view pictures of the target object through the COLMAP algorithm, thus retaining the detailed features in the original multi-view pictures. Therefore, the accuracy of point cloud quality assessment is effectively improved.

[0047] As another alternative implementation of the above step S110, for example: In the specific implementation process, other algorithms can also be used to extract feature points from multi-view pictures. For example: using the Scale-Invariant Feature Transform (SIFT) algorithm, Oriented fast and Rotated BRIEF (ORB) algorithm, AKAZE (Accelerated-KAZE) algorithm, etc. in OpenMVG (Open Multiple View Geometry) to extract multiple feature points from the multi-view pictures of the target object. OpenMVG is an open-source C++ library that provides a set of tools for three-dimensional reconstruction and image analysis.

[0048] As an alternative implementation of the above point cloud quality assessment method, before calculating the error values between multiple feature points and the point cloud to be evaluated of the target object, it may further include:

[0049] Step S112: Obtain the deviation value of each feature point among the multiple feature points from the processing result, or calculate the deviation value of each feature point among the multiple feature points.

[0050] The implementation manner of the above step S112 is as follows: It can be understood that in the specific practice process, various deviations may be caused by factors such as matching errors, image noise, and occlusion. Therefore, in most cases, the feature points calculated by the COLMAP tool have deviations, resulting in some feature points that cannot be directly used. While calculating the feature points, the COLMAP tool will also calculate the deviation value of each feature point among multiple feature points. At this time, the deviation value corresponding to the feature point can be directly obtained from the processing result of the COLMAP tool. The larger the deviation value, the greater the deviation between the feature point and the original real point cloud during the calculation process. Therefore, it is necessary to filter the feature points with too large deviation values, that is, remove some feature points with too large deviation values from multiple feature points.

[0051] Step S113: Filter multiple feature points according to the deviation values.

[0052] The implementation manner of the above step S113 is as follows: Determine whether the deviation value is greater than the deviation threshold. If the deviation value is greater than the deviation threshold, then the feature point corresponding to the deviation value can be removed from multiple feature points. The deviation threshold here can be an empirical data, that is, the result after multiple actual experiments under the influence of the actual application scenario. For example, it can be set to 20%. First, sort the deviation values of multiple feature points from large to small to obtain the sorted multiple deviation values, and then remove and filter the feature points corresponding to the first 20% of the multiple deviation values. Of course, it can also be a deviation threshold dynamically calculated by an algorithm. For example, one or a combination of the arithmetic mean, median, and geometric mean of the deviation values corresponding to multiple feature points, or three times the variance of the arithmetic mean or geometric mean can be used as the above deviation threshold.

[0053] In the implementation process of the above solution, by filtering and removing outliers with large deviations, these deviation values may be caused by factors such as matching errors, image noise, and occlusion, and then filtering multiple feature points according to the deviation values, thereby improving the adaptability of the point cloud quality assessment method to different lighting conditions, perspective changes, or image quality differences.

[0054] As an alternative implementation manner of the above step S120, the multi-view pictures include: target view pictures, and the target view pictures can be one or more pictures among the multi-view pictures; the implementation manner of calculating the error value between multiple feature points and the point cloud to be evaluated of the target object can include:

[0055] Step S121: For each feature point among the multiple feature points, obtain the three-dimensional coordinates of the feature point and the target pixel coordinates of the feature point in the target view picture from the processing result.

[0056] An implementation manner of the above step S121 is, for example: for each of the multiple feature points, directly obtain the three-dimensional coordinates of the feature point and the target pixel coordinates of the feature point in the target perspective image from the processing result of the COLMAP tool.

[0057] Step S122: Screen out the target three-dimensional point in the to-be-evaluated point cloud that is closest to the three-dimensional coordinates of the feature point.

[0058] An implementation manner of the above step S122 is, for example: for each three-dimensional point in the to-be-evaluated point cloud, calculate the distance between the three-dimensional point and the three-dimensional coordinates of each feature point, and then screen out the target three-dimensional point in the to-be-evaluated point cloud that is closest to the three-dimensional coordinates of the feature point.

[0059] Step S123: Use a rendering algorithm to map the target three-dimensional point to the target perspective image to obtain the mapped point coordinates.

[0060] Since the multi-perspective image has multiple perspectives, in the specific practice process, the to-be-evaluated point cloud can be projected onto all perspectives in the multi-perspective image, so as to calculate the mapped point coordinates of each point cloud in the to-be-evaluated point cloud mapped onto the 2D image. Here, the mapped point coordinates are the pixel coordinates of the to-be-evaluated point cloud on the target perspective image in the multi-perspective image, and the mapped point coordinates can be denoted as (P pred_x , P pred_y ).

[0061] Optionally, in the specific practice process, the to-be-evaluated point cloud can be mapped to each image in the multi-perspective image according to the pose information of each image in the multi-perspective image. It can be understood that each image in the multi-perspective image contains a projection mapping angle, and the information reflecting the projection mapping angle is the pose information. Therefore, the to-be-evaluated point cloud can be mapped to each image in the multi-perspective image according to the pose information of each image in the multi-perspective image.

[0062] Step S124: Calculate the coordinate error value between the target pixel coordinates and the mapped point coordinates, and then multiple coordinate error values of each feature point can be obtained.

[0063] An implementation manner of the above steps S123 to S124 is, for example: use rendering algorithms such as Ray Tracing algorithm and Scanline Rendering algorithm to map the target three-dimensional point to the target perspective image to obtain the mapped point coordinates, and obtain the target pixel coordinates of each of the multiple feature points in the multi-perspective image. If the feature point is calculated by the COLMAP tool, then the target pixel coordinates of the feature point in each multi-perspective image can be directly obtained from the processing result of the COLMA tool, and the target pixel coordinates can be denoted as (P GT_x, P GT_y ). Then, use the formula Err i = |P pred_x - P GT_x | + |P pred_t - P GT_y | to calculate the coordinate error value between the target pixel coordinates and the mapped point coordinates of each feature point among multiple feature points. Among them, Err i represents the coordinate error value between the target pixel coordinates and the mapped point coordinates of each feature point among multiple feature points, (P pred_x , P pred_y ) represents the mapped point coordinates of the point cloud to be evaluated mapped to multiple perspective pictures, (P GT_x , P GT_y ) represents the target pixel coordinates.

[0064] Optionally, after obtaining the multiple coordinate error values of each feature point, it is also possible to calculate the average value of the multiple coordinate error values obtained for each feature point and determine this average value as the error value of this feature point. Since the average value is determined as the error value of this feature point, it can effectively utilize the multi-perspective information in the multi-perspective pictures, thereby better evaluating the point cloud quality.

[0065] In the implementation process of the above solution, by mapping the point cloud to be evaluated to the mapped point coordinates of multiple perspective pictures, the coordinate error value between the mapped point coordinates and the feature point pixel coordinates is calculated, thereby improving the matching accuracy between the point cloud data and the image data.

[0066] As the second alternative implementation manner of the above step S120, the above implementation manner of calculating the error value between multiple feature points and the point cloud to be evaluated of the target object may include:

[0067] Step S123: Convert each feature point among multiple feature points into three-dimensional coordinates in the world coordinate system.

[0068] The implementation manner of the above step S123 is as follows: For each feature point among multiple feature points, COLMAP will generate a descriptor, which is a vector that can uniquely represent the local image features of this point. COLMAP searches for feature points with similar descriptors between different multi-perspective pictures, thereby establishing a matching relationship between the feature points. Once the matching relationship between the feature points is determined, COLMAP will use this information to estimate the relative camera pose between each pair of pictures. Then, through the method of triangulation, the matching feature points can be converted into spatial points in three-dimensional space, and the three-dimensional coordinates of this spatial point in the world coordinate system can be obtained.

[0069] Step S124: Calculate the error value between the three-dimensional coordinates and the world coordinates of the point cloud to be evaluated.

[0070] For example, the implementation of the above step S124 is as follows: There is an accurate reference coordinate system, and a point cloud registration algorithm can be used to align the point cloud to be evaluated with the reference coordinate system. The point cloud configuration algorithm here is, for example, the Iterative Closest Point (ICP) algorithm. During the registration process, the optimal transformation (including rotation and translation) will be found to minimize the difference between the point cloud to be evaluated and the reference coordinate system. After registration and alignment, for each point cloud in the registered point cloud to be evaluated, the error value between the three-dimensional coordinates and the world coordinates of the point cloud can be calculated in this reference coordinate system. For example, the distance to the corresponding point in the reference coordinate system can be calculated; this distance can be the Euclidean distance or other types of distance metrics, such as the Manhattan distance or the Chebyshev distance.

[0071] In the implementation process of the above solution, by converting the feature points into three-dimensional coordinates and calculating the error values between these coordinates and the world coordinates of the point cloud to be evaluated, the accuracy of accurately evaluating the point cloud data can be improved.

[0072] As an alternative implementation of the above step S130, the implementation of the above quality evaluation of the point cloud to be evaluated according to the error value may include:

[0073] Step S131: Count the number of differences less than the threshold from multiple error values.

[0074] For example, the implementation of the above step S131 is as follows: First, determine the threshold. This threshold can be set manually or calculated automatically. For example, one or a combination of the arithmetic mean, median, and geometric mean of multiple error values can be used, or three times the variance of the arithmetic mean or geometric mean can be used as the above threshold. This threshold is the standard for distinguishing acceptable errors and unacceptable errors. Then, collect all relevant error values to obtain multiple error values, and traverse all error values to count the number of differences less than the threshold from multiple error values. This number of differences represents how many error values are considered acceptable errors within the given error range.

[0075] Step S132: Determine the precision value in the evaluation result according to the number of differences less than the threshold and the number of multiple error values.

[0076] For example, the implementation of the above step S132 is as follows: Assume C abnorm represents the number of differences greater than the threshold among multiple error values, C norm represents the number of differences less than the threshold among multiple error values, and F num represents the number of multiple error values. The precision value in the evaluation result can be determined according to the number of differences less than the threshold and the number of multiple error values. Specifically, for example: using the formula Or Divide the number of errors less than the threshold by the total number of error values to obtain an accuracy ratio (ACCurate). This accuracy ratio can be used to characterize the accuracy level of multiple error values as a whole. Based on the accuracy ratio, an accuracy value can be determined. This accuracy value can be a percentage or a normalized score, depending on the evaluation system in the application scenario.

[0077] In the implementation process of the above solution, by counting the proportion of the number of errors less than the set threshold in the error values, the matching quality of the point cloud data can be intuitively quantified, thereby improving the intuitive effect of point cloud quality assessment.

[0078] Optionally, if there are a total of 100 error values and 70 of them are less than the set threshold, then the accuracy ratio will be 70 / 100 = 0.7, or 70%. This means that 70% of the errors are acceptable and the accuracy value is 70%. In the specific practice process, the counted number of errors less than the threshold, the total number of error values, and the calculated accuracy value report can also be visually output, or the counted number of errors less than the threshold, the total number of error values, and the calculated accuracy value report can be sent to other terminal devices (such as a large display screen, etc.) for visual output.

[0079] As an alternative implementation of the above point cloud quality assessment method, the method may further include:

[0080] Step S140: Determine the picture view with the largest number of feature points in the multi-view pictures as the target view.

[0081] It can be understood that in multi-view 3D reconstruction, selecting the picture view with the largest number of feature points as the target view can improve the accuracy and efficiency of reconstruction because feature points are the basis for estimating the camera pose and constructing the point cloud.

[0082] An implementation manner of the above step S140 is, for example: count the number of feature points detected in each view picture. This number can affect the quality of feature matching and the subsequent 3D reconstruction process. Then, determine the picture view with the largest number of feature points in the multi-view pictures as the target view, compare the number of feature points in all view pictures, and find the picture with the largest number of feature points. Determine the picture view with the largest number of feature points as the target view.

[0083] Step S150: Screen out the adjacent view pictures with the smallest angle with the target view from the multi-view pictures, and determine the picture view of the adjacent view pictures as the source view.

[0084] For example, the implementation of the above step S150 is as follows: calculate the angle between the pose angle corresponding to each perspective image in the multi-perspective image and the target perspective, and then, select the adjacent perspective image with the smallest angle between it and the target perspective from the multiple perspective images, and determine the image perspective of the adjacent perspective image as the source perspective.

[0085] Step S160: Visually output the evaluation result according to the source perspective and the target perspective.

[0086] For example, the implementation of the above step S160 is as follows: One or both of the above source perspective and target perspective may provide the most abundant information. The feature points corresponding to the target perspective and the evaluation result of whether the feature points are greater than the threshold can be visually output, so that the user can intuitively see the quality evaluation result of the point cloud.

[0087] Optionally, of course, it is also possible to perform three-dimensional reconstruction centered on the target perspective, combining information from other perspectives (such as the source perspective, etc.). Evaluate the use of the target perspective for three-dimensional reconstruction to obtain a reconstruction result. Then, use this point cloud quality evaluation method to evaluate and verify the point cloud data in this reconstruction result to confirm whether the quality and accuracy of the reconstruction are indeed improved. If the evaluation result indicates that the quality and accuracy of the reconstruction have not been improved, the target perspective can be reselected according to the reconstruction result for three-dimensional reconstruction.

[0088] In the implementation process of the above solution, by determining the image perspective with the largest number of feature points in the multi-perspective image as the target perspective, and visually outputting the image and feature points corresponding to the target perspective, the evaluation result of the point cloud quality can be effectively visually output from the image perspective with the largest number of feature points, enabling the user to intuitively see the evaluation result of the point cloud quality and enhancing the user experience.

[0089] As an alternative implementation of the above point cloud quality evaluation method, the method may further include:

[0090] Step S170: In response to the source perspective and the target perspective selected by the user, visually display the evaluation result of the above point cloud quality through the source perspective and the target perspective.

[0091] It can be understood that the above source perspective and target perspective are mainly for the visualization step. The source perspective can be the perspective corresponding to one of the input multi-perspective images, and the target perspective can be the perspective corresponding to another input multi-perspective image. That is to say, the source perspective and the target perspective are the perspectives corresponding to two types of images in the input multi-perspective images.

[0092] For example, the implementation of the above step S170 is as follows: Optionally, the user can select the source view and the target view of interest according to their own needs (such as focusing on the texture information or signs of a certain perspective), for example: in response to the selected source view and target view by the user, the front view in the scene is used as the source view, and the top view in the scene is used as the target view, and the evaluation result of the above point cloud quality is visualized through the source view and the target view.

[0093] Please refer to Figure 2 the schematic diagram of the visualization result provided by the embodiment of the present application shown; the sub - figure in the middle of the figure is the picture of the source view, and there are multiple feature points on the picture of the source view, and these feature points are represented by purple dots on the picture of the source view; the left sub - figure and the right sub - figure are respectively the rendering pictures of the point cloud data obtained by two different depth estimation algorithms on the target view. The left sub - figure is the picture rendered by the previous algorithm, and the right sub - figure is the picture rendered by the point cloud quality evaluation method on the target view. Mapping the pixels of the feature points from the source view, for example, from the middle sub - figure to the right sub - figure, the red feature points in the figure indicate that the mapping result is incorrect (that is, the coordinate error value between the pixel coordinates and the mapped point coordinates is greater than the threshold), and the green feature points in the figure indicate that the mapping result is correct (that is, the coordinate error value between the pixel coordinates and the mapped point coordinates is less than or equal to the threshold). From the whole figure, the number of red feature points in the left sub - figure is significantly higher than that in the right sub - figure. Therefore, without relying on the original real point cloud data, the above - mentioned point cloud quality evaluation method can effectively evaluate the point cloud quality, thereby improving the adaptability of the point cloud quality evaluation method.

[0094] Please refer to Figure 3 the schematic flowchart of the point cloud quality evaluation device provided by the embodiment of the present application shown; the embodiment of the present application provides a point cloud quality evaluation device 200, including:

[0095] A feature point extraction module 210, configured to extract a plurality of feature points from multi - perspective pictures of a target object.

[0096] An error value calculation module 220, configured to calculate the error values between the plurality of feature points and the point cloud to be evaluated of the target object.

[0097] An evaluation result obtaining module 230, configured to perform quality evaluation on the point cloud to be evaluated according to the error values and obtain an evaluation result.

[0098] As an optional implementation manner of the above device, the feature point extraction module includes:

[0099] A picture feature point extraction sub - module, configured to process the multi - perspective pictures of the target object using a 3D reconstruction tool to obtain a processing result; and extract a plurality of feature points from the processing result.

[0100] As an alternative implementation of the above device, the point cloud quality assessment device further includes:

[0101] A deviation value calculation module, configured to obtain the deviation value of each feature point among multiple feature points from the processing result.

[0102] A feature point filtering module, configured to filter multiple feature points according to the deviation value.

[0103] As an alternative implementation of the above device, the multi-view images include: target view images; the error value calculation module includes:

[0104] A feature point coordinate acquisition sub-module, configured to, for each feature point among the multiple feature points, obtain the three-dimensional coordinate of the feature point and the target pixel coordinate of the feature point in the target view image from the processing result;

[0105] An evaluation point cloud screening sub-module, configured to screen out the target three-dimensional point with the closest distance to the three-dimensional coordinate of the feature point from the point cloud to be evaluated;

[0106] A mapped point coordinate acquisition sub-module, configured to map the target three-dimensional point to the target view image by using a rendering algorithm to obtain the mapped point coordinate.

[0107] A coordinate error value calculation sub-module, configured to calculate the coordinate error value between the target pixel coordinate and the mapped point coordinate.

[0108] As an alternative implementation of the above device, the error value calculation module includes:

[0109] A three-dimensional coordinate conversion sub-module, configured to convert each feature point among multiple feature points into a three-dimensional coordinate in the world coordinate system.

[0110] A point cloud error calculation sub-module, configured to calculate the error value between the three-dimensional coordinate and the world coordinate of the point cloud to be evaluated.

[0111] As an alternative implementation of the above device, the evaluation result acquisition module includes:

[0112] A difference quantity statistics sub-module, configured to count the number of differences less than a threshold from multiple error values.

[0113] An evaluation result determination sub-module, configured to determine the precision value in the evaluation result according to the number of differences less than the threshold and the number of multiple error values.

[0114] As an alternative implementation of the above device, the point cloud quality assessment device further includes:

[0115] The target viewing angle determination module is used to determine the image viewing angle with the largest number of feature points in the multi-view image as the target viewing angle.

[0116] A source perspective determination module, configured to select an adjacent perspective picture having the smallest angle with the target perspective from the multi-perspective pictures, and determine the picture perspective of the adjacent perspective picture as the source perspective;

[0117] A visualization output module is used to visualize the evaluation result according to the source perspective and the target perspective.

[0118] It should be understood that the device corresponds to the above-mentioned point cloud quality assessment method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the above description, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device.

[0119] See also Figure 4 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0120] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0121] An embodiment of the present application further provides a computer program product, including: a computer program or computer instructions, which, when run by a processor, execute the method described above.

[0122] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple. For related parts, reference can be made to the partial description of the method embodiments.

[0123] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code. A module, a program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may also occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, mainly depending on the functions involved.

[0124] In addition, in each embodiment of the present application, the various functional modules may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part. Furthermore, in the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0125] The above description is only an optional implementation manner of the embodiments of the present application. However, the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the embodiments of the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present application.

Claims

1. A point cloud quality assessment method, characterized in that: include: Extract multiple feature points from multi-view images of the target object; Calculating coordinate error values ​​between the plurality of feature points and the point cloud to be evaluated of the target object; Performing a quality assessment on the point cloud to be assessed according to the coordinate error value to obtain an assessment result; Among them, the coordinate error value includes: the error value between the mapping point coordinates of the point cloud to be evaluated mapped to the multi-view image and the pixel coordinates of the feature point in the multi-view image, and / or the error value between the feature point converted into three-dimensional coordinates and the world coordinates of the point cloud to be evaluated.

2. The method according to claim 1, characterized in that The extracting of multiple feature points from the multi-view images of the target object includes: Use a three-dimensional reconstruction tool to process multi-view images of the target object to obtain processing results; A plurality of feature points are extracted from the processing result.

3. The method according to claim 2, characterized in that Before calculating the coordinate error values ​​between the plurality of feature points and the point cloud to be evaluated of the target object, the method further includes: Acquire a deviation value of each feature point in the plurality of feature points from the processing result; The plurality of feature points are filtered according to the deviation value.

4. The method according to claim 2, characterized in that: The multi-view picture includes: a target view picture; the calculating the coordinate error values ​​between the plurality of feature points and the point cloud to be evaluated of the target object includes: For each feature point among the multiple feature points, acquiring the three-dimensional coordinates of the feature point and the target pixel coordinates of the feature point in the target viewing angle image from the processing result; Filter out the target three-dimensional point with the closest three-dimensional coordinate distance to the feature point from the point cloud to be evaluated; Mapping the target three-dimensional point to the target perspective image using a rendering algorithm to obtain mapping point coordinates; The coordinate error value between the target pixel coordinates and the mapping point coordinates is calculated.

5. The method according to claim 1, characterized in that The calculating the coordinate error values ​​between the plurality of feature points and the point cloud to be evaluated of the target object includes: Convert each of the plurality of feature points into a three-dimensional coordinate in a world coordinate system; The coordinate error value between the three-dimensional coordinates and the world coordinates of the point cloud to be evaluated is calculated.

6. The method according to claim 1, characterized in that The step of performing quality assessment on the point cloud to be assessed according to the coordinate error value comprises: Counting the number of difference values ​​less than a threshold value from the plurality of coordinate error values; The accuracy value in the evaluation result is determined according to the number of the difference values ​​less than the threshold and the number of the plurality of coordinate error values.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the evaluation results, the method further includes: Determine the picture perspective with the largest number of feature points in the multi-view picture as the target perspective; Filtering out an adjacent perspective picture having the smallest angle with the target perspective from the multi-perspective pictures, and determining the picture perspective of the adjacent perspective picture as the source perspective; The evaluation result is visualized and outputted according to the source perspective and the target perspective.

8. A point cloud quality assessment device, characterized in that: include: A feature point extraction module is used to extract multiple feature points from multi-view images of the target object; An error value calculation module, used to calculate coordinate error values ​​between the plurality of feature points and the point cloud to be evaluated of the target object; An evaluation result obtaining module, used for performing a quality evaluation on the point cloud to be evaluated according to the coordinate error value to obtain an evaluation result; Among them, the coordinate error value includes: the error value between the mapping point coordinates of the point cloud to be evaluated mapped to the multi-view image and the pixel coordinates of the feature point in the multi-view image, and / or the error value between the feature point converted into three-dimensional coordinates and the world coordinates of the point cloud to be evaluated.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

11. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images