Point Cloud Quality Detection Method, Device, Equipment and Storage Medium

Abstract

The present invention relates to the field of autonomous driving control, and discloses a point cloud quality detection method, device, equipment and storage medium. The method includes: obtaining point cloud data and images corresponding to a driving scene, and extracting the dimensional attribute information in the point cloud data; using preset calibration parameters to project the point cloud data onto the image, and respectively performing point cloud coloring on the projected image according to the dimensional attribute information to correspondingly obtain a plurality of point cloud coloring images; according to each point cloud coloring image, using a preset deep learning model to perform multi-label classification on various quality indicators corresponding to the point cloud to obtain a quality detection result of the point cloud. The present invention realizes real-time monitoring of the quality of the point cloud.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving control, and particularly to a method, device, equipment and storage medium for point cloud quality detection. Background Art

[0002] With the development of autonomous driving technology, the application of lidar in high-precision and high-level autonomous driving levels has been increasingly recognized. Compared with cameras, lidar has a larger detection field of view for driving scenarios, and there are more and more applications using lidar to collect driving scenario data. The quality of the point cloud data collected by lidar directly determines the accuracy of subsequent autonomous driving systems in performing mapping, positioning, perception, planning and control. However, the point cloud data sets collected by existing lidars are prone to strong variability in local point density, missing data, overlapping points and noise. However, for the foregoing point cloud quality problems, they are usually not detected in time, so that the quality of the point cloud collected by lidar cannot be guaranteed. Summary of the Invention

[0003] The main purpose of the present invention is to solve the technical problem that the existing quality detection of the point cloud collected by lidar is not timely enough.

[0004] The first aspect of the present invention provides a method for point cloud quality detection, including: obtaining point cloud data and an image corresponding to a driving scenario, and extracting the dimensional attribute information in the point cloud data; using preset calibration parameters, projecting the point cloud data onto the image, and respectively performing point cloud coloring on the projected image according to the dimensional attribute information to obtain a plurality of point cloud coloring images correspondingly; according to each of the point cloud coloring images, using a preset deep learning model to perform multi-label classification on each quality index corresponding to the point cloud to obtain a quality detection result of the point cloud.

[0005] Optionally, in the first implementation manner of the first aspect of the present invention, the step of respectively performing point cloud coloring on the projected image according to the dimensional attribute information to obtain a plurality of point cloud coloring images correspondingly includes: extracting the spatial position information and depth information of the point cloud in the dimensional attribute information; performing binary point cloud coloring on the projected image according to the spatial position information to obtain a corresponding point cloud background image; performing color point cloud coloring on the projected image according to the depth information to obtain a plurality of corresponding point cloud depth images; and obtaining a point cloud coloring image according to the point cloud background image and each of the point cloud depth images.

[0006] Optionally, in the second implementation manner of the first aspect of the present invention, the point cloud depth image includes a first point cloud depth image, a second point cloud depth image, and a third point cloud depth image. The method of performing color point cloud staining on the projected image according to the depth information to obtain corresponding multiple point cloud depth images includes: determining the identification value of the lidar beam where each point cloud is located in the depth information, and staining the corresponding point cloud in the projected image according to the color value corresponding to each identification value to obtain the first point cloud depth image; determining the reflection intensity value of each point cloud in the depth information, and staining the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity value to obtain the second point cloud depth image; determining the relative distance between the point cloud and the camera optical center in the depth information, and staining the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain the third point cloud depth image.

[0007] Optionally, in the third implementation manner of the first aspect of the present invention, the method of performing multi-label classification on various quality indicators corresponding to the point cloud by using a pre-set deep learning model according to each point cloud staining image to obtain a quality detection result of the point cloud includes: classifying the integrity indicator corresponding to the point cloud by using a pre-set deep learning model according to the point cloud background image to obtain an integrity detection result of the point cloud; classifying various anomaly indicators corresponding to the point cloud by using a pre-set deep learning model according to each point cloud depth image to obtain an anomaly detection result of the point cloud.

[0008] Optionally, in the fourth implementation manner of the first aspect of the present invention, the method of classifying various anomaly indicators corresponding to the point cloud by using a pre-set deep learning model according to each point cloud depth image to obtain an anomaly detection result of the point cloud includes: classifying the noise indicator corresponding to the point cloud by using a pre-set deep learning model according to the first point cloud depth image in each point cloud depth image to obtain a noise anomaly detection result of the point cloud; classifying the beam calibration indicator corresponding to the point cloud by using a pre-set deep learning model according to the second point cloud depth image in each point cloud depth image to obtain a calibration anomaly detection result of the point cloud; classifying the reflection intensity indicator corresponding to the point cloud by using a pre-set deep learning model according to the third point cloud depth image in each point cloud depth image to obtain an intensity anomaly detection result of the point cloud.

[0009] Optionally, in the fifth implementation manner of the first aspect of the present invention, the step of projecting the point cloud data onto the image by using the preset calibration parameters includes: using the preset external parameter matrix from the lidar to the camera to convert the point cloud data from the point cloud coordinate system to the camera space coordinate system, and normalizing the point cloud coordinates in the camera space coordinate system; using the preset camera distortion parameters to convert the normalized point cloud coordinates to obtain distorted point cloud coordinates; and using the preset internal parameter matrix of the camera to convert the distorted point cloud coordinates from the camera space coordinate system to the camera plane coordinate system, so as to obtain the pixel coordinate information of the point cloud data projected onto the image.

[0010] The second aspect of the present invention provides a point cloud quality detection device, including: an extraction module, configured to obtain point cloud data and an image corresponding to a driving scene, and extract the dimensional attribute information of each dimension in the point cloud data; a point cloud coloring module, configured to project the point cloud data onto the image by using the preset calibration parameters, and respectively perform point cloud coloring on the projected image according to the dimensional attribute information to obtain a plurality of point cloud coloring images correspondingly; and a classification module, configured to perform multi-label classification on each quality index corresponding to the point cloud by using the preset deep learning model according to each of the point cloud coloring images to obtain a quality detection result of the point cloud.

[0011] Optionally, in the first implementation manner of the second aspect of the present invention, the point cloud coloring module includes: an extraction unit, configured to extract the spatial position information and depth information of the point cloud in the dimensional attribute information; a background coloring unit, configured to perform binary point cloud coloring on the projected image according to the spatial position information to obtain a corresponding point cloud background image; a depth coloring unit, configured to perform color point cloud coloring on the projected image according to the depth information to obtain a corresponding plurality of point cloud depth images; and a generation unit, configured to obtain a point cloud coloring image according to the point cloud background image and each of the point cloud depth images.

[0012] Optionally, in the second implementation manner of the second aspect of the present invention, the point cloud depth images include a first point cloud depth image, a second point cloud depth image, and a third point cloud depth image, and the depth coloring unit is further configured to: determine the identification numerical value of the lidar beam where each point cloud is located in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each identification numerical value to obtain the first point cloud depth image; determine the reflection intensity numerical value of each point cloud in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity numerical value to obtain the second point cloud depth image; and determine the relative distance between the point cloud and the camera optical center in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain the third point cloud depth image.

[0013] Optionally, in the third implementation manner of the second aspect of the present invention, the classification module includes: an integrity detection unit, configured to classify the integrity index corresponding to the point cloud according to the point cloud background image by using a preset deep learning model, so as to obtain an integrity detection result of the point cloud; and an anomaly detection unit, configured to classify each anomaly index corresponding to the point cloud according to each point cloud depth image by using a preset deep learning model, so as to obtain an anomaly detection result of the point cloud.

[0014] Optionally, in the fourth implementation manner of the second aspect of the present invention, the anomaly detection unit is further configured to: classify the noise index corresponding to the point cloud according to the first point cloud depth image in each point cloud depth image by using a preset deep learning model, so as to obtain a noise anomaly detection result of the point cloud; classify the harness calibration index corresponding to the point cloud according to the second point cloud depth image in each point cloud depth image by using a preset deep learning model, so as to obtain a calibration anomaly detection result of the point cloud; and classify the reflection intensity index corresponding to the point cloud according to the third point cloud depth image in each point cloud depth image by using a preset deep learning model, so as to obtain an intensity anomaly detection result of the point cloud.

[0015] Optionally, in the fifth implementation manner of the second aspect of the present invention, the point cloud coloring module further includes: a space conversion unit, configured to convert the point cloud data from the point cloud coordinate system to the camera space coordinate system by using a preset external parameter matrix from the lidar to the camera, and perform normalization processing on the point cloud coordinates in the camera space coordinate system; a distortion conversion unit, configured to convert the normalized point cloud coordinates by using a preset camera distortion parameter to obtain distorted point cloud coordinates; and a plane conversion unit, configured to convert the distorted point cloud coordinates from the camera space coordinate system to the camera plane coordinate system by using a preset internal parameter matrix of the camera, so as to obtain pixel coordinate information of the projection of the point cloud data on the image.

[0016] The third aspect of the present invention provides a point cloud quality detection device, including: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory, so that the point cloud quality detection device executes the above-mentioned point cloud quality detection method.

[0017] The fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a computer, the computer is made to execute the above-mentioned point cloud quality detection method.

[0018] In the technical solution provided by the present invention, each time the point cloud data collected by the lidar is obtained, it is directly projected onto the image of the driving scene by using the calibration parameters of the camera, including the external parameter matrix, distortion parameters, and internal parameter matrix; then, according to the bundle number, reflection intensity, and distance to the camera optical center of each point cloud, the point cloud is stained respectively to generate point cloud projection maps of different colors; finally, through a pre-trained deep learning model, multi-label classification is performed on each point cloud projection map, and the quality of the point cloud is evaluated according to the classification results, so as to realize the real-time detection of the point cloud quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 FIG. is a schematic diagram of an embodiment of the point cloud quality detection method according to an embodiment of the present invention;

[0020] Figure 2 FIG. is a schematic diagram of another embodiment of the point cloud quality detection method according to an embodiment of the present invention;

[0021] Figure 3 FIG. is a schematic diagram of an embodiment of the point cloud quality detection device according to an embodiment of the present invention;

[0022] Figure 4 FIG. is a schematic diagram of another embodiment of the point cloud quality detection device according to an embodiment of the present invention;

[0023] Figure 5 FIG. is a schematic diagram of an embodiment of the point cloud quality detection equipment according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The embodiments of the present invention provide a point cloud quality detection method, device, equipment and storage medium, which obtain point cloud data and images corresponding to a driving scene, and extract the dimensional attribute information in the point cloud data; use preset calibration parameters to project the point cloud data onto the image, and stain the projected image respectively according to the dimensional attribute information to obtain a plurality of point cloud stained images; according to each point cloud stained image, use a preset deep learning model to perform multi-label classification on various quality indicators corresponding to the point cloud to obtain the quality detection result of the point cloud. The present invention realizes the real-time monitoring of the point cloud quality.

[0025] In the description of the present invention, the claims, and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that illustrated or described herein. In addition, the term "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0026] For ease of understanding, the specific process of the embodiments of the present invention will be described below. Please refer to Figure 1 , the first embodiment of the point cloud quality detection method in the embodiments of the present invention includes:

[0027] 101. Obtain the point cloud data and images corresponding to the driving scenario, and extract the dimensional attribute information of each dimension in the point cloud data;

[0028] It can be understood that the execution subject of the present invention can be a point cloud quality detection device, or a terminal or a server. Specifically, it is not limited here. The embodiments of the present invention will be described by taking the server as the execution subject as an example.

[0029] In this embodiment, the lidar device and camera installed on the vehicle collect the point cloud data of the three-dimensional space and the images of the two-dimensional plane in real time during the driving of the vehicle. The present invention compares the attributes of the point cloud data with the images to detect the point cloud quality (that is, to detect the quality of the attributes of the point cloud data).

[0030] Specifically, the dimensional attribute information of each dimension extracted from the point cloud data includes (x, y, z), intensity (intensity information), ring (harness surrounding information), and depth (depth) in the three-dimensional space. Among them, ring refers to the harness to which the point cloud belongs among the multiple harnesses generated by the rotary mechanical radar, which is an integer value; intensity refers to the reflection intensity of the lidar point cloud; depth refers to the distance between the lidar point cloud and the optical center of the camera.

[0031] 102. Use the preset calibration parameters to project the point cloud data onto the image, and respectively perform point cloud coloring on the projected image according to the dimensional attribute information, and correspondingly obtain a plurality of point cloud coloring images;

[0032] In this embodiment, the conversion relationship between the point cloud coordinate axes and the camera coordinate axes can be determined through the calibration parameters corresponding to the camera, including the external parameter, distortion parameter, and internal parameter, so as to project the point cloud data onto the image and compare it with the image information at the corresponding projection position for quality detection of the point cloud.

[0033] In this embodiment, after projecting the point cloud data onto the image, the projected point cloud is further colored according to the extracted attribute information of the point cloud, including the values of (x, y, z), intensity, ring, and depth. For each item of attribute information, the corresponding color is used for coloring on the point cloud corresponding to its value according to the value size. Specifically, each image includes the same original image content and the point cloud colored with the corresponding item of attribute at this time, such as four point cloud coloring images corresponding to (x, y, z), intensity, ring, and depth.

[0034] 103. According to each of the point cloud coloring images, using a preset deep learning model, perform multi-label classification on each quality index corresponding to the point cloud to obtain the quality detection result of the point cloud.

[0035] In this embodiment, after performing point cloud coloring, multiple point cloud coloring images are obtained. At this time, through a trained deep learning model, such as VGGNet (Visual Geometry Group Net), ResNet (Residual Network), Inception network model, and Xception network model, each point cloud coloring image is respectively compared. Mainly, the comparison between the colored point cloud and the original image data is performed to classify each point cloud in each point cloud coloring image, determine whether it belongs to the abnormal category and the detection confidence, obtain the point cloud attribute quality detection result of each point cloud coloring image, and finally determine the quality detection result of the point cloud.

[0036] In the embodiment of the present invention, each time the point cloud data collected by the lidar is obtained, it is directly projected onto the image of the driving scene through the calibration parameters of the camera, including the external parameter matrix, distortion parameter, and internal parameter matrix; then, according to the belonging beam, reflection intensity, and distance to the camera optical center of each point cloud, the point cloud is colored respectively to generate point cloud projection maps of different colors; finally, through a pre-trained deep learning model, multi-label classification is performed on each point cloud projection map, and the quality of the point cloud is evaluated according to the classification result to realize real-time detection of the point cloud quality.

[0037] Please refer to Figure 2 , the second embodiment of the point cloud quality detection method in the embodiment of the present invention includes:

[0038] 201. Obtain the point cloud data and images corresponding to the driving scenario, and extract the dimensional attribute information in the point cloud data;

[0039] 202. Use the preset extrinsic parameter matrix from the lidar to the camera to convert the point cloud data from the point cloud coordinate system to the camera space coordinate system, and normalize the point cloud coordinates in the camera space coordinate system;

[0040] 203. Use the preset camera distortion parameters to convert the normalized point cloud coordinates to obtain distorted point cloud coordinates;

[0041] 204. Use the preset intrinsic parameter matrix of the camera to convert the distorted point cloud coordinates from the camera space coordinate system to the camera plane coordinate system, obtain the pixel coordinate information projected by the point cloud data on the image, and extract the spatial position information and depth information of the point cloud in the dimensional attribute information.

[0042] In this embodiment, when performing the projection of the point cloud data onto the image, first synchronize the collected point cloud data and the image in time so that both belong to the same driving scenario, which can be used for subsequent comparison between the two to accurately detect the quality of the point cloud. The projection process mainly includes the following four steps: Convert each point cloud P(lidar) located in the lidar coordinate system to P(camera) in the camera space coordinate system; then normalize P(camera) to P(normalize); then perform distortion processing on P(normalize) to obtain P(tortion), and finally convert P(tortion) to P(uv) of the pixel in the image under the camera plane coordinate system.

[0043] Specifically, when converting the point cloud P(lidar) to P(camera), calibration conversion can be performed through the preset extrinsic parameter matrix T(camera-to-lidar) using P(camera) = T(camera-to-lidar) * P(lidar); at this time, the coordinates of P(camera) are (x, y, z), then normalize its point cloud coordinates to obtain (x / z, y / z, 1), that is, normalize the z-axis coordinate of the camera space coordinates to make the coordinates of the point cloud on the same plane, converting from a three-dimensional spatial position to a two-dimensional plane; then convert P(x / z, y / z, 1) through the camera distortion parameters k1, k2, k3, where P(tortion, x) = x / z(1 + k1*r 4 + k2*r 4 + k3*r 6 ), P(tortion, y) = y / z(1 + k1*r 4 + k2*r 4+ k3 * r 6 );Finally, transform the distorted point cloud coordinates P(tortion, x, y) to the camera plane coordinate system through the internal parameter matrix K: = K * P(tortion, x, y).

[0044] 205. Perform binary point cloud coloring on the projected image according to the spatial position information to obtain a corresponding point cloud background image;

[0045] 206. Perform color point cloud coloring on the projected image according to the depth information to obtain corresponding multiple point cloud depth images;

[0046] 207. Obtain a point cloud colored image according to the point cloud background image and each of the point cloud depth images;

[0047] In this embodiment, among the various dimensional attribute information of the point cloud data, according to their roles in subsequent quality inspection, they can be divided into two types of information, namely spatial position information and depth information. Among them, the spatial position information is used to detect whether the position where the point cloud is located corresponds to the object captured in the image, and the deep learning information is used to detect whether the detailed parameters of the point cloud correspond to the details of the object captured in the image, so as to perform quality inspection on the point cloud data from two aspects.

[0048] Specifically, the spatial position information of each point cloud, including (x, y, z), can be used to perform point cloud coloring on the corresponding point cloud. Binary processing is performed on the position where the point cloud is located and the image background, the point cloud is colored white and the background is colored black, or the point cloud is colored black and the background is colored white, so as to obtain a point cloud background image, thereby determining which pixels have point clouds when the point cloud is projected onto the image.

[0049] In addition, the depth information of each point cloud, including intensity, ring, and depth, can be used for coloring respectively to obtain three corresponding point cloud depth images, as shown below:

[0050] 1.1) Determine the identification value of the lidar beam where each point cloud in the depth information is located, and color the corresponding point cloud in the projected image according to the color value corresponding to each identification value to obtain a first point cloud depth image;

[0051] 2.2) Determine the reflection intensity value of each point cloud in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity value to obtain a second point cloud depth image;

[0052] 3.3) Determine the relative distance between the point cloud and the camera optical center in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain a third point cloud depth image.

[0053] In this embodiment, intensity (the reflection intensity value of the point cloud), ring (the identification value of the lidar beam), and depth (the relative distance between the point cloud and the camera optical center) are mapped to the corresponding color values. According to the corresponding values, only one of the RGB color values can be taken for mutual mapping. For example, the preset (R, G, B) initial values are (225, 0, 0), (0, 225, 0), or (0, 0, 225). The corresponding R value, G value, or B value is assigned according to the reflection intensity value, identification value, and relative distance, and each point cloud is colored in turn.

[0054] In one implementation, the COLORMAP_JET algorithm of OpenCV can be used to perform point cloud coloring. This algorithm only needs to provide the upper and lower limits of a field, and then each point cloud maps the corresponding field value according to the values of its (x, y, z), intensity, ring, and depth. The corresponding color is calculated through the built-in formula of the COLORMAP_JET algorithm before to color the corresponding point cloud.

[0055] 208. According to the point cloud background image, use a preset deep learning model to classify the integrity index corresponding to the point cloud to obtain an integrity detection result for the point cloud;

[0056] 209. According to each of the point cloud depth images, use a preset deep learning model to classify each abnormal index corresponding to the point cloud to obtain an abnormal detection result for the point cloud.

[0057] In this embodiment, the deep learning model here can adopt a conventional multi-class neural network model to compare the point cloud color and the corresponding original image data, determine the difference between the two, and classify each quality index of the point cloud. The greater the difference, the more abnormal the detection result tends to be.

[0058] Among them, for the point cloud background image obtained by coloring the spatial position information of the point cloud, it can be detected whether the image itself captures an obstacle at the pixel point mapped by the point cloud, and the integrity index of the driving scene collected by the point cloud is determined by comparison. Each point cloud carries a corresponding detection result, including whether the point cloud is missing, and the corresponding probability or confidence. The code example is bool is_missing = 1 (missing) or 0 (not missing); double is_missing_prob = 0.5 (confidence).

[0059] In addition, for each depth image obtained by coloring the depth information of the point cloud, the depth parameters of each point cloud can be detected, and the semantic information of the corresponding pixels captured by the image can be compared to determine whether each point cloud collected conforms to the various obstacles in the driving scenario. The specific implementation process is as follows:

[0060] 2.1) According to the first point cloud depth image in each of the point cloud depth images, use a pre-set deep learning model to classify the noise point index corresponding to the point cloud, and obtain the noise point anomaly detection result of the point cloud;

[0061] 2.2) According to the second point cloud depth image in each of the point cloud depth images, use a pre-set deep learning model to classify the wire harness calibration index corresponding to the point cloud, and obtain the calibration anomaly detection result of the point cloud;

[0062] 2.3) According to the third point cloud depth image in each of the point cloud depth images, use a pre-set deep learning model to classify the reflection intensity index corresponding to the point cloud, and obtain the intensity anomaly detection result of the point cloud.

[0063] In one embodiment, examples of the classification of the noise point index: bool is_have_noise_points = 1 (there are noise points) or 0 (there are no noise points); double is_have_noise_points_prob = 0.3 (confidence level); examples of the classification of the wire harness calibration index: bool is_ring_valid = 1 (calibration is normal) or 0 (calibration is abnormal); double is_ring_valid_prob == 0.5 (confidence level); examples of the classification of the reflection intensity index: bool is_intensity_valid = 1 (reflection intensity is normal) or 0 (reflection intensity is abnormal); double is_intensity_valid_prob = 0.6 (confidence level).

[0064] The method for detecting the quality of the point cloud in the embodiment of the present invention is described above. Next, the device for detecting the quality of the point cloud in the embodiment of the present invention will be described. Please refer to Figure 3 In one embodiment, the device for detecting the quality of the point cloud in the embodiment of the present invention includes:

[0065] An extraction module 301, configured to obtain point cloud data and an image corresponding to a driving scenario, and extract the attribute information of each dimension in the point cloud data;

[0066] A point cloud coloring module 302, configured to project the point cloud data onto the image by using pre-set calibration parameters, and respectively perform point cloud coloring on the projected image according to the attribute information of each dimension, and correspondingly obtain a plurality of point cloud coloring images;

[0067] A classification module 303, configured to perform multi-label classification on various quality indicators corresponding to the point cloud by using a preset deep learning model according to each of the point cloud stained images, so as to obtain a quality detection result of the point cloud.

[0068] In the embodiment of the present invention, each time the point cloud data collected by the lidar is obtained, it is directly projected onto the image of the driving scene through the calibration parameters of the camera, including the external parameter matrix, distortion parameters, and internal parameter matrix; then, according to the bundle to which each point cloud belongs, the reflection intensity, and the distance to the camera optical center, the point cloud is stained respectively to generate different colored point cloud projection images; finally, through a pre-trained deep learning model, multi-label classification is performed on each point cloud projection image, and the quality of the point cloud is evaluated according to the classification result, so as to realize real-time detection of the point cloud quality.

[0069] Please refer to Figure 4 , another embodiment of the point cloud quality detection device in the embodiment of the present invention includes:

[0070] An extraction module 301, configured to obtain the point cloud data and the image corresponding to the driving scene, and extract the attribute information of each dimension in the point cloud data;

[0071] A point cloud staining module 302, configured to project the point cloud data onto the image by using preset calibration parameters, and perform point cloud staining on the projected image respectively according to the attribute information of each dimension, and correspondingly obtain a plurality of point cloud stained images;

[0072] A classification module 303, configured to perform multi-label classification on various quality indicators corresponding to the point cloud by using a preset deep learning model according to each of the point cloud stained images, so as to obtain a quality detection result of the point cloud.

[0073] Specifically, the point cloud staining module 302 includes:

[0074] An extraction unit 3021, configured to extract the spatial position information and depth information of the point cloud in the attribute information of each dimension;

[0075] A background staining unit 3022, configured to perform binary point cloud staining on the projected image according to the spatial position information to obtain a corresponding point cloud background image;

[0076] A depth staining unit 3023, configured to perform color point cloud staining on the projected image according to the depth information to obtain a corresponding plurality of point cloud depth images;

[0077] A generation unit 3024, configured to obtain a point cloud stained image according to the point cloud background image and each of the point cloud depth images.

[0078] Specifically, the point cloud depth image includes a first point cloud depth image, a second point cloud depth image, and a third point cloud depth image. The depth coloring unit 3023 is further configured to:

[0079] Determine the identification numerical value of the lidar beam where each point cloud in the depth information is located, and color the corresponding point cloud in the projected image according to the color value corresponding to each identification numerical value to obtain the first point cloud depth image;

[0080] Determine the reflection intensity numerical value of each point cloud in the depth information, and color the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity numerical value to obtain the second point cloud depth image;

[0081] Determine the relative distance between the point cloud in the depth information and the camera optical center, and color the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain the third point cloud depth image.

[0082] Specifically, the classification module 303 includes:

[0083] An integrity detection unit 3031, configured to classify the integrity index corresponding to the point cloud according to the point cloud background image by using a preset deep learning model to obtain an integrity detection result of the point cloud;

[0084] An anomaly detection unit 3032, configured to classify the anomaly indexes corresponding to the point cloud according to each point cloud depth image by using a preset deep learning model to obtain an anomaly detection result of the point cloud.

[0085] Specifically, the anomaly detection unit 3032 is further configured to:

[0086] Classify the noise index corresponding to the point cloud according to the first point cloud depth image in each point cloud depth image by using a preset deep learning model to obtain a noise anomaly detection result of the point cloud;

[0087] Classify the beam calibration index corresponding to the point cloud according to the second point cloud depth image in each point cloud depth image by using a preset deep learning model to obtain a calibration anomaly detection result of the point cloud;

[0088] Classify the reflection intensity index corresponding to the point cloud according to the third point cloud depth image in each point cloud depth image by using a preset deep learning model to obtain an intensity anomaly detection result of the point cloud.

[0089] Specifically, the point cloud coloring module 302 further includes:

[0090] A spatial transformation unit 3025 is configured to use a pre-set extrinsic parameter matrix from the lidar to the camera to transform the point cloud data from the point cloud coordinate system to the camera space coordinate system, and normalize the point cloud coordinates in the camera space coordinate system;

[0091] A distortion transformation unit 3026 is configured to use pre-set camera distortion parameters to transform the normalized point cloud coordinates to obtain distorted point cloud coordinates;

[0092] A plane transformation unit 3027 is configured to use a pre-set intrinsic parameter matrix of the camera to transform the distorted point cloud coordinates from the camera space coordinate system to the camera plane coordinate system, so as to obtain the pixel coordinate information of the projection of the point cloud data on the image.

[0093] Above Figure 3 And Figure 4 The point cloud quality detection device in the embodiments of the present invention is described in detail from the perspective of modular functional entities. Next, the point cloud quality detection device in the embodiments of the present invention is described in detail from the perspective of hardware processing.

[0094] Figure 5 FIG. is a schematic structural diagram of a point cloud quality detection device provided by an embodiment of the present invention. The point cloud quality detection device 500 may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPUs) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 for storing application programs 533 or data 532 (for example, one or more mass storage devices). Among them, the memory 520 and the storage media 530 may be transient storage or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the point cloud quality detection device 500. Further, the processor 510 may be configured to communicate with the storage media 530 and execute a series of instruction operations in the storage media 530 on the point cloud quality detection device 500.

[0095] The point cloud quality detection device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that Figure 5 The shown structure of the point cloud quality detection device does not constitute a limitation on the point cloud quality detection device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0096] The present invention also provides a point cloud quality detection device. The computer device includes a memory and a processor. Computer-readable instructions are stored in the memory. When the computer-readable instructions are executed by the processor, the processor is caused to execute the steps of the point cloud quality detection method in the above various embodiments.

[0097] The present invention also provides a computer-readable storage medium. The computer-readable storage medium can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is caused to execute the steps of the point cloud quality detection method.

[0098] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0100] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for detecting the quality of point clouds, characterized in that, The point cloud quality detection method includes: Obtain the point cloud data and images corresponding to the driving scenario, and extract the dimensional attribute information in the point cloud data; Using the preset calibration parameters, project the point cloud data onto the image, and according to the dimensional attribute information, perform point cloud coloring on the projected image respectively to obtain multiple point cloud coloring images correspondingly; According to each of the point cloud coloring images, use the preset deep learning model to perform multi-label classification on various quality indicators corresponding to the point cloud to obtain the quality detection result of the point cloud; The step of performing point cloud coloring on the projected image respectively according to the dimensional attribute information to obtain multiple point cloud coloring images correspondingly includes: extracting the spatial position information and depth information of the point cloud in the dimensional attribute information; performing binary point cloud coloring on the projected image according to the spatial position information to obtain the corresponding point cloud background image; performing color point cloud coloring on the projected image according to the depth information to obtain the corresponding multiple point cloud depth images; obtaining the point cloud coloring images according to the point cloud background image and each of the point cloud depth images; The point cloud depth images include a first point cloud depth image, a second point cloud depth image, and a third point cloud depth image. The step of performing color point cloud coloring on the projected image according to the depth information to obtain the corresponding multiple point cloud depth images includes: determining the identification numerical value of the lidar beam where each point cloud is located in the depth information, and coloring the corresponding point cloud in the projected image according to the color value corresponding to each identification numerical value to obtain the first point cloud depth image; determining the reflection intensity numerical value of each point cloud in the depth information, and coloring the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity numerical value to obtain the second point cloud depth image; determining the relative distance between the point cloud and the camera optical center in the depth information, and coloring the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain the third point cloud depth image.

2. The point cloud quality detection method according to claim 1, wherein The step of using the preset deep learning model to perform multi-label classification on various quality indicators corresponding to the point cloud according to each of the point cloud coloring images to obtain the quality detection result of the point cloud includes: According to the point cloud background image, use the preset deep learning model to classify the integrity indicator corresponding to the point cloud to obtain the integrity detection result of the point cloud; According to each of the point cloud depth images, use the preset deep learning model to classify various anomaly indicators corresponding to the point cloud to obtain the anomaly detection result of the point cloud.

3. The point cloud quality detection method according to claim 1, characterized in that The step of using the preset deep learning model to classify various anomaly indicators corresponding to the point cloud according to each of the point cloud depth images to obtain the anomaly detection result of the point cloud includes: According to the first point cloud depth image in each of the point cloud depth images, use the preset deep learning model to classify the noise indicator corresponding to the point cloud to obtain the noise anomaly detection result of the point cloud; According to the second point cloud depth image in each of the point cloud depth images, use the preset deep learning model to classify the beam calibration indicator corresponding to the point cloud to obtain the calibration anomaly detection result of the point cloud; Based on the third point cloud depth image in each of the point cloud depth images, using a pre-set deep learning model, classify the reflection intensity index corresponding to the point cloud to obtain the intensity anomaly detection result of the point cloud.

4. The point cloud quality detection method according to any one of claims 1-3, characterized in that, The projecting the point cloud data onto the image by using the pre-set calibration parameters includes: Using the pre-set external parameter matrix from the lidar to the camera, convert the point cloud data from the point cloud coordinate system to the camera space coordinate system, and normalize the point cloud coordinates in the camera space coordinate system; Using the pre-set camera distortion parameters, convert the normalized point cloud coordinates to obtain distorted point cloud coordinates; Using the pre-set internal parameter matrix of the camera, convert the distorted point cloud coordinates from the camera space coordinate system to the camera plane coordinate system to obtain the pixel coordinate information of the projection of the point cloud data onto the image.

5. A point cloud quality detection device, characterized in that, The point cloud quality detection device includes: An extraction module, configured to obtain the point cloud data and the image corresponding to the driving scene, and extract the dimensional attribute information in the point cloud data; A point cloud coloring module, configured to project the point cloud data onto the image by using the pre-set calibration parameters, and respectively perform point cloud coloring on the projected image according to the dimensional attribute information to correspondingly obtain a plurality of point cloud coloring images; A classification module, configured to perform multi-label classification on each quality index corresponding to the point cloud by using the pre-set deep learning model according to each of the point cloud coloring images to obtain the quality detection result of the point cloud; The point cloud coloring module includes: an extraction unit, configured to extract the spatial position information and depth information of the point cloud in the dimensional attribute information; a background coloring unit, configured to perform binary point cloud coloring on the projected image according to the spatial position information to obtain the corresponding point cloud background image; a depth coloring unit, configured to perform color point cloud coloring on the projected image according to the depth information to obtain the corresponding plurality of point cloud depth images; a generation unit, configured to obtain the point cloud coloring image according to the point cloud background image and each of the point cloud depth images; The point cloud depth image includes a first point cloud depth image, a second point cloud depth image, and a third point cloud depth image. The depth coloring unit is further configured to: determine the identification value of the lidar beam where each point cloud is located in the depth information, and perform coloring on the corresponding point cloud in the projected image according to the color value corresponding to each identification value to obtain the first point cloud depth image; determine the reflection intensity value of each point cloud in the depth information, and perform coloring on the corresponding point cloud in the projected image according to the color value corresponding to each reflection intensity value to obtain the second point cloud depth image; determine the relative distance between the point cloud and the camera optical center in the depth information, and perform coloring on the corresponding point cloud in the projected image according to the color value corresponding to each relative distance to obtain the third point cloud depth image.

6. A point cloud quality detection device, characterized in that, The point cloud quality detection device includes: a memory and at least one processor, and instructions are stored in the memory; The at least one processor invokes the instructions in the memory so that the point cloud quality detection device executes the steps of the point cloud quality detection method according to any one of claims 1-4.

7. A computer-readable storage medium, on which instructions are stored, characterized in that, When the instruction is executed by a processor, it implements the steps of the point cloud quality detection method according to any one of claims 1-4.

Images