Methods, apparatus, computer equipment and storage media for denoising point cloud data

Abstract

This disclosure provides a method, apparatus, computer device, and storage medium for denoising point cloud data. The method includes: acquiring point cloud data; dividing the point cloud data into raster segments to obtain local point cloud data contained in at least one target raster; determining the probability that the target raster includes noisy point cloud points based on the local point cloud data contained in each target raster; and updating the point cloud data based on the probability that each target raster includes noisy point cloud points to obtain updated point cloud data.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a method, apparatus, computer device, and storage medium for denoising point cloud data. Background Technology

[0002] Point cloud data acquired using lidar is typically used for object detection. However, point cloud data acquired under different environments often contains some environment-related noise points. For example, point cloud data acquired in heavy rain often contains noise points related to rain or splashes.

[0003] The presence of noisy point cloud points will affect the accuracy of object detection results. Therefore, how to identify and remove noisy point cloud points in point cloud data has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides at least one method, apparatus, computer device, and storage medium for denoising point cloud data.

[0005] In a first aspect, embodiments of this disclosure provide a method for denoising point cloud data, including:

[0006] Acquire point cloud data;

[0007] The point cloud data is rasterized to obtain local point cloud data contained in at least one target raster.

[0008] Based on the local point cloud data contained in each target grid, determine the probability that the target grid includes noisy point cloud points;

[0009] Based on the probability that each of the target grids includes noisy point cloud points, the point cloud data is updated to obtain updated point cloud data.

[0010] This implementation method, by rasterizing the acquired point cloud data, can obtain smaller local point cloud data corresponding to each target raster. Processing the local point cloud data contained in the target raster not only reduces computational burden and improves computational speed, but also accurately determines the probability that the target raster contains noisy point cloud points. Based on the probability that each target raster contains noisy point cloud points, the target raster containing noisy point cloud points can be accurately selected. Furthermore, by processing the local point cloud data corresponding to each target raster containing noisy point cloud points, the noisy point cloud points in the point cloud data can be deleted, resulting in updated point cloud data free of noisy point cloud points, thus completing the denoising process of the point cloud data.

[0011] In one possible implementation, updating the point cloud data based on the probability that each of the target grids includes noisy point cloud points to obtain updated point cloud data includes:

[0012] If the probability that the target grid includes noisy point cloud points is greater than a preset probability, the local point cloud data contained in the target grid is deleted to obtain updated point cloud data.

[0013] In this implementation, the higher the probability, the higher the likelihood that the point cloud points in the target raster are noisy point cloud points. By using the preset probability and the probability that the target raster includes noisy point cloud points, target rasteres containing noisy point cloud points can be accurately selected. By deleting the local point cloud data contained in the selected target rasteres, denoising processing of the point cloud data can be achieved.

[0014] In one possible implementation, determining the probability that a target raster includes noisy point cloud points based on the local point cloud data contained in each target raster includes:

[0015] For the local point cloud data contained in each target grid, noise features are extracted from the point cloud information of each point cloud point in the local point cloud data to obtain the intermediate feature information of each point cloud point.

[0016] The intermediate feature information of each point cloud point is fused to obtain the fused feature information of the target grid.

[0017] Based on the fused feature information, the probability that the target raster includes noisy point cloud points is determined.

[0018] This implementation extracts noise features from the point cloud information of each point cloud point, thereby extracting intermediate feature information related to noise. By fusing these intermediate feature information, a more noise-correlated fused feature information can be obtained while reducing the number of features and redundancy. Using this more noise-correlated fused feature information for probability prediction can improve the accuracy of the output probability.

[0019] In one possible implementation, after obtaining the updated point cloud data, the method further includes:

[0020] Based on the updated point cloud data, determine the object information for each object to be detected;

[0021] Based on the object information of each of the objects to be detected, the driving device is controlled to drive.

[0022] This implementation method performs object detection based on updated point cloud data, which can avoid the impact of noisy point cloud points on detection accuracy, thereby accurately obtaining object information of each object to be detected; and then, based on the object information of each object to be detected, controls the driving device to drive, which can improve the driving safety of the driving device and the safety of the objects to be detected.

[0023] In one possible implementation, the denoising method for the point cloud data is performed by a denoising neural network.

[0024] In this embodiment, since the trained denoising neural network has reliable prediction accuracy, the point cloud data denoising method provided in this embodiment can be executed using the trained denoising neural network to achieve accurate denoising of the point cloud data and improve the accuracy of the updated point cloud data.

[0025] In one possible implementation, the denoising neural network is trained according to the following steps:

[0026] Acquire sample point cloud data;

[0027] The sample point cloud data is rasterized to obtain local sample point cloud data contained in at least one sample raster; and the label information of each sample raster is determined.

[0028] The local sample point cloud data contained in each of at least one of the sample grids is input into the neural network to be trained to generate the predicted probability of the sample grid including noisy point cloud points;

[0029] Based on the predicted probability of each sample grid including noisy point cloud points and the label information of each sample grid, the neural network to be trained is iteratively trained until the training cutoff condition is met, and the denoising neural network is obtained.

[0030] This implementation method iteratively trains the neural network to be trained based on the predicted probability of the noisy point cloud points in the sample grid and the label information of the sample grid. This can improve the consistency between the predicted probability and the label information output by the network to be trained, and obtain a denoising neural network with reliable prediction accuracy.

[0031] In one possible implementation, determining the label information for each sample raster includes:

[0032] For each sample point in the sample point cloud data, determine the point type of the sample point; wherein, the point type includes noise type and non-noise type: the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type;

[0033] Determine the target number of noise-type sample point cloud points in the local sample point cloud data contained in each of the sample grids;

[0034] The label information of the sample grid is determined based on the target quantity and the preset quantity threshold corresponding to each sample grid.

[0035] This implementation achieves clear classification of potential noise point cloud samples by pre-setting multiple noise types. Determining the point type of the sample point cloud based on these pre-defined noise types avoids missing noise-related point cloud samples, improving the accuracy of the determined point type. Determining the labeling information based on the target number and a preset threshold of noise-type point cloud samples in the local sample point cloud data corresponding to the sample raster further improves the accuracy of the determined labeling information.

[0036] Secondly, embodiments of this disclosure also provide a noise reduction device for point cloud data, comprising:

[0037] The acquisition module is used to acquire point cloud data;

[0038] The segmentation module is used to perform rasterization segmentation on the point cloud data to obtain local point cloud data contained in at least one target raster.

[0039] The determination module is used to determine the probability that a target raster includes noisy point cloud points based on the local point cloud data contained in each target raster;

[0040] The update module is used to update the point cloud data based on the probability that each target grid includes noisy point cloud points, so as to obtain updated point cloud data.

[0041] In one possible implementation, when the update module updates the point cloud data based on the probability that each of the target grids includes noisy point cloud points to obtain updated point cloud data, it is used to delete the local point cloud data contained in the target grid if the probability that the target grid includes noisy point cloud points is greater than a preset probability, thereby obtaining updated point cloud data.

[0042] In one possible implementation, when determining the probability that a target grid includes noisy point cloud points based on the local point cloud data contained in each target grid, the determining module is used to extract noise features from the point cloud information of each point cloud point in the local point cloud data contained in each target grid to obtain intermediate feature information of each point cloud point.

[0043] The intermediate feature information of each point cloud point is fused to obtain the fused feature information of the target grid.

[0044] Based on the fused feature information, the probability that the target raster includes noisy point cloud points is determined.

[0045] In one possible implementation, the device further includes:

[0046] The control module is used to determine the object information of each object to be detected based on the updated point cloud data after obtaining the updated point cloud data.

[0047] Based on the object information of each of the objects to be detected, the driving device is controlled to drive.

[0048] In one possible implementation, the denoising method for the point cloud data is performed by a denoising neural network.

[0049] In one possible implementation, the device further includes:

[0050] The training module is used to train the denoising neural network according to the following steps:

[0051] Acquire sample point cloud data;

[0052] The sample point cloud data is rasterized to obtain local sample point cloud data contained in at least one sample raster; and the label information of each sample raster is determined.

[0053] The local sample point cloud data contained in each of at least one of the sample grids is input into the neural network to be trained to generate the predicted probability of the sample grid including noisy point cloud points;

[0054] Based on the predicted probability of each sample grid including noisy point cloud points and the label information of each sample grid, the neural network to be trained is iteratively trained until the training cutoff condition is met, and the denoising neural network is obtained.

[0055] In one possible implementation, the training module, when determining the label information of each sample grid, is used to determine the point type of each sample point in the sample point cloud data; wherein, the point type includes noise type and non-noise type: the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type.

[0056] Determine the target number of noise-type sample point cloud points in the local sample point cloud data contained in each of the sample grids;

[0057] The label information of the sample grid is determined based on the target quantity and the preset quantity threshold corresponding to each sample grid.

[0058] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0059] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0060] For a description of the effects of the denoising device, computer equipment, and computer-readable storage medium for the point cloud data mentioned above, please refer to the description of the point cloud data denoising method mentioned above, which will not be repeated here.

[0061] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0063] Figure 1 A flowchart of a point cloud data denoising method provided in an embodiment of this disclosure is shown;

[0064] Figure 2 A flowchart illustrating a method for training a neural network according to an embodiment of this disclosure is shown;

[0065] Figure 3 A schematic diagram of a point cloud data denoising device provided in an embodiment of this disclosure is shown;

[0066] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0068] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0069] In this article, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0070] Research has revealed that point cloud data acquired using lidar contains various types of noisy point clouds under specific weather conditions or road conditions. For example, point cloud data captured in rain often includes rain-related and splash-related noise point clouds. Similarly, point cloud data captured on dirt roads often includes dust-related noise point clouds. The presence of these noisy point clouds affects the accuracy of object detection based on the point cloud data. Therefore, denoising point cloud data is a pressing issue that needs to be addressed.

[0071] Based on the above research, this disclosure provides a denoising scheme for point cloud data. By rasterizing the acquired point cloud data, smaller local point cloud data corresponding to each target raster can be obtained. By processing the local point cloud data contained in the target raster, not only can the computational pressure be reduced and the computational speed improved, but the probability that the target raster contains noisy point cloud points can also be accurately determined. Based on the probability that each target raster contains noisy point cloud points, each target raster containing noisy point cloud points can be accurately screened. Furthermore, based on the processing of the local point cloud data corresponding to each target raster containing noisy point cloud points, the noisy point cloud points in the point cloud data can be deleted, resulting in updated point cloud data without noisy point cloud points, thus completing the denoising process of the point cloud data.

[0072] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0073] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0074] To facilitate understanding of this embodiment, a method for denoising point cloud data disclosed in this disclosure will first be described in detail. The execution subject of the point cloud data denoising method provided in this disclosure is generally a terminal device or other processing device with certain computing power. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computer device, etc. In some possible implementations, the point cloud data denoising method can be implemented by the processor calling computer-readable instructions stored in the memory.

[0075] The following describes the point cloud data denoising method provided in this disclosure embodiment, taking a computer device as the execution subject as an example.

[0076] like Figure 1 The flowchart shown is a method for denoising point cloud data provided in an embodiment of this disclosure, which may include the following steps:

[0077] S101: Acquire point cloud data.

[0078] Here, point cloud data can be acquired using a lidar sensor installed on the vehicle. The point cloud data can be a set of point cloud vectors, containing point cloud information for multiple point points. This point cloud information can include the coordinates of the point points in a three-dimensional world coordinate system, their color information, reflection intensity information, distance information, etc. The point cloud data may include noisy point cloud points. For example, noisy point cloud points could be rain-type point cloud points or exhaust-type point cloud points.

[0079] For example, point cloud data can be road point cloud data collected by lidar under normal weather conditions or normal road conditions, or it can be road point cloud data collected by lidar under special weather conditions and / or special road conditions; there is no limitation here. Special weather conditions may include, for example, rainy days, foggy days, windy days, sandstorm days, etc., and special road conditions may include, for example, dirt roads, construction roads, muddy roads, etc.

[0080] For example, when the driving device is traveling in special weather conditions or on special road conditions, it can use the installed lidar to collect point cloud data of the road it is traveling on. Thus, point cloud data can be obtained.

[0081] S102: Rasterize the point cloud data to obtain local point cloud data contained in at least one target raster.

[0082] Here, the point cloud data is rasterized to obtain multiple grids, which can include empty grids and non-empty target grids. Each target grid can contain local point cloud data.

[0083] Local point cloud data refers to a portion of the point cloud data, including at least one point in the point cloud data, and point cloud information for each point within that at least one point. The local point cloud data contained in each target raster can be combined to form the aforementioned point cloud data.

[0084] For example, the horizontal axis (x-axis) in the world coordinate system can be the direction of the road, the vertical axis (y-axis) in the world coordinate system can be perpendicular to the direction of the road, and the vertical axis (z-axis) in the world coordinate system can be perpendicular to the direction of the road and point towards the sky (or the ground).

[0085] After acquiring the point cloud data, it can be rasterized according to a preset grid size. For example, the preset grid size can be L meters * M meters * N meters, where L is the length along the x-axis, M is the length along the y-axis, and N is the length along the z-axis. For instance, the preset grid size could be 0.1 meters * 0.1 meters * 10 meters.

[0086] In practical applications, the values ​​corresponding to L, M, and N can be determined based on the parameters of the actual lidar used, and this embodiment does not impose specific limitations. For example, they can be determined based on the maximum x-value, maximum y-value, and maximum z-value of the coordinates of the point cloud points acquired by the lidar.

[0087] In practice, the number of target grids and the target grid containing each point cloud point can be determined according to the preset grid size and the coordinates of each point cloud point in the world coordinate system. Point cloud points within the same target grid are then considered as local point cloud data contained within that target grid. Based on the rasterization of the point cloud data, at least one target grid and the local point cloud data contained within each of the at least one target grid can be obtained.

[0088] S103: Based on the local point cloud data contained in each target raster, determine the probability that the target raster includes noisy point cloud points.

[0089] Here, noisy point cloud points can be point cloud points belonging to the noise type in local point cloud data. The noise type can include, but is not limited to, rain, fog, dust, exhaust fumes, and splashes. The specific noise type can be determined according to the actual denoising needs, and this embodiment does not impose a specific limitation. For example, when denoising tree-type point cloud points is required, the noise type can be tree type; when denoising pole-type point cloud points is required, the noise type can be pole type.

[0090] The above probability is used to characterize the probability that the local point cloud data contained in the target raster includes noisy point cloud points.

[0091] In practice, for each target raster, based on the point cloud information of each point cloud point in the local point cloud data contained in the target raster, the feature information related to the noise type of each point cloud point can be determined. Then, based on the feature information related to the noise type of each point cloud point, it can be determined whether each point cloud point belongs to the noise type of noise point cloud point.

[0092] Then, the probability that the target raster includes noisy point cloud points can be determined based on the number of noisy point cloud points in the local point cloud data. For example, if the number of noisy point cloud points in the local point cloud data is greater than a preset value, the probability that the target raster includes noisy point cloud points is determined to be 1; if the number of noisy point cloud points in the local point cloud data is not greater than the preset value, the probability that the target raster includes noisy point cloud points is determined to be 0. Alternatively, the probability that the target raster includes noisy point cloud points can be determined based on the ratio of the number of noisy point cloud points to the number of non-noisy point cloud points in the local point cloud data. Furthermore, after determining the noise type-related feature information for each point cloud point in the local point cloud data, the probability that the target raster includes noisy point cloud points can be directly determined based on the feature information of each point cloud point.

[0093] S104: Update the point cloud data based on the probability that each target grid includes noisy point cloud points to obtain updated point cloud data.

[0094] Here, the updated point cloud data can be point cloud data with noise removed.

[0095] For example, after obtaining the probability that each target grid includes noisy point cloud points, the target grids can be sorted in descending order of probability to obtain a sorting value for each target grid. For instance, if target grid 1 has a probability of 0.3 including noisy point cloud points, target grid 2 has a probability of 0.9 including noisy point cloud points, and target grid 3 has a probability of 0.8 including noisy point cloud points, then after sorting the target grids in descending order of probability, the sorting value for target grid 2 is 1, the sorting value for target grid 3 is 2, and the sorting value for target grid 1 is 3.

[0096] Next, each target raster with a sorting value less than a preset sorting value can be identified as a noise raster containing noise point cloud points. Each point cloud point in the local point cloud data contained in the noise raster is treated as a noise point cloud point, and all point cloud points in the local point cloud data contained in each noise raster are deleted to update the point cloud data and obtain the updated point cloud data.

[0097] In one embodiment, S104 can be implemented according to the following steps: if the probability that the target grid includes noisy point cloud points is greater than a preset probability, the local point cloud data contained in the target grid is deleted to obtain updated point cloud data.

[0098] Here, the preset probability can be a pre-defined minimum probability value. If the probability that the target includes noisy point cloud points is greater than the preset probability, it can be determined that the target raster belongs to the noisy raster, and every point cloud point in the local point cloud data contained in the target raster belongs to the noisy point cloud point; otherwise, it is determined that the target raster does not belong to the noisy raster, and every point cloud point in the local point cloud data contained in the target raster does not belong to the noisy point cloud point.

[0099] In practice, for each target raster, the probability that the target raster includes noisy point cloud points can be compared with a preset probability to determine whether the probability that the target raster includes noisy point cloud points is greater than the preset probability. If so, the target raster is determined to be a noisy raster, and the local point cloud data contained in the target raster is deleted from the point cloud data; if not, the target raster is determined not to be a noisy raster, and the local point cloud data contained in the target raster is retained in the point cloud data.

[0100] Based on this step, each noise grid with a probability greater than a preset probability can be filtered out, and the local point cloud data contained in each noise grid can be deleted, thereby achieving denoising processing of the point cloud data and obtaining updated point cloud data.

[0101] In this way, by rasterizing the acquired point cloud data, smaller local point cloud data corresponding to each target raster can be obtained. Processing the local point cloud data contained in the target raster not only reduces computational burden and improves computational speed, but also accurately determines the probability that the target raster contains noisy point cloud points. Based on the probability that each target raster contains noisy point cloud points, the target raster containing noisy point cloud points can be accurately selected. Furthermore, by processing the local point cloud data corresponding to each target raster containing noisy point cloud points, the noisy point cloud points can be deleted from the point cloud data, resulting in updated point cloud data free of noisy point cloud points, thus completing the denoising process of the point cloud data.

[0102] In one embodiment, the point cloud data denoising method provided in this disclosure can be executed by a trained denoising neural network. Specifically, the acquired point cloud data can be input into the denoising neural network, and the point cloud data can be rasterized using the denoising neural network to obtain local point cloud data contained in at least one target raster. Then, based on the local point cloud data contained in each target raster, the probability that the target raster includes noisy point cloud points is determined. Finally, based on the probability that each target raster includes noisy point cloud points, the point cloud data is updated, and the updated point cloud data is output.

[0103] Here, since the trained denoising neural network has reliable prediction accuracy, by using the trained denoising neural network to execute the point cloud data denoising method provided in the embodiments of this disclosure, the point cloud data can be accurately denoised to obtain accurate and updated point cloud data.

[0104] In one embodiment, S103 described above can be implemented according to the following steps:

[0105] S103-1: For the local point cloud data contained in each target grid, noise features are extracted from the point cloud information of each point in the local point cloud data to obtain the intermediate feature information of each point.

[0106] Here, the intermediate feature information refers to the feature information in the point cloud information that is related to the noise type.

[0107] In practice, for the local point cloud data contained in each target grid, the information related to the noise type in the point cloud information of each point cloud point can be determined based on the point cloud information of each point cloud point in the local point cloud data. The information related to the noise type in the point cloud information of each point cloud point is extracted to obtain the intermediate feature information of each point cloud point.

[0108] S103-2: Perform feature fusion on the intermediate feature information of each point cloud point to obtain the fused feature information of the target raster.

[0109] Here, the fused feature information is high-dimensional, such as 64-dimensional or 128-dimensional. The fused feature information is capable of characterizing whether the target raster is related to the noise type.

[0110] For example, for each target raster, target feature information related to the noise type can be further extracted from the intermediate feature information of each point cloud point in the local point cloud data contained in the target raster. Feature fusion operation is performed on the target feature information of each point cloud point to obtain a high-dimensional fused feature information, and the high-dimensional fused feature information is used as the fused feature information of the target raster.

[0111] S103-3: Based on fused feature information, determine the probability that the target raster includes noisy point cloud points.

[0112] In practice, after obtaining the fusion feature information of the target raster, the fusion feature information can be convolved and the result of the convolution can be upsampled. Based on the result of the upsampling, the probability that the target raster includes noisy point cloud points can be determined.

[0113] As can be seen from the above embodiments, the point cloud data denoising method provided in this disclosure can be executed using a denoising neural network. Therefore, S103-1 to S103-3 described above can also be executed using a denoising neural network.

[0114] Specifically, after acquiring the local point cloud data corresponding to each target grid, the denoising neural network can process the local point cloud data contained in each target grid sequentially in a concatenated manner, thereby outputting the probability that each target grid includes noisy point cloud points. In this way, the processing load on the target neural network can be reduced by using concatenated processing, making the target neural network more lightweight.

[0115] Optionally, after obtaining the local point cloud data contained in each target raster, the local point cloud data contained in each target raster can be processed in parallel to output the probability that each target raster includes noisy point cloud points.

[0116] For example, a denoising neural network can include multiple network layers. For the local point cloud data contained in each target grid, the fully connected layer in the denoising neural network can first perform fully connected processing on the point cloud information of each point in the local point cloud data to obtain the first feature information of each point. Then, the batch normalization layer in the denoising neural network can transform the range of the feature values ​​corresponding to each first feature information to obtain the second feature information of each point. Next, the linear rectification function (ReLU) layer in the denoising neural network can be used to numerically transform the feature values ​​corresponding to the second feature information of each point, setting feature values ​​less than 0 to 0 and retaining feature values ​​greater than 0, thereby obtaining the intermediate feature information of each point. Finally, the max pooling layer in the denoising neural network can be used to fuse the intermediate feature information of each point to obtain the high-dimensional fused feature information of the target grid.

[0117] Furthermore, a 2D convolutional layer in the denoising neural network can be used to perform 2D convolution on the road feature information to obtain the third feature information. Then, another batch normalization layer in the denoising neural network can be used to transform the range of the feature values ​​corresponding to the third feature information to obtain the fourth feature information. Next, the ReLU function in another ReLU function layer in the denoising neural network can be used to numerically transform the feature values ​​corresponding to the fourth feature information to obtain the fifth feature information. Then, an upsampling layer in the denoising neural network can be used to upsample the fifth feature information to obtain the sixth feature information. Finally, a sigmoid activation function layer in the denoising neural network can be used to process the sixth feature information, outputting a probability map of the target raster including noisy point cloud points. The probability interval of the probability map is (0, 1). Based on the probability map, the probability that the target raster includes noisy point cloud points can be obtained.

[0118] In this way, by using different network layers, the feature information that the network layer is interested in can be extracted from each network layer. Finally, based on the sigmoid activation function layer, the probability that the target raster includes noisy point cloud points can be accurately determined.

[0119] After obtaining the probability that each target grid contains noise point cloud points, the denoising neural network can use preset probabilities to filter out noise grids from the target grids and delete the local point cloud data contained in each noise grid from the acquired point cloud data, thereby obtaining updated point cloud data. Since the objects corresponding to the noise point cloud points of the various noise types mentioned above are objects that do not affect the driving device's operation, deleting the noise point cloud points of these various noise types can prevent the objects corresponding to these noise point cloud points from being identified as obstacles, thus affecting the driving device's operation. For example, if rainwater-type noise point cloud points are identified as obstacles, it may cause the driving device to experience sudden braking or sharp turns, affecting driving safety.

[0120] In one embodiment, after obtaining the updated point cloud data, the updated point cloud data can be input to the downstream LiDAR perception algorithm module to perform object detection on the updated point cloud data, determine each object to be detected corresponding to the updated point cloud data, and the object information of each object to be detected.

[0121] The objects to be detected may include, for example, lane lines, buildings, pedestrians, vehicles, and road poles. Object information may include, for example, object location, object category, distance between the object and the driving device in the aforementioned point cloud data, whether the object to be detected is a movable object, the direction of movement of movable objects, movement speed, and safe distance from movable objects.

[0122] Then, the driving device can be controlled to drive based on the object information of each object to be detected.

[0123] Here, the driving device can be a device equipped with a lidar unit that captures the aforementioned point cloud data. For example, the driving device can include any device capable of traveling on a road, such as an autonomous vehicle, a manually driven vehicle, or a robot.

[0124] For example, after obtaining the object information of each object to be detected, a safe driving route for the autonomous vehicle can be planned based on the object information of each object to be detected (e.g., object position, moving speed, etc.), and the autonomous vehicle can be controlled to drive along the safe driving route.

[0125] For example, if it is determined that the objects to be detected include pedestrians, the distance between the manually driven vehicle and the pedestrians can be determined based on the positions of the pedestrians and the manually driven vehicle. If the distance is less than the preset safe distance, a voice alarm can be triggered to prompt the pedestrians to give way to the vehicle and to remind the driver to slow down in time to give way to the pedestrians.

[0126] For example, if it is determined that other vehicles are included among the objects to be detected, the moving speed, moving direction, and moving trajectory of the other vehicles can be determined, the moving trajectory of the other vehicles can be predicted, and the driving device can be controlled to move based on the predicted trajectory of the other vehicles.

[0127] In one embodiment, where the point cloud data denoising method provided in this disclosure is executed by a denoising neural network, this disclosure also provides a method for training a neural network to be trained, such as... Figure 2 The flowchart shown is a method for training a neural network according to an embodiment of this disclosure, which may include the following steps:

[0128] S201: Obtain sample point cloud data.

[0129] Here, the sample point cloud data can be point cloud data acquired using any lidar. The sample point cloud data can be a set of sample point cloud vectors, which includes point cloud information for multiple sample point cloud points. This point cloud information can include the coordinates of the sample point cloud points in the three-dimensional world coordinate system, the color information, reflection intensity information, distance information, etc. The sample point cloud data may include sample point cloud points with noise, such as rain-type, exhaust-type, or dust-type sample point cloud points.

[0130] For example, point cloud data collected by lidar under special weather conditions and / or special road conditions can be used as sample point cloud data.

[0131] S202: Rasterize the sample point cloud data to obtain local sample point cloud data contained in at least one sample raster; and determine the label information of each sample raster.

[0132] Here, the label information can specifically be the label value corresponding to the sample raster. If the number of target noisy point cloud points included in the sample raster is greater than a preset threshold, the label information can be a first label value; if the number of target noisy point cloud points included in the sample raster is not greater than the preset threshold, the label information can be a second label value. The second label value can also be the label value corresponding to other types of labels, where different types of labels can correspond to different label values.

[0133] In practice, the number of sample grids and the grid to which each sample point cloud point is located can be determined according to the preset grid size and the coordinates of each sample point cloud point in the world coordinate system. Each sample point cloud point within the same grid is considered as part of the local sample point cloud data contained within that grid. Simultaneously, the label information for each sample grid can be pre-determined.

[0134] In one embodiment, the label information for each sample raster can be determined by following these steps:

[0135] Step 1: For each sample point in the sample point cloud data, determine the point type; the point type includes noise type and non-noise type: the noise type can include at least one of rain type, fog type, dust type, exhaust gas type, and splash type.

[0136] Here, "non-noise type" means that the sample point cloud points are of other types. For example, lane line type, building type, vehicle type, etc. Noise type can include, but is not limited to, rain type, fog type, dust type, exhaust gas type, and splash type. The specific noise type can be set according to the actual noise reduction needs, and no specific limitation is made here.

[0137] In practice, after obtaining the sample point cloud data, a 3D bounding box of a preset size can be used to divide the sample point cloud data into regions according to a preset step size using a sliding window, so as to obtain multiple 3D bounding boxes and at least one sample point cloud point contained in each 3D bounding box.

[0138] Subsequently, for each 3D annotation box, the point type of each sample point cloud point included in the 3D annotation box can be determined manually based on the point cloud information of each sample point cloud point included in the 3D annotation box.

[0139] Step 2: Determine the target number of noise-type sample point cloud points in the local sample point cloud data contained in each sample raster.

[0140] In practice, for each sample grid obtained based on S202, the target number of sample point cloud points belonging to the noise type in the local sample point cloud data can be determined according to the point type of each sample point cloud point in the local sample point cloud data contained in the sample grid.

[0141] Step 3: Determine the label information of the sample grid based on the target quantity and preset quantity threshold corresponding to each sample grid.

[0142] Here, the target number corresponding to a sample raster is the target number of noise-type sample point cloud points in the local sample point cloud data contained in the sample raster; or, the target number corresponding to a sample raster is the number of noise-type sample point cloud points included in the sample raster. The preset number threshold can be a pre-set minimum number threshold. If the target number of noise-type sample point cloud points in a sample raster is not less than the preset number threshold, then the sample raster can be determined as a noise raster; conversely, if the target number of noise-type sample point cloud points in a sample raster is less than the preset number threshold, then the sample raster can be determined as a non-noise raster, such as a background raster or any pre-set type of raster.

[0143] For example, for each sample raster, the number of targets corresponding to that sample raster can be compared with a preset threshold. If the number of targets corresponding to the sample raster is not less than the preset threshold (e.g., 1), the label value corresponding to the annotation information of that sample raster is determined to be the first label value, which means that the sample raster is a noise raster. If the number of targets corresponding to the sample raster is less than the preset threshold, the label value corresponding to the annotation information of that sample raster is determined to be the second label value, which means that the sample raster is a non-noise raster.

[0144] In one possible implementation, after determining the point type of each sample point cloud included in each 3D annotation frame, a corresponding semantic label can also be assigned to each sample point cloud based on its point type. Specifically, the semantic label for rain-type sample point cloud points is "rain," for fog-type sample point cloud points is "fog," for dust-type sample point cloud points is "dust," for exhaust gas-type sample point cloud points is "exhaust gas," for splash-type sample point cloud points is "splash," and for non-noise-type sample point cloud points, the semantic label is "other."

[0145] Then, based on the semantic label of each sample point cloud point, the sample point cloud points belonging to the noise type can be determined from the multiple sample point cloud points included in the sample point data. The label value of each sample point cloud point belonging to the noise type is set to a first preset value, which means that the sample point cloud point is a sample noise point cloud point. At the same time, the label values ​​of each sample point cloud point included in the sample point data that does not belong to the noise type can be set to a second preset value, which means that the sample point cloud point is a point cloud point of other types.

[0146] Next, for each sample raster, the target number of sample point clouds with a first label value in the local sample point cloud data can be determined based on the label values ​​of each sample point cloud point in the local sample point cloud data corresponding to that sample raster. This is equivalent to determining the target number of sample noise point clouds in the local sample point cloud data. The target number corresponding to that sample raster is compared with a preset threshold. If the target number corresponding to the sample raster is not less than the preset threshold, the label value corresponding to the annotation information of that sample raster is determined to be the first label value, indicating that the sample raster is a noise raster. If the target number corresponding to the sample raster is less than the preset threshold, the label value corresponding to the annotation information of that sample raster is determined to be the second label value, indicating that the sample raster is a non-noise raster.

[0147] S203: Input the local sample point cloud data contained in each sample grid in at least one sample grid into the neural network to be trained, and generate the predicted probability of the sample grid including noisy point cloud points.

[0148] Here, the neural network to be trained is the denoising neural network to be trained. The prediction probability is the probability that the sample grid includes noisy point cloud points, as output by the neural network to be trained.

[0149] In practice, the neural network to be trained can be used to process the local sample point cloud data of each sample grid in turn, and output the predicted probability of each sample grid including noisy point cloud points.

[0150] S204: Based on the predicted probability of each sample grid, including the noisy point cloud points, and the label information of each sample grid, the neural network to be trained is iteratively trained until the training cutoff condition is met, thus obtaining the denoising neural network.

[0151] Here, the training cutoff condition may include reaching a preset number of iterations and / or the prediction accuracy of the trained neural network reaching a preset accuracy.

[0152] In practice, the binary cross-entropy function can be used to determine the binary cross-entropy loss of the neural network to be trained, based on the predicted probability of noisy point cloud points in the sample grid and the label value corresponding to the annotation information of the sample grid. The binary cross-entropy loss can include a first loss and a second loss. For example, if the label value corresponding to the annotation information of the sample grid is the first label value, the binary cross-entropy function can be used to determine the first loss based on the predicted probability of noisy point cloud points in the sample grid and the first label value. If the label value corresponding to the annotation information of the sample grid is the second label value, the binary cross-entropy function can be used to determine the second loss based on the predicted probability of noisy point cloud points in the sample grid and the second label value.

[0153] Then, the binary cross-entropy loss can be used to iteratively train the neural network to be trained until the training cutoff condition is met, thus obtaining the denoised neural network.

[0154] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0155] Based on the same inventive concept, this disclosure also provides a point cloud data denoising device corresponding to the point cloud data denoising method. Since the principle of the device in this disclosure for solving the problem is similar to the point cloud data denoising method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0156] like Figure 3 The diagram shown is a schematic of a point cloud data denoising device provided in an embodiment of this disclosure, comprising:

[0157] Module 301 is used to acquire point cloud data;

[0158] The segmentation module 302 is used to perform rasterization segmentation on the point cloud data to obtain local point cloud data contained in at least one target raster.

[0159] The determining module 303 is used to determine the probability that the target grid includes noisy point cloud points based on the local point cloud data contained in each target grid;

[0160] The update module 304 is used to update the point cloud data based on the probability that each of the target grids includes noisy point cloud points, so as to obtain updated point cloud data.

[0161] In one possible implementation, when the update module 304 updates the point cloud data based on the probability that each of the target grids includes noisy point cloud points to obtain updated point cloud data, it is used to delete the local point cloud data contained in the target grid if the probability that the target grid includes noisy point cloud points is greater than a preset probability, so as to obtain updated point cloud data.

[0162] In one possible implementation, the determining module 303, when determining the probability that a target grid includes noisy point cloud points based on the local point cloud data contained in each target grid, is used to extract noise features from the point cloud information of each point cloud point in the local point cloud data contained in each target grid to obtain intermediate feature information of each point cloud point.

[0163] The intermediate feature information of each point cloud point is fused to obtain the fused feature information of the target grid.

[0164] Based on the fused feature information, the probability that the target raster includes noisy point cloud points is determined.

[0165] In one possible implementation, the device further includes:

[0166] The control module 305 is used to determine the object information of each object to be detected based on the updated point cloud data after obtaining the updated point cloud data.

[0167] Based on the object information of each of the objects to be detected, the driving device is controlled to drive.

[0168] In one possible implementation, the point cloud data denoising method provided in the above embodiments is executed by a denoising neural network.

[0169] In one possible implementation, the device further includes:

[0170] Training module 306 is used to train the denoising neural network according to the following steps:

[0171] Acquire sample point cloud data;

[0172] The sample point cloud data is rasterized to obtain local sample point cloud data contained in at least one sample raster; and the label information of each sample raster is determined.

[0173] The local sample point cloud data contained in each of at least one of the sample grids is input into the neural network to be trained to generate the predicted probability of the sample grid including noisy point cloud points;

[0174] Based on the predicted probability of each sample grid including noisy point cloud points and the label information of each sample grid, the neural network to be trained is iteratively trained until the training cutoff condition is met, and the denoising neural network is obtained.

[0175] In one possible implementation, the training module 306, when determining the label information of each sample grid, is used to determine the point type of each sample point in the sample point cloud data; wherein, the point type includes noise type and non-noise type: the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type.

[0176] Determine the target number of noise-type sample point cloud points in the local sample point cloud data contained in each of the sample grids;

[0177] The label information of the sample grid is determined based on the target quantity and the preset quantity threshold corresponding to each sample grid.

[0178] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0179] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 4 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising:

[0180] The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores machine-readable instructions executable by the processor 41. The processor 41 executes these machine-readable instructions, and when executed, it performs the following steps: S101: Acquire point cloud data; S102: Rasterize the point cloud data to obtain local point cloud data contained in at least one target raster; S103: Determine the probability that each target raster contains noisy point cloud points based on the local point cloud data contained in each target raster; and S104: Update the point cloud data based on the probability that each target raster contains noisy point cloud points to obtain updated point cloud data.

[0181] The aforementioned memory 42 includes a main memory 421 and an external memory 422. The main memory 421, also known as internal memory, is used to temporarily store the computational data in the processor 41, as well as the data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the computer device is running, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes the execution instructions mentioned in the above method embodiments.

[0182] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the point cloud data denoising method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0183] The computer program product of the point cloud data denoising method provided in this disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the point cloud data denoising method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0184] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0190] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A method for denoising point cloud data, characterized in that, include: Acquire point cloud data; The point cloud data is rasterized to obtain local point cloud data contained in at least one target raster. For the local point cloud data contained in each target grid, noise features are extracted from the point cloud information of each point cloud point in the local point cloud data to obtain the intermediate feature information of each point cloud point. The intermediate feature information of each point cloud point is fused to obtain the fused feature information of the target grid; based on the fused feature information, the probability that the target grid includes noise point cloud points is determined; the noise point cloud points are point cloud points corresponding to objects that do not affect the driving of the driving device; the point type of the noise point cloud points is a noise type, and the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type; Based on the probability that each of the target grids includes noisy point cloud points, the point cloud data is updated to obtain updated point cloud data.

2. The method according to claim 1, characterized in that, The step of updating the point cloud data based on the probability that each of the target grids includes noisy point cloud points to obtain updated point cloud data includes: If the probability that the target grid includes noisy point cloud points is greater than a preset probability, the local point cloud data contained in the target grid is deleted to obtain updated point cloud data.

3. The method according to claim 1, characterized in that, After obtaining the updated point cloud data, the following is also included: Based on the updated point cloud data, determine the object information for each object to be detected; Based on the object information of each of the objects to be detected, the driving device is controlled to drive.

4. The method according to any one of claims 1 to 3, characterized in that, The denoising method for the point cloud data is executed by a denoising neural network.

5. The method according to claim 4, characterized in that, The denoising neural network is trained according to the following steps: Acquire sample point cloud data; The sample point cloud data is rasterized to obtain local sample point cloud data contained in at least one sample raster; and the label information of each sample raster is determined. The local sample point cloud data contained in each of at least one of the sample grids is input into the neural network to be trained to generate the predicted probability of the sample grid including noisy point cloud points; Based on the predicted probability of each sample grid including noisy point cloud points and the label information of each sample grid, the neural network to be trained is iteratively trained until the training cutoff condition is met, and the denoising neural network is obtained.

6. The method according to claim 5, characterized in that, The determination of the label information for each sample grid includes: For each sample point in the sample point cloud data, determine the point type of the sample point; wherein, the point type includes noise type and non-noise type: the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type; Determine the target number of noise-type sample point cloud points in the local sample point cloud data contained in each of the sample grids; The label information of the sample grid is determined based on the target quantity and the preset quantity threshold corresponding to each sample grid.

7. A noise reduction device for point cloud data, characterized in that, include: The acquisition module is used to acquire point cloud data; The segmentation module is used to perform rasterization segmentation on the point cloud data to obtain local point cloud data contained in at least one target raster. The determination module is used to extract noise features from the point cloud information of each point cloud point in the local point cloud data contained in each target grid, so as to obtain the intermediate feature information of each point cloud point. The intermediate feature information of each point cloud point is fused to obtain the fused feature information of the target grid; based on the fused feature information, the probability that the target grid includes noise point cloud points is determined; the noise point cloud points are point cloud points corresponding to objects that do not affect the driving of the driving device; the point type of the noise point cloud points is a noise type, and the noise type includes at least one of rain type, fog type, dust type, exhaust gas type, and splash type; The update module is used to update the point cloud data based on the probability that each target grid includes noisy point cloud points, so as to obtain updated point cloud data.

8. A computer device, characterized in that, include: The processor and the memory, wherein the memory stores machine-readable instructions executable by the processor, the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the processor performs the steps of the point cloud data denoising method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer device, performs the steps of the point cloud data denoising method as described in any one of claims 1 to 6.

Images