A point cloud denoising method and device based on two-dimensional multi-modal range images

Abstract

The application discloses a kind of point cloud denoising method and device based on two-dimensional multi-modal range image.Therein, method includes obtaining to be denoised point cloud data;The to-be-denozed point cloud data is projected and handled, and projection image is obtained;The projection image is handled with feature information insertion, and two-dimensional multi-modal range image is obtained;According to the two-dimensional convolution point cloud denoising model that is pre-trained and completed, the two-dimensional multi-modal range image is handled with offset prediction, and predicted offset is obtained;The predicted offset is handled with reverse projection, and point cloud offset is obtained;The point cloud offset is added with the denoised point cloud data, and target point cloud data is obtained.The embodiment of the application can carry out denoising processing to complete point cloud data, improve the processing efficiency of point cloud denoising, and can be widely applied in point cloud denoising technical field.

Description

Technical Field

[0001] This invention relates to the field of point cloud denoising technology, and in particular to a point cloud denoising method and apparatus based on two-dimensional multimodal range images. Background Technology

[0002] With the rapid development of sensor devices in recent years, point cloud data has been widely used in an increasing number of fields. Point cloud denoising is a commonly used point cloud data preprocessing method that removes singular noise points from point cloud data, providing good data support for downstream tasks. Among related technologies, neural network-based point cloud denoising methods require inputting the point cloud into the network model as point-by-point patches, significantly increasing the denoising time required for a complete point cloud. In summary, the technical problems existing in these technologies urgently need to be solved. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a point cloud denoising method and apparatus based on two-dimensional multimodal range images to quickly denoise point cloud data.

[0004] On one hand, the present invention provides a point cloud denoising method based on two-dimensional multimodal range images, comprising:

[0005] Acquire cloud data containing noise points to be removed;

[0006] The point cloud data to be denoised is subjected to projection processing to obtain a projected image;

[0007] The projected image is subjected to feature information insertion processing to obtain a two-dimensional multimodal range image;

[0008] The offset prediction process is performed on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset.

[0009] The predicted offset is back-projected to obtain the point cloud offset;

[0010] The point cloud offset is added to the denoised point cloud data to obtain the target point cloud data.

[0011] Optionally, the projection processing of the point cloud data to be denoised to obtain a projected image includes:

[0012] Obtain the three-dimensional coordinates of the denoised point cloud data and the vertical field of view of the point cloud sensor;

[0013] The projection length and projection width are determined based on the point cloud sensor.

[0014] By combining the projection formula, the three-dimensional coordinates, vertical field of view, projection length, and projection width are calculated to obtain the projected image.

[0015] Optionally, the step of inserting feature information into the projected image to obtain a two-dimensional multimodal range image includes:

[0016] Obtain the intensity information of the point cloud data to be denoised;

[0017] The point cloud data to be denoised is subjected to surface fitting processing to obtain normal information;

[0018] The intensity information and the normal information are combined and processed to obtain feature information;

[0019] The feature information is inserted into the projected image to obtain a two-dimensional multimodal range image.

[0020] Optionally, the step of performing offset prediction processing on the two-dimensional multimodal range image based on a pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset includes:

[0021] The two-dimensional multimodal range image is input into the two-dimensional convolutional point cloud denoising model, which includes an input preprocessing convolutional layer, a ResNet50 neural network, and an output decoding convolutional layer.

[0022] The two-dimensional multimodal range image is updated by performing feature update processing through the input preprocessing convolutional layer to obtain an updated image;

[0023] The updated image is processed by the ResNet50 neural network to extract image features.

[0024] The image features are deconvolved by the output decoding convolutional layer to obtain the predicted offset.

[0025] Optionally, the back-projection process of the predicted offset to obtain the point cloud offset includes:

[0026] Obtain the focal length value of the point cloud sensor, as well as the pixel coordinates and depth values ​​of the projected image;

[0027] The point cloud offset is obtained by calculating the predicted offset, focal length, pixel coordinates, and depth value using the back projection formula.

[0028] Optionally, before performing offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model, the method further includes pre-training the two-dimensional convolutional point cloud denoising model, specifically including:

[0029] Acquire pre-training point cloud data and perform annotation processing on the pre-training point cloud data to obtain a training dataset;

[0030] The training dataset is input into the two-dimensional convolutional point cloud denoising model to obtain the point cloud offset prediction result;

[0031] The training loss value is determined based on the point cloud offset prediction results and the labels of the training dataset;

[0032] The parameters of the two-dimensional convolutional point cloud denoising model are updated based on the loss value.

[0033] Optionally, before performing feature extraction processing on the updated image using the ResNet50 neural network, the process further includes pre-training the ResNet50 neural network, including:

[0034] The ResNet50 neural network is initialized using an ImageNet pre-trained model.

[0035] The fully connected layers of the ResNet50 neural network are removed.

[0036] On the other hand, embodiments of the present invention also provide a point cloud denoising device based on two-dimensional multimodal range images, the device comprising:

[0037] The first module is used to acquire the point cloud data to be denoised;

[0038] The second module is used to perform projection processing on the point cloud data to be denoised to obtain a projected image.

[0039] The third module is used to perform feature information insertion processing on the projected image to obtain a two-dimensional multimodal range image;

[0040] The fourth module is used to perform offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset.

[0041] The fifth module is used to perform back projection processing on the predicted offset to obtain the point cloud offset;

[0042] The sixth module is used to add the point cloud offset to the denoised point cloud data to obtain the target point cloud data.

[0043] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory;

[0044] The memory is used to store programs;

[0045] The processor executes the program to implement the method described above.

[0046] On the other hand, embodiments of the present invention also disclose a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0047] On the other hand, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: The embodiments of the present invention obtain a projected image by projecting the point cloud data to be denoised, and then process the projected image by a two-dimensional convolutional point cloud denoising model, which can denoise the complete point cloud data and improve the processing efficiency of point cloud denoising. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a point cloud denoising method based on a two-dimensional multimodal range image provided in an embodiment of this application;

[0051] Figure 2 This is a schematic diagram of the structure of a point cloud denoising neural network model provided in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the structure of a two-dimensional convolutional point cloud denoising model provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the structure of a point cloud denoising device based on a two-dimensional multimodal range image provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] First, let's analyze some of the terms used in this application:

[0057] Two-dimensional multimodal range image: A three-dimensional point cloud is transformed into a two-dimensional image through spherical projection and stored. Each pixel can hold multimodal information.

[0058] Point cloud denoising: This is a point cloud data preprocessing method that uses algorithms to remove singular noise points from the original point cloud.

[0059] Point cloud denoising methods in related technologies can be divided into two main categories: The first category is traditional point cloud denoising methods, which can be further subdivided into filtering-based, geometry-based, and statistical-based methods. These methods are generally fast, but they are difficult to adapt to various types of data. The second category is methods based on deep learning models. These methods have better denoising effects than traditional methods, but they also have disadvantages such as requiring training data, limited versatility, and slow processing speed.

[0060] In view of this, this application provides a point cloud denoising method based on two-dimensional multimodal range images. This point cloud denoising method can be applied to a terminal, a server, or software running on either a terminal or a server. The terminal can be a tablet, laptop, desktop computer, etc., but is not limited to these. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0061] Reference Figure 1 This invention provides a point cloud denoising method based on two-dimensional multimodal range images, comprising:

[0062] S101. Obtain the cloud data of the points to be denoised;

[0063] S102. Perform projection processing on the point cloud data to be denoised to obtain a projected image;

[0064] S103. Perform feature information insertion processing on the projected image to obtain a two-dimensional multimodal range image;

[0065] S104. Perform offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset.

[0066] S105. Perform back projection processing on the predicted offset to obtain the point cloud offset;

[0067] S106. Add the point cloud offset to the denoised point cloud data to obtain the target point cloud data.

[0068] In this embodiment of the invention, point cloud data to be denoised is acquired using a point cloud sensor, which may include a LiDAR, a stereo camera, or a time-lapse camera. The point cloud sensor automatically measures information from a large number of points on the surface of an object, and then outputs the point cloud data to be denoised through a data file. The point cloud data to be denoised is then projected into 2D multimodal range images. Specifically, a spherical projection method is used to project the point cloud data to obtain a projected image. Feature information is then added to the projected image to obtain a 2D multimodal range image, which further improves the denoising effect of the neural network. The 2D multimodal range image is input into a 2D convolutional point cloud denoising model, and a predicted offset is output. Finally, the predicted offset is back-projected to obtain a point cloud offset, which is then added to the denoised point cloud data to obtain the target point cloud data, which is the point cloud data after denoising.

[0069] Reference Figure 2 The input to the point cloud denoising neural network model is a point cloud of size N×3, where N is the number of points and 3 represents the three coordinates (x, y, z). This point cloud is projected to obtain a 2D multimodal range image of size H×W×8, where H is the height and W is the width. This 2D multimodal range image is then input into a 2D convolutional point cloud denoising neural network, outputting a predicted offset of size H×W×3. This predicted offset is back-projected to obtain a point cloud offset of size N×3. This offset is then added to the original point cloud offset to obtain a denoised point cloud of size N×3.

[0070] As a further optional implementation, in step S102 above, the number of cloud points to be denoised is...

[0071] The image is obtained by projection processing, including:

[0072] Obtain the three-dimensional coordinates of the denoised point cloud data and the vertical field of view of the point cloud sensor;

[0073] The projection length and projection width are determined based on the point cloud sensor.

[0074] By combining the projection formula, the three-dimensional coordinates, vertical field of view, projection length, and projection width are calculated to obtain the projected image.

[0075] In this embodiment of the invention, the three-dimensional coordinates of the denoised point cloud data and the vertical field of view of the point cloud sensor are acquired. The projection length and projection width can be determined based on the point cloud sensor. In one feasible embodiment, a lidar is used as the point cloud sensor. The number of laser beams from the lidar is set as the width of the projected image, and the number of points acquired by the lasers is calculated based on the maximum horizontal resolution of the lidar. This number of points is then set as the width of the projected image. Alternatively, the projection length and projection width can be set to multiples of 2 to allow for subsequent neural network processing. Then, the three-dimensional coordinates, vertical field of view, projection length, and projection width are calculated using a projection formula to obtain the projected image. The projection formula is as follows:

[0076]

[0077] Where H and W represent the length and width of the generated two-dimensional multimodal range image, respectively. f = f u +f d , representing the vertical field of view of the sensor acquiring the point cloud, f u f represents the maximum field of view. d The minimum field of view is represented by . u and v represent the calculated coordinates of the two-dimensional projected image, respectively. d represents the Euclidean distance between each point in the point cloud data to be denoised and the origin, calculated as follows:

[0078]

[0079] This invention uses the projection formula to project the point cloud data to be denoised onto a two-dimensional plane to obtain a projected image, which can be denoised in one step, thereby improving the processing efficiency of point cloud denoising.

[0080] As a further optional implementation, in step S103 above, the step of inserting feature information into the projected image to obtain a two-dimensional multimodal range image includes:

[0081] Obtain the intensity information of the point cloud data to be denoised;

[0082] The point cloud data to be denoised is subjected to surface fitting processing to obtain normal information;

[0083] The intensity information and the normal information are combined and processed to obtain feature information;

[0084] The feature information is inserted into the projected image to obtain a two-dimensional multimodal range image.

[0085] In this embodiment of the invention, the feature information is additional information used to improve the denoising effect of the point cloud. This feature information can be set as (r, dx, dy, dz), and is added to each point in the point cloud data to be denoised. Here, r represents the intensity information of each point, and (dx, dy, dz) represents the normal information of each point. This embodiment of the invention uses a point cloud normal vector method based on surface fitting to obtain the normal information. The intensity information and the normal information are combined to obtain the feature information, which is then inserted into the projected image to obtain a two-dimensional multimodal range image. The normal information indicates the direction and orientation of the surface at the location of the point. Since the normal information of noise points is usually more prominent, adding normal information to the two-dimensional multimodal range image helps to further improve the denoising effect of the neural network. This method can insert arbitrary additional feature information into the two-dimensional multimodal range image, enhancing the robustness of the method.

[0086] As a further optional implementation, the step of performing offset prediction processing on the two-dimensional multimodal range image based on a pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset includes:

[0087] The two-dimensional multimodal range image is input into the two-dimensional convolutional point cloud denoising model, which includes an input preprocessing convolutional layer, a ResNet50 neural network, and an output decoding convolutional layer.

[0088] The two-dimensional multimodal range image is updated using the input preprocessing convolutional layer.

[0089] Get the updated image;

[0090] The updated image is processed by the ResNet50 neural network to extract image features.

[0091] The image features are deconvolved by the output decoding convolutional layer to obtain the predicted offset.

[0092] Reference Figure 3The two-dimensional convolutional point cloud denoising model includes an input preprocessing convolutional layer, a ResNet50 neural network, and an output decoding convolutional layer. The input preprocessing convolutional layer updates the features of the two-dimensional multimodal range image of the point cloud. The updated image is then used as input to the ResNet50 neural network. The input preprocessing convolutional layer consists of three 3×3 two-dimensional convolutional layers. The second part is the ResNet50 neural network, which contains 49 convolutional layers and one fully connected layer. The ResNet50 neural network extracts features from the updated image to obtain image features. The output decoding convolutional layer consists of five deconvolutional layers. Its main function is to reprocess the output of the ResNet50 neural network to the size of the input image, i.e., H×W×3. The output decoding convolutional layer deconvolves the image features to obtain the predicted offset.

[0093] As a further optional implementation, the back-projection processing of the predicted offset to obtain the point cloud offset includes:

[0094] Obtain the focal length value of the point cloud sensor, as well as the pixel coordinates and depth values ​​of the projected image;

[0095] The point cloud offset is obtained by calculating the predicted offset, focal length, pixel coordinates, and depth value using the back projection formula.

[0096] In this embodiment of the invention, the point cloud sensor is a stereo camera. First, the focal length value of the stereo camera, as well as the pixel coordinates and depth values ​​of the projected image, are acquired. Then, the predicted offset, focal length value, pixel coordinates, and depth values ​​are calculated using a back projection formula to obtain the point cloud offset. The back projection formula is as follows:

[0097]

[0098]

[0099] z′=z

[0100] Where (x, y) are the pixel coordinates in the image, (x', y', z') are the coordinates in the point cloud, (cx, cy) are the coordinates of the camera's optical center in the image, (fx, fy) is the camera's focal length, and z is the depth value. (x-cx) and (y-cy) can be represented as the predicted offsets. Finally, the point cloud offsets are added point-to-point to the input point cloud to be denoised, thus completing the point cloud denoising process.

[0101] As a further optional implementation, before performing offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model, the method further includes pre-training the two-dimensional convolutional point cloud denoising model, specifically including:

[0102] Acquire pre-training point cloud data and perform annotation processing on the pre-training point cloud data to obtain a training dataset;

[0103] The training dataset is input into the two-dimensional convolutional point cloud denoising model to obtain the point cloud offset prediction result;

[0104] The training loss value is determined based on the point cloud offset prediction results and the labels of the training dataset;

[0105] The parameters of the two-dimensional convolutional point cloud denoising model are updated based on the loss value.

[0106] In this embodiment of the invention, pre-trained point cloud data is acquired and labeled to obtain a training dataset. It should be noted that this embodiment can also divide the labeled pre-trained point cloud data dataset into a training set, a validation set, and a test set. The training set is used to train a point cloud denoising model based on two-dimensional multimodal range images until the model loss value converges. The model parameters are saved based on the performance of the validation set. The trained point cloud denoising model based on two-dimensional multimodal range images is then used to denoise the point cloud data in the test set. The training dataset can be input into an initialized point cloud pole recognition model for training. Specifically, after inputting the data from the training dataset into the initialized two-dimensional convolutional point cloud denoising model, the model's output prediction result, i.e., the point cloud offset prediction result, can be obtained. The accuracy of the recognition model's prediction can be evaluated based on the point cloud offset prediction result and the aforementioned labels, thereby updating the model's parameters. For 2D convolutional point cloud denoising models, the accuracy of the model's prediction results can be measured by a loss function. The loss function is defined on a single training data point and measures the prediction error of that data point. Specifically, the loss value is determined by the label of the individual training data point and the model's prediction result for that data point. However, in actual training, a training dataset contains many data points. Therefore, a cost function is generally used to measure the overall error of the training dataset. The cost function is defined on the entire training dataset and calculates the average prediction error of all training data points, providing a better measure of the model's prediction performance. For general machine learning models, the aforementioned cost function, plus a regularization term to measure model complexity, can serve as the training objective function. Based on this objective function, the loss value of the entire training dataset can be calculated. Many types of loss functions are commonly used, such as 0-1 loss, squared loss, absolute loss, logarithmic loss, and cross-entropy loss, which can all be used as loss functions for machine learning models. These will not be elaborated upon here. In this embodiment, the mean squared error loss function or any of the above loss functions can be used to determine the training loss value. Based on the training loss value, the backpropagation algorithm is used to update the model parameters. After several iterations, a well-trained point cloud tower recognition model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the accuracy requirement on the test set is met.

[0107] As a further optional implementation, before performing feature extraction processing on the updated image using the ResNet50 neural network, the process further includes pre-training the ResNet50 neural network, including:

[0108] The ResNet50 neural network is initialized using an ImageNet pre-trained model.

[0109] The fully connected layers of the ResNet50 neural network are removed.

[0110] In this embodiment of the invention, the parameters are initialized using an ImageNet pre-trained model, and the fully connected layers of the ResNet50 neural network are removed. By using the pre-trained model, this embodiment of the invention can improve training performance and make the training process faster and more stable.

[0111] On the other hand, refer to Figure 4 This invention also provides a point cloud denoising device based on a two-dimensional multimodal range image, the device comprising:

[0112] The first module 401 is used to acquire the point cloud data to be denoised;

[0113] The second module 402 is used to perform projection processing on the point cloud data to be denoised to obtain a projected image.

[0114] The third module 403 is used to perform feature information insertion processing on the projected image to obtain a two-dimensional multimodal range image;

[0115] The fourth module 404 is used to perform offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset.

[0116] The fifth module 405 is used to perform back projection processing on the predicted offset to obtain the point cloud offset;

[0117] The sixth module 406 is used to add the point cloud offset to the denoised point cloud data to obtain the target point cloud data.

[0118] Reference Figure 5 The present invention also provides an electronic device, including a processor 501 and a memory 502; the memory is used to store a program; the processor executes the program to implement the method described above.

[0119] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0120] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0121] In summary, the embodiments of the present invention have the following advantages: The embodiments of the present invention employ a point cloud denoising model based on a two-dimensional multimodal range image, improving the versatility and practicality of point cloud data denoising. Furthermore, in the point cloud denoising model based on a two-dimensional multimodal range image, the point cloud is projected as a 2D multimodal range image, enabling one-time denoising of the entire point cloud, thus improving the efficiency of point cloud denoising. Moreover, the point cloud denoising model of the embodiments of the present invention has a fast training speed; by using a mature 2D convolutional pre-trained model, a stable point cloud denoising model can be obtained more quickly.

[0122] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0123] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0126] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0127] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0128] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0129] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0130] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A point cloud denoising method based on two-dimensional multimodal range images, characterized in that, The method includes: Acquire cloud data containing noise points to be removed; The point cloud data to be denoised is subjected to projection processing to obtain a projected image; The projected image is subjected to feature information insertion processing to obtain a two-dimensional multimodal range image; The offset prediction process is performed on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset. The predicted offset is back-projected to obtain the point cloud offset; The point cloud offset is added to the denoised point cloud data to obtain the target point cloud data; The projection processing of the point cloud data to be denoised to obtain a projected image includes: Obtain the three-dimensional coordinates of the denoised point cloud data and the vertical field of view of the point cloud sensor; The projection length and projection width are determined based on the point cloud sensor. By combining the projection formula, the three-dimensional coordinates, vertical field of view, projection length, and projection width are calculated and processed to obtain the projected image; The expression for the projection formula is: Where H and W represent the length and width of the generated two-dimensional multimodal range image, respectively; f = f u +f d , representing the vertical field of view of the sensor acquiring the point cloud, f u f represents the maximum field of view. d denoted by minimum field of view; u and v represent the calculated coordinates of the two-dimensional projected image, respectively; d represents the Euclidean distance between each point in the point cloud data to be denoised and the origin.

2. The method according to claim 1, characterized in that, The step of inserting feature information into the projected image to obtain a two-dimensional multimodal range image includes: Obtain the intensity information of the point cloud data to be denoised; The point cloud data to be denoised is subjected to surface fitting processing to obtain normal information; The intensity information and the normal information are combined and processed to obtain feature information; The feature information is inserted into the projected image to obtain a two-dimensional multimodal range image.

3. The method according to claim 1, characterized in that, The step of performing offset prediction processing on the two-dimensional multimodal range image based on a pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset includes: The two-dimensional multimodal range image is input into the two-dimensional convolutional point cloud denoising model, which includes an input preprocessing convolutional layer, a ResNet50 neural network, and an output decoding convolutional layer. The two-dimensional multimodal range image is updated by performing feature update processing through the input preprocessing convolutional layer to obtain an updated image; The updated image is processed by the ResNet50 neural network to extract image features. The image features are deconvolved by the output decoding convolutional layer to obtain the predicted offset.

4. The method according to claim 1, characterized in that, The back projection processing of the predicted offset to obtain the point cloud offset includes: Obtain the focal length value of the point cloud sensor, as well as the pixel coordinates and depth values ​​of the projected image; The point cloud offset is obtained by calculating the predicted offset, focal length, pixel coordinates, and depth value using the back projection formula.

5. The method according to any one of claims 1 to 4, characterized in that, Before performing offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model, the method further includes pre-training the two-dimensional convolutional point cloud denoising model, specifically including: Acquire pre-training point cloud data and perform annotation processing on the pre-training point cloud data to obtain a training dataset; The training dataset is input into the two-dimensional convolutional point cloud denoising model to obtain the point cloud offset prediction result; The training loss value is determined based on the point cloud offset prediction results and the labels of the training dataset; The parameters of the two-dimensional convolutional point cloud denoising model are updated based on the loss value.

6. The method according to claim 3, characterized in that, Before performing feature extraction processing on the updated image using the ResNet50 neural network, the process further includes pre-training the ResNet50 neural network, including: The ResNet50 neural network is initialized using an ImageNet pre-trained model. The fully connected layers of the ResNet50 neural network are removed.

7. A point cloud denoising device based on two-dimensional multimodal range images, characterized in that, The device includes: The first module is used to acquire the point cloud data to be denoised; The second module is used to perform projection processing on the point cloud data to be denoised to obtain a projected image. The third module is used to perform feature information insertion processing on the projected image to obtain a two-dimensional multimodal range image; The fourth module is used to perform offset prediction processing on the two-dimensional multimodal range image based on the pre-trained two-dimensional convolutional point cloud denoising model to obtain the predicted offset. The fifth module is used to perform back projection processing on the predicted offset to obtain the point cloud offset; The sixth module is used to add the point cloud offset to the denoised point cloud data to obtain the target point cloud data; The second module is used to perform projection processing on the point cloud data to be denoised to obtain a projected image, including: Obtain the three-dimensional coordinates of the denoised point cloud data and the vertical field of view of the point cloud sensor; The projection length and projection width are determined based on the point cloud sensor. By combining the projection formula, the three-dimensional coordinates, vertical field of view, projection length, and projection width are calculated and processed to obtain the projected image; The expression for the projection formula is: Where H and W represent the length and width of the generated two-dimensional multimodal range image, respectively; f = f u +f d , representing the vertical field of view of the sensor acquiring the point cloud, f u f represents the maximum field of view. d denoted by minimum field of view; u and v represent the calculated coordinates of the two-dimensional projected image, respectively; d represents the Euclidean distance between each point in the point cloud data to be denoised and the origin.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store programs; The processor executes the program to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images