Methods, apparatus, equipment and storage media for generating point cloud representations of 3D point clouds

By training a network through hierarchical projection and feature fusion on 3D point clouds, a high-precision point cloud representation is generated, which solves the problem of low accuracy in representation learning in existing methods and realizes the effective acquisition and processing of 3D object information.

CN115170632BActive Publication Date: 2026-11-10PING AN TECH (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210822336.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-12
Publication Date
2026-11-10
Estimated Expiration
2042-07-12

AI Technical Summary

Technical Problem

Among existing methods for generating 3D point cloud representations, those based on point representation, voxel representation, and graph representation have low learning accuracy.

Method used

The target 3D point cloud is projected in layers using a preset number of layers and projection surface grid size to generate a projection feature map set. The point cloud representation is then generated using a point cloud representation generation model based on masking technology, deep autoencoder, and feature fusion network.

Benefits of technology

It improves the learning accuracy of point cloud representation, ensures the effectiveness of acquiring internal information of 3D objects and subsequent processing, and enhances the robustness of point cloud representation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115170632B_ABST
    Figure CN115170632B_ABST
Patent Text Reader

Abstract

This application relates to the field of artificial intelligence technology, and discloses a method, apparatus, device, and storage medium for generating point cloud representations of 3D point clouds. The method includes: performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set; inputting the projection feature map set into a preset point cloud representation generation model to generate a point cloud representation, thereby obtaining the target point cloud representation. The point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for the effective learning of subsequent point cloud representations. Feature fusion enables the utilization of multi-scale feature information, allowing the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, thus helping to improve the accuracy of representation learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for generating point cloud representations of three-dimensional point clouds. Background Technology

[0002] 3D point clouds are sets of (x, y, z) three-dimensional data obtained by scanning real-world objects using devices such as LiDAR and depth cameras. They provide complementary information to two-dimensional images, including geometric structure and spatial dimensions, further improving the accuracy and robustness of machine learning of the real world. The processing of 3D point clouds can be divided into upstream tasks of learning and modeling point cloud features, and downstream tasks of using point cloud feature representations for actual classification and segmentation. Point cloud representation is crucial for downstream tasks. There are three methods for modeling point cloud representations: point representation-based learning, voxel representation-based learning, and graph representation-based learning. These three methods suffer from low accuracy in representation learning. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, and storage medium for generating point cloud representations of three-dimensional point clouds, aiming to solve the technical problem that the accuracy of representation learning is low in the three methods of point cloud representation modeling: learning based on point representation, learning based on voxel representation, and learning based on graph representation.

[0004] To achieve the aforementioned objectives, this application proposes a method for generating point cloud representations of three-dimensional point clouds, the method comprising:

[0005] Obtain the target 3D point cloud;

[0006] Using a preset number of layers and a preset projection surface grid size, the target 3D point cloud is projected in layers to obtain a projection feature map set;

[0007] The projection feature map set is input into a preset point cloud representation generation model to generate a point cloud representation, thereby obtaining a target point cloud representation. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network.

[0008] Furthermore, the step of performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set includes:

[0009] The target 3D point cloud is normalized to obtain the 3D point cloud to be predicted.

[0010] Choose any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane;

[0011] Using the number of layers, the three-dimensional point cloud to be predicted is layered perpendicular to the first projection plane to obtain multiple single-layer three-dimensional point clouds corresponding to the first projection plane.

[0012] Using the projection surface grid size, each single-layer 3D point cloud corresponding to the first projection surface is projected onto the first projection surface to obtain a single-layer projection feature map corresponding to the first projection surface.

[0013] Repeat the step of obtaining any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane until the acquisition of each of the x-projection plane, y-projection plane, and z-projection plane is completed;

[0014] Each of the single-layer projection feature maps is used as the projection feature map set.

[0015] Furthermore, the step of inputting the projected feature map set into a preset point cloud representation generation model to generate a point cloud representation and obtain the target point cloud representation includes:

[0016] The single-layer projection feature maps corresponding to each projection surface in the projection feature map set are stitched together in the channel dimension to obtain a stitched feature map.

[0017] Each of the stitched feature maps is input into the point cloud representation generation model to generate point cloud representation, thereby obtaining single projection surface feature data.

[0018] Each of the single projection surface feature data is used as the target point cloud representation.

[0019] Furthermore, the point cloud representation generation model includes: two information heatmap generation modules and a feature extraction module corresponding to each projection surface;

[0020] The feature extraction module includes: a CNN skeleton submodule and two feature fusion submodules, wherein the first feature fusion submodule is located between the first residual block and the second residual block of the CNN skeleton submodule, the second feature fusion submodule is located between the third residual block and the fourth residual block of the CNN skeleton submodule, the first information heatmap generation module is connected to each of the first feature fusion submodules, and the second information heatmap generation module is connected to each of the second feature fusion submodules.

[0021] The first feature fusion submodule of the x-projection plane, the first feature fusion submodule of the y-projection plane, and the first feature fusion submodule of the z-projection plane are connected to each other;

[0022] The second feature fusion submodule of the x-projection plane, the second feature fusion submodule of the y-projection plane, and the second feature fusion submodule of the z-projection plane are connected to each other.

[0023] Furthermore, the feature fusion submodule includes: an instance normalization unit and a feature fusion unit;

[0024] The output of the instance normalization unit of the first feature fusion submodule is used as the input of the first information heatmap generation module;

[0025] The output of the first information heatmap generation module is used as the input of the feature fusion unit of the first feature fusion submodule;

[0026] The output of the instance normalization unit of the second feature fusion submodule is used as the input of the second information heatmap generation module;

[0027] The output of the second information heatmap generation module is used as the input of the feature fusion unit of the second feature fusion submodule.

[0028] Furthermore, the feature fusion unit of the first feature fusion submodule is used to obtain any one of the projection planes from the x-projection plane, the y-projection plane, and the z-projection plane as the second projection plane, add the element-values ​​at the same position of the outputs of the instance normalization unit of the first feature fusion submodule for each of the projection planes other than the second projection plane to obtain a first feature map, multiply the first feature map by the element-values ​​at the same position of the output of the first information heatmap generation module to obtain a second feature map, and add the element-values ​​at the same position of the second feature map and the output of the instance normalization unit of the first feature fusion submodule of the second projection plane to obtain the fused feature corresponding to the second projection plane, wherein the fused feature corresponding to the second projection plane is the second residual block corresponding to the second projection plane;

[0029] The feature fusion unit of the second feature fusion submodule is used to obtain any one of the projection planes from the x-projection plane, the y-projection plane, and the z-projection plane as the third projection plane. The outputs of the instance normalization units of the second feature fusion submodule for each of the projection planes other than the third projection plane are summed at the same position to obtain a third feature map. The third feature map is multiplied at the same position by the output of the second information heatmap generation module to obtain a fourth feature map. The fourth feature map is summed at the same position by the output of the instance normalization unit of the second feature fusion submodule for the third projection plane to obtain the fused feature corresponding to the third projection plane. The fused feature corresponding to the third projection plane is the input of the fourth residual block on the third projection plane.

[0030] Furthermore, the step of inputting the projected feature map set into a preset point cloud representation generation model to generate a point cloud representation and obtain the target point cloud representation includes:

[0031] Multiple training samples and a model to be trained are obtained, wherein the model to be trained includes, in sequence, an initial model and a decoder;

[0032] Use any one of the training samples as the target sample;

[0033] The projection feature map sample set in the target sample is input into the model to be trained to reconstruct the projection map, and the projection map reconstruction result is obtained.

[0034] The perceptual loss is calculated based on the reconstruction result of the projection map and the projection map labels in the target sample;

[0035] Update the network parameters of the model to be trained based on the perceptual loss;

[0036] Repeat the step of using any one of the training samples as the target sample until the preset model training termination condition is reached, and use the initial model in the model to be trained that has reached the model training termination condition as the point cloud representation generation model.

[0037] The method for generating the projection feature map sample set includes: acquiring a three-dimensional point cloud sample, masking the three-dimensional point cloud sample to obtain an unmasked three-dimensional point cloud, and performing layered projection on the unmasked three-dimensional point cloud using the number of layers and the projection surface grid size to obtain the projection feature map sample set.

[0038] This application also proposes a point cloud representation generation device for three-dimensional point clouds, the device comprising:

[0039] The data acquisition module is used to acquire the target's 3D point cloud.

[0040] The layered projection module is used to perform layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set.

[0041] The target point cloud representation determination module is used to input the projection feature map set into a preset point cloud representation generation model to generate point cloud representations and obtain target point cloud representations. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network.

[0042] This application also proposes a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0043] This application also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0044] This application discloses a method, apparatus, device, and storage medium for generating point cloud representations of 3D point clouds. The method involves performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set. Layered projection represents the 3D data in two dimensions. Projection information from any two directions can essentially determine the point cloud's position information. Simultaneously, the information from the two-dimensional image facilitates processing using currently mature and powerful skeleton networks. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for effective learning of subsequent point cloud representations. The projection feature map set is input into a preset point cloud representation generation model to generate the target point cloud representation. The point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. The feature fusion network improves the interaction between features of various projection surfaces, realizing the utilization of multi-scale feature information. This enables the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, helping to improve the accuracy of representation learning. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a method for generating a point cloud representation of a three-dimensional point cloud according to an embodiment of this application.

[0046] Figure 2 This is a schematic block diagram of a three-dimensional point cloud representation generation device according to an embodiment of this application.

[0047] Figure 3 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0048] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] Reference Figure 1 This application provides a method for generating point cloud representations of three-dimensional point clouds, the method comprising:

[0051] S1: Obtain the target 3D point cloud;

[0052] S2: Using a preset number of layers and a preset projection surface grid size, the target 3D point cloud is projected in layers to obtain a projection feature map set;

[0053] S3: Input the projection feature map set into a preset point cloud representation generation model to generate point cloud representations and obtain the target point cloud representation. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network.

[0054] This embodiment employs a preset number of layers and a preset projection surface grid size to perform layered projection on the target 3D point cloud, obtaining a projection feature map set. Layered projection represents the 3D data in 2D form. Projection information from any two directions can essentially determine the point cloud's position information. Simultaneously, the information from the 2D image facilitates processing using currently mature and powerful skeleton networks. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for effective learning of subsequent point cloud representations. The projection feature map set is input into a preset point cloud representation generation model to generate the target point cloud representation. This point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. The feature fusion network improves the interaction between features of various projection surfaces, realizing the utilization of multi-scale feature information. This allows the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, contributing to improved accuracy in representation learning.

[0055] For S1, it can obtain the target 3D point cloud input by the user, obtain the target 3D point cloud from the database, or obtain the target 3D point cloud from a third-party application.

[0056] A target 3D point cloud is a 3D point cloud that needs to be represented by point cloud data. A target 3D point cloud is a 3D point cloud generated for the same object.

[0057] For S2, the target 3D point cloud is layered using a preset number of layers. The projection surface grid size is used to project each data obtained from the layering onto a projection surface, and the resulting projection feature maps are used as a projection feature map set.

[0058] The number of layers is an integer greater than 0.

[0059] The projection surface grid size is the size of a projection surface. The projection surface grid size is an n×m grid. n can be equal to m or not equal to m. Both n and m are integers greater than 0.

[0060] Optionally, the projection surface grid size is a 256×256 grid.

[0061] The range of values ​​for the projection planes includes: the x-projection plane along the x-axis, the y-projection plane along the y-axis, and the z-projection plane along the z-axis. The x-axis is the x-axis in the three-dimensional coordinate system, the y-axis is the y-axis in the three-dimensional coordinate system, and the z-axis is the z-axis in the three-dimensional coordinate system.

[0062] It is understood that the projection surface grid sizes of the x-projection plane in the x-axis direction, the y-projection plane in the y-axis direction, and the z-projection plane in the z-axis direction are all the same.

[0063] The number of projection feature maps in the projection feature map set is equal to the product of the number of layers and the number of projection surfaces. For example, if the projection surfaces include three projection surfaces: the x-projection surface in the x-axis direction, the y-projection surface in the y-axis direction, and the z-projection surface in the z-axis direction, and the number of layers is 3, then the number of projection feature maps in the projection feature map set is equal to 9.

[0064] Projection involves calculating the number of point clouds in the grid that falls onto the projection surface, and using the calculated number as the pixel value of that grid. In other words, the pixel value of a pixel in the projected feature map is the number of point clouds in the grid corresponding to that pixel.

[0065] Optionally, a preset number of layers will be used to divide the target 3D point cloud into layers along the vertical z-axis direction. Each layer of data will be projected onto each projection plane, and the data obtained from the projection onto a projection plane will be used as a projection feature map.

[0066] For S3, the projection feature map set is input into a preset point cloud representation generation model to generate point cloud representations, and the generated point cloud representations are used as target point cloud representations.

[0067] In this process, multiple training samples are used to train the model based on a deep autoencoder, a feature fusion network, and a decoder. The part of the trained model other than the decoder is used as the point cloud representation generation model.

[0068] The training samples include a set of projected feature maps and projected map labels. The set of projected feature maps is a collection of feature maps obtained based on masking techniques and hierarchical projection. The projected map labels are the complete set of feature maps corresponding to the hierarchical projection feature map collection. This enables self-supervised learning of the deep autoencoder. It is understood that the hierarchical projection method for the training samples is the same as the hierarchical projection method for the target 3D point cloud.

[0069] A deep autoencoder is an encoder based on a CNN (convolutional neural network) skeleton.

[0070] Feature fusion networks are network models derived from convolutional neural networks.

[0071] In one embodiment, the step of performing layered projection on the target 3D point cloud to obtain a projection feature map set by using a preset number of layers and a preset projection surface grid size includes:

[0072] S21: Normalize the target 3D point cloud to obtain the 3D point cloud to be predicted;

[0073] S22: Select any one of the x-projection plane, y-projection plane and z-projection plane as the first projection plane;

[0074] S23: Using the number of layers, the three-dimensional point cloud to be predicted is layered perpendicular to the first projection plane to obtain multiple single-layer three-dimensional point clouds corresponding to the first projection plane.

[0075] S24: Using the projection surface grid size, project each of the single-layer three-dimensional point clouds corresponding to the first projection surface onto the first projection surface to obtain a single-layer projection feature map corresponding to the first projection surface.

[0076] S25: Repeat the step of obtaining any one of the x-projection plane, y-projection plane and z-projection plane as the first projection plane until the acquisition of each of the x-projection plane, y-projection plane and z-projection plane is completed;

[0077] S26: Each of the single-layer projection feature maps is used as the projection feature map set.

[0078] This embodiment determines the projection feature map set by normalizing the target 3D point cloud, layering it on the vertical projection plane, and projecting it onto the projection plane. The 3D data is represented in two dimensions by layered projection. The projection information in any two directions can basically determine the position information of the point cloud. At the same time, the information of the two-dimensional image helps to process it using the mature and powerful skeleton network. Layered projection can further ensure the acquisition of the internal information of the 3D object, providing a guarantee for the effective learning of the subsequent point cloud representation.

[0079] For S21, the target three-dimensional point cloud is normalized, and the normalized target three-dimensional point cloud is used as the three-dimensional point cloud to be predicted. That is, the value of each point in the target three-dimensional point cloud is normalized to between 0 and 1. It can be 0, 1, or a decimal between 0 and 1.

[0080] For S22, the x-projection plane is a projection plane parallel to the x-axis, the y-projection plane is a projection plane parallel to the y-axis, and the z-projection plane is a projection plane parallel to the z-axis. The x-projection plane, y-projection plane, and z-projection plane are perpendicular to each other.

[0081] For S23, using the specified number of layers, the 3D point cloud to be predicted is divided into equal layers perpendicular to the first projection plane. Each layer of the 3D point cloud is then used as a single-layer 3D point cloud corresponding to the first projection plane. Due to the three-dimensional nature of point cloud data, a simple top-down projection cannot effectively represent the internal information of an object, as internal projections may be covered by external projections. Through layering, the internal information of the object can be fully and effectively represented, ensuring effective learning of subsequent point cloud representations.

[0082] Equal division and layering means dividing the three-dimensional point cloud to be predicted into the same number of equal parts perpendicular to the first projection plane as the number of layers. Each part of the three-dimensional point cloud is a single-layer three-dimensional point cloud corresponding to the first projection plane.

[0083] Understandably, the greater the number of layers, the clearer the internal information of the object will be, but it will also place a greater burden on data storage and model training.

[0084] For S24, the size of the first projection surface is the same as the size of the projection surface grid; each of the single-layer three-dimensional point clouds corresponding to the first projection surface is projected onto the first projection surface, and each feature map obtained by projection is used as the single-layer projection feature map corresponding to the first projection surface.

[0085] In other words, the single-layer 3D point cloud and the single-layer projection feature Figure 1 One-to-one correspondence.

[0086] For S25, the step of obtaining any one of the x, y, and z projection planes as the first projection plane is repeated, that is, steps S22 to S25 are repeated until each of the x, y, and z projection planes is obtained. When the acquisition of each of the x, y, and z projection planes is completed, it means that the single-layer projection feature map of each of the x, y, and z projection planes has been determined. At this time, the repeated execution of steps S22 to S25 needs to be stopped, and step S26 needs to be started.

[0087] For S26, each of the single-layer projection feature maps is directly used as the projection feature map set, thereby achieving full acquisition of the internal information of the three-dimensional object and providing a guarantee for the effective learning of subsequent point cloud representation.

[0088] In one embodiment, the step of inputting the projected feature map set into a preset point cloud representation generation model to generate a point cloud representation and obtain a target point cloud representation includes:

[0089] S311: The single-layer projection feature maps corresponding to each projection surface in the projection feature map set are stitched together in the channel dimension to obtain a stitched feature map;

[0090] S312: Input each of the stitched feature maps into the point cloud representation generation model to generate point cloud representations and obtain single projection surface feature data;

[0091] S313: Use the feature data of each single projection surface as the representation of the target point cloud.

[0092] This embodiment provides a basis for the interaction between features of different projection surfaces by stitching together the single-layer projection feature maps corresponding to the same projection surface in the channel dimension.

[0093] For S311, the single-layer projection feature maps corresponding to each projection surface in the projection feature map set are stitched together along the channel dimension, and the stitched feature map is used as the stitched feature map. That is, the stitched feature map corresponds one-to-one with the projection surface.

[0094] The number of channels in the stitched feature map is the same as the number of layers. Each single-layer projection feature map corresponding to the projection plane is located in one channel of the stitched feature map.

[0095] For S312, each of the stitched feature maps is input into the point cloud representation generation model to generate a point cloud representation, and the generated point cloud representation is used as single-projection-plane feature data. That is, the single-projection-plane feature data corresponds one-to-one with the projection plane.

[0096] For S313, the feature data of each single projection surface are concatted along the channel dimension, and the concatenated data is used as the representation of the target point cloud.

[0097] In one embodiment, the point cloud representation generation model includes: two information heatmap generation modules and a feature extraction module corresponding to each projection surface;

[0098] The feature extraction module includes: a CNN skeleton submodule and two feature fusion submodules, wherein the first feature fusion submodule is located between the first residual block and the second residual block of the CNN skeleton submodule, the second feature fusion submodule is located between the third residual block and the fourth residual block of the CNN skeleton submodule, the first information heatmap generation module is connected to each of the first feature fusion submodules, and the second information heatmap generation module is connected to each of the second feature fusion submodules.

[0099] The first feature fusion submodule of the x-projection plane, the first feature fusion submodule of the y-projection plane, and the first feature fusion submodule of the z-projection plane are connected to each other;

[0100] The second feature fusion submodule of the x-projection plane, the second feature fusion submodule of the y-projection plane, and the second feature fusion submodule of the z-projection plane are connected to each other.

[0101] This embodiment uses a deep autoencoder and feature fusion network to obtain a point cloud representation generation model, enabling the deep autoencoder to effectively learn point cloud information representations from the projected feature map set, thus improving the accuracy of representation learning. The first feature fusion submodule of the two submodules is located between the first and second residual blocks of the CNN skeleton submodule, and the second feature fusion submodule is located between the third and fourth residual blocks of the CNN skeleton submodule. The first information heatmap generation module of the two heatmap generation modules is connected to each of the first feature fusion submodules, and the second information heatmap generation module is connected to each of the second feature fusion submodules, thereby improving the interaction between features of different projection planes and realizing the utilization of multi-scale feature information.

[0102] The information heatmap generation module uses convolutional layers. This module is used to extract information heatmaps.

[0103] The CNN skeleton submodule is an encoder based on the CNN (Convolutional Neural Network) skeleton.

[0104] The CNN skeleton submodule can use ResNet50. ResNet50 is a residual network.

[0105] ResNet50 contains four consecutive residual blocks.

[0106] The range of values ​​for the projection surfaces includes: the x-projection surface in the x-axis direction, the y-projection surface in the y-axis direction, and the z-projection surface in the z-axis direction. In other words, the point cloud representation generation model contains a feature extraction module corresponding to the x-projection surface, a feature extraction module corresponding to the y-projection surface, and a feature extraction module corresponding to the z-projection surface.

[0107] In one embodiment, the feature fusion submodule includes: an instance normalization unit and a feature fusion unit;

[0108] The output of the instance normalization unit of the first feature fusion submodule is used as the input of the first information heatmap generation module;

[0109] The output of the first information heatmap generation module is used as the input of the feature fusion unit of the first feature fusion submodule;

[0110] The output of the instance normalization unit of the second feature fusion submodule is used as the input of the second information heatmap generation module;

[0111] The output of the second information heatmap generation module is used as the input of the feature fusion unit of the second feature fusion submodule.

[0112] In one embodiment, the feature fusion unit of the first feature fusion submodule is used to obtain any one of the projection planes (x, y, and z) as a second projection plane, add the element-values ​​at the same position to the outputs of the instance normalization units of the first feature fusion submodule for each of the projection planes other than the second projection plane to obtain a first feature map, multiply the first feature map by the element-values ​​at the same position to obtain a second feature map, and add the element-values ​​at the same position to the outputs of the instance normalization units of the first feature fusion submodule for the second projection plane to obtain a fused feature corresponding to the second projection plane, wherein the fused feature corresponding to the second projection plane is the second residual block corresponding to the second projection plane;

[0113] The feature fusion unit of the second feature fusion submodule is used to obtain any one of the projection planes from the x-projection plane, the y-projection plane, and the z-projection plane as the third projection plane. The outputs of the instance normalization units of the second feature fusion submodule for each of the projection planes other than the third projection plane are summed at the same position to obtain a third feature map. The third feature map is multiplied at the same position by the output of the second information heatmap generation module to obtain a fourth feature map. The fourth feature map is summed at the same position by the output of the instance normalization unit of the second feature fusion submodule for the third projection plane to obtain the fused feature corresponding to the third projection plane. The fused feature corresponding to the third projection plane is the input of the fourth residual block on the third projection plane.

[0114] Since the feature maps of the three projection planes have complementary information, in order to better learn the point cloud representation, this embodiment performs instance normalization and feature fusion on the output feature maps of the residual blocks of the three projection planes respectively, thereby improving the interaction between the features of each projection plane and realizing the utilization of multi-scale feature information.

[0115] The instance normalization unit is used to perform instance normalization on the feature map. The implementation method of the instance normalization unit will not be described in detail here.

[0116] The feature fusion unit is used to fuse the feature maps of the other two projection surfaces for the first time, fuse the feature map obtained from the first fusion with the information heatmap extracted by the information heatmap generation module for the second fusion, and fuse the feature map obtained from the second fusion with the feature map of this projection surface to obtain the overall feature fusion result corresponding to this projection surface.

[0117] In one embodiment, the step of inputting the projected feature map set into a preset point cloud representation generation model to generate a point cloud representation and obtain a target point cloud representation includes:

[0118] S321: Obtain multiple training samples and a model to be trained, wherein the model to be trained includes, in sequence, an initial model and a decoder;

[0119] S322: Use any one of the training samples as the target sample;

[0120] S323: Input the projection feature map sample set in the target sample into the model to be trained to reconstruct the projection map and obtain the projection map reconstruction result;

[0121] S324: Calculate the perceptual loss based on the reconstruction result of the projection map and the projection map labels in the target sample;

[0122] S325: Update the network parameters of the model to be trained based on the perceptual loss;

[0123] S326: Repeat the step of taking any one of the training samples as the target sample until the preset model training termination condition is reached, and take the initial model in the model to be trained that has reached the model training termination condition as the point cloud representation generation model.

[0124] The method for generating the projection feature map sample set includes: acquiring a three-dimensional point cloud sample, masking the three-dimensional point cloud sample to obtain an unmasked three-dimensional point cloud, and performing layered projection on the unmasked three-dimensional point cloud using the number of layers and the projection surface grid size to obtain the projection feature map sample set.

[0125] This embodiment uses masking and hierarchical projection techniques to determine the projection feature map sample set of training samples, which effectively helps the model learn truly specific information rather than memorizing noise. Therefore, the point cloud representation determined in this way can be robustly applied to various downstream tasks. By calculating the perceptual loss of the projection map reconstruction result and the projection map label, the trained model can perceive images from a higher dimension that are closer to human thinking, further improving the accuracy of the model's representation learning.

[0126] For S321, it can obtain multiple training samples and models to be trained from user input, or it can obtain multiple training samples and models to be trained from a database, or it can obtain multiple training samples and models to be trained from a third-party application.

[0127] For S322, any one of the training samples is taken as the target sample;

[0128] For S323, the projection feature map sample set in the target sample is input into the initial model of the model to be trained to generate point cloud representation, and the point cloud representation to be analyzed is obtained. The point cloud representation to be analyzed is input into the decoder of the model to be trained to reconstruct the projection map, and each projection map obtained by reconstruction is used as the projection map reconstruction result.

[0129] For S324, the method and steps for calculating the perceptual loss based on the projection map reconstruction result and the projection map labels in the target sample are not described in detail here.

[0130] For S325, the network parameters of the model to be trained are updated according to the perceptual loss, and the updated model to be trained is used for the next calculation of the projection map reconstruction result.

[0131] For S326, the step of using any one of the training samples as the target sample is repeated, that is, steps S322 to S326 are repeated until the preset model training termination condition is reached; when the preset model training termination condition is reached, it means that the performance of the model to be trained has reached the expected requirements. Therefore, the initial model in the model to be trained that has reached the model training termination condition is used as the point cloud representation generation model.

[0132] Optionally, the preset model training termination condition is a preset value of the convergence region of the perceptual loss of the model to be trained.

[0133] The step of masking the 3D point cloud sample to obtain an unmasked 3D point cloud involves dividing the unmasked 3D point cloud into 20 equal parts along the three dimensions (x, y, and z), resulting in 8000 grids. 40% of these 8000 grids are randomly masked, and the point cloud data corresponding to all the remaining grids is considered the unmasked 3D point cloud. It is understood that the number of parts used for masking can be other values, and this is not limited here. Similarly, the grid ratio used for masking can also be other values, and this is not limited here.

[0134] The steps of performing layered projection on the unmasked 3D point cloud using the number of layers and the projection surface grid size to obtain the projection feature map sample set are the same as the steps of performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain the projection feature map set.

[0135] It is understood that the three-dimensional point cloud sample is layered and projected using the number of layers and the size of the projection surface grid to obtain the projection map label.

[0136] The three-dimensional point cloud sample is a three-dimensional point cloud generated for the same object.

[0137] Reference Figure 2 This application also proposes a point cloud representation generation device for three-dimensional point clouds, the device comprising:

[0138] Data acquisition module 100 is used to acquire the target 3D point cloud;

[0139] The layered projection module 200 is used to perform layered projection on the target three-dimensional point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set.

[0140] The target point cloud representation determination module 300 is used to input the projection feature map set into a preset point cloud representation generation model to generate point cloud representations and obtain target point cloud representations. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network.

[0141] This embodiment employs a preset number of layers and a preset projection surface grid size to perform layered projection on the target 3D point cloud, obtaining a projection feature map set. Layered projection represents the 3D data in 2D form. Projection information from any two directions can essentially determine the point cloud's position information. Simultaneously, the information from the 2D image facilitates processing using currently mature and powerful skeleton networks. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for effective learning of subsequent point cloud representations. The projection feature map set is input into a preset point cloud representation generation model to generate the target point cloud representation. This point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. The feature fusion network improves the interaction between features of various projection surfaces, realizing the utilization of multi-scale feature information. This allows the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, contributing to improved accuracy in representation learning.

[0142] Reference Figure 3 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data such as methods for generating point cloud representations of 3D point clouds. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for generating point cloud representations of 3D point clouds. The method includes: acquiring a target 3D point cloud; performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature set; inputting the projection feature set into a preset point cloud representation generation model to generate a point cloud representation, thereby obtaining a target point cloud representation, wherein the point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network.

[0143] This embodiment employs a preset number of layers and a preset projection surface grid size to perform layered projection on the target 3D point cloud, obtaining a projection feature map set. Layered projection represents the 3D data in 2D form. Projection information from any two directions can essentially determine the point cloud's position information. Simultaneously, the information from the 2D image facilitates processing using currently mature and powerful skeleton networks. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for effective learning of subsequent point cloud representations. The projection feature map set is input into a preset point cloud representation generation model to generate the target point cloud representation. This point cloud representation generation model is a model trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. The feature fusion network improves the interaction between features of various projection surfaces, realizing the utilization of multi-scale feature information. This allows the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, contributing to improved accuracy in representation learning.

[0144] An embodiment of this application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements a method for generating point cloud representations of three-dimensional point clouds, including the steps of: acquiring a target three-dimensional point cloud; performing layered projection on the target three-dimensional point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set; inputting the projection feature map set into a preset point cloud representation generation model to generate point cloud representations to obtain a target point cloud representation, wherein the point cloud representation generation model is a model trained based on masking technology, layered projection technology, deep autoencoder, and feature fusion network.

[0145] The aforementioned method for generating 3D point cloud representations involves performing layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size. This yields a projection feature map set, which represents the 3D data in two dimensions. The projection information from any two directions can essentially determine the positional information of the point cloud. Simultaneously, the information from the two-dimensional image facilitates processing using mature and powerful skeleton networks. Layered projection further ensures the acquisition of internal information of the 3D object, providing a guarantee for effective learning of subsequent point cloud representations. The projection feature map set is then input into a preset point cloud representation generation model to generate the target point cloud representation. This model is trained based on masking technology, layered projection technology, a deep autoencoder, and a feature fusion network. The feature fusion network improves the interaction between features on each projection surface, enabling the utilization of multi-scale feature information. This allows the deep autoencoder to effectively learn the representation of point cloud information from the projection feature map set, contributing to improved accuracy in representation learning.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0148] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for generating point cloud representations of three-dimensional point clouds, characterized in that, The method includes: Obtain the target 3D point cloud; Using a preset number of layers and a preset projection surface grid size, the target 3D point cloud is projected in layers to obtain a projection feature map set; The projection feature map set is input into a preset point cloud representation generation model to generate point cloud representations, thereby obtaining the target point cloud representation. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network. The step of performing layered projection on the target 3D point cloud to obtain a projection feature map set by using a preset number of layers and a preset projection surface grid size includes: The target 3D point cloud is normalized to obtain the 3D point cloud to be predicted. Choose any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane; Using the number of layers, the three-dimensional point cloud to be predicted is divided into equal layers perpendicular to the first projection plane to obtain multiple single-layer three-dimensional point clouds corresponding to the first projection plane. Using the projection surface grid size, each single-layer 3D point cloud corresponding to the first projection surface is projected onto the first projection surface. The projection is calculated by counting the number of point clouds falling onto the projection surface grid, and the calculated number is used as the pixel value of the grid to obtain the single-layer projection feature map corresponding to the first projection surface. Repeat the step of obtaining any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane until the acquisition of each of the x-projection plane, y-projection plane, and z-projection plane is completed; Each of the single-layer projection feature maps is used as the projection feature map set; The point cloud representation generation model includes: two information heatmap generation modules and a feature extraction module corresponding to each projection surface; The feature extraction module includes: a CNN skeleton submodule and two feature fusion submodules, wherein the first feature fusion submodule is located between the first residual block and the second residual block of the CNN skeleton submodule, the second feature fusion submodule is located between the third residual block and the fourth residual block of the CNN skeleton submodule, the first information heatmap generation module is connected to each of the first feature fusion submodules, and the second information heatmap generation module is connected to each of the second feature fusion submodules. The first feature fusion submodule of the x-projection plane, the first feature fusion submodule of the y-projection plane, and the first feature fusion submodule of the z-projection plane are connected to each other; The second feature fusion submodule of the x-projection plane, the second feature fusion submodule of the y-projection plane, and the second feature fusion submodule of the z-projection plane are connected to each other.

2. The method for generating point cloud representations of three-dimensional point clouds according to claim 1, characterized in that, The feature fusion submodule includes: an instance normalization unit and a feature fusion unit; The output of the instance normalization unit of the first feature fusion submodule is used as the input of the first information heatmap generation module; The output of the first information heatmap generation module is used as the input of the feature fusion unit of the first feature fusion submodule; The output of the instance normalization unit of the second feature fusion submodule is used as the input of the second information heatmap generation module; The output of the second information heatmap generation module is used as the input of the feature fusion unit of the second feature fusion submodule.

3. The method for generating point cloud representations of three-dimensional point clouds according to claim 1, characterized in that, The step of inputting the projected feature map set into a preset point cloud representation generation model to generate a point cloud representation and obtain a target point cloud representation includes: Multiple training samples and a model to be trained are obtained, wherein the model to be trained includes, in sequence, an initial model and a decoder; Use any one of the training samples as the target sample; The projection feature map sample set in the target sample is input into the model to be trained to reconstruct the projection map, and the projection map reconstruction result is obtained. The perceptual loss is calculated based on the reconstruction result of the projection map and the projection map labels in the target sample; Update the network parameters of the model to be trained based on the perceptual loss; Repeat the step of using any one of the training samples as the target sample until the preset model training termination condition is reached, and use the initial model in the model to be trained that has reached the model training termination condition as the point cloud representation generation model. The method for generating the projection feature map sample set includes: acquiring a three-dimensional point cloud sample, masking the three-dimensional point cloud sample to obtain an unmasked three-dimensional point cloud, and performing layered projection on the unmasked three-dimensional point cloud using the number of layers and the projection surface grid size to obtain the projection feature map sample set.

4. A point cloud representation generation device for three-dimensional point clouds, characterized in that, The device includes: The data acquisition module is used to acquire the target's 3D point cloud. The layered projection module is used to perform layered projection on the target 3D point cloud using a preset number of layers and a preset projection surface grid size to obtain a projection feature map set. The target point cloud representation determination module is used to input the projection feature map set into a preset point cloud representation generation model to generate point cloud representations and obtain target point cloud representations. The point cloud representation generation model is a model trained based on masking technology, hierarchical projection technology, deep autoencoder and feature fusion network. The step of performing layered projection on the target 3D point cloud to obtain a projection feature map set by using a preset number of layers and a preset projection surface grid size includes: The target 3D point cloud is normalized to obtain the 3D point cloud to be predicted. Choose any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane; Using the number of layers, the three-dimensional point cloud to be predicted is divided into equal layers perpendicular to the first projection plane to obtain multiple single-layer three-dimensional point clouds corresponding to the first projection plane. Using the projection surface grid size, each single-layer 3D point cloud corresponding to the first projection surface is projected onto the first projection surface. The projection is calculated by counting the number of point clouds falling onto the projection surface grid, and the calculated number is used as the pixel value of the grid to obtain the single-layer projection feature map corresponding to the first projection surface. Repeat the step of obtaining any one of the x-projection plane, y-projection plane, and z-projection plane as the first projection plane until the acquisition of each of the x-projection plane, y-projection plane, and z-projection plane is completed; Each of the single-layer projection feature maps is used as the projection feature map set; The point cloud representation generation model includes: two information heatmap generation modules and a feature extraction module corresponding to each projection surface; The feature extraction module includes: a CNN skeleton submodule and two feature fusion submodules, wherein the first feature fusion submodule is located between the first residual block and the second residual block of the CNN skeleton submodule, the second feature fusion submodule is located between the third residual block and the fourth residual block of the CNN skeleton submodule, the first information heatmap generation module is connected to each of the first feature fusion submodules, and the second information heatmap generation module is connected to each of the second feature fusion submodules. The first feature fusion submodule of the x-projection plane, the first feature fusion submodule of the y-projection plane, and the first feature fusion submodule of the z-projection plane are connected to each other; The second feature fusion submodule of the x-projection plane, the second feature fusion submodule of the y-projection plane, and the second feature fusion submodule of the z-projection plane are connected to each other.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Target detection method and device, and computer storage medium

    CN113205515A

  • Real-time three-dimensional target detection method based on point cloud cross-view feature conversion

    CN114511846A