Large-scale point cloud completion method and device

Abstract

The invention provides a large-scale point cloud completion method and device, and the method comprises the steps: carrying out the resampling of each small-scale point cloud data segmented from large-scale point cloud data, and obtaining first point cloud data and second point cloud data; inputting the first point cloud data, the second point cloud data and RGB-D point cloud data corresponding to the small-scale point cloud data into a trained generative adversarial network, and obtaining complete point cloud data generated by the generative adversarial network; and fusing the RGB-D point cloud data and the complete point cloud data to obtain complemented point cloud data. According to the large-scale point cloud completion method and device provided by the invention, the multi-scale point cloud features and the generated complete point cloud data are deeply extracted through the generative adversarial network, and the generated three-dimensional point cloud data and the high-resolution two-dimensional image data are fused, so that the understanding and adaptability to regular and irregular object structures are improved; and the accuracy and the reliability of the complemented point cloud data are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large-scale point cloud completion method and device. Background Art

[0002] In the actual 3D modeling process, the limitation of sensor resolution and occlusion factors often lead to partial loss of geometric features and semantic information. Although point cloud data contains detailed 3D details, the existence of outliers and blurred edges significantly increases the difficulty of object recognition.

[0003] In the existing technology, the PointNet architecture captures and integrates the local features and overall shape features of a single point in the point cloud data through its unique shared multi-layer perceptron (Shared-MLP) module. However, its performance is not satisfactory when processing point cloud datasets containing a variety of complex topological structures. The Point Completion Network (PCN) extracts point cloud feature vectors based on the multi-layer perceptron (MLP) mechanism in PointNet, and innovatively combines a voxel-based generator in the decoding stage to gradually refine and improve the reconstruction of the point cloud. Nevertheless, due to its reliance on voxel generators that tend to produce uniformly distributed grids, PCN exhibits significant limitations when applied to dense and non-uniformly distributed point clouds.

[0004] Therefore, the point cloud completion method in the prior art still has technical problems such as inaccurate point cloud data and poor reliability. Summary of the invention

[0005] The present invention provides a large-scale point cloud completion method and device, which are used to solve the technical problems of inaccurate point cloud data obtained by the point cloud completion method in the prior art and poor reliability.

[0006] The present invention provides a large-scale point cloud completion method, comprising: Obtain incomplete large-scale point cloud data collected in actual urban scenes; Segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; For each small-scale point cloud data, first point cloud data with a first resolution and second point cloud data with a second resolution are obtained by resampling; Inputting the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The RGB-D point cloud data and the complete point cloud data are fused to obtain completed point cloud data.

[0007] In some embodiments, the generative adversarial network includes a generator; the generator includes a multi-resolution point cloud encoder and a point cloud decoder; The point cloud encoder includes an input transformation layer, a shared multi-layer perceptron layer and a feature transformation layer; After the first point cloud data, the second point cloud data and the RGB-D point cloud data are sequentially passed through the input transformation layer, the shared multi-layer perceptron layer and the feature transformation layer, a feature vector of a specific dimension is generated; The point cloud decoder includes three decoders with different feature decoding levels; The feature vector is input into the decoders of the three different feature decoding levels, and different detail levels of the missing points are predicted from coarse to fine to obtain complete point cloud data.

[0008] In some embodiments, the generative adversarial network includes a discriminator; The method further comprises: Using sample data to train the generator of the generative adversarial network; The complete point cloud data is compared with the true value by the discriminator to evaluate the accuracy of the generator.

[0009] In some embodiments, the generative adversarial network includes a discriminator including a six-layer fully connected layer with dimensions of (256, 128, 64, 32, 16, 1) and an average pooling layer.

[0010] In some embodiments, the feature vector includes local geometric information, global geometric information, information at different scales, and XYZ coordinate information.

[0011] In some embodiments, the method further comprises: Based on each small-scale point cloud data, a view transformer is used to generate the corresponding RGB image and depth map from a bird's-eye view. The RGB image and the depth map are fused to obtain RGB-D point cloud data corresponding to the small-scale point cloud data.

[0012] The present invention also provides a large-scale point cloud completion device, comprising: The acquisition module is used to obtain incomplete large-scale point cloud data collected in actual urban scenes; A segmentation module, used for segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; A resampling module, for obtaining, for each small-scale point cloud data, first point cloud data having a first resolution and second point cloud data having a second resolution by resampling; A completion module, used to input the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The fusion module is used to fuse the RGB-D point cloud data with the complete point cloud data to obtain completed point cloud data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large-scale point cloud completion method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the large-scale point cloud completion methods described above.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the large-scale point cloud completion methods described above.

[0016] The large-scale point cloud completion method and device provided by the present invention deeply extract multi-scale point cloud features through a generative adversarial network, generate complete point cloud data, and fuse the generated three-dimensional point cloud data with high-resolution two-dimensional image data, thereby improving the understanding and adaptability of regular and irregular object structures and improving the accuracy and reliability of the completed point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the large-scale point cloud completion method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the overall framework of the large-scale point cloud completion network provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the corresponding relationship between the image and the point cloud provided by the present invention in different coordinate systems.

[0021] Figure 4is a schematic diagram of the fused RGB-D point cloud provided by the present invention.

[0022] Figure 5 Schematic diagram of the structure of the improved GAN provided by the present invention.

[0023] Figure 6 It is a schematic diagram for comparing the visual completion effects provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the large-scale point cloud completion device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] Nowadays, the importance of high-quality communication environment in social life is becoming increasingly prominent. Under ideal conditions, personal handheld devices (such as mobile phones) can effectively access network services through ground base station facilities. However, in high-density hotspots such as stadiums, limited communication resources are often insufficient to support the simultaneous connection of a large number of user terminal devices.

[0027] With the continuous advancement of unmanned aerial vehicle (UAV) technology, it has become a reality to deploy aerial signal relay nodes using the unique hovering capabilities of UAVs. This method is gradually evolving into a promising integrated air-ground communication strategy. Due to the characteristics of UAV mobility, flexibility and rapid deployment, UAVs are increasingly becoming an ideal supplementary means to enhance communication coverage capabilities, especially when complex geological conditions make it inconvenient or impractical to establish traditional ground base stations. In order to quickly deploy UAVs in urban environments, it is essential to perform three-dimensional modeling of urban landscapes to achieve environmental perception. At present, the mainstream three-dimensional modeling method is to use laser scanning (LiDAR) technology to laser scan urban building structures, terrain and vehicles to obtain point cloud data containing spatial XYZ information.

[0028] However, in real urban environments, laser scanning can generate point clouds consisting of millions or at least tens of thousands of points. However, existing technologies are usually designed to process much smaller point clouds, often limited to about 2048 points. This difference requires adaptive adjustments to real urban scene representation and point cloud completion networks to cope with large-scale challenges.

[0029] In addition, in the actual 3D modeling process, the limitations of sensor resolution and occlusion factors often lead to partial loss of geometric features and semantic information. Although point cloud data contains detailed 3D details, the existence of outliers and blurred edges significantly increases the difficulty of object recognition. Existing point cloud completion techniques usually rely on voxel-based generators to infer missing data points within the point cloud region. This method tends to produce matrix-like sparse output point sets and has proven to be effective when processing regular-shaped objects. However, when faced with complex irregular structures such as high-voltage towers or bridges, the performance of voxel-based generators often drops significantly in practical applications, and there are technical problems such as inaccurate and unreliable completed point cloud data.

[0030] Based on the above technical problems, the present invention provides a large-scale point cloud completion method and device, which deeply extracts multi-scale point cloud features through a generative adversarial network, generates complete point cloud data, and fuses the generated three-dimensional point cloud data with high-resolution two-dimensional image data, thereby improving the understanding and adaptability of regular and irregular object structures, and improving the accuracy and reliability of the completed point cloud data.

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Figure 1 is a flow chart of the large-scale point cloud completion method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Acquire incomplete large-scale point cloud data collected in actual urban scenes.

[0033] Step 102: segment the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data.

[0034] Specifically, in order to address the problem that large-scale point cloud data cannot be processed in a real urban environment, the present invention obtains incomplete large-scale point cloud data collected in an actual urban scene, and then segments the incomplete large-scale point cloud data to obtain multiple small-scale point cloud data.

[0035] Figure 2 is a schematic diagram of the overall framework of the large-scale point cloud completion network provided by the present invention, such as Figure 2As shown, the present invention proposes a new large-scale point cloud completion network (PIFC-Net) for point cloud and image fusion. The network mainly includes a segmentation module, a completion module and a fusion module, which aims to complete the cropped point cloud and combine the point cloud with the image to generate a colored point cloud in a large-scale real city 3D modeling scene.

[0036] In some embodiments, the method further comprises: Based on each small-scale point cloud data, a view transformer is used to generate the corresponding RGB image and depth map from a bird's-eye view. The RGB image and the depth map are fused to obtain RGB-D point cloud data corresponding to the small-scale point cloud data.

[0037] Figure 3 is a schematic diagram of the corresponding relationship between the image and the point cloud provided by the present invention in different coordinate systems, such as Figure 3 As shown, taking the sun as an example, the world coordinate system can be expressed as , the camera coordinate system can be expressed as , the image coordinate system can be expressed as , define the depth map coordinates as .

[0038] From the world coordinate system to the camera coordinate system, the transformation formula is as follows:

[0039] From the camera coordinate system to the image coordinate system, the transformation formula is as follows:

[0040] From the image coordinate system to the pixel coordinate system, the transformation formula is as follows:

[0041] Therefore, the transformation formula from the world coordinate system to the pixel coordinate system is as follows:

[0042] At this time, the image coordinates are , the depth map coordinates are ,and and The expression is as follows:

[0043] During the segmentation process, the camera uses the cropped colored object point cloud to generate the RGB image and depth map from the bird's eye view (BEV) perspective, and generates the corresponding camera external parameter matrix and internal parameter matrix. The coordinates of the RGB image can be upgraded to , the coordinates of the depth map can be upgraded to Therefore, the coordinate information and color information of the colored point cloud are fused.

[0044] Figure 4 is a schematic diagram of the fused RGB-D point cloud provided by the present invention, such as Figure 4 As shown, the RGB-D point cloud coordinates can be obtained from the following formula: , corresponding to the coordinates in the world coordinate system :

[0045] Because the objects are segmented from the real city point cloud, the XYZ coordinate range is very large, and the point cloud completion network can only accept normalized data, the present invention normalizes the point cloud data through the following steps: Calculate the centroid of the point cloud using the following formula:

[0046] Calculate the point cloud offset, the formula is as follows:

[0047] Calculate the scaling factor using the following formula:

[0048] Then the expression of the regularized coordinates is as follows:

[0049] The coordinates can be inverse normalized using the following formula:

[0050] Step 103 : for each small-scale point cloud data, obtain first point cloud data with a first resolution and second point cloud data with a second resolution by resampling.

[0051] Step 104: Input the first point cloud data, the second point cloud data, and the RGB-D point cloud data corresponding to the small-scale point cloud data into the trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network.

[0052] Step 105: Fusing the RGB-D point cloud data with the complete point cloud data to obtain completed point cloud data.

[0053] In some embodiments, the generative adversarial network includes a generator; the generator includes a multi-resolution point cloud encoder and a point cloud decoder; The point cloud encoder includes an input transformation layer, a shared multi-layer perceptron layer and a feature transformation layer; After the first point cloud data, the second point cloud data and the RGB-D point cloud data are sequentially passed through the input transformation layer, the shared multi-layer perceptron layer and the feature transformation layer, a feature vector of a specific dimension is generated; The point cloud decoder includes three decoders with different feature decoding levels; The feature vector is input into the decoders of the three different feature decoding levels, and different detail levels of the missing points are predicted from coarse to fine to obtain complete point cloud data.

[0054] In some embodiments, the generative adversarial network includes a discriminator; The method further comprises: Using sample data to train the generator of the generative adversarial network; The complete point cloud data is compared with the true value by the discriminator to evaluate the accuracy of the generator.

[0055] In some embodiments, the generative adversarial network includes a discriminator including a six-layer fully connected layer with dimensions of (256, 128, 64, 32, 16, 1) and an average pooling layer.

[0056] In some embodiments, the feature vector includes local geometric information, global geometric information, information at different scales, and XYZ coordinate information.

[0057] Specifically, Figure 5 : is a schematic diagram of the structure of the improved generative adversarial network GAN provided by the present invention, such as Figure 5 As shown in the figure, the network architecture of the completion module adopts GAN, which consists of a point cloud completion generator and a completion result discriminator (discriminator). Among them, the generator contains a multi-resolution point cloud encoder and decoder. The original point cloud from the segmentation module (XYZ data of 2048 points from the incomplete RGB-D point cloud) and two resampled (downsampled) low-resolution point clouds (for example, 512 and 256 points, respectively) are input into the encoder. The encoder has three branches for processing point clouds of different resolutions. These point clouds are first converted into feature vectors through three levels of downsampling. Then, they pass through the input transformation layer, the shared multi-layer perceptron (shared-MLP) layer, and the feature transformation layer to generate a feature vector F of a specific dimension. The output feature vector F contains local and global geometric information, information of different scales, and XYZ coordinate information.

[0058] The feature vector F is fed into a decoder with three different feature decoding levels and then used to predict different levels of detail of the missing points. The coarse-to-fine decoding strategy ensures that node coordinates are decoded step by step starting from the coarsest level until the finest points. The generated points are combined with the original points and jointly trained adversarially in the discriminator, resulting in better completion results.

[0059] After the point cloud is generated, the result discriminator compares it with the true value to evaluate whether the error range is acceptable. The present invention removes batch normalization to reduce the complexity of the discriminator, expands the fully connected layer from four layers with dimensions (256, 128, 16, 1) to six layers with dimensions (256, 128, 64, 32, 16, 1), and changes the maximum pooling (Max Pooling) to the average pooling (Average Pooling). These improvements make the generated point cloud have better generation results, more accurate and reliable.

[0060] The purpose of the fusion module is to integrate RGB information into the point cloud and arrange the normalized point cloud according to its original macro position. After coordinate normalization, different objects show obvious RGB differences, which helps subsequent tasks such as object detection and object segmentation.

[0061] Using offset and scaling , we can get the coordinates of the object in the city after inverse normalization as follows:

[0062] From RGB-D point cloud coordinates and the initial point cloud coordinates , we can get the completed point cloud coordinates .

[0063] The loss function of PIFC-Net in the present invention consists of completion loss and adversarial loss. The expression of completion loss is as follows: +

[0064] The completion loss is used to measure the loss between the generator and the GT. Predicted completion point The error between .

[0065] The expression of adversarial loss is as follows:

[0066] Among them, E represents the expected value, from the ground measured point cloud Y gtx is sampled from the distribution of , z is sampled from the original point cloud p1(z), and G(z) is a generator used to convert z into a preliminary completed point cloud.

[0067] The expression of the total loss function is as follows:

[0068] Among them, L represents the total loss function, w represents the weight, and L c represents the completion loss, L a Represents resistance to loss.

[0069] The large-scale point cloud completion method provided by the present invention deeply extracts multi-scale point cloud features through a generative adversarial network, generates complete point cloud data, and fuses the generated three-dimensional point cloud data with high-resolution two-dimensional image data, thereby improving the understanding and adaptability of regular and irregular object structures, and improving the accuracy and reliability of the completed point cloud data.

[0070] The technical effect of the present invention is verified by experimental data below.

[0071] UrbanBIS provides a high-precision color city model under ideal conditions, from which the present invention simulates a single drone scanning operation to extract more than 9,000 sets of RGB images, depth maps, and bird's-eye view (BEV) point cloud data. The RGB information in the ideal point cloud is cleared, FPS (furthest point sampling) downsampling is performed to one-tenth to one-hundredth of the original number of points, and the point cloud is cropped to simulate the roughly colorless point cloud with missing areas obtained by a single drone LiDAR scan of the city in an emergency. All experiments are implemented using PyTorch 1.0.1 on an NVIDIA GeForce RTX 3090 Ti GPU with 24GB of video memory.

[0072] The PIFC-Net in this paper is divided into three sub-networks: Generator, Discriminator and LiDAR-RGB fusion module, aiming to explore the effect of different sub-network combinations on the point cloud completion effect. The results are shown in Table 1. Compared with other modules, the point cloud completion network combining the generator, discriminator and LiDAR-RGB fusion significantly improves the completion performance, which is mainly due to the use of color information. For large top-down objects such as cubes and pyramid-shaped buildings, the performance of the complete network is better than that of the network without RGB fusion; but for complex geometric shapes such as cylindrical structures, the effect is reduced. Analysis shows that the network performs well in completing objects with sufficient bird's-eye visibility, such as buildings and bridges. LiDAR-RGB fusion extracts RGB and depth data from the bird's-eye view, and enhances the bird's-eye view representation with finer features, structural details and rich topological structures. The RGB-D data is integrated with the standard point cloud data through a multi-layer feature extraction encoder to achieve excellent point cloud completion from a bird's-eye view, which is particularly important for drone-based urban modeling tasks.

[0073] Table 1 Comparison of the average performance of different models in the test set Table 1

[0074] The experiments focused on five types of buildings with different geometric shapes: cube buildings, pyramid buildings, strip buildings, cylindrical buildings, and irregular buildings. The results of PIFC-Net highlight the better reconstruction effect of cube and cylindrical buildings, which is attributed to their predictable regularity. However, buildings based on pyramid and strip structures pose challenges to reconstruction due to their small top area, resulting in lower completion. Expanding the scope of research to different types of objects, PIFC-Net performs best on building and bridge datasets, because simpler geometric shapes help improve reconstruction performance. In addition, the execution time is shown in Table 2. Compared with the state-of-the-art (SOTA) Shape-Inversion method and SnowFlakeNet, the method proposed in this application achieves better completion results in a shorter time.

[0075] Table 2 Comparison of average performance of different models in the test set Table 2

[0076] Figure 6 is a schematic diagram for comparing the visual completion effects provided by the present invention. Figure 6 As shown, the PIFC-Net proposed in the present invention has a better point cloud completion effect, and the completed point cloud data is more accurate and reliable.

[0077] The point cloud completion device provided by the present invention is described below. The point cloud completion device described below and the point cloud completion method described above can be referenced to each other.

[0078] Figure 7 is a schematic diagram of the structure of the large-scale point cloud completion device provided by the present invention, such as Figure 7 As shown, the present invention provides a large-scale point cloud completion device, comprising: The acquisition module 701 is used to acquire incomplete large-scale point cloud data collected in actual urban scenes; The segmentation module 702 is used to segment the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; The resampling module 703 is used for obtaining first point cloud data with a first resolution and second point cloud data with a second resolution by resampling each small-scale point cloud data; The completion module 704 is used to input the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into the trained generative adversarial network to obtain the complete point cloud data generated by the generative adversarial network; The fusion module 705 is used to fuse the RGB-D point cloud data with the complete point cloud data to obtain completed point cloud data.

[0079] Specifically, the above-mentioned large-scale point cloud completion device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned large-scale point cloud completion method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0080] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the large-scale point cloud completion method, which includes: Obtain incomplete large-scale point cloud data collected in actual urban scenes; Segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; For each small-scale point cloud data, first point cloud data with a first resolution and second point cloud data with a second resolution are obtained by resampling; Inputting the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The RGB-D point cloud data and the complete point cloud data are fused to obtain completed point cloud data.

[0081] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0082] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the large-scale point cloud completion method provided by the above methods, the method comprising: Obtain incomplete large-scale point cloud data collected in actual urban scenes; Segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; For each small-scale point cloud data, first point cloud data with a first resolution and second point cloud data with a second resolution are obtained by resampling; Inputting the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The RGB-D point cloud data and the complete point cloud data are fused to obtain completed point cloud data.

[0083] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the large-scale point cloud completion method provided by the above methods, the method comprising: Obtain incomplete large-scale point cloud data collected in actual urban scenes; Segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; For each small-scale point cloud data, first point cloud data with a first resolution and second point cloud data with a second resolution are obtained by resampling; Inputting the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The RGB-D point cloud data and the complete point cloud data are fused to obtain completed point cloud data.

[0084] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-scale point cloud completion method, characterized in that: include: Obtain incomplete large-scale point cloud data collected in actual urban scenes; Segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; For each small-scale point cloud data, first point cloud data with a first resolution and second point cloud data with a second resolution are obtained by resampling; Inputting the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The RGB-D point cloud data and the complete point cloud data are fused to obtain completed point cloud data.

2. The large-scale point cloud completion method according to claim 1, characterized in that: The generative adversarial network includes a generator; the generator includes a multi-resolution point cloud encoder and a point cloud decoder; The point cloud encoder includes an input transformation layer, a shared multi-layer perceptron layer and a feature transformation layer; After the first point cloud data, the second point cloud data and the RGB-D point cloud data are sequentially passed through the input transformation layer, the shared multi-layer perceptron layer and the feature transformation layer, a feature vector of a specific dimension is generated; The point cloud decoder includes three decoders with different feature decoding levels; The feature vector is input into the decoders of the three different feature decoding levels, and different detail levels of the missing points are predicted from coarse to fine to obtain complete point cloud data.

3. The large-scale point cloud completion method according to claim 2, characterized in that: The generative adversarial network includes a discriminator; The method further comprises: Using sample data to train the generator of the generative adversarial network; The complete point cloud data is compared with the true value by the discriminator to evaluate the accuracy of the generator.

4. The large-scale point cloud completion method according to claim 3, characterized in that: The generative adversarial network includes a discriminator including a six-layer fully connected layer with a dimension of (256, 128, 64, 32, 16, 1) and an average pooling layer.

5. The large-scale point cloud completion method according to claim 2, characterized in that: The feature vector includes local geometric information, global geometric information, information of different scales, and XYZ coordinate information.

6. The large-scale point cloud completion method according to claim 1, characterized in that: The method further comprises: Based on each small-scale point cloud data, a view transformer is used to generate the corresponding RGB image and depth map from a bird's-eye view. The RGB image and the depth map are fused to obtain RGB-D point cloud data corresponding to the small-scale point cloud data.

7. A large-scale point cloud completion device, characterized in that: include: The acquisition module is used to obtain incomplete large-scale point cloud data collected in actual urban scenes; A segmentation module, used for segmenting the incomplete large-scale point cloud data to obtain a plurality of small-scale point cloud data; A resampling module, for obtaining, for each small-scale point cloud data, first point cloud data having a first resolution and second point cloud data having a second resolution by resampling; A completion module, used to input the first point cloud data and the second point cloud data, as well as the RGB-D point cloud data corresponding to the small-scale point cloud data, into a trained generative adversarial network to obtain complete point cloud data generated by the generative adversarial network; The fusion module is used to fuse the RGB-D point cloud data with the complete point cloud data to obtain completed point cloud data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the large-scale point cloud completion method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large-scale point cloud completion method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the large-scale point cloud completion method as described in any one of claims 1 to 6 is implemented.

Images