A 3D point cloud data enhancement method based on viewpoint occlusion simulation

Abstract

The application discloses a 3D point cloud data enhancement method based on a viewpoint occlusion simulation, and the method comprises the following steps: selecting a virtual viewpoint direction based on an original point cloud; performing original point cloud depth projection and sorting based on the virtual viewpoint direction; simulating occlusion and selecting a visible point set according to the depth projection and sorting results; obtaining a supplementary point set based on the visible point set; and merging the visible point set and the supplementary point set to obtain an enhanced point cloud. The method disclosed by the application simulates a single sensor viewpoint, simulates real physical occlusion, and generates a point cloud with structuralization and continuity loss, which is highly consistent with the occlusion condition in the real world and is much better than a traditional random point discarding method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D point cloud data enhancement, and particularly relates to a 3D point cloud data enhancement method based on viewpoint occlusion simulation. BACKGROUND

[0002] 3D point cloud classification and analysis, as a core technology in the field of computer vision and autonomous driving, is a key support for machines to understand three-dimensional objects and realize environmental perception. With the rapid development of deep learning technology, especially deep neural networks (DNNs), models for processing point cloud data have continuously broken through performance bottlenecks on various benchmark datasets such as ShapeNet and ModelNet40, with classification accuracy, segmentation precision and other indicators continuously refreshed, showing strong three-dimensional data analysis capabilities.

[0003] However, the excellent performance of these models is highly dependent on clean and complete training datasets, and once faced with real-world complex scenarios, their performance will often drop dramatically. In actual applications, when 3D sensors such as LiDAR and depth cameras collect data, structural and continuous point cloud occlusion problems are common due to two main factors: one is object self-occlusion, where the geometric structure of three-dimensional objects causes some areas to be occluded, such as the chassis of a car occluding the ground point cloud below and the arms of a human occluding the point cloud on the sides of the torso. The second is the limited view of the sensor, which can only capture the surface information of objects within its field of view, and areas outside the view (such as the back of an object or areas occluded by other objects) cannot be detected, resulting in data missing.

[0004] This non-random, spatially related data missing is a common occurrence in real-world scenarios such as autonomous driving and robot navigation, directly leading to significant deviations between the features learned by the model during training and the data distribution in actual applications, severely affecting the reliability of the model.

[0005] To alleviate this problem, existing research has proposed various point cloud data enhancement techniques, with random point dropout being one of the most widely used methods. This method removes some points from the point cloud independently and uniformly, increasing the diversity of training data and attempting to improve the model's generalization ability. However, this approach has fundamental flaws:

[0006] It can only simulate random sparsification and cannot reproduce the structural and continuous occlusion caused by a single viewpoint in real-world scenarios. The discarded points are discretely distributed in space and cannot reflect the physical reality of invisible object backs or completely occluded local areas. Therefore, models trained on such non-physical augmented data still struggle to adapt to occluded scenes in the real world: for example, in autonomous driving, the point cloud captured by a laser radar may lack leg information due to occlusion, and the model may not be able to accurately identify it; in the robot grasping task, partial occlusion of the object may lead the model to misjudge the object class or pose. This gap between training and application scenarios greatly limits the application of deep learning point cloud models in safety-critical fields and becomes a core bottleneck restricting the improvement of their robustness.

[0007] Currently, some related technologies also focus on point cloud data augmentation in specific environments, for example: the patent document with publication number CN117574799B discloses a method and system for laser radar point cloud data augmentation in adverse weather scenarios. This application focuses on adverse weather scenarios and generates a simulation dataset by constructing a rain-splashing physical model and a laser radar model, and optimizing the point cloud intensity features to solve the problem of poor rain point cloud intensity authenticity. However, this technology only addresses weather interference and does not address structural data loss caused by viewpoint occlusion. For another example, the patent with publication number CN117872316A discloses a multi-scale physical simulation sandstorm weather laser radar point cloud data augmentation method. This application constructs a three-dimensional distribution model of sand particles based on the Monte Carlo algorithm to simulate the influence of sand on laser radar echoes, achieving point cloud augmentation in sandstorm weather. The core of this technology is to address environmental medium interference with sensor signals, which is completely different from the physical mechanism of object self-occlusion or viewpoint occlusion.

[0008] In summary, existing technologies do not provide effective solutions to the core problem of structural occlusion caused by a single viewpoint. Therefore, there is a need for a point cloud data augmentation method that can simulate the real physical observation process and generate point cloud data with continuous occlusion characteristics to narrow the gap between model training and real-world application and improve the robustness and reliability of 3D point cloud analysis models in occluded scenarios. SUMMARY

[0009] To address the above problems, the purpose of the present application is to provide a 3D point cloud data augmentation method based on viewpoint occlusion simulation to solve the problem of structural occlusion caused by a single viewpoint and generate point cloud data with continuous occlusion characteristics by simulating the real physical observation process.

[0010] The first aspect: a 3D point cloud data augmentation method based on viewpoint occlusion simulation, comprising the steps of:

[0011] S1, based on the original point cloud, selecting a virtual viewpoint direction;

[0012] S2. Perform original point cloud depth projection and sorting based on the virtual viewpoint direction;

[0013] S3. According to the depth projection and sorting result, simulate occlusion, and select a visible point set;

[0014] S4. Based on the visible point set, obtain a supplementary point set;

[0015] S5. Merge the visible point set and the supplementary point set to obtain an enhanced point cloud.

[0016] In an embodiment of the present application, the S1 comprises the steps of:

[0017] S11. Sampling a random direction vector z from a three-dimensional standard Gaussian distribution ;

[0018] S12. Normalizing the direction vector z and calculating the L2 norm of the direction vector z;

[0019] S13. Based on the L2 norm, calculating the unit direction vector v of the direction vector z as the virtual viewpoint direction.

[0020] In an embodiment of the present application, the unit direction vector v is expressed as:

[0021]

[0022]

[0023] wherein, is the direction vector, is the L2 norm of the direction vector z.

[0024] In an embodiment of the present application, the S2 comprises the steps of:

[0025] S21. Calculating the point product of the original point cloud and the unit direction vector v to obtain a depth set , which is expressed as:

[0026]

[0027] wherein, , is the bit direction vector, is the depth value of ;

[0028] S22. Sorting the depth set to obtain an original point cloud P sorting point list.

[0029] In an embodiment of the present application, the S3 comprises the steps of:

[0030] S31, determining the reserved ratio ;

[0031] S32, determining the number of points in the visible point set according to the reserved ratio , the formula is:

[0032]

[0033] wherein, is the number of points in the original point cloud, is the floor operation;

[0034] S33, selecting the ordered point list , and constructing the visible point set .

[0035] In an embodiment of the present application, the reserved ratio is determined by randomly sampling from a uniform distribution in the range of 0 to 1 .

[0036] In an embodiment of the present application, the S4 comprises the steps of:

[0037] S41, calculating the number of points in the supplementary point set , the formula is:

[0038]

[0039] wherein, is the number of points in the original point cloud, is the number of points in the visible point set, is the number of points in the supplementary point set;

[0040] S42, based on the visible point set , generating supplementary points by using an interpolation method to form the supplementary point set .

[0041] In an embodiment of the present application, the supplementary point set in the S42 is obtained based on the interpolation method, comprising the steps of:

[0042] S42a, randomly and uniformly selecting a seed point from the visible point set ;

[0043] ​​​​​​​S42b, find the k nearest neighbors of the seed point in the visible point set using a nearest neighbor search algorithm (KD-Tree);

[0044] S42c, perform a weighted average of the seed point and its k neighbors to generate a new point;

[0045] S42d, repeat S42a-S42c a total of times to form a supplementary point set .

[0046] In an embodiment of the present application, the enhanced point cloud in S5 is represented by the formula:

[0047]

[0048] wherein, is the enhanced point cloud, is the visible point set, is the supplementary point set, the number of points in , the number of points in , and .

[0049] In an embodiment of the present application, the enhanced point cloud is randomly shuffled to disrupt its arrangement order.

[0050] Second aspect: An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method provided in the first aspect.

[0051] Third aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the method provided in the first aspect.

[0052] Advantages of the present application:

[0053] 1. The method proposed in the present application simulates the viewpoint of a single sensor and simulates real physical occlusion to generate a point cloud with structured and continuous missing, which is highly consistent with the occlusion in the real world and is much better than the traditional random point discarding method.

[0054] 2. The deep learning model trained on the data set enhanced by the method of the present application can better cope with partially occluded data encountered during testing, significantly improving the generalization ability and robustness of the model in real application scenarios.

[0055] ​3、The method of the present application is a data preprocessing / enhancement strategy that is independent of specific downstream models (model-agnostic) and can be seamlessly integrated into the training process of any point cloud-based classification model, making it widely applicable.

[0056] 4、The method of the present application ensures that the enhanced point cloud has the same number of points as the original point cloud through a unique point number preservation mechanism, eliminating the need to modify subsequent network structures or training processes, and facilitating batch training in existing deep learning frameworks.

[0057] 5、The method of the present application adjusts the range of the retention ratio , which can flexibly control the severity of occlusion to adapt to the needs of different tasks and datasets. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The principle flowchart of the 3D point cloud data enhancement method of the present application;

[0059] Figure 2 The principle diagram of occlusion simulation of the 3D point cloud data enhancement method of the present application;

[0060] Figure 3 The effect comparison chart of the method of the present application and the prior art (random point deletion);

[0061] Figure 4 The structural diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0063] The existing 3D point cloud data enhancement method (such as random point deletion) can increase the generalization ability of the model to a certain extent, but its strategy is to remove points in the point cloud independently and uniformly. This way cannot truly simulate the structural and continuous occlusion caused by a single viewpoint. Therefore, the model trained on this non-physical reality enhanced data still performs poorly for the common viewpoint occlusion problem in the real world. This greatly limits the reliability and robustness of deep learning models in safety-critical fields such as autonomous driving and robot navigation.

[0064] To solve the above problems, the present application provides a 3D point cloud data enhancement method based on viewpoint occlusion simulation, Figure 1 The principle flowchart of the 3D point cloud data enhancement method of the present application.

[0065] Embodiment 1:

[0066] The embodiment discloses a 3D point cloud data enhancement method based on viewpoint occlusion simulation, comprising the steps of:

[0067] S1, selecting a virtual viewpoint direction based on the original point cloud.

[0068] The purpose of this step is to determine a random and unbiased observation direction in three-dimensional space to simulate the scenario that the sensor may observe the target object from any angle. In order to ensure the diversity and fairness of the enhancement process, that is, to avoid excessive occlusion simulation for certain specific directions, the selection of the viewpoint direction must achieve uniform distribution on the unit sphere.

[0069] The embodiment adopts a computationally efficient and mathematically proven uniform sampling method to achieve this purpose. The specific operation is as follows:

[0070] First, define a three-dimensional standard Gaussian distribution, denoted as where the mean is a zero vector and the covariance matrix is a three-order unit matrix Randomly sample a three-dimensional direction vector from this distribution. The direction of this direction vector is random in space.

[0071] Then, in order to convert this direction vector to a standard, distance-independent direction representation, normalize it, including calculating the L2 norm (i.e. Euclidean norm) of the direction vector z, the formula is:

[0072]

[0073] Finally, divide the direction vector z by its own L2 norm to obtain the final unit direction vector v, the formula is:

[0074]

[0075] where the length of the unit direction vector v is 1, and its direction is uniformly distributed on the unit sphere. The unit direction vector v serves as the basis for all subsequent calculations, representing the virtual sensor's line of sight direction, i.e. the virtual viewpoint direction.

[0076] S2, based on the virtual viewpoint direction, perform original point cloud depth projection and sorting.

[0077] After determining the unit direction vector v of observation, the purpose of this step is to quantify the visibility or exposure of each point in the original point cloud relative to the viewpoint.

[0078] In geometric intuition, the "convex" part of a point cloud facing the viewpoint is easier to see, while the "concave" or "back" part is prone to be occluded.

[0079] This quantization process is efficiently implemented by vector dot product operation in this embodiment. For each point in the original point cloud P (which is a three-dimensional coordinate vector itself), the dot product between it and the unit direction vector v of the viewpoint is calculated, resulting in a scalar value as the depth value, the formula is expressed as:

[0080]

[0081] where the depth value has the physical meaning of the projection length of the point on the observation axis defined by v. The larger the value of , the more the point faces the virtual viewpoint on the object surface, and thus the higher its priority of visibility.

[0082] After calculating the respective depth values for all N points, all points in the original point cloud P are arranged in descending order according to their corresponding depth values .

[0083] After sorting, an ordered point list is obtained, where the point at the front of the point list is the point with the largest depth value, i.e. the most "visible" point; the point at the end is the most "invisible" point. This step simplifies the complex three-dimensional visibility problem to an efficient one-dimensional sorting problem.

[0084] S3, according to the depth projection and sorting results, simulate occlusion and select the visible point set.

[0085] This step is the core link to realize structured occlusion. It simulates different degrees of occlusion according to an adjustable parameter, and selects the visible point set from the sorted point list.

[0086] First, in order to make the data augmentation have diversified occlusion strength, this embodiment introduces a hyperparameter - the retention ratio range , for example, it can be set to [0.5, 0.8].

[0087] The ratio range defines the lower and upper limits of the proportion of the original point number retained after augmentation. In each augmentation operation, a specific numerical value is randomly sampled from the uniform distribution of the range as the retention ratio .

[0088] Then, according to the sampled ratio and the original point cloud total number of points calculate the number of points of the visible point set that need to be reserved in this operation the calculation method is to take the integer part of the product of and to ensure an integer result:

[0089]

[0090] Finally, from the descendingly arranged point list, directly select the first points. These points, because of their highest depth values, constitute the visible surface of the object from the viewpoint.

[0091] This point set is defined as the visible point set Since these points are adjacent in the original three-dimensional space, the removed points (i.e., the last points in the list) also constitute a continuous and structured region, thus perfectly simulating the effect of physical occlusion.

[0092] S4, based on the visible point set, obtain a supplementary point set.

[0093] The purpose of this step is to solve the training compatibility problem caused by the change in the number of point clouds, and to cleverly use the process to further enhance the features of the data.

[0094] In deep learning practice, in order to utilize GPU for efficient parallel computing, data is usually processed in batches of fixed size (batch), requiring all samples in a batch to have exactly the same dimensions, i.e., the same number of points.

[0095] The detailed operation process of this step is as follows:

[0096] First, calculate the number of supplements, calculate the number of points removed from the original point cloud, that is, the number of points that need to be supplemented back The calculation formula is:

[0097]

[0098] Then, generate points to form a supplementary point set. To achieve this goal, an interpolation-based method can be used to ensure that the newly generated points can be smoothly filled on the visible surface.

[0099] In this embodiment, these supplementary points are not sampled from the original point cloud P, but from the just-generated visible point set The reference points for interpolation are derived from the visible point set Instead of the original point cloud P. By interpolating between the visible points to increase the point cloud density of the visible surface, which can be physically more accurately explained as the sensor has scanned the visible area more finely and with higher resolution.

[0100] Obtaining the supplementary point set based on the interpolation method , the process includes:

[0101] First, a "seed point" is randomly and uniformly selected from .

[0102] Then, the k-nearest neighbors of the seed point in are found using an efficient nearest neighbor search algorithm (KD-Tree).

[0103] Finally, a new point is generated by weighted averaging (i.e. linear interpolation) of the seed point and its k neighbors.

[0104] The above process is repeated times, and the final supplementary point set is formed.

[0105] S5, merging the visible point set and the supplementary point set to obtain the enhanced point cloud.

[0106] First, the visible point set of is merged with the newly generated supplementary point set to form the final enhanced point cloud , which is expressed by the formula

[0107]

[0108] At this time, the total number of points in the enhanced point cloud P' is exactly , which restores the number of the original point cloud, thus solving the compatibility problem of batch training.

[0109] Further, the enhanced point cloud is randomly shuffled, as the merging operation may cause the points in the enhanced point cloud to have an order (for example, all points are in front, and all points are at the back), in order to prevent the deep learning model from learning useless bias from this artificial order, a random shuffle (shuffle) must be performed on all points in the enhanced point cloud to completely disrupt their arrangement order, forming the final enhanced point cloud .

[0110] Embodiment 2:

[0111] This example is based on the method of example 1, and extracts an original point cloud sample of the "airplane" category from the ModelNet40 training set To illustrate the enhancement process of the method of the present application completely, step by step.

[0112] Original point cloud is a NumPy array with shape (1024, 3), representing the coordinates of 1024 three-dimensional points, and the point cloud is complete and unoccluded.

[0113] First, virtual viewpoint selection is performed, which is dynamically executed every time a single sample is enhanced to ensure the diversity of the enhancement.

[0114] Generate a random vector, use the random.randn function of the NumPy library to sample a random direction vector z from a three-dimensional standard Gaussian distribution Assuming that in this specific execution, the sampled vector is:

[0115]

[0116] Normalization, calculate the L2 norm (i.e. Euclidean length) of vector z, formula:

[0117]

[0118]

[0119] Determine the viewpoint direction, divide each component of the direction vector z by its L2 norm to get the unit direction vector v, formula:

[0120]

[0121] This unit direction vector v is the virtual viewpoint direction used in this enhancement operation.

[0122] Then, based on the viewpoint unit direction vector v, perform original point cloud Depth projection and sorting. The goal of this step is to calculate a "visibility score" for all 1024 points in .

[0123] Depth value calculation, use the matrix multiplication function of NumPy (@ operator or np.dot) to perform dot product operation between the point cloud matrix with shape (1024, 3) and the viewpoint direction vector v (after transposition) with shape (3, 1). Calculate the depth value for each point to get a depth value vector D with shape (1024, 1), formula:

[0124]

[0125] Data sorting, original point cloud is associated with a depth value vector D. According to the values in the depth value vector D, it is sorted in descending order. After sorting, a new point cloud set is obtained, the first row corresponds to the point with the maximum depth value, and the last row corresponds to the point with the minimum depth value. At the same time, a sorted point list is obtained, which records the position of each point in the new list.

[0126] Then, occlusion simulation and point selection are performed. According to the preset rules, part of the points are removed to form a structured occlusion.

[0127] Determine the retention ratio, and set the hyperparameter range of the retention ratio in this embodiment as . A value is randomly sampled in this range using the random.uniform function of Python, and it is assumed that the sampled value is .

[0128] Calculate the number of points to be retained, according to the sampled ratio and the total number of points , the number of visible points to be retained is calculated .

[0129]

[0130] Screening visible point set, based on the sorted point cloud set , the first 737 rows are directly selected. These 737 points constitute the visible point set .

[0131] These points correspond to a continuous surface of the aircraft model facing the viewpoint v in the three-dimensional space, for example, the belly and one side of the wing of the aircraft. The discarded points correspond to the back of the aircraft and the occluded wing on the other side, forming a continuous and structured missing area, as shown in the principle shown in the right side of the attached Figure 2 .

[0132] Then, the point cloud number is kept, which is the key to ensure that the enhanced data can be directly sent to the model for training, and it also plays a role in increasing the feature density of the visible area.

[0133] Calculate the number of points to be supplemented, and the formula is:

[0134]

[0135] To generate these 287 additional points, a method based on local neighborhood interpolation can be used instead of simple resampling.

[0136] The method based on local neighborhood interpolation generates completely new points on the visible surface, thus more realistically simulating high-density scans, which specifically includes:

[0137] Constructing a search structure, a KD-Tree is constructed on the visible point set (containing 737 points) to quickly query the nearest neighbors of any point. This is repeated 287 times: Randomly and uniformly select a point from as a "seed point".

[0138] Find the k nearest neighbors (e.g., k = 3) of the seed point using the KD-Tree. Then perform interpolation: generate a set of random weights that sum to 1, then take a weighted average of the seed point and its k neighbors to calculate a completely new three-dimensional coordinate, which is the additional point.

[0139] Collect the 287 points generated by the interpolation method to form the additional point set . In this way, each point in is "created" based on the local geometry in

[0140] , which smoothly fills between the visible points, effectively improving the density and surface details of the point cloud, and avoiding the problem of point overlap and local aggregation caused by simple resampling. Finally, merge the point cloud, use the concatenate function of NumPy to merge the visible point set (shape (737, 3)) and the additional point set

[0141] (shape (287, 3)) in the first dimension.

[0142] At this time, the shape is (1024, 3), and the number of point clouds has returned to the original size.

[0143] Random shuffle, shuffle all 1024 rows of This can be achieved by generating a random index from 0 to 1023, and then using the index to rearrange the rows of This step is crucial as it eliminates the artificial order caused by the merging operation (i.e., all original visible points first, all additional points last), ensuring that the data input to the neural network is unordered, consistent with the basic nature of point clouds as unordered sets.

[0144] The above steps together constitute a complete "on-the-fly" data augmentation function. During the PointNet++ model's training loop, this augmentation function is called independently for each point cloud sample in the batch each time a batch of data is retrieved from the DataLoader.

[0145] This means that for the same "aircraft" sample :

[0146] In the first training cycle (epoch 1), it may be obscured by a viewpoint from "directly above".

[0147] In the second training cycle (epoch2), it may be obscured by a viewpoint from a "45-degree angle to the left front".

[0148] Even within the same period, if the dataset is iterated multiple times, the occlusion viewpoints and occlusion ratios it encounters each time are entirely new and random.

[0149] like Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the occlusion simulation principle in this embodiment. Figure 2 (a) in the image is the original point cloud: showing a complete, unobstructed 3D point cloud of the aircraft. Figure 2 (b) Viewpoint and Visible Surface: The red arrows indicate randomly selected viewpoint vectors, which are opposite to the real viewpoint direction. The darker the point cloud color, the greater the projection depth value and the higher the visibility score. Figure 2 (c) Point cloud after occlusion. Blue represents visible points, and gray represents occluded points. Figure 2 (d) Final augmented point cloud: Showing the final result P' after executing S4. It can be observed that... Figure 2 The point cloud density of the visible part in (d) has increased, and the interpolated orange points have been added.

[0150] Figure 2 This embodiment demonstrates how to generate a sample with realistic, continuous occlusion from a complete object, and clearly illustrates the effect of the "point cloud quantity preservation" step.

[0151] Figure 3 This is a comparison chart of the effects of this embodiment and the prior art (randomly dropping points). The results of the method in this embodiment are as follows: Figure 3 As shown in (a), this is the result after applying the method of this embodiment. Structured continuous occlusion samples. The results of the random drop-point method are as follows... Figure 3 As shown in (b), the entire aircraft model becomes uniformly sparse, but the overall outline is still visible.

[0152] By Figure 3 It can be seen that the method of the embodiment can simulate real physical occlusion, which is much better than the traditional random point discarding method.

[0153] This dynamic and diverse augmentation approach allows the model to see occlusions of all possible severities from almost all possible angles throughout the training process. This forces the model to not rely on any specific local features (as they can be occluded at any time), but to learn to find more essential and robust global structure information from the remaining, incomplete views to make the "airplane" classification decision.

[0154] Compared with the prior art (such as random point discarding), the advantages of the embodiment are obvious. If P airplane Using random point discarding (for example, discarding 25% of the points), the result will be that the entire airplane model becomes uniformly sparse, but the overall outline is still clearly visible. Using the method of the present application, the result is that part of the airplane (such as the entire right wing and tail) completely disappears, and the point cloud density of the visible part (the left wing and the nose) may even increase. The latter is obviously a more difficult and more realistic identification task.

[0155] In summary, the embodiment details and reproducibly demonstrates the complete process of the method proposed by the present application. Through this simulation based on physical reality, the method of the present application can generate high-quality, challenging augmented data, so that after the training is completed, the deep learning model can perform much better than the robustness and accuracy that can be achieved by the prior art when facing various occlusions in the real world. The principle and implementation of the present application are described in the specific examples in the embodiment, and the above description of the embodiment is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

[0156] The present application starts from a physical process, and innovatively converts the data augmentation problem into a process of simulating physical observation, that is, "taking pictures" of a 3D object from a random viewpoint, and only retaining the visible part. At the same time, unlike randomly removing individual points, the present method uses structured information removal, which removes a continuous and structured point set, forcing the model to learn the ability to infer the overall structure from an incomplete local view. At the same time, the neighborhood-based interpolation method for visible points is used to keep the total number of points unchanged, which ingeniously solves the problem of different sample sizes after data augmentation, so that it can be directly used in the standard training process.

[0157] In summary, the application effectively solves the deficiencies of the prior art in handling point cloud occlusion problems by a novel, physically realistic viewpoint occlusion simulation method, and provides an efficient and general solution for training more robust and reliable 3D point cloud classification models.

[0158] The application also provides an electronic device, Figure 4 The structural schematic diagram of the electronic device provided by the embodiment of the application is shown in the figure, Figure 4 As shown in the figure, the electronic device can include a processor, a communications interface, a memory and a communications bus, wherein the processor, the communications interface and the memory complete mutual communication through the communications bus. The processor can invoke the logical instructions in the memory, for example, to execute the following method:

[0159] S1, selecting a virtual viewpoint direction based on the original point cloud;

[0160] S2, performing original point cloud depth projection and sorting based on the virtual viewpoint direction;

[0161] S3, simulating occlusion and selecting a visible point set according to the depth projection and sorting results;

[0162] S4, obtaining a supplementary point set based on the visible point set;

[0163] S5, merging the visible point set and the supplementary point set to obtain an enhanced point cloud.

[0164] In addition, the logical instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

[0165] The embodiment of the application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method provided by each of the above embodiments, for example, including:

[0166] S1, selecting a virtual viewpoint direction based on the original point cloud;

[0167] S2, performing depth projection and sorting of the original point cloud based on the virtual viewpoint direction;

[0168] S3, simulating occlusion and selecting a visible point set according to the depth projection and sorting result;

[0169] S4, obtaining a supplementary point set based on the visible point set;

[0170] S5, merging the visible point set and the supplementary point set to obtain an enhanced point cloud.

[0171] The apparatus embodiments described above are merely illustrative, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0172] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0173] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A 3D point cloud data enhancement method based on viewpoint occlusion simulation, characterized in that, The method comprises the following steps: S1, selecting a virtual viewpoint direction based on an original point cloud; S2, performing depth projection and sorting of the original point cloud based on the virtual viewpoint direction; S3, simulating occlusion and selecting a visible point set according to the depth projection and sorting result; S4, obtaining a supplementary point set based on the visible point set; S5, merging the visible point set and the supplementary point set to obtain an enhanced point cloud. The S3 comprises the following steps: S31, determining a reserved proportion r; S32、According to the reserved proportion r, the point number N of the visible point set to be reserved is calculated keep The formula is: N keep = floor(r*N) wherein N is the number of points of the original point cloud P, and floor() is a down rounding operation; S33, selecting the ordered point list N keep points, and constructing the visible point set P vis ; The S4 comprises the following steps: S41, calculate the supplementary point set P fill The number of points is calculated by the formula: N fill = N - N keep where N is the number of points of the original point cloud P, N keep is the number of points of the visible point set P vis , N fill is the number of points of the supplementary point set P fill ; S42, based on the visible point set P vis , take interpolation method, generate N fill supplementary points, constitute the supplementary point set P fill ; The S42 obtains the supplementary point set P based on an interpolation method fill comprising the steps of: S42a, selecting a seed point randomly and uniformly from the set of visible points P vis S42b, selecting a seed point randomly and uniformly from the set of visible points P S42b, find the k nearest neighbors of the seed point in the set of visible points P using a nearest neighbor search algorithm; vis S42b, find the k nearest neighbors of the seed point in the set of visible points P using a nearest neighbor search algorithm; S42c, performing weighted average on the seed point and the k neighboring points to generate a new point; S42d, repeat S42a-S42c for N fill Next, a set of supplementary points P is formed fill .

2. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 1, characterized in that, The S1 comprises the following steps: S11, sampling a random direction vector z from a three-dimensional standard Gaussian distribution N(0, I3); S12, performing normalization processing on the direction vector z to calculate the L2 norm of the direction vector z; S13, calculating a unit direction vector v of the direction vector z as the virtual viewpoint direction based on the L2 norm.

3. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 2, characterized in that, The unit direction vector v is expressed by the following formula: where z = [z x ,z y ,z z ] is a direction vector and ||z||2is the L2 norm of the direction vector z.

4. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 2, characterized in that, The S2 comprises the following steps: S21, calculating the dot product of the original point cloud P and the unit direction vector v to obtain a depth set D, which is expressed by the following formula: d i = p i · v where p i ∈ P, v is a bit direction vector, d i ∈ D is a depth value of p i . S22, sorting the depth set D to obtain an original point cloud P sorting point list.

5. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 1, characterized in that, The enhanced point cloud in the S5 is expressed by the following formula: P' = P vis ∪P fill where P' is the enhanced point cloud, P vis is the visible point set, P fill is the supplemental point set, P vis has N keep points, P fill has N fill points, and N keep +N fill =N.

6. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 5, characterized in that, At each enhancement operation, the retention ratio r is determined by random sampling from a uniform distribution U(r min ,r max ) over the range [r min ,r max ].

7. The 3D point cloud data enhancement method based on viewpoint occlusion simulation according to claim 5, characterized in that, Randomly shuffling the enhanced point cloud P' to break its arrangement order.

Images