Open wagon surface defect point cloud data enhancement method based on improved 3DGAN
By improving the 3D GAN model, the problem of point cloud data enhancement under small sample data is solved, and the generalization performance and robustness of the network are improved, laying the foundation for the automatic identification and positioning system of surface defects of the vehicle Technical foundation.
Patent Information
- Application Number
- CN202411595071.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2025-02-14
AI Technical Summary
In practical application scenarios, it is difficult to obtain a sufficient number of point cloud data for open car surface defects, especially the data of holes and superlimit deformation is small, which leads to difficulty in enhancing point cloud data under small sample point cloud data.
Using an improved 3D GAN method, sample amplification and data enhancement of the existing open car defect point cloud dataset is used to increase the number of layers of the adversarial network, and the introduction of the conv_layer helper function and torch.squeeze function to generate high-quality open car defect points. Cloud dataset.
It improves the generalization performance and robustness of the target detection network, increases the diversity of data samples, and helps to establish a technical foundation for automatic identification and positioning system for open vehicle surface defects.
Smart Images
Figure CN119444597A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a point cloud data enhancement method, and in particular to a gondola defect point cloud data enhancement method based on an improved 3D GAN, and belongs to the technical field of machine learning. Background Art
[0002] As one of the classic tasks in the field of computer vision, three-dimensional point cloud target detection has developed rapidly in recent years. Point cloud defect detection technology based on deep learning has significant advantages in detection accuracy compared with traditional two-dimensional defect detection algorithms (such as those based on texture features, color features, Haar feature classifiers, adaptive clustering, etc.). This technology can improve detection accuracy as the scale of point cloud data sets increases, effectively reducing missed detection and false detection rates. A large number of point cloud data samples are the prerequisite for ensuring the accuracy of three-dimensional defect detection. Data sets are a crucial part of the field of deep learning, but this condition is often difficult to achieve in actual application scenarios. It is difficult and costly to collect point cloud data of surface defects of open cars in car factories. Due to safety production factors, it is difficult to obtain point cloud data containing defects. There are very few point cloud data of open car surfaces containing holes and excessive deformation defects. Therefore, point cloud data enhancement based on small sample point cloud data has become an effective means to improve network performance. For this reason, efficient and robust methods are needed to enhance point cloud data of open car surface defect point cloud data sets.
[0003] In order to obtain a sufficient number of open car defect point cloud data at a low cost, traditional sample augmentation methods were used in early work. Traditional sample augmentation methods include rotation, scaling, adding noise, and random discarding, but the defect point clouds obtained by these methods are difficult to simulate the many details of the open car defect point clouds in real scenes, and the effect is too simplistic. Three-dimensional generative adversarial network (3D GAN) is a technology that learns to generate three-dimensional samples based on high-dimensional noise vectors. Compared with two-dimensional GANs, 3D GANs are able to generate more complex three-dimensional structures and shapes, which makes them excellent in generating diversity. However, due to the complexity of three-dimensional data, three-dimensional models directly generated by random noise may encounter problems of insufficient resolution or blurred details, which may affect the generalization ability of the model and reduce its performance when processing real-world three-dimensional data. Summary of the invention
[0004] In view of the above problems, the present invention proposes a gondola defect point cloud data enhancement method based on improved 3D GAN, which can efficiently and robustly perform sample amplification and data enhancement on the existing gondola defect point cloud dataset, increase the diversity of training samples, and thereby improve the generalization performance and robustness of the target detection network, laying a technical foundation for the ultimate establishment of an automatic recognition and positioning system for gondola surface defects.
[0005] The present invention performs data enhancement on the open wagon defect point cloud data set and makes full use of the 3D GAN network to realize the generation of differentiated open wagon defect point cloud data, thereby achieving the purpose of data enhancement.
[0006] The open car defect point cloud data enhancement method based on improved 3D GAN includes the following steps: (1) Obtain a certain number of point cloud data of two types of defects, holes and excessive deformation, on open-top cars to form an original data set; (2) Point cloud preprocessing; (3) Normalize the spatial scale of the point cloud to form a normalized data set; (4) Using traditional methods to expand the point cloud dataset to form an extended dataset; (5) Build an improved 3D GAN model; (6) Using the extended dataset as input, a point cloud dataset of holes and excessive deformation defects on the surface of open cars is generated by improving the 3D GAN model.
[0007] The improvements to the 3D GAN model in step (5) are as follows: 1) Increase the number of adversarial network layers to make the original defect point cloud completely and clearly presented; 2) Introduce the conv_layer auxiliary function at the beginning of the __init__ method in the adversarial network; 3) Introduce the torch.squeeze function at the end of the forward method of the adversarial network.
[0008] Beneficial technical effects of the present invention: An improved 3D GAN model is proposed. More convolutional layers are introduced into the adversarial network to solve the problem of feature loss or distortion when the source code generates point cloud datasets. The conv_layer auxiliary function is introduced to uniformly modify the parameters of the convolution layer. The torch.squeeze function is introduced to remove the redundant dimensions of the adversarial network output shape to ensure that the tensor shape meets the requirements. Through iterative operation, a high-quality open car defect point cloud dataset is effectively generated, which increases the diversity of data samples. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 It is a structural schematic diagram of the original point cloud acquisition device adopted by the present invention; Figure 2 It is a flow chart of the method for enhancing defect point cloud data of open car based on improved 3D GAN of the present invention; Figure 3 Schematic diagram of the improved 3D GAN network structure of the present invention; Figure 4 It is a schematic diagram of the structure of the improved generator of the present invention; Figure 5 is a schematic diagram of the structure of the improved discriminator of the present invention; Figure 6 This is a point cloud effect diagram of excessive deformation defects on the surface of a gondola car generated by the present invention; Figure 7 It is a point cloud effect diagram of hole defects on the surface of an open car generated by the present invention. DETAILED DESCRIPTION
[0010] Combined with Figure 1-7 , describing the steps and specific operations of the present invention.
[0011] The present invention performs data enhancement on the open wagon defect point cloud data set and makes full use of the 3D GAN network to realize the generation of differentiated open wagon defect point cloud data, thereby achieving the purpose of data enhancement.
[0012] The open car defect point cloud data enhancement method based on improved 3D GAN includes the following steps: (1) Obtain a certain number of point cloud data of two types of defects, holes and excessive deformation, on open-top cars to form an original data set; (2) Point cloud preprocessing; (3) Normalize the spatial scale of the point cloud to form a normalized data set; (4) Using traditional methods to expand the point cloud dataset to form an extended dataset; (5) Build an improved 3D GAN model; (6) Using the extended dataset as input, a point cloud dataset of holes and excessive deformation defects on the surface of open cars is generated by improving the 3D GAN model.
[0013] The improvements to the 3D GAN model in step (5) are as follows: 1) Increase the number of adversarial network layers to make the original defect point cloud completely and clearly presented; 2) Introduce the conv_layer auxiliary function at the beginning of the __init__ method in the adversarial network; 3) Introduce the torch.squeeze function at the end of the forward method of the adversarial network.
[0014] Specifically, the main steps of the open car surface defect point cloud data enhancement method based on improved 3D GAN in the present invention are as follows: 1. Obtain the original point cloud dataset of the surface defects of the open car The original point cloud acquisition device of the present invention comprises a horizontal linear module 1, a vertical linear module 2, a camera bracket 3 and a 3D industrial camera 4. The horizontal linear module 1 is horizontally fixedly installed on the ground along the length direction of the open car. The main body of the vertical linear module 2 is vertically fixedly installed on the slider of the horizontal linear module 1. The camera bracket 3 is fixedly installed on the slider of the vertical linear module 2. The 3D industrial camera 4 is fixedly installed on the camera bracket 3 facing the open car. The square meter motion range of the horizontal linear module 1 and the vertical linear module 2 covers the side of the open car.
[0015] The horizontal linear module 1 and the vertical linear module 2 of the original point cloud acquisition device are controlled to make the 3D industrial camera 4 reach the defect area of the open car, and the horizontal linear module 1 moves horizontally to drive the 3D industrial camera 4 to scan and obtain the defect point cloud. Repeat the above operation, and then use the point cloud segmentation method to process the scanned point cloud for a certain number of defect point clouds to obtain the defect original point cloud data set.
[0016] 2. Point cloud preprocessing The denoising filtering method is mainly used in point cloud preprocessing.
[0017] (1) Through filtering: Use CloudCompare software to determine the range of points to be retained in the Z-axis direction, then remove points outside the specific range in the Z-axis direction and focus on the area of interest; (2) Statistical filtering: set filtering parameters, calculate the distance between each point and its neighboring points, and remove outliers based on the set threshold; (3) Radius filtering: Set a radius and point number threshold to remove points that do not have enough neighboring points within a given radius, thereby removing outliers and noise, thus completing point cloud preprocessing.
[0018] Point cloud preprocessing is achieved through the open3d, numpy, and os code libraries in the Python program code, specifically by using the z_pass_through function for Z-axis through filtering, the pcd.remove_statistical_outlier function for statistical filtering, and the o3d.geometry.PointCloud function for radius filtering.
[0019] 3. Normalization of point cloud spatial scale The point cloud dataset in pcd format is converted into a .mat format dataset with a spatial shape of 120×120×120 int8 for training the 3D generative adversarial network.
[0020] This step is implemented through the open3d, numpy, os, and scipy code libraries in the Python program code. The voxel_down_sample function is used to downsample the point cloud, and the voxel_size value is set to reduce the density of the point cloud and increase the processing speed. The np.amin and np.amax functions are used to find the minimum and maximum boundaries of the point cloud, and padding is added to the boundaries to ensure that the point cloud does not exceed the grid range when scaling. The scale function is used to scale the point cloud to the specified grid size. The np.zeros function is used to create a three-dimensional array with a spatial shape of 120×120×120, and the data type is np.int8, which is used to store point cloud data. The scipy.io.savemat function is used to save the processed point cloud data as a mat file. So far, the spatial scale normalization of the defect point cloud dataset has been achieved to form a normalized dataset.
[0021] 4. Use traditional point cloud data enhancement methods to expand the defect point cloud dataset The specific steps are as follows: (1) Each point cloud of the original data set is randomly rotated, and the original data set is doubled; (2) Each point cloud of the original dataset is randomly reduced and enlarged, and the original dataset is expanded twice; (3) Random noise is added to each point cloud of the original dataset, and the original dataset is doubled; (4) Each point cloud in the original data set is randomly discarded, and the original data set is doubled.
[0022] This step is implemented through the open3d, numpy, and os code libraries in the Python program code, including the random function, scale function, array function, radians function, delete function, utility function, write_point_cloud function, and read_point_cloud function to achieve batch sample expansion of point clouds. Specifically, the rotation matrix is defined by using the np.array function, the point_cloud.rotate function is used to achieve random rotation of the point cloud, the scale_factor function is used to set the scaling ratio, the pcd.scale function is used to achieve random scaling of the point cloud and keep the center of mass unchanged, the np.random.normal and o3d.utility.Vector3dVector functions are used to add random noise to the point cloud, and the np.random.choice function is used to randomly select the index of the point to be discarded to achieve random discarding of the point cloud. After a series of operations, the defective point cloud data set is expanded to form an extended data set.
[0023] 5. Build an improved 3D GAN model First, build a 3D GAN model. The specific steps are as follows: (1) Configure the environment: Download the official source code of 3D GAN from GitHub, then use it to create a virtual environment and install scipy, scikit-image, floydhub, and Pytorch according to the requirements file of the 3D GAN source code; (2) Data input: Input the constructed data set into the 3D GAN network structure model, and input the established open car defect point cloud data set into the 3D GAN network structure model; (3) Configure hyperparameters: adjust the learning rate to ensure the effective tolerance of the model, modify the loss function to optimize the model output, adjust the number of batch samples to adapt to the hardware capacity to avoid memory explosion, adjust the number of iterations to avoid underfitting or overfitting of the model, etc. Set the optimal hyperparameter combination before running the program; (4) Run the program to train the 3D GAN network model. In the PyCharm terminal page of the 3D GAN project, run cd ..\3D_GAN, switch to the 3D_GAN folder, and then run python main.py –cube_len 122 to start training. High-quality open car defect point cloud data can be obtained through iterative training.
[0024] Improve the 3D GAN network structure model. The specific improvements are as follows: (1) Increase the number of adversarial network layers to make the original defect point cloud appear complete and clear The official source code of 3D GAN downloaded from GitHub requires a .mat file in 30×30×30 int8 spatial shape. When the defect point cloud dataset is converted to a 30×30×30 int8 spatial shape format, the point cloud size will be squeezed, resulting in severe distortion and loss of features. Therefore, in order to expand the spatial shape of the 3D generative adversarial network model, it is necessary to add additional layers to adapt to the larger defect point cloud spatial shape. After continuous testing and adjustment, it is finally determined to convert the defect point cloud dataset to a 120×120×120 int8 spatial shape format. The generator_G adds three convolutional layers, layer6, layer7, and layer8, and the discriminator_D adds a transposed convolutional layer of layer6. This can not only make the defect point cloud features complete and clear, but also ensure the simplicity and conciseness of the 3D generative adversarial network. The defect generation effect reaches an ideal level.
[0025] (2) Introduce the conv_layer auxiliary function at the beginning of the __init__ method in the adversarial network The conv_layer function is used in the construction process of the generator_G and the discriminator_D to create and configure the convolution layer, which simplifies the creation process of the convolution layer and makes the network definition clearer and easier to manage.
[0026] The parameters that the conv_layer auxiliary function needs to input include: input_dim (the number of channels of the input feature map), output_dim (the number of channels of the output feature map), kernel_size (the size of the convolution kernel), stride (the step size of the convolution), padding (the padding method of the input feature map, used to control the size of the output feature map), bias (a Boolean value indicating whether to add a bias term to the convolution layer). The network configuration is adjusted by passing these convolution layer parameters as input to the conv_layer function.
[0027] The conv_layer function significantly improves code reusability by providing a unified interface to create convolutional layers. This design allows the same convolutional layer configuration to be used in different places without having to repeatedly write the same code blocks, thus reducing code redundancy.
[0028] More importantly, the conv_layer function enhances the readability of the code. By encapsulating the creation details of the convolutional layer, it makes the definition of the network architecture more intuitive and clear. If the convolutional layer needs to be modified, such as changing the activation function or adjusting the batch normalization parameters, it only needs to be modified once in the conv_layer function, instead of having to adjust one by one in multiple locations in the network definition, which greatly reduces the maintenance workload and reduces the possibility of errors.
[0029] In summary, the conv_layer helper function provides a flexible and reusable building block for both the generator and the discriminator, helping to keep the code clean and maintainable.
[0030] (3) Introduce the torch.squeeze function at the end of the forward method of the adversarial network Each dimension in a tensor has a size, and if a dimension has a size of 1, then that dimension has no actual effect on the shape of the tensor, it just adds an extra axis. The torch.squeeze function checks all dimensions of a tensor and removes any dimension if it has a size of 1.
[0031] At the end of the generator's forward method, the output out is a tensor of shape [batch_size, 1, cube_len, cube_len, cube_len], where the second dimension (size 1) is redundant because it does not carry any additional information. After using torch.squeeze(out), the output shape becomes [batch_size, cube_len, cube_len, cube_len], removing the dimension of size 1.
[0032] In the forward method of the discriminator, the final output out is a tensor with a shape of [batch_size, 1, 1, 1, 1]. After using torch.squeeze(out), the output shape becomes [batch_size], removing all dimensions of size 1.
[0033] Removing these extra dimensions can make the shape of the output tensor more concise and ensure that the tensor shape meets the requirements. In subsequent operations, such as calculating the loss function, the absence of extra dimensions can reduce confusion and errors.
[0034] 6. Use improved 3D GAN training to expand the dataset and generate defect point cloud dataset The specific steps are as follows: (1) Obtaining point cloud data information of holes and excessive deformation defects on the surface of the open car, including the point cloud space shape size, quantity and number of channels. The point cloud space shape size of the present invention is 120×120×120 int8, the number of hole defect point clouds is 301, the number of excessive deformation defect point clouds is 550, and the number of channels is 120; (2) Set the improved 3D GAN network hyperparameters, and set the number of iterations n_epochs, batch processing batch_size, generator initial learning rate g_lr, discriminator initial learning rate d_lr and other parameters according to the device conditions. The hyperparameters used in this paper are Epoch=500, Batchsize=10, Channels=1, g_lr=0.0025, d_lr=0.001; (3) The open car surface hole and over-limit deformation defect point cloud data set is input into the improved 3D GAN model with set hyperparameters for training. The configuration used in the present invention is CPU: Intel Xeon W-2150B processor, CPU main frequency: 4.5GHz, memory: DDR4 32G, GPU: NVIDIA GeForce RTX 3090Ti, video memory: 24G, deep learning framework is PyTorch, and the development environment is Pytorch2.0.1, Python3.9, Cuda12.3; (4) After training is completed, point cloud data of holes and excessive deformation defects on the surface of the open car are generated.
Claims
1. The open-top car defect point cloud data enhancement method based on improved 3D GAN is used to perform sample amplification and data enhancement on the open-top car defect point cloud dataset, including the following steps: (1) Obtain a certain number of point cloud data of two types of defects, holes and excessive deformation, on open-top cars to form an original data set; (2) Point cloud preprocessing; (3) Normalize the spatial scale of the point cloud to form a normalized data set; (4) Using traditional methods to expand the point cloud dataset to form an extended dataset; (5) Build an improved 3D GAN model; (6) Using the extended dataset as input, a point cloud dataset of holes and excessive deformation defects on the surface of open cars is generated by improving the 3D GAN model.
2. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: The original point cloud acquisition device for point cloud data of two types of defects, namely holes and over-limit deformation, of the open car in step (1) comprises a horizontal linear module (1), a vertical linear module (2), a camera bracket (3) and a 3D industrial camera (4), wherein the horizontal linear module (1) is fixedly mounted horizontally on the ground along the length direction of the open car, the main body of the vertical linear module (2) is fixedly mounted vertically on a slider of the horizontal linear module (1), the camera bracket (3) is fixedly mounted on the slider of the vertical linear module (2), and the 3D industrial camera (4) is fixedly mounted on the camera bracket (3) facing the open car, and the square meter movement range of the horizontal linear module (1) and the vertical linear module (2) covers the side of the open car.
3. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: The improvements to the 3D GAN model in step (5) include: 1) Increase the number of adversarial network layers to make the original defect point cloud completely and clearly presented; 2) Introduce the conv_layer auxiliary function at the beginning of the __init__ method in the adversarial network; 3) Introduce the torch.squeeze function at the end of the forward method of the adversarial network.
4. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: The specific methods of point cloud preprocessing in step (2) include: through filtering, statistical filtering and radius filtering.
5. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: Step (3) The method for normalizing the spatial scale of the point cloud is as follows: convert the point cloud dataset in pcd format into a .mat format dataset with a spatial shape of 120×120×120 int8.
6. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: Step (4) The specific steps of using the traditional method to expand the point cloud dataset are as follows: 1) Each point cloud of the original data set is randomly rotated, and the original data set is doubled; 2) Each point cloud of the original dataset is randomly reduced and enlarged, and the original dataset is expanded twice; 3) Random noise is added to each point cloud of the original dataset, and the original dataset is doubled; 4) Each point cloud of the original data set is randomly discarded, and the original data set is doubled.
7. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 1, characterized in that: Step (6) The specific steps of generating a point cloud dataset of holes and over-limit deformation defects on the surface of a gondola by improving the 3D GAN model are as follows: 1) Obtain point cloud data information of holes and excessive deformation defects on the surface of open wagons; 2) Set the improved 3D GAN network hyperparameters; 3) Input the point cloud dataset of holes and over-limit deformation defects on the surface of open-top cars into the improved 3D GAN model with set hyperparameters for training; 4) After training is completed, point cloud data of holes and excessive deformation defects on the surface of the open car are generated.
8. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN as claimed in claim 3, characterized in that: Improvement point 1) Increase the number of layers of the adversarial network to make the original defect point cloud complete and clear. The specific method is: convert the defect point cloud dataset into a 120×120×120 int8 spatial shape format, add three convolution layers, layer6, layer7, and layer8 to the generator_G, and add a transposed convolution layer of layer6 to the discriminator_D.
9. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 3, characterized in that: Improvement 2) Introduce the conv_layer auxiliary function at the beginning of the __init__ method in the adversarial network. The parameters required by the conv_layer auxiliary function include: input_dim (the number of channels of the input feature map), output_dim (the number of channels of the output feature map), kernel_size (the size of the convolution kernel), stride (the step size of the convolution), padding (the padding method of the input feature map, used to control the size of the output feature map), bias (Boolean value, indicating whether to add a bias term in the convolution layer). The network configuration is adjusted by passing these convolution layer parameters as input to the conv_layer function.
10. The method for enhancing defect point cloud data of open wagon based on improved 3D GAN according to claim 3, characterized in that: Improvement point 3) Introduce the torch.squeeze function at the end of the forward method of the adversarial network. At the end of the forward method of the generator, the output out is a tensor with a shape of [batch_size, 1, cube_len, cube_len, cube_len]. After using torch.squeeze(out), the output shape becomes [batch_size, cube_len, cube_len, cube_len], and the dimension of size 1 is removed. In the forward method of the discriminator, the final output out is a tensor with a shape of [batch_size, 1,1,1,1]. After using torch.squeeze(out), the output shape becomes [batch_size], and all dimensions of size 1 are removed.