A method for completing laser radar point cloud data

Abstract

The present invention relates to a method for completing laser radar point cloud data, which belongs to the technical field of image completion, including a dual-light sensor collecting laser radar images and visible light images in the same space and time, and performing preprocessing; constructing an adversarial network, the adversarial network including a generator and a discriminator; inputting the preprocessed laser radar images and visible light images into the adversarial network for training, and training to the optimum through multiple iterations; the output result of the generator trained to the optimum adversarial network is the final completed image of the laser radar point cloud data. The present invention is based on the target contour obtained from the visible light image collected in the dual-light coaxial system to counter-generate the data required to complete the point cloud data obtained by the laser radar, which can effectively increase the sampling point density in the edge area while maintaining the accuracy of the data structure, thereby improving the data quality and sensor performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image completion, and specifically relates to a method for laser radar image completion, which is based on a coaxial dual-light sensor to collect visible light images and laser radar images, and is implemented through a generative adversarial network (GAN) and a residual network (ResNet). Background Art

[0002] LiDAR can quickly obtain high-precision, long-distance three-dimensional information of targets in harsh environments, and is widely used in high-tech fields such as three-dimensional perception and environmental modeling. However, due to the limitations of its working principle and the fact that existing hardware often cannot achieve high-rich details and high clarity, LiDAR will generate unstable data points when scanning the edges of objects, resulting in blurred edge positions and errors in distance estimation. In the coaxial dual-light LiDAR system, the edge contours of typical object features obtained from visible light images are used as criteria, and the missing data of LiDAR is continuously generated through corresponding interpolation, which helps to further improve the quality of LiDAR point cloud data, improve the accuracy and reliability of data, and enhance target recognition capabilities.

[0003] Traditional LiDAR image completion methods include point cloud interpolation, plane fitting, voxel gridding, and constrained optimization methods. The above methods are highly practical in some occasions, but usually require manual data preprocessing and feature selection. These require experienced professionals and are relatively cumbersome, which to a certain extent limits the development of LiDAR image completion methods. In addition, the existing technology usually uses the same transformation or representation for different source images during the completion process. However, it may not be applicable to LiDAR and visible light images because the point cloud in the LiDAR image and the pixel in the visible light image are two completely different data structures.

[0004] In recent years, the research on generative adversarial networks (GANs) has gradually expanded and deepened, with important breakthroughs and applications in the fields of image generation and processing, video generation and processing, and speech synthesis and conversion. However, there has been no research on using generative adversarial networks to achieve the completion of lidar point cloud images based on visible light images. Summary of the invention

[0005] The present invention provides a method for completing laser radar point cloud data to solve the problem of manual preprocessing and feature selection of laser radar images in the prior art, and at the same time solve the problem that laser radar and visible light images cannot be effectively fused. The present invention is based on the target contour obtained from the visible light image collected in the dual-light coaxial system to counteract the data needed to complete the point cloud data obtained by the laser radar. It can also effectively increase the sampling point density in the edge area while maintaining the accuracy of the data structure, thereby improving the data quality and sensor performance.

[0006] The technical solution of the present invention is:

[0007] A method for completing laser radar point cloud data comprises the following steps:

[0008] S1: The dual-light sensor collects lidar images and visible light images in the same time and space and performs preprocessing;

[0009] S2: Build an adversarial network, which includes a generator and a discriminator;

[0010] S3: Input the preprocessed lidar image and visible light image into the adversarial network for training, and train to the optimal state through multiple iterations;

[0011] S4: The output result of the generator of the optimal adversarial network is the final completed image of the lidar point cloud data.

[0012] Preferably, in step S1, the preprocessing includes downsampling the visible light image, registering and cropping the lidar image and the visible light image, and performing depth denoising and edge detection on the visible light image.

[0013] The purpose of downsampling is to reduce the size or complexity of the visible light image and retain the main features and details of the image. After downsampling the visible light image, a visible light image with typical feature samples is obtained.

[0014] Preferably, the registration process is:

[0015] Get the lidar image to be registered I r , with visible light images I v As a reference image; using the local transformation model to register the lidar image I r Perform registration processing; the local transformation model includes a predefined objective function, a similarity metric, and setting key parameters. During the registration process, template matching is performed by optimizing the predefined objective function, where the template is located on a dense regular grid to cover all possible local deformation areas; the objective function is expressed as a formula for the lowest seam line of the two images , where the optimal seam is defined as the set of registration points with the smallest structural difference, Indicates the structural difference value of the position, and Represents the visible light image and lidar image in The gray value at , Respectively represent the 3*3 Sobel operator in the horizontal and vertical directions for feature extraction, which is used to measure the changes in the local structure and the position of the edge in the image.

[0016] When determining the transformation model, the control points between the templates are extracted according to the similarity metric between the templates. Specifically, local templates are extracted from the two images respectively, and the similarity between the templates, such as the normalized correlation coefficient, is calculated. The best matching control point pair is determined according to the similarity. The key parameters include the window size and the maximum transformation amount. The setting of key parameters is used to adjust the template matching process to ensure the best registration result. The specific settings of the window size and the maximum transformation amount need to be adjusted according to the actual situation. For example, for a 1920*1080 image, the window size can be set to 64×64. Usually, a larger window and transformation range are set first, and the best empirical parameters are determined through experiments. These parameters will affect the accuracy and robustness of the registration.

[0017] The cropping process is as follows: multiple pairs of lidar and visible light images that have been downsampled and registered are selected and cut into corresponding images of the same size. In addition, in order to increase the data set, multiple pairs of corresponding images are cropped into small image pairs one by one to obtain multiple groups of corresponding images with doubled number.

[0018] Preferably, deep denoising is achieved using a deep denoising autoencoder;

[0019] The Sobel operator is used for edge extraction to obtain the edge information of each area in the image, that is, texture and contour information.

[0020] Preferably, the generator includes an input layer, a residual network 1, a residual network 2, an output layer 1 and an output layer 2 in sequence; wherein the input layer includes a convolution layer 1, which is composed of a convolution operation, a batch normalization and an activation function, and extracts and amplifies the features of the source image;

[0021] The residual network 1 includes a residual block 1, a transfer layer 1 and a skip connection layer 1. The residual block 1 consists of two layers of convolution operations, two layers of batch normalization and two activation functions, which are: convolution operation, batch normalization, activation function, convolution operation, batch normalization, activation function; the transfer layer 1 consists of convolution operation, batch normalization and activation function; the role of the residual network 1 is to improve the network learning ability, retain the residual information, and enable the deep learning network to have the ability of backpropagation.

[0022] Residual network 2 includes residual block 2, transfer layer 2 and skip connection layer 2. Residual block 2 consists of two layers of convolution operations, two layers of batch normalization and two activation functions, which are: convolution operation, batch normalization, activation function, convolution operation, batch normalization, activation function; transfer layer 2 consists of convolution operation, batch normalization and activation function; residual network 2 is connected in series with residual network 1 to further enhance the learning ability of the network, and can transfer and retain the features of the previous layer to further complete the features before decoding.

[0023] The output layer 1 includes the convolution layer 2, which consists of convolution operations, batch normalization, and activation functions. The output layer 1 is responsible for converting the data into an intermediate representation, which can be regarded as an expression in the latent space, capturing the main features of the input data and trying to map the input data to a higher-level feature representation, so that subsequent layers can more effectively generate realistic images.

[0024] The output layer 2 includes the convolution layer 3, which consists of a convolution operation and an activation function. The output layer 2 further processes the intermediate representation and converts it into the final generated image or data.

[0025] Preferably, the activation function of convolution layer 1 and convolution layer 2 is LReLU, and the activation function of convolution layer 3 is tanh. LReLU performs well in high-resolution image modeling, and the tanh activation function is used in the last convolution layer to ensure stable results.

[0026] The generator outputs the completed point cloud data image as follows: first, multiple groups of corresponding images are extracted using convolution layer one, and input into a residual block one and a jump connection layer one; secondly, residual block one is used to extract and retain features, and then a transfer layer one is used to further extract features; then, the preliminary features extracted by residual block one are superimposed with the input original image as the input of residual block two, while retaining the jump connection; then, the residual block and convolution operation are repeated once, and the feature information of the original image is processed and superimposed multiple times (the number is obtained through training); finally, the completed point cloud data image is output through the convolution layer.

[0027] The generator works as follows:

[0028] The convolution layer 1 of the input layer directly transforms the input lidar image into a channel feature map, performs feature extraction and encoding, gradually amplifies the features, and then inputs it into the residual network 1 and residual network 2 for feature extraction to retain more residual information. There are two jump connections inside the two residual networks to transfer and retain the features of the previous layer, and finally input it into the convolution layer of the two output layers to extract features layer by layer and reduce the channel dimension, and finally output a single-channel completed image.

[0029] The functions of each structure of the generator are as follows:

[0030] Convolutional layer of the input layer: performs dimensionality increase on the input two-channel source image and outputs a 64-channel feature map.

[0031] Residual block: Maintains the integrity of the input data, while the learned residual information helps to supplement the multi-scale features, paving the way for further feature extraction or dimensionality reduction. In addition, the addition of residual blocks makes network training easier, and the problem of gradient disappearance during back propagation is solved.

[0032] Transfer layer: transforms and integrates the features of the activated residual block output to facilitate superposition with the module input.

[0033] Skip connection layer: It can further promote the interactive fusion of the front-layer and back-layer features of the source image, obtain more useful features before dimensionality reduction, and lay the foundation for image decoding output.

[0034] Output layer: includes output layer 1 and output layer 2, including two convolutional layers, to achieve feature dimensionality reduction operation and transform the 64-channel feature map into a single-channel completed image.

[0035] Preferably, the discriminator includes convolution layer four, convolution layer five, convolution layer six, convolution layer seven, linear transformation layer and fully connected layer in sequence, and convolution layer four is composed of convolution operation and activation function, and its function is to extract low-level features of point cloud data, introduce nonlinearity and improve the expressive ability of the model; convolution layer five is composed of convolution operation, batch normalization and activation function, and its function is to extract higher-level features, among which batch normalization further stabilizes the learning process, and activation function increases nonlinear features; convolution layer six is ​​composed of convolution operation, batch normalization and activation function, and its function is to further extract complex features, batch normalization can stabilize the learning process, and activation function enhances nonlinear features; convolution layer seven is composed of convolution operation, batch normalization and activation function, and its function is to extract complex features again, batch normalization can stabilize the learning process, and activation function can enhance nonlinear features. Except for the first convolutional layer, all four convolutional layers have batch normalization operations to improve the convergence speed of the model; the linear transformation layer consists of linear expansion and series operations to form a long column of feature outputs of the feature map; the fully connected layer consists of a group of linear calculations and outputs a discrimination result.

[0036] In the discriminator, a convolutional layer is first used to extract initial features from the input complement image or visible light image; then, by continuously using multiple convolutional layers, the size of the feature map is gradually reduced and the number of channels of the feature map is increased; then, a fully connected layer is used to flatten the output feature map of the last convolutional layer into a one-dimensional vector, and then this one-dimensional vector is mapped to a single vector through a fully connected layer; finally, a unique possible discriminant value is output through linear multiplication.

[0037] In the discriminator, there are four convolutional layers. The first convolutional layer extracts features through a larger receptive field and performs dimensionality increase. The following three convolutional layers use lower convolution kernels for internal feature extraction and perform batch normalization of parameters to improve the convergence speed of the model.

[0038] Linear conversion layer: The discriminator takes a single-channel image as input and passes through four layers of convolution to output a small channel feature map. To facilitate the subsequent calculation of a single discriminant value output, each feature map containing multiple neurons is expanded into a linear form, and then all linear feature maps are connected in series to form a "long column" of feature output.

[0039] Fully connected layer: According to the number of feature points of linear transformation, the corresponding number weights are matched, the two are multiplied one by one and then added, and then the average value is obtained according to the total number of connection points, which is the final judgment result.

[0040] The weight adjustment of the discriminator of the present invention is also performed through the back propagation algorithm.

[0041] The goal of this loss function is to maximize the probability that the discriminator correctly judges the real data as true and the synthetic data as false. Therefore, through back propagation, the optimizer adjusts the parameters of the discriminator according to this loss function to achieve its optimization goal.

[0042] Preferably, in step S3, the preprocessed lidar image is input into the generator to generate a completed image, and the loss function of the generator is calculated. L G , and continuously adjust the weight value of the generator through back propagation λ ;

[0043] When the generator generates a completed image, it also generates a loss function. The adversarial loss function and the content loss function are combined to adjust the parameters of the generator, so that the adversarial loss function tends to a high value and the content loss function tends to a low value, thus completing the optimization of the generator.

[0044] The weight value of the generator λ The adjustment is mainly carried out through the back-propagation algorithm. The loss function of the generator consists of two parts: one is the adversarial loss between the generator and the discriminator, that is, the similarity loss between the generated image and the real image, and the other is the content loss between the completed image and the visible light image, that is, the loss of the generated image being judged as a real image by the discriminator. The weights of these two parts of the loss function are adjusted according to the specific situation to balance the training effect of the generator. During the training process, the loss function of the generator will be minimized. By continuously adjusting the parameters of the generator, the output image of the generator gradually approaches the real image, while minimizing the judgment error of the discriminator on the generated image, so that the output image of the generator is more realistic.

[0045] The preprocessed visible light image and the completed image obtained by the generator are input into the discriminator. The parameters of the discriminator are updated by calculating the loss function and performing back propagation. The discriminator is continuously optimized, and the edge information in the visible light image is added to the completed image. After multiple iterations, the network parameters of the generator and discriminator are continuously updated until they are stable.

[0046] When adding the edge information in the visible light image to the completed image, the edge information of the visible light image is first extracted by the sobel operator, and then the above visible light image is put into the discriminator. The completed image is in the generator. After adversarial training, the generator continuously generates fake samples for the completed image that may deceive the discriminator. In the generator network, there is a suitable network structure and loss function designed so that the generator can generate the texture and contour information in the visible light image in the discriminator as much as possible. In addition, the weight value and bias value of the generator network are adjusted through back propagation, which is a process to make the completed image have the texture and contour information of the visible light image. After multiple iterations, the generator and discriminator networks are continuously updated until they are stable. When the discriminator cannot distinguish whether the fake samples generated by the generator are true or false, it is considered stable. Specifically, when the loss function of the generator and the discriminator reaches a balance state and no longer fluctuates violently during the training process, it can be considered that the network has reached a stable performance state.

[0047] Preferably, in step S3, the loss function of the generator is L G for:

[0048] (1)

[0049] (2)

[0050] (3)

[0051] in, is the adversarial loss between the generator and the discriminator; To complement the content loss between the image and the visible light image; λ is the weight value of the generator, used for balancing and , which can be adjusted through cross-validation, and the initial choice λ =0.001, and then observe the quality of the point cloud data generated by the generator and the accuracy of the discriminator according to the experimental results, adjust and optimize, and fine-tune λ value to achieve the best generation effect and training stability.

[0052] N is the number of completed images; is the discriminant output of the discriminator for the completed image; express The completed image of c is the value of the fake data that the generator wants the discriminator to believe; H and W Represent the height and width of the input image respectively; is the completed image after each iteration; is the matrix Frobenius norm; ∇ is the gradient operator; in formula (3), The purpose of the first item is to preserve the depth information of the lidar image in the completed image. The purpose of the second term is to preserve the gradient information contained in the visible light image. To control the positive parameter of the trade-off between the two items, different Values ​​can be set from 0.01 to 10.0 in steps of 0.1 or 0.5. value, train the GAN model, and look at the quality of the generated results to choose the appropriate The value can be, for example, 0.01 to 0.1.

[0053] When the generator is trained, the losses of the generator and discriminator are calculated through forward propagation, and then the gradient of the loss function to the data interpolation weight is calculated through back propagation. Finally, the weights and bias values ​​of the network data interpolation are updated according to the gradient.

[0054] Preferably, in step S3, the loss function of the discriminator is L D for:

[0055] (4)

[0056] in, is the discriminant output of the discriminator for the visible light image; is the discriminant output of the completed image; the training of the discriminator is to update the discriminator parameters through optimization algorithms such as gradient descent , so that L D As small as possible to improve the discriminator's ability to distinguish between real samples and generated samples.

[0057] b is the label corresponding to the visible light image; a To complete the label corresponding to the image, b Approaching 1, a Approaching 0.

[0058] For any details not provided in the present invention, please refer to the prior art.

[0059] The beneficial effects of the present invention are:

[0060] 1. The present invention improves the quality of lidar data by using the texture contour information of visible light images and adding it to the completed lidar data, making the completed data more realistic and usable. There is currently no relevant domestic and foreign research on this work, which will help promote technological progress and application development in related fields.

[0061] 2. The image preprocessing of the present invention includes deep denoising, downsampling and other processing, which solves the problem that the resolution of domestic laser radar is lower than visible light. The algorithm can be used with input source images of any size and has strong network generalization ability.

[0062] 3. The present invention uses multiple groups of corresponding images to input into the generator, extracts features through convolutional layers and uses residual blocks and jump connections to retain features, obtains and retains as many visible light image sources as possible, so that the generator can better generate accurate completion data.

[0063] 4. Currently, the image generation problem based on generative adversarial networks (GANs) is mainly used to generate visible light images, which are usually characterized by a single pixel value. This invention sets the main generation object as lidar point cloud data, and the pixel value of the visible light image is used as the generation criterion, which can accurately complete the missing data of the lidar point cloud. This invention expands the application of GAN in different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0065] Figure 1 This is the overall structure diagram of the algorithm of the present invention;

[0066] Figure 2 A flowchart of specific implementation steps of the present invention;

[0067] Figure 3 This is a diagram of the network architecture model of the generator of the present invention;

[0068] Figure 4 This is a model diagram of the discriminator network architecture of the present invention;

[0069] Figure 5 Schematic diagram of the residual block structure of the present invention;

[0070] Figure 6 A visible light image in the image completion method of the present invention;

[0071] Figure 7 is a visible light image preprocessed by the present invention;

[0072] Figure 8 is the preprocessed lidar image;

[0073] Fig. 9 is the final completed image;

[0074] Fig.10 It is a schematic diagram of the processing effect of the prior art and the method of the present invention on the standard database. DETAILED DESCRIPTION

[0075] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the implementation of this specification, but are not limited to this. Anything not fully described in the present invention shall be based on the conventional technology in the art.

[0076] Example 1

[0077] A method for completing lidar point cloud data, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0078] S1: The dual-light sensor collects lidar images and visible light images in the same time and space and performs preprocessing;

[0079] S2: Build an adversarial network, which includes a generator and a discriminator;

[0080] S3: Input the preprocessed lidar image and visible light image into the adversarial network for training, and train to the optimal state through multiple iterations;

[0081] S4: The output result of the generator of the optimal adversarial network is the final completed image of the lidar point cloud data.

[0082] Example 2

[0083] A method for completing lidar point cloud data, as described in Example 1, except that in step S1, preprocessing includes downsampling the visible light image, aligning and cropping the lidar image and the visible light image, and performing depth denoising and edge detection on the visible light image.

[0084] The purpose of downsampling is to reduce the size or complexity of the visible light image and retain the main features and details of the image. After downsampling the visible light image, a visible light image with typical feature samples is obtained.

[0085] The registration process is:

[0086] Get the lidar image to be registered I r, with visible light images I v As a reference image; using the local transformation model to register the lidar image I r Perform registration processing; the local transformation model includes a predefined objective function, a similarity metric, and setting key parameters. During the registration process, template matching is performed by optimizing the predefined objective function, where the template is located on a dense regular grid to cover all possible local deformation areas; the objective function is expressed as a formula for the lowest seam line of the two images , where the optimal seam is defined as the set of registration points with the smallest structural difference, Indicates the structural difference value of the position, and Represents the visible light image and lidar image in The gray value at , Respectively represent the 3*3 Sobel operator in the horizontal and vertical directions for feature extraction, which is used to measure the changes in the local structure and the position of the edge in the image.

[0087] When determining the transformation model, the control points between the templates are extracted according to the similarity metric between the templates. Specifically, local templates are extracted from the two images respectively, and the similarity between the templates, such as the normalized correlation coefficient, is calculated. The best matching control point pair is determined according to the similarity. The key parameters include the window size and the maximum transformation amount. The setting of key parameters is used to adjust the template matching process to ensure the best registration result. The specific settings of the window size and the maximum transformation amount need to be adjusted according to the actual situation. For example, for a 1920*1080 image, the window size can be set to 64×64. Usually, a larger window and transformation range are set first, and the best empirical parameters are determined through experiments. These parameters will affect the accuracy and robustness of the registration.

[0088] The cropping process is as follows: multiple pairs of lidar and visible light images that have been downsampled and registered are selected and cut into corresponding images of the same size. In addition, in order to increase the data set, multiple pairs of corresponding images are cropped into small image pairs one by one to obtain multiple groups of corresponding images with doubled number.

[0089] Deep denoising is achieved using a deep denoising autoencoder;

[0090] The Sobel operator is used for edge extraction to obtain the edge information of each area in the image, that is, texture and contour information.

[0091] Figure 6 The figure shows a visible light image. The preprocessed visible light image is as follows Figure 7 As shown;

[0092] The preprocessed lidar image is as follows Figure 8 shown.

[0093] Example 3

[0094] A method for completing laser radar point cloud data, as described in Example 2, except that Figure 3 As shown, the generator includes an input layer, a residual network 1, a residual network 2, an output layer 1 and an output layer 2 in sequence; wherein the input layer includes a convolutional layer 1, which is composed of a convolution operation, a batch normalization and an activation function, and extracts and amplifies the features of the source image;

[0095] The residual network 1 includes a residual block 1, a transfer layer 1 and a skip connection layer 1. The residual block 1 consists of two layers of convolution operations, two layers of batch normalization and two activation functions, which are: convolution operation, batch normalization, activation function, convolution operation, batch normalization, activation function; the transfer layer 1 consists of convolution operation, batch normalization and activation function; the role of the residual network 1 is to improve the network learning ability, retain the residual information, and enable the deep learning network to have the ability of backpropagation.

[0096] Residual network 2 includes residual block 2, transfer layer 2 and skip connection layer 2. Residual block 2 consists of two layers of convolution operations, two layers of batch normalization and two activation functions, which are: convolution operation, batch normalization, activation function, convolution operation, batch normalization, activation function; transfer layer 2 consists of convolution operation, batch normalization and activation function; residual network 2 is connected in series with residual network 1 to further enhance the learning ability of the network, and can transfer and retain the features of the previous layer to further complete the features before decoding.

[0097] The output layer 1 includes the convolution layer 2, which consists of convolution operations, batch normalization, and activation functions. The output layer 1 is responsible for converting the data into an intermediate representation, which can be regarded as an expression in the latent space, capturing the main features of the input data and trying to map the input data to a higher-level feature representation, so that subsequent layers can more effectively generate realistic images.

[0098] The output layer 2 includes the convolution layer 3, which consists of a convolution operation and an activation function. The output layer 2 further processes the intermediate representation and converts it into the final generated image or data.

[0099] Preferably, the activation function of convolution layer 1 and convolution layer 2 is LReLU, and the activation function of convolution layer 3 is tanh. LReLU performs well in high-resolution image modeling, and the tanh activation function is used in the last convolution layer to ensure stable results.

[0100] The generator outputs the completed point cloud data image as follows: first, multiple groups of corresponding images are extracted using convolution layer one, and input into a residual block one and a jump connection layer one; secondly, residual block one is used to extract and retain features, and then a transfer layer one is used to further extract features; then, the preliminary features extracted by residual block one are superimposed with the input original image as the input of residual block two, while retaining the jump connection; then, the residual block and convolution operation are repeated once, and the feature information of the original image is processed and superimposed multiple times (the number is obtained through training); finally, the completed point cloud data image is output through the convolution layer.

[0101] The generator works as follows:

[0102] The convolution layer 1 of the input layer directly transforms the input lidar image into a channel feature map, performs feature extraction and encoding, gradually amplifies the features, and then inputs it into the residual network 1 and residual network 2 for feature extraction to retain more residual information. There are two jump connections inside the two residual networks to transfer and retain the features of the previous layer, and finally input it into the convolution layer of the two output layers to extract features layer by layer and reduce the channel dimension, and finally output a single-channel completed image.

[0103] The functions of each structure of the generator are as follows:

[0104] Convolutional layer of the input layer: performs dimensionality increase on the input two-channel source image and outputs a 64-channel feature map.

[0105] Residual block: maintains the integrity of the input data, and the learned residual information helps to supplement the multi-scale features, paving the way for further feature extraction or dimensionality reduction. Figure 5 As shown in Figure 2, the addition of residual blocks makes network training easier and solves the problem of gradient disappearance during back propagation.

[0106] Transfer layer: transforms and integrates the features of the activated residual block output to facilitate superposition with the module input.

[0107] Skip connection layer: It can further promote the interactive fusion of the front-layer and back-layer features of the source image, obtain more useful features before dimensionality reduction, and lay the foundation for image decoding output.

[0108] Output layer: includes output layer 1 and output layer 2, including two convolutional layers, to achieve feature dimensionality reduction operation and transform the 64-channel feature map into a single-channel completed image.

[0109] Figure 3In k3n64s1, k3n32s1, and k1n1s1, n represents the number of channels, k represents the convolution kernel size, and s represents the moving step, which respectively represent the parameters of the image at this step. Specifically, k3n64s1 indicates that the convolution operation of this layer uses 3×3 convolution for feature extraction, the number of convolution kernels is 64, and the moving step is 1.

[0110] Example 4

[0111] A method for completing laser radar point cloud data, as described in Example 3, except that Figure 4 As shown in the figure, the discriminator includes convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, linear transformation layer and fully connected layer in sequence. Convolution layer 4 is composed of convolution operation and activation function. Its function is to extract low-level features of point cloud data, introduce nonlinearity and improve the expressiveness of the model; convolution layer 5 is composed of convolution operation, batch normalization and activation function. Its function is to extract higher-level features. Batch normalization further stabilizes the learning process, and activation function increases nonlinear features; convolution layer 6 is composed of convolution operation, batch normalization and activation function. Its function is to further extract complex features. Batch normalization can stabilize the learning process, and activation function enhances nonlinear features; convolution layer 7 is composed of convolution operation, batch normalization and activation function. Its function is to extract complex features again. Batch normalization can stabilize the learning process, and activation function can enhance nonlinear features. Except for the first convolutional layer, all four convolutional layers have batch normalization operations to improve the convergence speed of the model; the linear transformation layer consists of linear expansion and series operations to form a long column of feature outputs of the feature map; the fully connected layer consists of a group of linear calculations and outputs a discrimination result.

[0112] In the discriminator, a convolutional layer is first used to extract initial features from the input complement image or visible light image; then, by continuously using multiple convolutional layers, the size of the feature map is gradually reduced and the number of channels of the feature map is increased; then, a fully connected layer is used to flatten the output feature map of the last convolutional layer into a one-dimensional vector, and then this one-dimensional vector is mapped to a single vector through a fully connected layer; finally, a unique possible discriminant value is output through linear multiplication.

[0113] In the discriminator, there are four convolutional layers. The first convolutional layer extracts features through a larger receptive field and performs dimensionality increase. The following three convolutional layers use lower convolution kernels for internal feature extraction and perform batch normalization of parameters to improve the convergence speed of the model.

[0114] Linear conversion layer: The discriminator takes a single-channel image as input and passes through four layers of convolution to output a small channel feature map. To facilitate the subsequent calculation of a single discriminant value output, each feature map containing multiple neurons is expanded into a linear form, and then all linear feature maps are connected in series to form a "long column" of feature output.

[0115] Fully connected layer: According to the number of feature points of linear transformation, the corresponding number weights are matched, the two are multiplied one by one and then added, and then the average value is obtained according to the total number of connection points, which is the final judgment result.

[0116] Figure 4 The k5n64s2, k3n128s2, k3n256s2, k3n512s2, and n1 in the figure represent the image parameters at this step, respectively, where n represents the number of channels, k represents the convolution kernel size, and s is the moving step size.

[0117] The weight adjustment of the discriminator of the present invention is also performed through the back propagation algorithm.

[0118] The goal of this loss function is to maximize the probability that the discriminator correctly judges the real data as true and the synthetic data as false. Therefore, through back propagation, the optimizer adjusts the parameters of the discriminator according to this loss function to achieve its optimization goal.

[0119] Example 5

[0120] A method for completing laser radar point cloud data, as described in Example 4, except that in step S3, the preprocessed laser radar image is input into the generator to generate a completed image, and the loss function of the generator is calculated L G , and continuously adjust the weight value of the generator through back propagation λ ;

[0121] When the generator generates a completed image, it also generates a loss function. The adversarial loss function and the content loss function are combined to adjust the parameters of the generator, so that the adversarial loss function tends to a high value and the content loss function tends to a low value, thus completing the optimization of the generator.

[0122] The weight value of the generator λ The adjustment is mainly carried out through the back-propagation algorithm. The loss function of the generator consists of two parts: one is the adversarial loss between the generator and the discriminator, that is, the similarity loss between the generated image and the real image, and the other is the content loss between the completed image and the visible light image, that is, the loss of the generated image being judged as a real image by the discriminator. The weights of these two parts of the loss function are adjusted according to the specific situation to balance the training effect of the generator. During the training process, the loss function of the generator will be minimized. By continuously adjusting the parameters of the generator, the output image of the generator gradually approaches the real image, while minimizing the judgment error of the discriminator on the generated image, so that the output image of the generator is more realistic.

[0123] The preprocessed visible light image and the completed image obtained by the generator are input into the discriminator. The parameters of the discriminator are updated by calculating the loss function and performing back propagation. The discriminator is continuously optimized, and the edge information in the visible light image is added to the completed image. After multiple iterations, the network parameters of the generator and discriminator are continuously updated until they are stable.

[0124] When adding the edge information in the visible light image to the completed image, the edge information of the visible light image is first extracted by the sobel operator, and then the above visible light image is put into the discriminator. The completed image is in the generator. After adversarial training, the generator continuously generates fake samples for the completed image that may deceive the discriminator. In the generator network, there is a suitable network structure and loss function designed so that the generator can generate the texture and contour information in the visible light image in the discriminator as much as possible. In addition, the weight value and bias value of the generator network are adjusted through back propagation, which is a process to make the completed image have the texture and contour information of the visible light image. After multiple iterations, the generator and discriminator networks are continuously updated until they are stable. When the discriminator cannot distinguish whether the fake samples generated by the generator are true or false, it is considered stable. Specifically, when the loss function of the generator and the discriminator reaches a balance state and no longer fluctuates violently during the training process, it can be considered that the network has reached a stable performance state.

[0125] Example 6

[0126] A method for completing laser radar point cloud data, as described in Example 5, except that the loss function of the generator L G for:

[0127] (1)

[0128] (2)

[0129] (3)

[0130] in, is the adversarial loss between the generator and the discriminator; To complement the content loss between the image and the visible light image; λ is the weight value of the generator, used for balancing and , which can be adjusted through cross-validation, and the initial choice λ =0.001, and then observe the quality of the point cloud data generated by the generator and the accuracy of the discriminator according to the experimental results, adjust and optimize, and fine-tune λ value to achieve the best generation effect and training stability.

[0131] N is the number of completed images; is the discriminant output of the discriminator for the completed image; express The completed image of c is the value of the fake data that the generator wants the discriminator to believe; H and W Represent the height and width of the input image respectively; is the completed image after each iteration; is the matrix Frobenius norm; ∇ is the gradient operator; in formula (3), The purpose of the first item is to preserve the depth information of the lidar image in the completed image. The purpose of the second term is to preserve the gradient information contained in the visible light image. To control the positive parameter of the trade-off between the two items, different Values ​​can be set from 0.01 to 10.0 in steps of 0.1 or 0.5. value, train the GAN model, and look at the quality of the generated results to choose the appropriate , for example, 0.01 to 0.1.

[0132] When the generator is trained, the losses of the generator and discriminator are calculated through forward propagation, and then the gradient of the loss function to the data interpolation weight is calculated through back propagation. Finally, the weights and bias values ​​of the network data interpolation are updated according to the gradient.

[0133] Loss function of the discriminator L D for:

[0134] (4)

[0135] in, is the discriminant output of the discriminator for the visible light image; is the discriminant output of the completed image; the training of the discriminator is to update the discriminator parameters through optimization algorithms such as gradient descent , so that L D As small as possible to improve the discriminator's ability to distinguish between real samples and generated samples.

[0136] b is the label corresponding to the visible light image; a To complete the label corresponding to the image, b Approaching 1, a Approaching 0.

[0137] The final completed image is as follows Fig. 9 shown.

[0138] like Fig.10 As shown, the four images of the standard database are processed by the method of the present invention, the traditional method PCN, and the traditional method AtlasNet, respectively. PCN represents a partial convolutional network, and AtlasNet represents an atlas network. Both are common image completion algorithm networks. The processing effects are shown in FIG. Fig.10 , and calculate the chamfer distance (CD), earth movement distance (EMD) and F score respectively. The results are shown in Table 1:

[0139] Table 1 Calculation results

[0140]

[0141] As can be seen from Table 1, the method of the present invention obtains lower chamfer distance (CD) and earth movement distance (EMD) scores and higher F score, which indicates that the present invention can generate more accurate and complete point clouds without introducing new noise.

[0142] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for completing laser radar point cloud data, characterized in that: The method comprises the following steps: S1: The dual-light sensor collects lidar images and visible light images in the same time and space and performs preprocessing; S2: Build an adversarial network, which includes a generator and a discriminator; S3: Input the preprocessed lidar image and visible light image into the adversarial network for training, and train to the optimal state through multiple iterations; S4: The output result of the generator trained to the optimal adversarial network is the final completed image of the lidar point cloud data; In step S2, the generator includes an input layer, a residual network 1, a residual network 2, an output layer 1 and an output layer 2 in sequence; wherein the input layer includes a convolutional layer 1, which is composed of a convolution operation, a batch normalization and an activation function; The residual network 1 includes a residual block 1, a transfer layer 1 and a jump connection layer 1. The residual block 1 consists of two layers of convolution operations, two layers of batch normalization and two activation functions; the transfer layer 1 consists of a convolution operation, batch normalization and an activation function; The residual network 2 includes residual block 2, transfer layer 2 and skip connection layer 2. The residual block 2 consists of two layers of convolution operations, two layers of batch normalization and two activation functions. The transfer layer 2 consists of convolution operations, batch normalization and activation functions. The output layer 1 includes the convolution layer 2, which consists of convolution operations, batch normalization, and activation functions; the output layer 1 is responsible for converting the data into an intermediate representation; The output layer 2 includes the convolution layer 3, which consists of a convolution operation and an activation function. The output layer 2 further processes the intermediate representation and converts it into the final generated image or data; In step S2, the discriminator includes convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, linear transformation layer and full connection layer in sequence. Convolution layer 4 consists of convolution operation and activation function; convolution layer 5 consists of convolution operation, batch normalization and activation function; convolution layer 6 consists of convolution operation, batch normalization and activation function; convolution layer 7 consists of convolution operation, batch normalization and activation function; linear transformation layer consists of linear expansion and series operation, and its function is to form a long column of feature output from feature map; full connection layer consists of a group of linear calculations, and outputs a discrimination result; In step S3, the preprocessed lidar image is input into the generator to generate a completed image and calculate the loss function of the generator L G , and continuously adjust the weight value of the generator through back propagation λ ; The preprocessed visible light image and the completed image obtained by the generator are input into the discriminator, and the discriminator is continuously optimized to add the edge information in the visible light image to the completed image. After multiple iterations, the network parameters of the generator and the discriminator are continuously updated until they are stable. In step S3, the loss function of the generator is L G for: （1）； （2）； （3）； in, is the adversarial loss between the generator and the discriminator; To complement the content loss between the image and the visible light image; λ is the weight value of the generator; N is the number of completed images; is the discriminant output of the discriminator for the completed image; express The completed image of c is the value of the fake data that the generator wants the discriminator to believe; H and W Represent the height and width of the input image respectively; is the completed image after each iteration; is the matrix Frobenius norm; ∇ is the gradient operator; is a positive parameter; I r is the lidar image to be registered; In step S3, the loss function of the discriminator is L D for: （4）； in, is the discriminant output of the discriminator for the visible light image; is the discriminant output of the discriminator for the completed image; b is the label corresponding to the visible light image; a is the label corresponding to the completed image.

2. The method for completing laser radar point cloud data according to claim 1, characterized in that: In step S1, the preprocessing includes downsampling the visible light image, registering and cropping the lidar image and the visible light image, and performing depth denoising and edge detection on the visible light image.

3. The method for completing laser radar point cloud data according to claim 2, characterized in that: The registration process is: Get the lidar image to be registered I r , with visible light images I v As a reference image; Using local transformation model to register lidar images I r Perform registration processing; the local transformation model includes a predefined objective function, a similarity metric, and setting key parameters. During the registration processing, template matching is performed by optimizing the predefined objective function, wherein the template is located on a dense regular grid to cover all possible local deformation areas; when determining the transformation model, control points are extracted based on the similarity metric between the templates; key parameters include window size and maximum transformation amount, which are used to determine the predefined objective function of the optimal template matching process.

4. The method for completing laser radar point cloud data according to claim 3, characterized in that: In step S1, deep denoising is implemented using a deep denoising autoencoder; The Sobel operator is used for edge extraction to obtain the edge information of each area in the image.

5. The method for completing laser radar point cloud data according to claim 4, characterized in that: The activation function of convolutional layer 1 and convolutional layer 2 is LReLU, and the activation function of convolutional layer 3 is tanh.

Images