An RGB-D camera depth module calibration method based on an adversarial neural network

Abstract

The present application relates to the technical field of deep learning image processing, and particularly relates to a kind of RGB-D camera depth module calibration method based on adversarial neural network, calculates depth camera distortion function and distortion correction function;According to the real depth value of scene and the calculated observation depth value, construct neural network dataset;Adversarial depth map correction neural network model is constructed;Training set data is sent into the generator network constructed and is trained;Using the trained generator to carry out depth surrounding correction prediction, and being sent into the discriminator network constructed and being discriminated;The parameters of generator network and the parameters of discriminator network are optimized;The network model obtained by training is carried out depth surrounding correction and depth camera calibration.An end-to-end depth map correction network based on conditional adversarial neural network is trained, which improves the accuracy of depth information.A parameter regression network is designed to further improve the calibration accuracy of the depth module.

Description

Technical Field

[0001] This invention relates to the field of deep learning image processing technology, and in particular to a method for calibrating the depth module of an RGB-D camera based on an adversarial neural network. Background Technology

[0002] An RGB-D camera is an integrated optical system that can capture optical images and distance / depth information of a target scene in real time. It typically consists of an RGB color camera and an (infrared) depth camera and is widely used in many fields such as 3D scene reconstruction, human-computer interaction, object recognition, and robot navigation. However, factors such as prolonged use, changes in external temperature or environmental conditions, or collisions can cause changes in the parameters of the RGB-D camera, thus affecting the accuracy of depth map acquisition. An effective way to solve this problem is to perform high-precision calibration of the RGB-D camera.

[0003] Traditional camera calibration methods mainly fall into two categories: The first category utilizes distinctive artificial references, such as a checkerboard pattern, and calculates calibration parameters using the camera's imaging model. This method is cumbersome and cannot meet the requirements for online, self-calibration. The second category attempts to obtain the correspondence between the 3D scene and the pixel plane through geometric features such as points, lines, and surfaces. While overcoming the limitations of the first category, it requires the scene to have structured features. Furthermore, these two methods primarily address the calibration of RGB color cameras and are difficult to directly apply to RGB-D depth camera calibration. Two indirect methods are typically used: one is to use a high-resolution RGB camera and indirectly obtain the relative extrinsic parameters between the depth camera and the calibration plane by calculating the relative extrinsic parameters between the depth camera and the RGB camera, and between the RGB camera and the calibration plane, thus deriving the depth value; the other is to directly obtain the depth values ​​between the calibration plane and the LiDAR using high-precision depth measurement equipment (such as LiDAR), and then calculate the high-precision depth value between the depth module and the calibration plane by combining the relative extrinsic parameters between the structured light depth module and the LiDAR. Both of these methods are cumbersome to operate, and the depth camera itself has limited calibration accuracy.

[0004] In recent years, deep learning technologies, represented by Convolutional Neural Networks (CNNs), have achieved great success in various computer vision tasks such as classification and segmentation. Some scholars have attempted to introduce them into the camera calibration process to achieve end-to-end camera self-calibration. Currently, research results on high-precision calibration of RGB depth cameras (modules) using CNNs are limited, and a key problem is the difficulty in obtaining sufficient sample datasets. In view of this, this invention proposes an RGB-D camera depth module calibration method based on adversarial neural networks. It introduces adversarial training into the RGB-D camera depth module calibration network construction process and generates a large sample dataset based on the statistical depth distortion patterns during RGB-D camera imaging, showing broad market application prospects. Summary of the Invention

[0005] The purpose of this invention is to provide a calibration method for RGB-D camera depth modules based on adversarial neural networks, which solves the problem of insufficient research results in the current research on high-precision calibration of RGB depth cameras (modules) using convolutional neural networks.

[0006] To achieve the above objectives, this invention provides a method for calibrating the depth module of an RGB-D camera based on an adversarial neural network, comprising the following steps:

[0007] S01: Obtain depth data captured by a real RGB-D camera and the true depth value of the scene;

[0008] S02: Based on the observed depth value and the actual depth value, analyze the trend of depth distortion of the depth camera, and calculate its distortion function and distortion correction function;

[0009] S03: Build a scene in the simulation software, load the depth renderer to obtain the true depth value of the scene, calculate the observed depth value according to the depth distortion function in S02, construct a neural network dataset based on the true depth value of the scene and the calculated observed depth value, use 70% of the data in this dataset as the training set, and use 30% of the data in this dataset as the test set.

[0010] S04: Construct an adversarial deep map correction neural network model, including a generator and a discriminator, and randomly initialize the parameters of the backbone network;

[0011] S05: Input the training set data from S03 into the generator network constructed in S04 for training, and save the training model parameters;

[0012] S06: Use the generator trained in S05 to perform depth perimeter correction prediction, feed the correction result and the currently used dataset into the discriminator network built in S04 for discrimination, and save the training model parameters.

[0013] S07: Calculate the loss functions of S05 and S06, optimize the parameters of the generator network and the discriminator network, and save the training model parameters.

[0014] S08: Use the test set data to test the correction effect of the generator network. If the accuracy requirements are met, the training is complete.

[0015] S09: Perform depth correction and depth camera calibration on the trained network model.

[0016] The steps include constructing an adversarial deep map correction neural network model, comprising a generator and a discriminator, and randomly initializing the parameters of the backbone network.

[0017] The generator's task is to progressively learn and correct the observed depth data to approximate the distribution of the true depth data, while the discriminator is used to evaluate the quality of the generator's output depth map.

[0018] The step of feeding the training set data from S03 into the generator network constructed in S04 for training and saving the trained model parameters further includes:

[0019] During training, the model continuously adjusts its parameters to minimize the difference between the observed depth data and the actual depth data.

[0020] The process includes using a generator trained in S05 to perform depth circumference correction prediction, feeding the correction result and the currently used dataset into a discriminator network built in S04 for discrimination, and saving the trained model parameters. The steps also include:

[0021] The discriminator network evaluates the quality of the generated depth map completion results and provides feedback to guide further optimization of the generator model.

[0022] The steps include calculating the loss functions for S05 and S06, optimizing the parameters of the generator network and the discriminator network, and saving the trained model parameters.

[0023] The loss function value of the model is obtained by comparing the difference between the generated depth map prediction results and the actual depth data, as well as the discriminator's evaluation of the generated results.

[0024] Using the backpropagation algorithm, the parameters of the generator network and the discriminator network are optimized based on the gradient information of the loss function, so that they are continuously adjusted to minimize the loss function.

[0025] Save the optimized training model parameters.

[0026] The process includes testing the generator network's correction effect using test set data; if the accuracy requirements are met, the training is complete. The steps also include:

[0027] The test set data is input into the trained generator network, the difference between the generated depth map and the real depth data is observed, and the correction effect is evaluated to see if it meets the preset accuracy requirements.

[0028] If the generator network performs well on the test set and reaches the required level of accuracy, the training task is considered complete.

[0029] If the model performs poorly or overfitting occurs, return to S05, adjust hyperparameters or data augmentation methods, and retrain.

[0030] This invention discloses an RGB-D camera depth module calibration method based on adversarial neural networks. The method obtains depth data captured by a real RGB-D camera and the true depth values ​​of the scene. Based on the observed and true depth values, it analyzes the trend of depth distortion in the depth camera and calculates its distortion function and distortion correction function. A scene is built in simulation software, a depth renderer is loaded to obtain the true depth values ​​of the scene, and the observed depth values ​​are calculated based on the depth distortion function. A neural network dataset is constructed based on the true depth values ​​of the scene and the calculated observed depth values. 70% of the data in this dataset is used as the training set, and 30% is used as the test set. An adversarial depth map correction mechanism is then constructed. A neural network model, including a generator and a discriminator, is constructed. The parameters of the backbone network are randomly initialized. Training data is fed into the constructed generator network for training, and the training model parameters are saved. The trained generator is used to perform depth perimeter correction prediction. The correction result and the currently used dataset are fed into the constructed discriminator network for discrimination, and the training model parameters are saved. The loss function is calculated, and the parameters of the generator and discriminator networks are optimized, and the training model parameters are saved. The correction effect of the generator network is tested using test data; if the accuracy requirements are met, training is complete. Depth perimeter correction and depth camera calibration are performed on the trained network model. An end-to-end depth map correction network based on a conditional adversarial neural network is trained, capable of correcting the original depth map, obtaining a distortion-free corrected depth map, and improving the accuracy of depth information. A parameter regression network is designed and combined with the correction network. Through joint training, the calibration parameters of the depth module are successfully regressed, further improving the calibration accuracy of the depth module. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0032] Figure 1 This is a flowchart illustrating the steps of the RGB-D camera depth module calibration method based on adversarial neural networks of the present invention.

[0033] Figure 2 This invention presents a statistical and analytical graph of the depth data after collecting the depth dataset.

[0034] Figure 3 The neural network structure for depth graph completion constructed in this invention is a generator network graph in an adversarial neural network.

[0035] Figure 4 This is a diagram of the discriminator structure in the adversarial neural network constructed in this invention.

[0036] Figure 5 This is a diagram of the neural network structure constructed by the present invention for depth sensor calibration.

[0037] Figure 6 This is a simplified illustration of the dataset created by this invention. Detailed Implementation

[0038] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0039] Please see Figures 1 to 6 ,in, Figure 1 This is a flowchart illustrating the steps of the RGB-D camera depth module calibration method based on adversarial neural networks of the present invention. Figure 2 This invention presents a statistical and analytical graph of the depth data after collecting the depth dataset. Figure 3 The neural network structure for depth graph completion constructed in this invention is a generator network graph in an adversarial neural network. Figure 4 This is a diagram of the discriminator structure in the adversarial neural network constructed in this invention. Figure 5 This is a diagram of the neural network structure constructed by the present invention for depth sensor calibration. Figure 6 This is a simplified illustration of the dataset generated by this invention. This invention provides a method for calibrating the depth module of an RGB-D camera based on an adversarial neural network, comprising the following steps:

[0040] S01: Obtain depth data captured by a real RGB-D camera and the true depth value of the scene;

[0041] S02: Based on the observed depth value and the actual depth value, analyze the trend of depth distortion of the depth camera, and calculate its distortion function and distortion correction function;

[0042] Specifically, depth images captured by RGB-D cameras are obtained from publicly available datasets. By statistically analyzing the trends between observed and true depth data, the mapping relationship from observed depth data to true depth data is calculated. Furthermore, the mapping function from true depth data to observed depth data is derived in reverse. This process helps reveal the complex correlations between depth data, providing an effective mathematical foundation and methodological support for the accurate calibration and standardization of depth modules.

[0043] S03: Build a scene in the simulation software, load the depth renderer to obtain the true depth value of the scene, calculate the observed depth value according to the depth distortion function in S02, construct a neural network dataset based on the true depth value of the scene and the calculated observed depth value, use 70% of the data in this dataset as the training set, and use 30% of the data in this dataset as the test set.

[0044] Specifically, a virtual environment with diverse scenarios is built in simulation software, and the real depth values ​​of the scenarios are loaded using a depth rendering module. Based on the mapping relationship from real data to observed data established in step S02, corresponding observed data is generated. Subsequently, the real data and observed data are combined to form the dataset required by the neural network, with 70% of the data used to train the neural network model and 30% used to test the model's accuracy and generalization ability. This process provides sufficient training samples for the deep learning algorithm, thereby effectively optimizing model performance and verifying its robustness under different scenarios.

[0045] S04: Construct an adversarial deep map correction neural network model, including a generator and a discriminator, and randomly initialize the parameters of the backbone network;

[0046] Specifically, an adversarial depth map correction neural network model is constructed, consisting of a generator and a discriminator. First, the parameters of the generator and discriminator are randomly initialized, and the network structure is built. The generator's task is to progressively learn how to correct the observed depth data to approximate the distribution of the real depth data, while the discriminator is used to evaluate the quality of the generator's output depth map. This step aims to establish the network framework, providing a foundation for subsequent adversarial training.

[0047] S05: Input the training set data from S03 into the generator network constructed in S04 for training, and save the training model parameters;

[0048] Specifically, the training dataset created in step S03 is input into the adversarial depth map correction neural network model built in step S04 for training. During training, the model continuously adjusts its parameters to minimize the difference between the observed depth data and the actual depth data, thereby achieving effective depth map correction. Once training is complete, the present invention saves the parameters of the trained model, providing a fully trained model foundation for subsequent depth map correction tasks.

[0049] S06: Use the generator trained in S05 to perform depth perimeter correction prediction, feed the correction result and the currently used dataset into the discriminator network built in S04 for discrimination, and save the training model parameters.

[0050] Specifically, the generator trained in step S05 is used for depth map correction prediction, and the generator model is used to predict and fill in regions lacking depth information. Then, the generated depth map prediction results and the currently used dataset are fed into the discriminator network constructed in step S04 for discrimination. The discriminator network evaluates the quality of the generated depth map completion results and provides feedback to guide further optimization of the generator model. Finally, this invention saves the parameters of the adversarially trained depth map correction model for future use and deployment in practical applications. This process aims to continuously improve the accuracy and robustness of the depth map completion model to meet the depth information requirements in real-world scenarios.

[0051] S07: Calculate the loss functions of S05 and S06, optimize the parameters of the generator network and the discriminator network, and save the training model parameters.

[0052] Specifically, the loss function in steps S05 and S06 is calculated by comparing the difference between the generated depth map prediction results and the actual depth data, as well as the discriminator's evaluation of the generated results, to obtain the model's loss function value. Subsequently, using the backpropagation algorithm, the parameters of the generator network and the discriminator network are optimized based on the gradient information of the loss function, continuously adjusting them to minimize the loss function, thereby improving the model's performance and generalization ability. Finally, the optimized training model parameters are saved for future use and deployment in practical applications. This process aims to continuously improve the accuracy and robustness of the depth map correction model through repeated parameter optimization, adapting to the depth information processing needs of various complex scenarios.

[0053] S08: Use the test set data to test the correction effect of the generator network. If the accuracy requirements are met, the training is complete.

[0054] Specifically, the depth map correction performance of the generator network is evaluated using test set data. By inputting the test set data into the trained generator network, the difference between the generated depth map and the real depth data can be observed, and the accuracy of the correction can be assessed. If the generator network performs well on the test set and reaches the required accuracy level, the training task can be considered complete, and the model has a high depth map correction capability. This evaluation process helps verify the effectiveness of the training structure and the reliability of the model, providing an important reference for subsequent deployment and application in practical applications. If the model performs poorly or overfitting occurs, it is necessary to return to step S05, adjust the hyperparameters or data augmentation methods, and retrain. This iterative process will continue until the model converges and reaches a satisfactory performance level.

[0055] S09: Perform depth correction and depth camera calibration on the trained network model.

[0056] Specifically, the trained network model is used for depth map correction and depth camera calibration. First, by inputting depth map data into the trained network model, the depth map can be corrected quickly and accurately, thereby improving the accuracy and reliability of depth information. Second, the trained model can be used to calibrate the depth camera, ensuring that the depth data acquired by the depth camera matches the distance information of the actual scene, thus improving the overall performance and stability of the depth perception system. This application process effectively combines deep learning technology with practical depth perception tasks, providing powerful support and solutions for various depth information processing-based application scenarios.

[0057] Among them, such as Figure 2 As shown in the figure, this invention illustrates the distortion patterns of the depth camera. By capturing depth maps of a plane at distances ranging from 1 to 5 meters using the depth camera and converting them into point clouds, the depth distortion trend along the z-axis was observed. The differences between the point cloud and the average depth at different z-planes were calculated, along with metrics such as root mean square error (RMSE), mean error (MAE), and relative error (REL). Analysis revealed that the structured light depth camera exhibits local distortion when measuring depth, and the average depth of the observed point cloud differs from the true value. Furthermore, this difference increases with increasing distance.

[0058] Since the observed depth value of a structured light depth camera can be corrected by a quadratic function, this invention first uses simulation software to obtain the true depth value of the simulated scene, and then calculates the inverse function of the correction function to obtain the observed depth value corresponding to the true depth value.

[0059] observe Figure 2 The observed depth map Z(u,v) and the true depth value Z * By observing the change in (u,v), we can see that as the observed depth value Z(u,v) increases, the true depth value Z... * (u,v) will also increase. That is, the true depth value Z * (u,v) is a monotonically increasing function relative to the observed depth map Z(u,v). Therefore, it satisfies the condition of an inverse function.

[0060] Considering the difference between the observed depth value Z(u,v) and the true depth value Z * The correction function for (u,v) can be expressed by the following formula:

[0061] z * =f u,v (z)≈a u,v z 2 +b u,v z+c u,v

[0062] Because within the range of the observed depth value Z(u,v) (the measurement range of the RGB-D depth camera), it corresponds to the true depth value Z. * There is a one-to-one correspondence between (u,v), therefore the corresponding inverse function can be obtained:

[0063]

[0064] The correction function has different values ​​for parameters a, b, and c for different pixel values. The above formula allows for a mapping from true depth to observed depth, thus enabling the acquisition of a dataset.

[0065] While the above formula can achieve a numerical mapping from true depth to observed depth, this mapping is only derived from the structured light depth module correction model. In practical applications, depth sensors face various hardware and environmental noises, including but not limited to system noise, ambient light noise, shadow noise, multipath interference noise, and artifact noise. Therefore, to simulate these situations, various noises still need to be added for simulation. This invention randomly adds 10% to 20% of Gaussian noise, Poisson noise, multiplicative noise, and salt-and-pepper noise to the observed depth map dataset to simulate the observation noise that may be encountered when capturing depth maps in real-world environments.

[0066] like Figure 3As shown, the generator structure used in this invention consists of a neural network composed of an encoder, a decoder, and jumpers. The encoder is responsible for feature extraction and dimensionality reduction of the input data, while the decoder performs upsampling to restore the original size and generate pixel-level output. Jumpers add connections to the mirror layer of the encoder and decoder stacks, bypassing the bottleneck layer and directly passing low-level information to the decoder. This generator network accepts a single-channel depth map, i.e., a simulated depth map with noise, as input. The generator network output is a corrected depth map that closely approximates the real depth map based on the given input. During adversarial training, the generator and the discriminator compete with each other, making it impossible for the discriminator to distinguish between the real depth map and the corrected depth map, thereby achieving a depth map correction effect close to reality. The encoder part of the generator consists entirely of convolution (Conv), batch normalization (BN), activation functions, and max pooling (MaxPooling). The convolution operation selects a kernel size of 3×3, a stride of 1, and padding of 1. When the feature map passes through this convolutional layer, its resolution remains unchanged. For activation functions, ReLU activation is used for all operations. For max pooling, a 2×2 kernel size is chosen, so that after one pooling operation, the length and width of the feature map are reduced to half. The decoder consists entirely of upsampling, concatenation, convolution, batch normalization, and activation functions. Upsampling is performed using deconvolution (DeConv) with a 2×2 kernel size, and blank areas are filled using linear interpolation. Concatenation connects the feature maps from the encoder (axis=3) to the feature map generated after deconvolution. The subsequent convolution, batch normalization, and activation function operations are consistent with those in the encoder section.

[0067] like Figure 4 As shown, the discriminator used in this invention is a Markov discriminator, which is a discriminator structure based on image patches. Its input is a pair of matching images. When the input is a true depth map and a raw depth map, the discriminator can classify it as true; conversely, when the input is a completed depth map and an observed depth map, it is classified as false. In calculating the judgment result, the input image is first divided into multiple image patches, and each image patch is judged individually. If the judgment result is biased towards 1, the image patch is considered true; otherwise, it is classified as false. In this way, an evaluation matrix can be calculated from a feature map, and the final value is calculated based on the average of the evaluation matrix. When processing a single image patch, it uses a structure of a fully convolutional neural network plus a fully connected neural network, ultimately outputting the probability of the image patch being true or false. The input of the discriminator is obtained by concatenating two depth maps, with a size x∈R. 512×512×2 Therefore, it is decomposed into 16 image patches x'∈R based on resolution.128 ×128×2 For each sub-image patch, it is fed into a discriminator to determine its authenticity. The discriminator consists of an encoder and a fully connected neural network. The encoder is composed of modules consisting of convolutions, regularization, neural networks, and max pooling. The structure of this module is the same as the convolutional layer structure used in the encoder part of the generator. For each image patch, eight convolutional blocks are used, ultimately compressing it into a feature vector x∈R. 1×1×1024 The feature vector is then flattened and fed into a fully connected neural network to predict the probability of being true or false. The fully connected neural network has 1024 input nodes, 1 output node, and a hidden layer with 512 nodes. The sigmoid activation function is then used to calculate the probability of being true or false. If the result is true, the value is set to 1 in the subsequent matrix; otherwise, it is set to 0. For 16 image patches, this method yields a 4×4 matrix, where the values ​​represent the true or false status of each patch. Finally, the average of this matrix is ​​calculated to obtain the overall image's true / false score.

[0068] like Figure 5As shown, the calibrator of this invention is based on the encoder-decoder structure in the generator branch, extracting feature maps calculated in each downsampling and upsampling process. Upsampling is performed on feature maps of different dimensions to restore each layer's feature map to the same resolution as the input image, i.e., the same size as the input image. The reason for restoring it to the input image size is that pixels in traditional correction models are independent of each other; therefore, when the size of the calibrator's feature map is the same as the resolution of the input image, point-to-point calibration can be achieved. Then, feature vectors at the same row and column indices of the upsampled feature map are extracted and input into a fully connected neural network, ultimately outputting three calibration parameters for that position. Then, by traversing all rows and columns, calibration parameters for all positions can be obtained. For each layer's feature map entering the encoder, an upsampling module is first used to restore its size. The number of upsampling operations differs for feature maps of different dimensions. For the feature maps of layers 1 and 7 in the generator, whose dimensions are [dimension missing], since the resolution is the same, no upsampling operation is performed; instead, the calibrator's features are calculated using a 3×3 convolution with regularization and a ReLU activation function layer. The generator's layers 2 and 6, with their feature dimensions, undergo upsampling via deconvolution to restore them to their initial dimensions, followed by non-linear operations (convolutional regularization activation function). Layers 3 and 5 are upsampled twice, while layer 4 is upsampled three times. Ultimately, the seven upsampled feature maps yield seven 512×512 feature maps with 64 channels. These seven feature maps are concatenated to obtain the calibration feature map. Feature vectors at the same row and column indices are extracted and input into a fully connected neural network to calculate the calibration coefficients at those positions. The fully connected network used in this invention has four hidden layers, 49 input nodes, three output nodes, and 13 hidden nodes each. The first four hidden layers use the ReLU activation function, and the last layer uses the Tanh activation function. This structure was obtained through experimental testing.

[0069] In the depth map correction CGAN network, this invention uses L1 loss to evaluate the difference between the corrected depth map predicted by the generator and the true depth map. Its objective function can be described by the following equation:

[0070]

[0071] The generator G aims to minimize the loss function, while the discriminator D aims to maximize the loss function.

[0072] In the calibrator network, the goal is to obtain more accurate calibration parameters. Therefore, the objective function combining global and local errors is:

[0073]

[0074] Where i represents the data sequence number used in this training, and n represents the data sequence number of all training data. This represents the true value of this training session. This represents the output of the generator during this training process.

[0075] Therefore, the total loss function of the calibration adversarial network is the sum of the loss function of the calibrator and the loss function of the adversarial network, and the final result is as follows:

[0076]

[0077] An end-to-end depth map correction network based on conditional adversarial neural networks was trained, which can correct the original depth map, obtain a distortion-free corrected depth map, and improve the accuracy of depth information.

[0078] A parametric regression network was designed and combined with a correction network. Through joint training, the calibration parameters of the depth module were successfully regressed, further improving the calibration accuracy of the depth module.

[0079] Example 1:

[0080] The dataset used in this invention was obtained through the simulation software GAZEBO, a powerful software platform widely used in robot simulation that provides rich and diverse scene representation capabilities. Combined with the Intel D435I camera model, multi-view images, including depth maps, color images, and infrared images, can be acquired. Simultaneously, the observed depth map is calculated based on the rendered real depth map. Based on this, this invention aims to construct realistic scenes in the GAZEBO simulation environment and use the D435I camera model for image acquisition, subsequently organizing these images into a comprehensive dataset. All operations are performed in a simulation environment, thus easily acquiring both observational and real data, facilitating dataset creation. The resulting dataset has high accuracy and can well reflect the effect and accuracy of network training. In this invention, 100,000 depth maps were collected at different angles and heights ranging from 0.5m to 6m from the target, with 70,000 used as the training set and 30,000 used as the test set. Figure 6 A brief overview of the dataset created for this invention:

[0081] This invention uses common evaluation metrics to assess the calibration results and also compares the correction effects of the neural network. The evaluation metrics used in this invention include RMSE, MAE, and REL. The descriptions of each evaluation metric are as follows:

[0082] RMSE is the root mean square error between the corrected depth map and the true depth map, and its formula is as follows:

[0083]

[0084] In the formula, T represents the true depth map; P represents the depth map predicted by the generator, i.e., the corrected depth map; p represents the depth value of a pixel; and roi represents all pixels in the completed region.

[0085] REL stands for Relative Error, which is the ratio of the absolute error to the true depth map. Its formula is:

[0086]

[0087] MAE is the mean absolute error, and its formula is:

[0088]

[0089] Different backbone networks play a crucial role in the training results of neural networks. A suitable backbone network should possess strong feature extraction capabilities, good contextual understanding capabilities, and strike an appropriate trade-off between model complexity and computational efficiency. Therefore, this invention first uses a common convolutional neural network architecture to test its ability to correct depth maps within a conditional adversarial neural network structure. This invention tested three networks: UNet, ResNet, and DenseNet. These three networks were chosen because each has its strengths. UNet can extract rich semantic information from images, thus better understanding the image content. ResNet focuses on solving the gradient vanishing problem in neural networks to learn at higher dimensions. DenseNet, by introducing dense connections, allows the network to contribute information across multiple feature layers. This invention trains these three networks separately until the loss function of the neural network converges. The following metrics evaluate their performance as correction networks using these three networks only for generator training. The experimental results of this invention are shown in the table below. As can be seen from the table, UNet performs better in all three metrics. Therefore, this invention selects the UNet network as the backbone network of the CGAN network.

[0090]

[0091] As explained earlier, to achieve point-to-point correction, the resolution of the feature map in the calibrator network should be the same as the resolution of the input image. However, different numbers of channels in the feature map still affect the calibration results. Furthermore, the number of layers and nodes in the fully connected network also affects the accuracy of the calibration coefficient regression. Therefore, this invention explores different combinations to determine a calibrator structure that achieves high-precision calibration without consuming excessive computational resources and without overfitting.

[0092] To improve experimental efficiency, this invention first tests the impact of different numbers of hidden layers on the calibration results. This is because, compared to the number of hidden layer nodes, the number of hidden layers has a greater impact on the representational power and performance of the neural network, and also a more significant impact on computational cost and the risk of overfitting. This invention first fixes the number of channels in the calibrator feature map to 1, then fixes the number of hidden layer nodes to 7, and then tests the impact of different numbers of hidden layers on the calibration results. Tests were conducted with 1, 2, 3, 4, and 5 layers respectively.

[0093]

[0094]

[0095] As can be seen, with the increase of the number of hidden layers, the three indicators finally reach their lowest values ​​at the 4th layer. Therefore, this invention selects 4 hidden layers as the final number of hidden layers for the calibrator. After confirming that the number of hidden layers is 4, this invention continues to test the effect of the number of nodes on the calibration results. Based on previous experience, the number of hidden layer nodes is chosen to be between 1 and 2 times that of the input nodes. Therefore, the number of hidden layer nodes tested ranges from 7 to 14.

[0096]

[0097] Observing the experimental data, it can be found that a number of nodes of 13 achieves better results; therefore, a hidden layer node count of 13 is chosen. After confirming a node count of 13 and a hidden layer count of 4, this invention continues to test the impact of the number of channels in the calibrator feature map on the calibration results. This invention tested channel counts from 1 to 8, and the experimental results are shown below:

[0098]

[0099]

[0100] Therefore, the fully connected neural network structure used in this invention comprises 49 input nodes and 4 hidden layers, with 13 nodes in each hidden layer. The feature map of the calibrator is based on the feature map of the generator, and after upsampling to obtain 7 channels, it is then input into the fully connected neural network.

[0101] To test the network's performance on real data, this invention tested its calibration effect on the BIDCD dataset.

[0102]

[0103]

[0104] Observing the experimental data, regardless of whether it's the correction network branch or the calibration network branch, the training effect on the real dataset is weaker than the training effect of a mixture of the simulation dataset and the real dataset. In the correction network branch, the network cannot effectively correct the depth values ​​in the real data. This may be because there are errors in calculating the true depth values ​​in the real data, thus interfering with the network training, while the simulation dataset does not have this problem.

[0105] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A method for calibrating the depth module of an RGB-D camera based on an adversarial neural network, characterized in that, Includes the following steps: S01: Obtain depth data captured by a real RGB-D camera and the true depth value of the scene; S02: Based on the observed depth value and the actual depth value, analyze the trend of depth distortion of the depth camera, and calculate its distortion function and distortion correction function; S03: Build a scene in the simulation software, load the depth renderer to obtain the true depth value of the scene, calculate the observed depth value according to the depth distortion function in S02, construct a neural network dataset based on the true depth value of the scene and the calculated observed depth value, and use 70% of the data in the dataset as the training set and 30% of the data in the dataset as the test set. S04: Construct an adversarial deep map correction neural network model, including a generator and a discriminator, and randomly initialize the parameters of the backbone network; S05: Input the training set data from S03 into the generator network constructed in S04 for training, and save the training model parameters. S06: Use the generator trained in S05 to perform depth perimeter correction prediction, feed the correction result and the currently used dataset into the discriminator network built in S04 for discrimination, and save the training model parameters. S07: Calculate the loss functions of S05 and S06, optimize the parameters of the generator network and the discriminator network, and save the training model parameters. S08: Use the test set data to test the correction effect of the generator network. If the accuracy requirements are met, the training is complete. S09: Perform depth bounding correction and depth camera calibration on the trained network model. Specifically, depth camera calibration includes: Based on the encoder-decoder structure in the generator branch, feature maps calculated in each downsampling and upsampling process are extracted; feature maps of different dimensions are upsampled to restore each layer of feature maps to the same resolution as the input image; feature vectors at the same row and column index positions of the upsampled feature maps are extracted, and the feature vectors are input into a fully connected neural network to finally output three calibration parameters at the positions; all rows and all columns are traversed to obtain calibration parameters at all positions.

2. The RGB-D camera depth module calibration method based on adversarial neural networks as described in claim 1, characterized in that, Constructing an adversarial deep map correction neural network model, including a generator and a discriminator, and randomly initializing the parameters of the backbone network, the steps further include: The generator's task is to progressively learn and correct the observed depth data to approximate the distribution of the true depth data, while the discriminator is used to evaluate the quality of the generator's output depth map.

3. The RGB-D camera depth module calibration method based on adversarial neural networks as described in claim 2, characterized in that, The training set data in S03 is fed into the generator network constructed in S04 for training, and the training model parameters are saved. The steps also include: During training, the model continuously adjusts its parameters to minimize the difference between the observed depth data and the actual depth data.

4. The RGB-D camera depth module calibration method based on adversarial neural networks as described in claim 3, characterized in that, The generator trained in S05 is used to perform depth perimeter correction prediction. The correction result and the currently used dataset are fed into the discriminator network built in S04 for discrimination, and the training model parameters are saved. The steps also include: The discriminator network evaluates the quality of the generated depth map completion results and provides feedback to guide further optimization of the generator model.

5. The RGB-D camera depth module calibration method based on adversarial neural networks as described in claim 4, characterized in that, The steps include calculating the loss functions for S05 and S06, optimizing the parameters of the generator network and the discriminator network, and saving the trained model parameters. The loss function value of the model is obtained by comparing the difference between the generated depth map prediction results and the actual depth data, as well as the discriminator's evaluation of the generated results. Using the backpropagation algorithm, the parameters of the generator network and the discriminator network are optimized based on the gradient information of the loss function, so that they are continuously adjusted to minimize the loss function. Save the optimized training model parameters.

6. The RGB-D camera depth module calibration method based on adversarial neural networks as described in claim 5, characterized in that, The generator network's correction effect is tested using test set data. If the accuracy requirements are met, the training is complete. The steps also include: The test set data is input into the trained generator network, the difference between the generated depth map and the real depth data is observed, and the correction effect is evaluated to see if it meets the preset accuracy requirements. If the generator network performs well on the test set and reaches the required level of accuracy, the training task is considered complete. If the model performs poorly or overfitting occurs, return to S05, adjust hyperparameters or data augmentation methods, and retrain.

Images