A camera calibration and correction method based on multi-neural network

Abstract

The present invention discloses a method for obtaining the internal and external parameters of a camera based on a neural network optimized using a BP neural network for linear parameters and an ant colony algorithm with strong nonlinear fitting capability for nonlinear parameters. The outputs of the BP and GA-BP neural networks are used as the input of a convolutional neural network AlexNet. AlexNet adopts a dropout strategy and an activation function (ReLU) to alleviate the overfitting problem. The method can autonomously learn camera calibration and distorted image correction in a variety of scenarios and complex environments. This method not only improves the simplicity and applicability of the camera calibration method, but also improves the accuracy and reliability of the camera calibration to meet the needs of camera calibration.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a camera calibration method based on multiple neural networks. Background Art

[0002] With the rapid development of science and technology, people are beginning to use machine vision to obtain corresponding three-dimensional object information from two-dimensional images, thereby better understanding the world. However, due to factors such as camera production and assembly processes, the captured images may have varying degrees of distortion, resulting in significant errors in the information extracted from the images. Therefore, camera calibration and distortion correction technology are essential in many areas of machine vision.

[0003] In order to obtain accurate image information, the current camera calibration is mainly divided into the following methods:

[0004] 1. Traditional calibration method: This method requires the use of a precisely manufactured geometric calibration plate and the use of relevant calibration algorithms to obtain the internal and external parameters of the camera. This method is not suitable for complex environments, and the calibration steps are cumbersome and the calibration results are unstable.

[0005] 2. Camera self-calibration method: This method can calibrate the camera online and is more flexible than traditional methods, but its robustness is particularly poor in some environments.

[0006] 3. Calibration method based on active vision: This method calibrates the camera by moving the camera or calibration object relative to each other on a specially made calibration platform. Compared with the first two methods, this method has higher accuracy, but the experimental conditions are special and demanding, the cost is high, and it is difficult to implement.

[0007] Convolutional neural network (CNN) is an important method for computer vision tasks. Existing convolutional neural networks most commonly use square convolution kernels, or use convolution with holes (called atrous convolution) to increase the receptive area (also known as the receptive field) of neurons. In order to improve the computing speed while maintaining a large receptive field, separable convolution kernels in vector form (1×k and k×1, where k is the convolution kernel size) have also been proposed and used. However, as an important method for visual tasks, convolutional neural networks have not shown the expected recognition ability on fisheye images with large distortion. Currently, convolutional neural networks have been widely used in image distortion correction in various fields, with excellent correction accuracy.

[0008] Based on the defects of the above existing technologies such as complex calibration steps, poor robustness, harsh calibration conditions, high cost, and difficulty in implementation, there is an urgent need to provide a camera calibration method with wide applicability, simple operation, and low cost. Summary of the Invention

[0009] The present invention aims to provide a camera calibration method based on multiple neural networks, which greatly improves the operational simplicity, applicability, and accuracy of the camera calibration method and better corrects camera distortion. To achieve the above objectives, the present invention provides the following technical solutions:

[0010] A camera calibration method based on multiple neural networks, the specific implementation steps include:

[0011] S1. Obtain the pixel coordinates (u, v) on the two-dimensional image through a computer, and then determine the corresponding point in the world coordinate system (x, y, z);

[0012] S2. The obtained pixel coordinates (u, v) and the corresponding world coordinate points (x, y, z) are respectively input into the BP neural network and the GA-BP neural network as inputs, and the BP neural network and the GA-BP neural network optimize the calculation to obtain the camera calibration parameters;

[0013] S3. The obtained camera calibration parameters and the obtained distorted image are used as the input of the convolutional neural network AlexNet, and AlexNet adopts the dropout strategy and activation function (ReLU) function to alleviate the overfitting problem, so as to better achieve the correction of the distorted image.

[0014] Specifically, the workflow of the GA-BP neural network in step S2 mainly includes the following steps:

[0015] S21. Evaluate the fitness of the individual corresponding to each chromosome;

[0016] S22. Following the principle that the higher the fitness, the greater the selection probability, two individuals are selected from the population as the father and mother;

[0017] S23, extract chromosomes from both parents, perform crossover, and produce offspring;

[0018] S24, mutating the chromosomes of the offspring;

[0019] S25. Repeat steps S22, S23, and S24 until a new population is generated as the camera calibration parameters.

[0020] Specifically, the convolutional neural network AlexNet in step S3 includes two convolution kernels, six pooling layers, one fully connected layer, and one softmax layer. The two convolution kernels are set to 11×11 with a stride of 4 and 5×5 with a stride of 1, respectively. The pooling layers all have a window size of 3×3, with a stride of 2 or 1. The fully connected layers consist of two fully connected layers with dropout.

[0021] Furthermore, the convolutional neural network AlexNet parameter setting mainly includes the following steps:

[0022] Hyperparameter settings: The model was trained using stochastic gradient descent, batch size = 128, momentum = 0.9, weight decay = 0.0005 (used as a regularizer to reduce the training error of the model), and learning rate = 0.01.

[0023] Initialization settings: ① Initialize the weights of each layer from a Gaussian distribution with standard deviation 0.01 and mean 0; ② Initialize the neuron biases of the second, fourth, and fifth convolutional layers and the fully connected hidden layer to a constant of 1 (this initialization accelerates the learning of the network in the early stages by providing positive inputs to the ReLU); ③ Initialize the neuron biases of the remaining layers to a constant of 0.

[0024] Learning rate settings: All layers are equal and are manually adjusted during training. The heuristic method followed is to divide the learning rate by 10 when the error rate of the validation set no longer improves with the current learning rate. The learning rate is initialized to 0.01 and reduced three times before termination.

[0025] Technical effects and advantages of the present invention:

[0026] The method provided by the present invention obtains the internal and external parameters of the camera based on a neural network optimized using a BP neural network for linear parameters and an ant colony algorithm with strong nonlinear fitting ability for nonlinear parameters. The outputs of the BP and GA-BP neural networks are used as the input of the convolutional neural network AlexNet. AlexNet adopts a dropout strategy and an activation function (ReLU) function to alleviate the overfitting problem. It can autonomously learn camera calibration and distorted image correction in a variety of scenarios and complex environments. It not only improves the simplicity and applicability of the camera calibration method, but also improves the accuracy and reliability of the camera calibration to meet the needs of camera calibration.

[0027] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of the camera calibration method provided by the present invention;

[0029] Figure 2 This is the workflow diagram of BP neural network;

[0030] Figure 3 This is the workflow diagram of the convolutional neural network AlexNet.

[0031] In the figure, (u, v) is the pixel coordinate in the pixel coordinate system, (x, y, z) is the world coordinate in the world coordinate system, Network1 is the BP neural network, Network2 is the GA-BP neural network, Network3 is the convolutional neural network AlexNet, Distorted image is the distorted image, and Corrected image is the corrected image. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] To address the shortcomings of the existing technology, the present invention discloses a camera calibration method based on multiple neural networks. By using BP neural network and GA-BP neural network to obtain the internal and external parameters of the camera, and using the outputs of BP and GA-BP neural networks as the input of convolutional neural network AlexNet, AlexNet adopts dropout strategy and activation function (ReLU) function to alleviate the overfitting problem. It can autonomously learn camera calibration and distorted image correction in a variety of scenarios and complex environments.

[0034] like Figure 1 As shown in FIG, a camera calibration method based on multiple neural networks is implemented, and the specific implementation steps include:

[0035] S1. Obtain the pixel coordinates (u, v) on the two-dimensional image through a computer, and then determine the corresponding point in the world coordinate system (x, y, z);

[0036] S2, send the obtained pixel coordinates (u, v) and the corresponding world coordinate points (x, y, z) to Network 1 and Network 2 as input respectively, and obtain the camera calibration parameters from Network 1 and Network 2;

[0037] S3: The obtained camera calibration parameters and the obtained distorted image are used as inputs of Network3 to obtain the corrected image output.

[0038] Specifically, such as Figure 2 As shown, Network 1 and Network 2 in step S2 are BP neural networks respectively, and Network 2 is a GA-BP neural network.

[0039] BP (back propagation) neural network is a commonly used algorithm mathematical model. Its working process is mainly divided into two stages. The first stage is the forward propagation of the signal, from the input layer through the hidden layer, and finally to the output layer; the second stage is the reverse propagation of the error, from the output layer to the hidden layer.

[0040] Finally, at the input layer, the weights and biases from the hidden layer to the output layer, and from the input layer to the hidden layer, are adjusted sequentially. Specifically, error output calculation proceeds from input to output, while weights and thresholds are adjusted from output to input. During forward propagation, the input signal acts on the output node through the hidden layer, undergoing a nonlinear transformation to generate an output signal. If the actual output does not match the expected output, the error proceeds to the backward propagation process. Error backpropagation involves propagating the output error back through the hidden layer to the input layer layer by layer, distributing the error to all units in each layer. The error signals obtained from each layer serve as the basis for adjusting the weights of each unit. By adjusting the connection strengths between input nodes and hidden layer nodes, the connection strengths between hidden layer nodes and output nodes, and the thresholds, the error is reduced along the gradient. After repeated learning and training, the network parameters (weights and thresholds) corresponding to the minimum error are determined, and training ceases. At this point, the trained neural network can independently process the nonlinearly transformed information with the minimum output error for similar sample inputs.

[0041] Because the BP algorithm converges slowly during network training and is prone to falling into local minima, the genetic optimization algorithm can initialize the system to a set of random solutions and iteratively search for the optimal value, often used to find the global optimal solution. Therefore, this paper incorporates a genetic algorithm (i.e., a GA-BP neural network) into the BP neural network to select the optimal parameters for camera calibration.

[0042] The purpose of adding genetic algorithms for optimization is to optimize the network parameter configuration and minimize the test set prediction error. This can usually be achieved in the following ways:

[0043] (1) The initial values ​​of the network parameters can be changed, and then the back propagation of the BP neural network can be used to find the optimal parameters of the entire network;

[0044] (2) The entire network parameters can be optimized directly using genetic algorithms without introducing BP to obtain the best network parameter settings;

[0045] (3) You can change the initial values ​​of the parameters, use genetic algorithms to optimize better initial parameters in forward propagation, and then use back propagation to optimize the parameters.

[0046] The workflow of the GA-BP neural network provided by the present invention mainly includes the following steps:

[0047] S21. Evaluate the fitness of the individual corresponding to each chromosome;

[0048] S22. Following the principle that the higher the fitness, the greater the selection probability, two individuals are selected from the population as the father and mother;

[0049] S23, extract chromosomes from both parents, perform crossover, and produce offspring;

[0050] S24, mutating the chromosomes of the offspring;

[0051] S25. Repeat steps S22, S23, and S24 until a new population is generated as the camera calibration parameters.

[0052] The Network3 in step S3 is specifically the convolutional neural network AlexNet. In 2012, the AlexNet network proposed by Alex et al. won the ImageNet competition by a wide margin, garnering widespread attention again for convolutional neural networks and even deep learning. AlexNet has the following key features:

[0053] (1) AlexNet deepens the network structure based on LeNet and learns richer and higher-dimensional image features. Features of AlexNet:

[0054] (2) Deeper network structure;

[0055] (3) Use stacked convolutional layers, i.e. convolutional layer + convolutional layer + pooling layer to extract image features

[0056] (4) Use Dropout to suppress overfitting;

[0057] (5) Use data augmentation to suppress overfitting;

[0058] (6) Use Relu to replace the previous sigmoid as the activation function;

[0059] (7) Multi-GPU training.

[0060] like Figure 3As shown in the figure, the convolutional neural network AlexNet provided by the present invention includes 2 convolution kernels, 6 pooling layers, 1 fully connected layer, and 1 softmax layer.

[0061] The convolution kernel is the core of the convolutional neural network, and the size of the convolution kernel is set according to actual needs, such as Figure 3 As shown, the two convolution kernels of the present invention are set to be 11×11 in size, 4 in stride, and 5×5 in size, 1 in stride; the window size of the pooling layer is 3×3, and the stride is 2 or 1; the fully connected layer is 2 fully connected layers with dropout.

[0062] The convolutional neural network AlexNet parameter setting mainly includes the following steps:

[0063] Hyperparameter settings: The model was trained using stochastic gradient descent, batch size = 128, momentum = 0.9, weight decay = 0.0005 (used as a regularizer to reduce the training error of the model), and learning rate = 0.01.

[0064] Initialization settings: ① Initialize the weights of each layer from a Gaussian distribution with standard deviation 0.01 and mean 0; ② Initialize the neuron biases of the second, fourth, and fifth convolutional layers and the fully connected hidden layer to a constant of 1 (this initialization accelerates the learning of the network in the early stages by providing positive inputs to the ReLU); ③ Initialize the neuron biases of the remaining layers to a constant of 0.

[0065] Learning rate settings: All layers are equal and are manually adjusted during training. The heuristic method followed is to divide the learning rate by 10 when the error rate of the validation set no longer improves with the current learning rate. The learning rate is initialized to 0.01 and reduced three times before termination.

[0066] Based on the above method, the present invention obtains the internal and external parameters of the camera by using a BP neural network for linear parameters and a GA-BP neural network for nonlinear parameters. The outputs of the BP and GA-BP neural networks are then used as the input of a convolutional neural network AlexNet. AlexNet adopts a dropout strategy and an activation function (ReLU) function to alleviate the overfitting problem, thereby better achieving the correction of distorted images.

[0067] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A camera calibration method based on multiple neural networks, characterized in that: The specific implementation steps of the method include: S1. Obtain the pixel coordinates (u, v) on the two-dimensional image through a computer, and then determine the corresponding point in the world coordinate system (x, y, z); S2, send the obtained pixel coordinates (u, v) and the corresponding world coordinate points (x, y, z) to Network1 and Network2 as input respectively, and obtain the camera calibration parameters from Network1 and Network2; S3, taking the obtained camera calibration parameters and the obtained distorted image as input to Network3, thereby obtaining the corrected image output; Wherein, Network1 and Network2 in step S2 are BP neural networks respectively, and Network2 is a GA-BP neural network; The Network3 of step S3 is specifically a convolutional neural network AlexNet; The convolutional neural network AlexNet includes 2 convolution kernels, 6 pooling layers, 1 fully connected layer, and 1 softmax layer.

2. The camera calibration method according to claim 1, wherein: A genetic algorithm is added to the BP neural network as an option for optimizing camera calibration parameters. The main implementation steps of the genetic algorithm are as follows: S21. Evaluate the fitness of the individual corresponding to each chromosome; S22. Following the principle that the higher the fitness, the greater the selection probability, two individuals are selected from the population as the father and mother; S23, extract chromosomes from both parents, perform crossover, and produce offspring; S24, mutating the chromosomes of the offspring; S25. Repeat steps S22, S23, and S24 until a new population is generated as the camera calibration parameters.

3. The camera calibration method according to claim 1, wherein: The convolution kernels are set to be 11×11 in size, 4 in stride, and 5×5 in size, 1 in stride; the window size of the pooling layer is 3×3, with a stride of 2 or 1; the fully connected layers are 2 fully connected layers with dropout.

4. The camera calibration method according to claim 1, wherein: The hyperparameter settings of the convolutional neural network AlexNet are as follows: the model is trained using the stochastic gradient descent method, batch size = 128, momentum = 0.9, weight decay = 0.0005, and learning rate = 0.

01.

5. The camera calibration method according to claim 1, wherein: The convolutional neural network AlexNet initialization settings include: initializing the weights of each layer from a Gaussian distribution with a standard deviation of 0.01 and a mean of 0; initializing the neuron biases of the second, fourth, and fifth convolutional layers and the fully connected hidden layer to a constant of 1; and initializing the neuron biases of the remaining layers to a constant of 0.

6. The camera calibration method according to claim 1, wherein: The learning rate setting of the convolutional neural network AlexNet includes: all layers are equal, and they will be manually adjusted during training. The heuristic method followed is to divide the learning rate by 10 when the error rate of the validation set no longer increases with the current learning rate.

7. The camera calibration method according to claim 6, wherein: The learning rate is initialized to 0.01 and decreased three times before termination.

Images