Wide-angle lens image distortion correction method based on deep learning

Abstract

The invention discloses a wide-angle lens image distortion correction method based on deep learning. The method comprises the following steps: receiving an original distortion image shot by a wide-angle lens as input; the original distorted image is preprocessed; neural network training: training the distorted image correction deep network model to obtain an optimal model; preparing model training: inputting data, training a network model by using an AdamW optimizer, scheduling a learning rate by using cosine annealing, and defining and using an integral loss function to carry out effective wide-angle distortion image correction; inputting the preprocessed image into the optimal distorted image correction deep network model, and carrying out distortion correction; and outputting the corrected distortionless high-quality image. A global convolutional neural network with a self-attention mechanism is adopted to correct a wide-angle lens image, and an expanded convolutional neural network is adopted to expand a filter and a self-attention acceptance field so as to discover which features of an input image should pay more attention.

Description

Technical Field

[0001] The present invention relates to a correction method, and in particular to a wide-angle lens image distortion correction method based on deep learning. Background Art

[0002] Generally speaking, when a lens with an ideal shape is kept strictly parallel to the image sensor, the image captured by the camera does not have any nonlinear distortion. However, due to manufacturing processes, installation errors and other reasons, there is no ideal lens in reality, and it is very difficult to keep the lens and image sensor strictly parallel, which leads to a certain degree of nonlinear distortion in the actual imaging effect of the camera. Wide-angle lens images have been widely used in many computer vision tasks because they can capture a wide field of view (WFoV). However, due to the limitations of lenses and sensors and the need to capture a wide field of view, images taken with wide-angle lenses will also suffer from severe geometric distortion. Distorted images will reduce the geometric position accuracy of objects in the image in some quantitative analysis and image measurement fields, so it is absolutely necessary to correct the distortion of wide-angle lenses.

[0003] At present, there are two methods for wide-angle lens image correction. The first is a pattern-based method, and the second is a deep learning-based method. Generally speaking, the mathematical model for wide-angle lens image correction requires professional knowledge and complex calculations. In addition, since it only calculates the distortion rate of a specific pattern of a wide-angle lens image, it is not possible to correct wide-angle lens images of various patterns. The deep learning model can correct various patterns of wide-angle lens images. The main input is the wide-angle lens image to obtain the corrected image without complex calculations. Therefore, the distortion correction method based on deep learning is more commonly used.

[0004] Most deep learning-based methods use convolutional neural networks (CNNs), which involve many parameters and simplify the features of each region through the pooling layer. However, when performing the pooling operation, the existing features are lost, affecting the accuracy of distortion correction. In addition, these features are calculated on local neighborhoods, which means that repeated convolution operations are required to extract the overall features of the input image. Summary of the invention

[0005] In order to solve the shortcomings of the above-mentioned technology, the present invention provides a wide-angle lens image distortion correction method based on deep learning.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a wide-angle lens image distortion correction method based on deep learning, comprising:

[0007] Receiving the original distorted image taken by the wide-angle lens;

[0008] Preprocessing the original distorted image;

[0009] Neural network training: Train the deep network model for distorted image correction to obtain the optimal model;

[0010] The training process is as follows: prepare the input data for model training, use the AdamW optimizer to train the network model, use cosine annealing to schedule the learning rate, and define and use the integral loss function for effective wide-angle distortion image correction;

[0011] Input the preprocessed image into the trained distortion image correction deep network model for distortion correction;

[0012] Output the rectified distortion-free high-quality image.

[0013] As a further implementation of this plan:

[0014] Preprocessing the original distorted image includes: normalizing the pixel values ​​of the original distorted image and adjusting the image size, so that the pixel value and size of the preprocessed image are consistent with the input data of the model training.

[0015] As a further implementation of this plan:

[0016] The input data for model training is obtained by resizing the images of the Place2 dataset to 256×256 and generating wide-angle distorted images, as shown in the following formula:

[0017]

[0018] Among them, r d is the distortion radius, r u is the normal radius, k i is the i-th polynomial coefficient.

[0019] As a further implementation of this plan:

[0020] k3 is set to 0, k1 and k2 are between 0.01 and 0.2, and then the calculated distortion radius r is used. d , transform the pixels of the image from polar coordinates to Cartesian coordinates to obtain a wide-angle distorted image.

[0021] As a further implementation of this plan:

[0022] The initial learning rate of model training is set to 0.001, and the cosine annealing scheduling learning rate is performed according to the following formula:

[0023]

[0024] Among them, LRt The current learning rate calculated for cosine annealing, and is the maximum and minimum value of the learning rate; T cur is the current decay step size in the loop, T i is the loop that performs cosine annealing.

[0025] As a further implementation of this plan:

[0026] The integral loss function is defined as:

[0027] L=αL2+(1-α)L SSIM

[0028] Where L is the integral loss function; α is a hyperparameter used to balance the weight between L2 loss and SSIM loss; L2 refers to L2 loss, also known as mean square error; L SSIM Refers to the loss function based on structural similarity;

[0029] First, use L2loss, which is the sum of the squares of the difference between the input and the true value:

[0030]

[0031] Among them, y i and are the ground truth value and the predicted value from the network, i is used to specify the sample currently being processed; n is the total number of samples in the dataset;

[0032] Introducing L SSIM loss,L SSIM Used to produce visually pleasing images, SSIM is defined as:

[0033]

[0034] Among them, μ x and μ y is a photometric measure of the luminous intensity per unit area of ​​light propagating in a given direction, σ x and σ y is the difference in brightness or color that makes an object or its representation in an image or display distinguishable; xy is the cross covariance of x and y, where x and y represent two images, and C1 and C2 are constants;

[0035] L SSIM =1-SSIM(x,y)

[0036] The coefficient α is set to 0.85 based on experience, and the training goal is to minimize this loss function.

[0037] As a further implementation of this plan:

[0038] The deep network model for distorted image correction consists of the following three layers: base layer, dilated convolution layer, and self-attention layer;

[0039] The output of the dilated convolutional layer is used as the input of the base layer, and the self-attention layer is connected to the output of each convolutional layer to extract global features.

[0040] As a further implementation of this plan:

[0041] With CNN as the base layer, there are five base layers used to extract features from the image, and the number of filters are 16, 32, 64, 128 and 3 respectively.

[0042] As a further implementation of this plan:

[0043] The dilated convolutional neural network d-CNN is used as the dilated convolutional layer. The d-CNN is obtained by expanding the CNN using amplification filters to expand the receptive field of the filters.

[0044] As a further implementation of this plan:

[0045] The self-attention layer calculates the attention score in a single image by focusing on all positions within the same image through the self-attention mechanism; given an input feature X, three different sets of features Q, K, and V are calculated using linear transformations, as shown in the following calculation formula:

[0046] Q=W Q X+b Q

[0047] K=W K X+b K

[0048] V=W V X+b V

[0049] Among them, W Q , W K , W V Respectively represent the weight matrix; b Q , b K , b V They represent the bias vectors respectively;

[0050] Then, the self-attention weights are calculated by the scaled dot product between features Q and K, where the scaling factor d is the dimension of the vectors in features Q and K; the self-attention layer uses the scaled dot product of Q and K to calculate the obtained self-attention weights and the value of the matrix V to calculate its output, as shown below:

[0051]

[0052] Among them, Softmax represents the Softmax function; T represents the transpose of the matrix.

[0053] A wide-angle lens image distortion correction device based on deep learning, comprising:

[0054] A receiving module, used for receiving an original distorted image taken by a wide-angle lens;

[0055] A preprocessing module, used for preprocessing the original distorted image;

[0056] Training module, used for neural network training: training the deep network model for distorted image correction to obtain the optimal model; the training process is: preparing the input data for model training, using the AdamW optimizer to train the network model, using cosine annealing to schedule the learning rate, and defining and using the integral loss function for effective wide-angle distorted image correction;

[0057] The correction module is used to input the preprocessed image into the trained distortion image correction deep network model for distortion correction; and output the corrected distortion-free high-quality image.

[0058] As a further implementation of this plan:

[0059] The preprocessing module is specifically used for normalizing the pixel values ​​of the original distorted image and adjusting the image size, so that the pixel value and size of the preprocessed image are consistent with the input data of the model training.

[0060] The present invention discloses a method for wide-angle lens image distortion correction based on deep learning, which uses a global convolutional neural network with a self-attention mechanism to correct wide-angle lens images, and uses a dilated convolutional neural network to expand the receptive field of the filter and self-attention to find out which features of the input image should be paid more attention to. This method is suitable for fields such as automated production and quality control, industrial monitoring and safety, scientific research and development, and specific industrial applications, and has the following technical advantages compared with the prior art:

[0061] 1) Design a deep neural network consisting of a base layer, a dilated convolutional layer, and a self-attention layer (i.e., a deep network model for distorted image correction). Since the network is supervised, it can perform global feature extraction and realize end-to-end wide-angle lens image correction from wide-angle lens images to corrected images.

[0062] 2) In the process of image distortion processing, the d-CNN of the dilated convolution layer uses amplification filters to expand the receptive field. In the convolution operation, the expanded receptive field has a small dimensionality loss, so global features can be extracted. Combining CNN and d-CNN, and extracting the most important features by the self-attention layer, the self-attention mechanism is applied to all convolution layer outputs to correct wide-angle distorted images.

[0063] 3) The loss function defined by model training introduces L SSIM loss, which can better capture image structure information, improve image quality, realize multi-scale analysis, and improve model stability and convergence.

[0064] 4) The present invention uses cosine annealing to schedule the learning rate during model training, so that the image distortion correction method of the present invention has dynamic adaptability, can avoid local optimality, and at the same time improves training efficiency, and also has flexibility and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the deep neural network structure of the present invention.

[0066] Figure 2 It is a schematic diagram of the correction process of the present invention.

[0067] Figure 3 is the original wide-angle distorted image in the embodiment of the present invention.

[0068] Figure 4 It is the image after correction in the embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] In view of the shortcomings of existing deep learning methods in processing wide-angle lens image distortion, the present invention proposes a global convolutional neural network with a self-attention mechanism to correct wide-angle lens images, and adopts an extended convolutional neural network to expand the receptive field of filters and self-attention to discover which features of the input image should be paid more attention to, thereby realizing an end-to-end process from wide-angle lens images to corrected images.

[0071] First, a deep neural network, namely the distorted image correction deep network model, is designed to extract image features, learn distortion patterns, and generate corrected images. Figure 1As shown in the figure, the deep neural network consists of the following three layers: base layer, dilated convolution layer and self-attention layer. The deep neural network is supervised, and it uses base layer, dilated convolution layer and self-attention layer to extract global features and realize the end-to-end correction process from wide-angle lens image to corrected image. The dilated convolution layer is a dilated convolutional neural network (d-CNN), which uses a magnification filter to expand the receptive field. In the convolution transport, the dilated receptive field has a small dimensionality loss, so global features can be extracted. The self-attention layer extracts the most important features, and the residual mapping improves the training efficiency of the network.

[0072] The following will introduce the detailed settings of each layer one by one:

[0073] (1) Base layer

[0074] The purpose of the base layer design is to extract features from the input for wide-angle lens image correction. Many deep learning studies have shown that convolutional neural networks (CNNs) trained with a large amount of data perform well in various computer vision tasks, such as image classification and target detection. Therefore, the present invention adopts CNN as the base layer. There are five base layers to extract features from the image. The number of filters in the convolutional layer is 16, 32, 64, 128 and 3 respectively. The output of the expanded convolutional layer is used as the input of the base layer so that the filter of the CNN matches the receptive field range of the d-CNN. In addition, a self-attention layer is connected to the output of each CNN to extract global features, and residual mapping is applied to optimize the deep neural network.

[0075] (2) Dilated Convolutional Layer

[0076] Dilated convolutional layers use dilated convolutional neural networks (d-CNNs), which are extensions of CNNs using magnified filters. One way to improve the performance of CNN networks is to expand the receptive field. However, in traditional CNNs, expanding the receptive field increases the amount of computation required by the network. So generally, by stacking layers, the size of the filters is reduced and the depth of the network is increased. In contrast, d-CNNs use magnified filters, called dilated convolutional filters, with a dilation rate. The dilation rate is the spacing between the median values ​​of the filter.

[0077] (3) Self-attention layer

[0078] The self-attention layer computes the attention score in a single image by focusing on all locations within the same image via the self-attention mechanism.

[0079] Self-attention Given an input feature X, three different sets of features Q, K and V are calculated using linear transformation, as shown in the following calculation formula:

[0080] Q=W Q X+b Q

[0081] K=W K X+b K

[0082] V=W V X+b V

[0083] in,

[0084] W Q : Weight matrix, used to transform the input X into a query vector;

[0085] W K : Weight matrix, used to transform the input X into a key vector;

[0086] W V : Weight matrix, used to transform the input X into a value vector;

[0087] b Q : Bias vector, related to the calculation of the query vector;

[0088] b K : Bias vector, related to the calculation of the key vector;

[0089] b V : Bias vector, related to the calculation of the value vector.

[0090] Then, the self-attention weights are calculated by the scaled dot product between features Q and K, where the scaling factor d is the dimension of the vectors in features Q and K. The self-attention layer uses the scaled dot product of Q and K to calculate the obtained self-attention weights and the value of the matrix V to calculate its output, as shown below:

[0091]

[0092] Among them, Softmax represents the Softmax function, which is a function that converts the output of a multi-classification neural network into a probability distribution. It is used to normalize the similarity score in the attention mechanism to generate the attention weight; T represents the transpose of the matrix.

[0093] Therefore, for the deep neural network designed by the present invention, a global convolutional neural network based on self-attention is adopted, in which CNN is used as the base layer, and an expanded convolutional layer and a self-attention layer are introduced to expand the receptive field of the filter, and also allow the input images to interact with each other and find features that the input should pay more attention to.

[0094] Secondly, after constructing the deep network model for distorted image correction, network training is required to obtain the optimal model. In order to train the network, a loss function is defined to measure the difference between the network output (corrected image) and the true or expected output (undistorted image).

[0095] The loss function is defined as:

[0096] L=αL2+(1-α)L SSIM

[0097] Where L is the integral loss function; α is a hyperparameter used to balance the weight between L2 loss and SSIM loss; L2 refers to L2 loss, also known as mean square error, which is a common loss function used to measure the difference between the predicted value and the actual value; L SSIM Refers to the loss function based on structural similarity index (SSIM).

[0098] First, use L2 loss, which is the sum of the squares of the difference between the input and the true value:

[0099]

[0100] Among them, y i and are the ground truth and the predicted value from the network, i is used to specify the sample currently being processed; n is the total number of samples in the dataset. However, when the network is trained with L2loss, it produces speckle artifacts. To compensate for the speckle artifacts, L SSIM loss should be connected. SSIM Used to produce visually pleasing images. SSIM is defined as:

[0101]

[0102] Among them, μ x and μ y is a photometric measure of the luminous intensity per unit area of ​​light propagating in a given direction, σ x and σ y is the difference in brightness or color that makes an object (or its representation in an image or display) distinguishable; xy is the cross covariance of x and y, where x and y represent two images, and C1 and C2 are constants;

[0103] L SSIM =1-SSIM(x,y)

[0104] The coefficient α is set to 0.85 based on experience, and the training goal is to minimize this loss function.

[0105] When defining the loss function, the present invention introduces L2loss based on the traditional L2loss. SSIM loss is used to define the loss function, which can better capture the image structure information, improve the image quality, realize multi-scale analysis, and improve the model stability and convergence.

[0106] After defining the loss function, the model training process is performed to obtain the optimal deep network model for distorted image correction.

[0107] The model training process includes:

[0108] The Place2 dataset is a large-scale image database that is widely used in scene classification and recognition. This paper resizes the images of the Place2 dataset to 256×256, then generates wide-angle lens images and marks them as real images. The AdamW optimizer is used to train our network, and cosine annealing is used to schedule the learning rate. The initial learning rate is set to 0.001.

[0109]

[0110] Among them, LR t The current learning rate calculated for cosine annealing, and is the maximum and minimum value of the learning rate; T cur is the current decay step size in the loop, T i is the loop that performs cosine annealing.

[0111] The network is trained using the prepared training data and the defined loss function. During the training process, the weights and biases of the network are updated through the back propagation algorithm to minimize the loss function. At this point, a deep network model that can be applied to the correction of distorted images taken with a wide-angle lens is obtained.

[0112] The present invention uses cosine annealing to schedule the learning rate during model training, so that the image distortion correction method of the present invention has dynamic adaptability, can avoid local optimality, and at the same time improves training efficiency, and also has flexibility and wide applicability.

[0113] The deep neural network designed based on the above process, namely the deep network model for distortion image correction, is further introduced in the process of wide-angle lens image distortion correction processing. Figure 2 As shown, the wide-angle lens image distortion correction method based on deep learning includes the following steps:

[0114] Step 1: Receive the original distorted image captured by a wide-angle lens as input.

[0115] Step 2: Preprocess the original distorted image; preprocessing includes: normalizing the pixel values ​​of the original distorted image and adjusting the image size, so that the pixel values ​​and size of the preprocessed image are consistent with the input data of the model training. Specifically, the image pixel values ​​are normalized to the range used in model training, usually [0,1] or [-1,1], and the image size is adjusted according to the model requirements to ensure the consistency of the input data.

[0116] Step 3: Neural network training: Train the distorted image correction deep network model to obtain the optimal model;

[0117] First, prepare the input data for model training. Resize the images of the Place2 dataset to 256×256 to generate wide-angle distorted images. These images are used as input for training the network model. The wide-angle distorted images are obtained according to the following formula:

[0118]

[0119] Among them, r d is the distortion radius, r u is the normal radius, k i is the i-th polynomial coefficient.

[0120] At the same time, since k3 has little effect on radial distortion, k3 is set to 0, and the degree of distortion is adjusted by adjusting the values ​​of k1 and k2.

[0121] The present invention limits the values ​​of k1 and k2 to between 0.01 and 0.2, and then uses the calculated distortion radius r d , transform the pixels of the image from polar coordinates to Cartesian coordinates to obtain a wide-angle distorted image.

[0122] Secondly, we use the AdamW optimizer to train our network model, use cosine annealing to schedule the learning rate, and define and use an integral loss function for effective wide-angle distortion image correction.

[0123] The initial learning rate of model training is set to 0.001, and the cosine annealing scheduling learning rate is performed according to the following formula:

[0124]

[0125] Among them, LR t The current learning rate calculated for cosine annealing, and is the maximum and minimum value of the learning rate; T cur is the current decay step size in the loop, T i is the loop that performs cosine annealing.

[0126] The integral loss function is defined as:

[0127] L=αL2+(1-α)L SSIM

[0128] Where L is the integral loss function; α is a hyperparameter used to balance the weight between L2 loss and SSIM loss; L2 refers to L2 loss, also known as mean square error, which is a common loss function used to measure the difference between the predicted value and the actual value; L SSIM Refers to the loss function based on structural similarity index (SSIM).

[0129] First, use L2 loss, which is the sum of the squares of the difference between the input and the true value:

[0130]

[0131] Among them, y i and are the ground truth and the predicted value from the network, i is used to specify the sample currently being processed; n is the total number of samples in the dataset. However, when the network is trained with L2loss, it produces speckle artifacts. To compensate for the speckle artifacts, L SSIM loss should be connected. SSIM Used to produce visually pleasing images. SSIM is defined as:

[0132]

[0133] Among them, μ x and μ y is a photometric measure of the luminous intensity per unit area of ​​light propagating in a given direction, σ x and σ y is the difference in brightness or color that makes an object (or its representation in an image or display) distinguishable; xy is the cross covariance of x, y and C1, where x and y represent two images, and C1 and C2 are constants.

[0134] L SSIM =1-SSIM(x,y)

[0135] The coefficient α is set to 0.85 based on experience, and the training goal is to minimize this loss function.

[0136] For the Place2 dataset, the batch size is set to 16 and the total number of training data is set to 60K.

[0137] The constructed deep network model for distorted image correction consists of the following three layers: base layer, dilated convolutional layer and self-attention layer.

[0138] The output of the dilated convolutional layer is used as the input of the base layer, and the self-attention layer is connected to the output of each convolutional layer to extract global features.

[0139] With CNN as the base layer, there are five base layers used to extract features from the image, and the number of filters are 16, 32, 64, 128 and 3 respectively.

[0140] The dilated convolutional neural network d-CNN is used as the dilated convolutional layer. The d-CNN is obtained by expanding the CNN row using amplification filters, which has the advantage of expanding the receptive field of the filter.

[0141] The self-attention layer calculates the attention score in a single image by focusing on all positions within the same image through the self-attention mechanism. Given an input feature X, three different sets of features Q, K, and V are calculated using linear transformations, as shown in the following calculation formula:

[0142] Q=W Q X+b Q

[0143] K=W K X+b K

[0144] V=W V X+b V

[0145] in,

[0146] W Q : Weight matrix, used to transform the input X into a query vector;

[0147] W K : Weight matrix, used to transform the input X into a key vector;

[0148] W V : Weight matrix, used to transform the input X into a value vector;

[0149] b Q : Bias vector, related to the calculation of the query vector;

[0150] b K : Bias vector, related to the calculation of the key vector;

[0151] b V : Bias vector, related to the calculation of the value vector.

[0152] Then, the self-attention weights are calculated by the scaled dot product between features Q and K, where the scaling factor d is the dimension of the vectors in features Q and K. The self-attention layer uses the scaled dot product of Q and K to calculate the obtained self-attention weights and the value of the matrix V to calculate its output, as shown below:

[0153]

[0154] Among them, Softmax represents the Softmax function, which is a function that converts the output of a multi-classification neural network into a probability distribution. It is used to normalize the similarity score in the attention mechanism to generate the attention weight; T represents the transpose of the matrix.

[0155] The present invention specifically applies the rectified linear unit (ReLU) activation function to all base layers. Next, the LeakyReLU activation function is applied to the dilated convolutional layer.

[0156] Since the deep network model for distorted image correction consisting of a base layer, an expanded convolutional layer, and a self-attention layer is supervised, it uses three layers to extract global features, thereby achieving end-to-end wide-angle lens image correction from wide-angle lens images to corrected images.

[0157] Step 4: Input the preprocessed image into the trained distorted image correction deep network model for distortion correction.

[0158] The deep network model for distorted image correction uses a dilated convolutional neural network (d-CNN) to expand the receptive field of the filter. In the convolution operation, the expanded receptive field has a small dimensionality loss, so global features can be extracted. Then, a method combining CNN and d-CNN is used to apply self-attention to all convolutional layer outputs to correct wide-angle distorted images.

[0159] Step 5: Output the corrected distortion-free high-quality image.

[0160] Therefore, the wide-angle lens image distortion correction method based on deep learning disclosed in the present invention has the following technical advantages compared with the prior art:

[0161] 1) Design a deep neural network consisting of a base layer, a dilated convolutional layer, and a self-attention layer (i.e., a deep network model for distorted image correction). Since the network is supervised, it can perform global feature extraction and realize end-to-end wide-angle lens image correction from wide-angle lens images to corrected images.

[0162] 2) In the process of image distortion processing, the d-CNN of the dilated convolution layer uses amplification filters to expand the receptive field. In the convolution operation, the expanded receptive field has a small dimensionality loss, so global features can be extracted. Combining CNN and d-CNN, and extracting the most important features by the self-attention layer, the self-attention mechanism is applied to all convolution layer outputs to correct wide-angle distorted images.

[0163] 3) The loss function defined by model training introduces L SSIM loss, which can better capture image structure information, improve image quality, realize multi-scale analysis, and improve model stability and convergence.

[0164] 4) The present invention uses cosine annealing to schedule the learning rate during model training, so that the image distortion correction method of the present invention has dynamic adaptability, can avoid local optimality, and at the same time improves training efficiency, and also has flexibility and wide applicability.

[0165] Example:

[0166] This embodiment demonstrates the specific application of the distortion processing of the wide-angle lens image distortion correction method based on deep learning disclosed in the present invention. First, the neural network is trained: the image size of the Place2 data set is adjusted to 256×256, and a wide-angle distorted image is generated, as shown in the following formula:

[0167]

[0168] Among them, r d is the distortion radius, r u is the normal radius, k i is the i-th polynomial coefficient.

[0169] In the training module, k3 is set to 0, k1 and k2 are between 0.01 and 0.2, and then the calculated distortion radius r is used. d , transform the pixels of the image from polar coordinates to Cartesian coordinates to obtain a wide-angle distorted image.

[0170] In the training module, the initial learning rate of the model training is set to 0.001, and the cosine annealing scheduling learning rate is performed according to the following formula:

[0171]

[0172] Among them, LR t The current learning rate calculated for cosine annealing, and is the maximum and minimum value of the learning rate; T cur is the current decay step size in the loop, T i is the loop that performs cosine annealing.

[0173] In the training module, the integral loss function is defined as:

[0174] L=αL2+(1-α)L SSIM

[0175] Where L is the integral loss function; α is a hyperparameter used to balance the weight between L2 loss and SSIM loss; L2 refers to L2 loss, also known as mean square error; L SSIM Refers to the loss function based on structural similarity;

[0176] First, use L2loss, which is the sum of the squares of the difference between the input and the true value:

[0177]

[0178] Among them, y i and are the ground truth value and the predicted value from the network, i is used to specify the sample currently being processed; n is the total number of samples in the dataset;

[0179] Introducing L SSIM loss,L SSIM Used to produce visually pleasing images, SSIM is defined as:

[0180]

[0181] Among them, μ x and μ y is a photometric measure of the luminous intensity per unit area of ​​light propagating in a given direction, σ x and σ y is the difference in brightness or color that makes an object or its representation in an image or display distinguishable; xy is the cross covariance of x and y, where x and y represent two images, and C1 and C2 are constants;

[0182] L SSIM =1-SSIM(x,y)

[0183] The coefficient α is set to 0.85 based on experience, and the training goal is to minimize this loss function.

[0184] In the training module, the deep network model for distorted image correction consists of the following three layers: base layer, dilated convolution layer, and self-attention layer;

[0185] The output of the dilated convolutional layer is used as the input of the base layer, and the self-attention layer is connected to the output of each convolutional layer to extract global features.

[0186] With CNN as the base layer, there are five base layers used to extract features from the image, and the number of filters are 16, 32, 64, 128 and 3 respectively.

[0187] The dilated convolutional neural network d-CNN is used as the dilated convolutional layer. The d-CNN is obtained by expanding the CNN using amplification filters to expand the receptive field of the filters.

[0188] The self-attention layer calculates the attention score in a single image by focusing on all positions within the same image through the self-attention mechanism; given an input feature X, three different sets of features Q, K, and V are calculated using linear transformations, as shown in the following calculation formula:

[0189] Q=W Q X+b Q

[0190] K=W K X+b K

[0191] V=W V X+b V

[0192] Among them, W Q , W K , W V Respectively represent the weight matrix; b Q 、b K 、b V They represent the bias vectors respectively;

[0193] Then, the self-attention weights are calculated by the scaled dot product between features Q and K, where the scaling factor d is the dimension of the vectors in features Q and K; the self-attention layer uses the scaled dot product of Q and K to calculate the obtained self-attention weights and the value of the matrix V to calculate its output, as shown below:

[0194]

[0195] Among them, Softmax represents the Softmax function; T represents the transpose of the matrix.

[0196] Figure 3 The original distorted image taken by the wide-angle lens is Figure 3 After the pixel value of the original distorted image is normalized and the image size is adjusted, it is input into the trained distorted image correction deep network model for image distortion correction. The output corrected image is as follows Figure 4 As shown. Figure 4The displayed image results show that the image processed by the correction method of the present invention has solved the distortion problem. This shows that the image processed by the wide-angle lens image distortion correction method based on deep learning proposed by the present invention has excellent visual performance and good image quality in local areas.

[0197] The system to which the above-mentioned distortion processing method is applied is an end-to-end system that can directly take wide-angle distorted images as input and output corrected images without complex pre-processing or post-processing steps. The design and training process of this model involves a lot of calculations and data processing, and usually requires high-performance computing resources, such as GPUs, to achieve fast training and reasoning. Therefore, corresponding hardware support is required to run. The hardware / software support used in this patent is the following devices.

[0198]

[0199] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0204] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for wide-angle lens image distortion correction based on deep learning, characterized in that: include: Receiving the original distorted image taken by the wide-angle lens; Preprocessing the original distorted image; Neural network training: Train the deep network model for distorted image correction to obtain the optimal model; The training process is as follows: prepare the input data for model training, use the AdamW optimizer to train the network model, use cosine annealing to schedule the learning rate, and define and use the integral loss function for effective wide-angle distortion image correction; Input the preprocessed image into the trained distortion image correction deep network model for distortion correction; Output the rectified distortion-free high-quality image.

2. The wide-angle lens image distortion correction method based on deep learning according to claim 1, characterized in that: Preprocessing the original distorted image includes: normalizing the pixel values ​​of the original distorted image and adjusting the image size, so that the pixel value and size of the preprocessed image are consistent with the input data of the model training.

3. The wide-angle lens image distortion correction method based on deep learning according to claim 1, characterized in that: The input data for model training is obtained by resizing the images of the Place2 dataset to 256×256 and generating wide-angle distorted images, as shown in the following formula: Among them, r d is the distortion radius, r u is the normal radius, k i is the i-th polynomial coefficient.

4. The wide-angle lens image distortion correction method based on deep learning according to claim 3, characterized in that: k3 is set to 0, k1 and k2 are between 0.01 and 0.2, and then the calculated distortion radius r is used. d , transform the pixels of the image from polar coordinates to Cartesian coordinates to obtain a wide-angle distorted image.

5. The wide-angle lens image distortion correction method based on deep learning according to claim 1, characterized in that: The initial learning rate of model training is set to 0.001, and the cosine annealing scheduling learning rate is performed according to the following formula: Among them, LR t The current learning rate calculated for cosine annealing, and is the maximum and minimum value of the learning rate; T cur is the current decay step size in the loop, T i is the loop that performs cosine annealing.

6. The method for wide-angle lens image distortion correction based on deep learning according to claim 5, characterized in that: The integral loss function is defined as: L=αL2+(1-α)L SSIM Where L is the integral loss function; α is a hyperparameter used to balance the weight between L2 loss and SSIM loss; L2 refers to L2 loss, also known as mean square error; L SSIM Refers to the loss function based on structural similarity; First, use L2 loss, which is the sum of the squares of the difference between the input and the true value: Among them, y i and are the ground truth value and the predicted value from the network, i is used to specify the sample currently being processed; n is the total number of samples in the dataset; Introducing L SSIM loss,L SSIM Used to produce visually pleasing images, SSIM is defined as: Among them, μ x and μ y is a photometric measure of the luminous intensity per unit area of ​​light propagating in a given direction, σ x and σ y is the difference in brightness or color that makes an object or its representation in an image or display distinguishable; xy is the cross covariance of x and y, where x and y represent two images, and C1 and C2 are constants; L SSIM =1-SSIM(x,y) The coefficient α is set to 0.85 based on experience, and the training goal is to minimize this loss function.

7. The wide-angle lens image distortion correction method based on deep learning according to claim 1, characterized in that: The deep network model for distorted image correction consists of the following three layers: base layer, dilated convolution layer, and self-attention layer; The output of the dilated convolutional layer is used as the input of the base layer, and the self-attention layer is connected to the output of each convolutional layer to extract global features.

8. The wide-angle lens image distortion correction method based on deep learning according to claim 7, characterized in that: With CNN as the base layer, there are five base layers used to extract features from the image, and the number of filters are 16, 32, 64, 128 and 3 respectively.

9. The wide-angle lens image distortion correction method based on deep learning according to claim 7, characterized in that: The dilated convolutional neural network d-CNN is used as the dilated convolutional layer. The d-CNN is obtained by expanding the CNN using amplification filters to expand the receptive field of the filters.

10. The wide-angle lens image distortion correction method based on deep learning according to claim 7, characterized in that: The self-attention layer calculates the attention score in a single image by focusing on all positions within the same image through the self-attention mechanism; given an input feature X, three different sets of features Q, K, and V are calculated using linear transformations, as shown in the following calculation formula: Q=W Q X+b Q K=W K X+b K V=W V X+b V Among them, W Q , W K , W V Respectively represent the weight matrix; b Q , b K , b V They represent the bias vectors respectively; Then, the self-attention weights are calculated by the scaled dot product between features Q and K, where the scaling factor d is the dimension of the vectors in features Q and K; the self-attention layer uses the scaled dot product of Q and K to calculate the obtained self-attention weights and the value of the matrix V to calculate its output, as shown below: Among them, Softmax represents the Softmax function; T represents the transpose of the matrix.

11. A wide-angle lens image distortion correction device based on deep learning, characterized in that: include: A receiving module, used for receiving an original distorted image taken by a wide-angle lens; A preprocessing module, used for preprocessing the original distorted image; Training module, used for neural network training: training the deep network model for distorted image correction to obtain the optimal model; the training process is: preparing the input data for model training, using the AdamW optimizer to train the network model, using cosine annealing to schedule the learning rate, and defining and using the integral loss function for effective wide-angle distorted image correction; The correction module is used to input the preprocessed image into the trained distortion image correction deep network model for distortion correction; and output the corrected distortion-free high-quality image.

12. The wide-angle lens image distortion correction device based on deep learning according to claim 11, characterized in that: The preprocessing module is specifically used for normalizing the pixel values ​​of the original distorted image and adjusting the image size, so that the pixel value and size of the preprocessed image are consistent with the input data of the model training.

Images