A method for correcting image geometric distortion based on Unet network

Abstract

This invention discloses an image geometric distortion correction method based on the Unet network, which solves the problem that current methods mainly target single distortion types and have significant limitations when correcting mixed distortions. The key technical points are: constructing a distortion correction model based on the Unet network; selecting place365 images and a pre-generated checkerboard image as image data sources; considering radial and tangential distortion to generate distorted images and their coordinate difference vectors as datasets; setting the distortion correction model according to the experimental environment parameters; training and testing the distortion correction model using the constructed dataset; acquiring the actual dataset and predicting the coordinate difference vectors of the distorted images; and using an image resampling algorithm to resample the corrected image based on the predicted coordinate difference vectors and the actual distorted image. This Unet network-based image geometric distortion correction method considers both radial and tangential distortions, has fewer limitations, and achieves better overall correction results.

Description

Technical Field

[0001] This invention relates to image distortion correction technology, and in particular to an image geometric distortion correction method based on Unet network. Background Technology

[0002] Traditional distortion correction methods include camera calibration methods, which require specific calibration boards to solve for the camera's intrinsic, extrinsic, and distortion parameters. Although these methods have achieved high correction accuracy, the correction process is complex and requires additional calibration instruments. Furthermore, different cameras require different calibration procedures to obtain camera parameters, which limits their widespread application in various scenarios and imposes certain limitations. Secondly, there are image-based geometric feature correction methods, which rely on detecting contour lines, parallel lines, etc., in the image. However, many images do not meet this requirement.

[0003] Deep learning-based methods primarily perform correction by extracting image features, reducing the limitation of known camera parameters and thus having broader applicability. However, current deep learning-based research mainly focuses on correcting images with a single distortion type (radial distortion), and the training datasets constructed also represent a single distortion type. In contrast, images acquired by cameras often exhibit mixed radial and tangential distortions. Therefore, current methods have significant limitations in addressing mixed distortion correction. Summary of the Invention

[0004] The purpose of this invention is to provide an image geometric distortion correction method based on Unet network, which considers radial and tangential distortion, has fewer limitations, and achieves better overall correction results.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] An image geometric distortion correction method based on Unet network includes the following steps:

[0007] A distortion correction model is constructed based on the Unet network;

[0008] A dataset was constructed, selecting place365 images and a pre-generated checkerboard image as the image data source. The image data source was preprocessed, considering radial and tangential distortion, based on the actual image point p(x,y) and the ideal distortion-free image point in the physical coordinate system. The mathematical relationship formula is used to generate distorted images and their corresponding coordinate difference vectors as a dataset.

[0009] The distortion correction model is set according to the parameter configuration of the experimental environment. The distortion correction model is trained and tested using the constructed dataset. Image features are extracted, the mapping from the distorted image domain to the correction coordinate domain is learned, and the trained distortion correction model is saved.

[0010] Collect actual datasets, input actual distorted images into distortion correction models, and predict the coordinate difference vector of distorted images;

[0011] The corrected image is obtained by resampling based on the predicted coordinate difference vector and the actual distorted image using an image resampling algorithm.

[0012] In summary, the present invention has the following beneficial effects:

[0013] Deep learning-based methods correct distortion by extracting image features, thus mitigating limitations imposed by known camera conditions and offering broader applicability. Using place and checkerboard images as data sources preserves more distortion features, reduces feature loss, and provides the most significant geometric features. Simultaneous consideration of radial and tangential distortion better reflects the distortions inherent in images actually captured by the camera, addressing the significant limitations of current distortion correction methods. Furthermore, through distortion correction models and resampling algorithms, geometric distortion correction can be achieved, providing insights and references for subsequent correction of various types of mixed distortions. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the method.

[0015] Figure 2 This is a diagram of the Unet network architecture used in this application;

[0016] Figure 3 The image shows the experimental results of distortion correction.

[0017] Figure 4 To correct the image detail analysis diagram;

[0018] Figure 5 The results of the calibration experiment for images taken with a GoPro camera;

[0019] Figure 6 (a) is an image of "fisheye" distortion taken with a GoPro camera, and (b) is the result after correction using this method. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings.

[0021] According to one or more embodiments, an image geometric distortion correction method based on Unet network is disclosed, such as... Figure 1 As shown, it includes the following steps:

[0022] S1. Construct a distortion correction model based on the Unet network.

[0023] The network architecture of the distortion correction model is as follows Figure 2 As shown. Images containing radial distortion typically have specific structural features. A deep learning network model is used to extract these features, and the network learns a mapping from the distorted image domain to the corrected coordinate domain. The corrected coordinate domain is a two-dimensional vector with the same size as the input distorted image. The corrected image is obtained by resampling the image using the predicted corrected coordinate domain vector.

[0024] The distortion correction model is based on an Encoder-Decoder structure, a U-shaped symmetric structure. Its main components include downsampling (compression path), upsampling (expansion path), and skip connections. In the compression path, the image is encoded using 3×3 convolutional layers, a ReLU function, and 2×2 pooling layers. The input image undergoes four downsampling operations, each reducing the feature map size while doubling the number of channels. In the expansion path, the image is decoded using four 2×2 deconvolutional layers to upsampling and expanding the feature map size to match the input image size. The skip connection structure fuses high-level features from the decoding layer with low-level features from the encoding layer, combining abstract and detailed features to achieve multi-scale feature fusion. This results in captured feature maps containing both global and local information, improving feature expressiveness.

[0025] Choosing the Unet network allows for better feature extraction from distorted images, and also improves model training speed. A three-channel distorted image (Input) is input into the network, and after learning, the model outputs a two-dimensional coordinate difference vector V with the same size as the input image. pred This transforms the image correction problem into a problem of predicting the pixel-by-pixel coordinate displacement changes of the distorted image.

[0026] S2. Construct a dataset, selecting place365 images and the generated checkerboard image as image data sources. Preprocess the image data sources, considering radial and tangential distortion, based on the actual image point p(x,y) and the ideal distortion-free image point in the physical coordinate system. The mathematical relationship formula is used to generate distorted images and their corresponding coordinate difference vectors as a dataset.

[0027] The basic principle of camera imaging is based on the pinhole camera model. However, deviations in camera manufacturing precision and assembly processes introduce distortion, causing the image imaging model to deviate from the pinhole camera model. Therefore, it is necessary to consider the problem of imaging geometric distortion. Lens distortion is mainly divided into radial distortion and tangential distortion. Radial distortion is one of the main factors causing image distortion, caused by the radial position deviation of the image point; tangential distortion is caused by the lens itself not being parallel to the camera sensor plane. Considering both types of distortion, and referring to the distortion description in Zhang's calibration method, the actual image point p(x,y) and the ideal distortion-free image point in the physical coordinate system are compared... The mathematical relationship is expressed as follows:

[0028]

[0029] In the formula, r 2 =x 2 +y 2 k1, k2, and k3 are radial distortion coefficients; p1 and p2 are tangential distortion coefficients.

[0030] To train the network to perform geometric distortion correction, a large dataset is needed as a preliminary dataset. Current research on distortion correction using deep learning methods typically uses datasets with only radial distortion, and the distorted images are often low-resolution (256×256 pixels), resulting in the loss of features at the edges of images with significant distortion. Considering that images containing radial distortion exhibit greater distortion at the edges and less distortion in the center, it is necessary to generate higher-resolution distorted images to preserve the distortion features at the image edges.

[0031] First, an image data source needs to be selected. Considering the above, the place365 image dataset is chosen, and the images in the dataset are cropped to 512×512 pixels. Images with obvious geometric structures, such as straight lines, arcs, and horizons, can provide a wealth of geometric features for the network to learn. Therefore, further image filtering is performed, using Hough transform to filter out images with obvious straight-line structures. The filtering criteria are defined as follows:

[0032] ∑(l>α)≥λ×C (2)

[0033] In the formula, l represents the length of the detected line, α is the length of the shortest line segment, λ is the selection factor for the dataset (a larger value indicates more lines are selected from the images), and C is the image resolution, which is 512 pixels. Through multiple experiments, 1800 images with α = 80 and λ = 4 were selected as one of the image data sources.

[0034] Secondly, based on the idea of ​​traditional camera calibration, checkerboard images can provide the most significant geometric features for image geometric distortion correction. Therefore, by setting different square sizes and image rotation angles, the features of the checkerboard are diversified, generating 1800 checkerboard images (512×512 pixels) as one of the image data sources.

[0035] The dataset used in this application contains two types of distortion. The description of radial distortion is determined by multiple distortion parameters k, and can be extended to handle non-radial symmetric distortion by adding tangential components. The single-parameter model, however, can only describe radial symmetric distortion. The specific construction process of the dataset is as follows:

[0036] First, camera calibration was performed for different camera modes, obtaining four different camera intrinsic parameters fx, fy, cx, cy and distortion parameters k1, k2, k3, p1, p2. This was based on the actual image point p(x,y) and the ideal distortion-free image point in the physical coordinate system. The mathematical formula generates a dataset I of 512×512 pixels with radial and tangential mixed distortion and its corresponding coordinate difference dataset V. The coordinate difference dataset V records the pixel p(x,y) in the distorted image and its corresponding pixel in the undistorted image in the pixel coordinate system. The difference was calculated. Because the generated distorted images have black borders, and to preserve edge features as much as possible, a 352×352 pixel image was cropped from the center of the original image, along with the coordinate difference vector, to form the final dataset. This resulted in 14,400 distorted images, of which 12,000 were used to train the model and 2,400 were used to test it. Figure 3 The image shown is a partial image of the synthetic dataset.

[0037] S3. Configure the distortion correction model according to the parameters of the experimental environment, train and test the distortion correction model using the constructed dataset, extract image features, learn the mapping from the distorted image domain to the correction coordinate domain, and save the trained distortion correction model.

[0038] S4. Collect the actual dataset, input the actual distorted image into the distortion correction model, and predict the coordinate difference vector of the distorted image.

[0039] The accuracy of the predicted values ​​in the corrected coordinate domain is related to the accuracy of the corrected image; therefore, a loss function is designed to improve the model's prediction accuracy.

[0040] The loss function (EPE) is expressed as the Euclidean distance between the predicted coordinate difference vector and the true value, and the coordinate difference vector V estimated by the network is... predIt predicts pixel-by-pixel from the distorted image, indicating how a pixel in the distorted image should be moved to its corresponding pixel in the original undistorted image. First, it uses the Mean Absolute Error (MAE) as the loss function, defined as:

[0041]

[0042] In the formula, n is the total number of elements in the coordinate difference vector.

[0043] Analysis of the predicted coordinate difference vector revealed a trend of larger prediction errors around the edges and smaller errors at the center. Therefore, a weight vector M was designed to optimize the loss function, improving the network's accuracy in predicting the edges of the coordinate difference vector. The ground truth vector V corresponding to the coordinate difference vector is then used. pred The Euclidean distance from each pixel to the coordinate (0,0) is used as the weight metric. After normalizing the Euclidean distance, the weight vector M is obtained. M has the difference vector between the predicted coordinates and the target coordinates V. pred The same shape is defined by the following formula:

[0044]

[0045] In the formula, G is the normalization function.

[0046] Finally, the coordinate difference loss function is defined as:

[0047]

[0048] In the formula, n is the total number of elements in the coordinate difference vector, and M is the weight vector.

[0049] The final corrected image is obtained by resampling the distorted image using the predicted coordinate difference vector. To further improve the correction effect, an image resampling loss function L2 is designed, representing the corrected image I obtained after resampling from the predicted coordinate difference vector. pred With distortion-free image I gt The pixel-by-pixel Euclidean distance between them. It is defined as:

[0050]

[0051] In the formula, n represents the number of pixels.

[0052] In summary, the final loss function is defined as:

[0053] L=L1+μL2 (7)

[0054] The weight μ provides a trade-off between coordinate difference prediction and resampling loss, and can be set to u = 0.5.

[0055] S5. Using an image resampling algorithm, a corrected image is obtained by resampling based on the predicted coordinate difference vector and the actual distorted image.

[0056] Specifically, the experiment was conducted on a Tesla-P100 GPU, using the PyTorch deep learning framework for algorithm implementation, with a runtime environment of Python 3.6, PyTorch 1.8, and CUDA 10.1. The network was trained for 200 epochs with a batch size of 24, using the Adam optimizer and a learning rate of 1×10⁻⁴.

[0057] This method is compared with other deep learning-based geometric distortion image correction methods, including Li and Hosono. The comparison methods and their test results are as follows:

[0058] 1. In 2019, Li proposed a method for blindly correcting multiple types of geometric distortions. It uses an Encoder-Decoder network to learn the features of the distorted image, and uses the Euclidean distance between the predicted coordinate difference vector and the ground truth as the loss function. The distorted image is then resampled using the predicted coordinate difference vector to achieve correction.

[0059] 2. In 2021, Hosono proposed a distortion parameter evaluation network to correct radially distorted images. This method uses a single-parameter distortion model and builds a distortion parameter evaluation network model based on the VGG11 network to predict the distortion parameters of the distorted image and then achieve correction. It also proposes a line reconstruction error loss to improve the prediction accuracy of the network by utilizing the characteristic that the distorted straight line is still a straight line after correction.

[0060] 3. Using the dataset proposed in this method, 1000 images were selected from the test set for geometric distortion correction experiments.

[0061] The correction effects of different methods were quantitatively evaluated using three evaluation metrics: MAE, PSNR, and SSIM. The results are presented in Table 1.

[0062]

[0063] Table 1

[0064] MAE represents the predicted coordinate difference vector V. pred With truth value V gt The mean absolute error between the two values, expressed in pixels, indicates that the closer this value is to 0, the higher the accuracy of the coordinate difference predicted by the network, and the better the correction effect of the resampled image.

[0065] The PSNR and SSIM metrics are the same image quality evaluation metrics used in traditional work: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). A PSNR value above 20 is considered acceptable for image quality; a higher value indicates less image distortion. The SSIM value ranges from 0 to 1; a higher value indicates that the corrected image quality is closer to the original image.

[0066] The quantitative results in Table 1 show that our method outperforms other methods in terms of accuracy in predicting coordinate difference vectors, image correction, and structure restoration, achieving the highest score in the quantitative evaluation. The reasons are as follows: the Unet network can better extract the geometric features of distorted images, improving the network's learning ability; and the two loss functions enhance the network's prediction accuracy.

[0067] Further comparisons were made by selecting images with different degrees of distortion from the test set for correction experiments. This allowed for a direct comparison of the correction effects of different methods. The experimental results are as follows: Figure 4 As shown. From Figure 4 As can be seen intuitively, this method has achieved good correction results in both ordinary images and checkerboard images, and the edge parts of images with a large degree of distortion have also been well corrected.

[0068] To further analyze the correction effects of the three methods, two sets of images were selected for local magnification to analyze details, such as... Figure 5 As shown. From Figure 5 In the first row of images, it can be observed that Li's method has poor edge correction performance and some pixel loss (top left corner), while Figure 5 The second row of images shows that the checkerboard image corrected by Hosono's method has obvious jagged edges.

[0069] To evaluate the effectiveness of the proposed loss function method, ablation experiments were conducted. Keeping other training conditions constant, only the loss function was changed. The effects of different loss functions on the accuracy of network prediction and distortion correction were quantitatively compared using three metrics: MAE, PSNR, and SSIM. The experimental results are shown in Table 2.

[0070]

[0071] Table 2

[0072] The results in the table show that using the mean absolute error loss L0 achieves higher prediction accuracy than the EPE loss used in the Li method. Furthermore, optimizing the coordinate difference loss through the weight vector M further improves the model's prediction accuracy and reduces the edge prediction error of the predicted coordinate difference vector. The image resampling loss L2 also plays a role, complementing the coordinate difference loss L1 and further improving the correction effect of distorted images.

[0073] In addition to the performance verification on the test set, the trained model was also applied to the correction of real images acquired by GoPro cameras, and good correction results were achieved. Figure 6 (a) Distorted images are "fisheye" distorted images taken with a GoPro camera (compressed to 352×352 pixels). Figure 6 (b) Corrected images are the results after correction using this method. It can be seen that this method has good generalization performance and can be directly applied to the correction of GoPro camera images.

[0074] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A Unet network-based image geometric distortion correction method, characterized in that, The method comprises the following steps: A distortion correction model is constructed based on an Unet network; The dataset is constructed by selecting place365 images and setting the generated checkerboard images as image data sources. The image data sources are preprocessed, radial distortion and tangential distortion are considered, and according to the mathematical relationship formula of the actual image point p(x, y) in the physical coordinate system and the ideal non-distortion image point , a distortion image and its corresponding coordinate difference vector are generated as a dataset; The distortion correction model is set according to the parameter configuration of the experimental environment, and the distortion correction model is trained and tested through the constructed data set, image features are extracted, mapping from the distortion image domain to the corrected coordinate domain is learned, and the trained distortion correction model is saved; Actual data sets are collected, actual distortion images are input into the distortion correction model, and a coordinate difference vector of the distortion image is predicted and obtained; A corrected image is resampled according to the predicted coordinate difference vector and the actual distortion image through an image resampling algorithm. 2.The Unet network-based image geometric distortion correction method according to claim 1, characterized in that, The image data source is specifically: The place365 image data set is selected, and the image is cropped to 512*512 pixels in size; The image is screened by using a Hough transform, and the screening condition is defined as: ∑(l>α)≥λ×C In the formula, l represents the length of the detected straight line, α is the length of the shortest straight line segment, λ is a selection factor of the data set, and C is the resolution size of the image. α is selected as 80, λ is selected as 4, and a certain number of place365 images are screened; Different grid sizes and image rotation angles are set to diversify the features of the checkerboard, and a certain number of checkerboard images are generated; The screened place365 images and the generated checkerboard images constitute the image data source. 3.The Unet network-based image geometric distortion correction method according to claim 2, characterized in that, The construction of the data set is specifically: Camera calibration is performed on different modes of the camera to obtain different camera intrinsic parameters and distortion parameters; According to the mathematical relationship formula of the actual image point p(x, y) and the ideal non-distorted image point under the physical coordinate system, the distorted image data set I with radial and tangential mixed distortion and its coordinate difference data set V are generated for each image data source wherein r 2 = x 2 + y 2 , k1, k2, k3 are radial distortion coefficients; p1, p2 are tangential distortion coefficients; the coordinate difference dataset V records the difference between the pixel point p(x, y) in the distorted image under the pixel coordinate system and the pixel point in the undistorted image corresponding thereto. An image of 352*352 pixels in size and a coordinate difference vector are cropped at the center of the image as the final data set. 4.The Unet network-based image geometric distortion correction method of claim 1, characterized in that: The constructed distortion model comprises A compression path, which encodes the image and is composed of a convolution layer with a size of 3*3, a ReLU function and a pooling layer with a size of 2*2, performs 4 times of downsampling operation on the input image, and the size of the feature map is reduced and the number of channels is doubled after each downsampling operation; An expansion path, which decodes the image, realizes upsampling through 4 times of deconvolution layers with a size of 2*2, expands the size of the feature map, and restores to the size of the input image; A skip connection, which fuses high-level features in the decoding layer and low-level features in the encoding layer, fuses abstract features and detailed features. 5.The Unet network-based image geometric distortion correction method according to claim 1, characterized in that: The coordinate difference loss function in the distortion correction model is optimized The true value vector V corresponding to the coordinate difference vector pred The Euclidean distance of each pixel point to the coordinate (0, 0) of the image as a weight index; The Euclidean distance is normalized to obtain a weight vector M, M has and the predicted coordinate difference vector V pred The same shape; In the formula, G is a normalization function; The coordinate difference loss function is defined as In the formula, n is the total number of coordinate difference vector elements, and M is a weight vector. 6.The Unet network-based image geometric distortion correction method according to claim 5, characterized in that: The resampling loss function L2 is set for the distortion correction model, representing the corrected image I obtained by resampling the predicted coordinate difference vector pred The pixel-by-pixel Euclidean distance between the non-distorted image I gt is defined as In the formula, n represents the number of pixels; The final loss function is: L=L1+μL2 Wherein, μ is a weight, which provides a trade-off between the coordinate difference prediction and the resampling loss.

Images