Distortion image correction method based on fast multi-scale residual network

Abstract

The invention discloses a distorted image correction method based on a fast multi-scale residual network. Relates to the field of image geometric distortion correction, in particular to the technical field of distortion image correction based on a fast multi-scale residual network. Comprising the following steps: S1, constructing a data set, and generating a paired data set of a distorted image and a standard image and a deformation displacement field file of a corresponding number of the distorted image by using a geometric distortion model; s2, a distorted image correction model based on a fast multi-scale residual network is constructed, the model is divided into a backbone network and a tributary network, the backbone network is used for recovering image features, and the tributary network is used for outputting image classification features; s3, training the distorted image correction model based on the fast multi-scale residual network by using the pairwise data set of the distorted image and the standard image and the deformation displacement field file; and S4, processing the to-be-corrected image by using the trained distortion image correction model based on the fast multi-scale residual network to obtain the corrected image and the corresponding classification thereof.

Description

Technical Field

[0001] The present invention relates to the field of image geometric distortion correction, and in particular to the technical field of distorted image correction based on a fast multi-scale residual network. Background Art

[0002] Geometric distortion is a common problem in digital image processing, which affects image quality in many areas. Its causes include the acquisition system (such as optical lenses, imaging sensors), the imaging environment (such as platform or target motion, observation geometry), and image processing operations (such as image warping). For example, camera lenses are often distorted by optical aberrations, barrel distortion (image magnification changes with distance from the optical axis) in wide-angle lenses, or pincushion distortion (when using a telephoto lens or the telephoto end of a zoom lens). Although lens distortion is an inherent problem of the camera, external factors can also cause geometric distortion, such as rotation, shear, and perspective distortion, which are usually caused by improper camera posture or movement. In addition, image processing tools may also introduce distortion effects such as waveform distortion.

[0003] There are several types of existing image correction methods:

[0004] Traditional image correction methods: such as boundary detection, baseline detection, connected domain segmentation and other image processing methods to achieve image correction. Such algorithms can only process slightly distorted images or certain types of distortion and are not adaptive.

[0005] Deep learning algorithm based on automatic calibration: Take multiple images while the camera is moving, use the algorithm to automatically calibrate the camera parameters, and correct image distortion in real time without human intervention. It can handle various types of distortion more efficiently and accurately, but it requires multiple images and can only be performed when the camera is moving.

[0006] Correction technology based on fisheye lens images: It is a method for dealing with the wide-angle distortion problem unique to fisheye lenses. A calibration plate is usually used to calculate the distortion parameters (including radial distortion coefficient and tangential distortion coefficient). The distortion parameters obtained by calibration are used to correct the image using a distortion model. This method is often used in 360-degree panoramic photography, autonomous driving and other fields, but there are still problems such as low correction accuracy, distortion or blurring of the image edge areas, inability to process in real time, and the robustness of the algorithm in complex scenes. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a distorted image correction method based on a fast multi-scale residual network, and the method comprises the following steps:

[0008] S1. Dataset construction: by using the geometric distortion model, a paired dataset of distorted images and standard images is generated, as well as a deformation displacement field file corresponding to the number of the distorted image;

[0009] S2. Construct a distorted image correction model based on a fast multi-scale residual network. The model is divided into a trunk network and a tributary network. The trunk network is used to restore image features, and the tributary network is used to output image classification features.

[0010] S3, using the paired data set of the distorted image and the standard image and the deformation displacement field file to train the distorted image correction model based on the fast multi-scale residual network;

[0011] S4. Process the image to be corrected using the trained distorted image correction model based on the fast multi-scale residual network to obtain a corrected image and its corresponding classification.

[0012] Furthermore, the geometric distortion model includes a barrel distortion model, a pincushion distortion model, a perspective distortion model, a rotation distortion model, a shear distortion model and a waveform distortion model.

[0013] Furthermore, the backbone network includes, from input to output, a 3×3 convolutional layer, 5 FMRB and LCAB module groups, and a model fitting and resampling unit, wherein the output end of the 3×3 convolutional layer is connected to the input end of the fifth FMRB and LCAB module group through a residual connection, and the output end of the first FMRB and LCAB module group is connected to the input end of the fourth FMRB and LCAB module group through a residual connection.

[0014] Furthermore, the tributary network is a derived network of the trunk network, the input end of the tributary network is connected to the output end of the second FMRB and LCAB module group in the trunk network, and the output end of the tributary network is connected to the model fitting and resampling unit in the trunk network; the tributary network includes two 3×3 convolutional layers and one fully connected layer from input to output.

[0015] Furthermore, the data processing flow of the FMRB module in the FMRB and LCAB module group is as follows: the original input data first passes through a 1×1 convolutional layer, and is divided into two data paths after output. One data path is input into a 1×5 convolutional layer, and the other data path is input into a 1×3 convolutional layer. The data output by the two convolutional layers are added together, and the added data is divided into two paths again. One data path is input into a 1×5 convolutional layer, and the other data path is input into a 1×3 convolutional layer. The data output by the two convolutional layers are added together again, and the obtained data is added to the original input data. The added data is input into a 1×1 convolutional layer and then output.

[0016] Furthermore, the data processing flow of the LCAB module in the FMRB and LCAB module group is as follows: the original input data first passes through the global average pooling layer, and the output enters the fully connected layer, and then enters the 1×1 convolutional layer after the output from the fully connected layer. The data output from the 1×1 convolutional layer is processed by Sigmoid, and then the processed data is added to the original input data and output.

[0017] The beneficial effects of the method of the present invention are:

[0018] The FMRB module is designed to better extract the global and local semantic information of the image and expand the network receptive field. The module has a carefully designed residual structure, and its topological reasoning speed is only equivalent to the largest scale convolution kernel (1×5 convolution) in the structure.

[0019] The LCAB module focuses on global channel information, effectively integrates feature semantics and adds nonlinear mapping (Sigmoid), which accelerates the convergence of the network on the one hand; on the other hand, it prevents the problem of gradient disappearance in deep networks and better retains feature information.

[0020] The dual optimization of EPE loss and cross entropy loss enables the network to focus not only on the accuracy of image correction but also on the classification accuracy of image content.

[0021] Innovation of fast and lightweight network architecture: Lightweight is reflected in two aspects. First, the careful residual structure design fully integrates multi-grained image information and accelerates the topological reasoning of the overall network. On the other hand, from the perspective of module design, the fast multi-scale residual structure of the present invention carefully uses rectangular convolution to extract image features, which reduces the amount of image parameters on the one hand and reduces the amount of image reasoning computation through re-parameterization on the other hand. For lightweight channel attention, compared with most cumbersome attention modules, the design of the present invention is extremely simplified, with low computational efficiency, while focusing on the fusion of global information.

[0022] Strong algorithm generalization ability: The algorithm can handle various types of geometric distortions, including barrel distortion, pincushion distortion, perspective distortion, rotation distortion, shear distortion, and waveform distortion. This shows that the algorithm has strong generalization ability and can adapt to different distortion conditions. At the same time, it provides a new type of distortion mapping model, which can further construct a new data set and realize more types of distorted image correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the method described in an embodiment of the present invention;

[0024] Figure 2 This is a structural diagram of a distorted image correction model based on a fast multi-scale residual network according to an embodiment of the present invention;

[0025] Figure 3 The FMRB structure diagram of the embodiment of the present invention;

[0026] Figure 4 LCAB structure diagram described in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0028] Embodiment 1,

[0029] This embodiment provides a method for correcting distorted images based on a fast multi-scale residual network. Figure 1 As shown, the method comprises the following steps:

[0030] S1. Dataset construction: by using the geometric distortion model, a paired dataset of distorted images and standard images is generated, as well as a deformation displacement field file corresponding to the number of the distorted image;

[0031] S2. Construct a distorted image correction model based on a fast multi-scale residual network. The model is divided into a trunk network and a tributary network. The trunk network is used to restore image features, and the tributary network is used to output image classification features.

[0032] S3, using the paired data set of the distorted image and the standard image and the deformation displacement field file to train the distorted image correction model based on the fast multi-scale residual network;

[0033] S4. Process the image to be corrected using the trained distorted image correction model based on the fast multi-scale residual network to obtain a corrected image and its corresponding classification.

[0034] Embodiment 2,

[0035] This embodiment is a further limitation of Embodiment 1. In step S1, the dataset used in constructing the dataset is the dataset in Places: A 10 Million Image Database for Scene Recognition.

[0036] In step S1, in order to achieve the generalization of data processing, a data set is constructed starting from six types of basic camera distortion models. Six types of distortion: barrel distortion, pincushion distortion, perspective distortion, rotational distortion, shear distortion and waveform distortion. Through multiple types of geometric distortion models, paired data sets of distorted images and standard images are generated, and at the same time, corresponding numbered deformation displacement field files are constructed, named in the form of distortion + number. During the generation process, if there are problems such as inconsistent size / blank in the image size, the integer grid and bilinear interpolation method are used to calibrate the distorted image pixels to maintain its mapping relationship with the standard image. This method is not only applicable to given geometric distortion models, but can also be effectively expanded through new data distortion types.

[0037] The barrel-shaped and pincushion-shaped distortion models both belong to radial distortion models, and their model formulas are:

[0038]

[0039] (x 0 ,y 0 ): represents the coordinates of the center point of the image, that is, the original optical axis center; (x u ,y u ): represents the pixel coordinates in the uncorrected image;

[0040] (x dis ,y dis ): represents the actual output coordinates in the corrected image;

[0041] The adjustment coefficient indicating the degree of distortion;

[0042] Output: represents the coefficient of radial distortion.

[0043] The model formula of the rotation distortion model is:

[0044]

[0045] (x dis ,y dis ): represents the new pixel position in the rotated image;

[0046] (x u ,y u ): represents the pixel coordinates in the image before rotation;

[0047] θ ro : The angle by which the image is rotated (in radians);

[0048] ω: The horizontal coordinate used to calculate the center of the image;

[0049] (sin(θ ro ), cos(θro )): Used to rotate the pixel coordinates around the center point; the model formula of the wave distortion model is:

[0050]

[0051] (x dis ,y dis ): represents the pixel coordinates in the distorted image;

[0052] (x u ,y u ): represents the pixel coordinates in the uncorrected image;

[0053] θ shear : Shear coefficient, used to control the degree of wave deformation in the horizontal direction;

[0054] ω: represents the wave frequency or period;

[0055] The formula of the shear distortion model is:

[0056]

[0057] (x u ,y u ): represents the pixel coordinates in the uncorrected image;

[0058] (x dis ,y dis ): represents the pixel coordinates in the distorted image;

[0059] θ mag : represents the magnification factor, which controls the amplitude of shear distortion;

[0060] W: represents the image width, which determines the period length of the fluctuation;

[0061] The model formula of the perspective distortion model is:

[0062]

[0063] (i m , j m ): represents the pixel index in the original image;

[0064] (x dis ,y dis ): represents the pixel coordinates in the distorted image;

[0065] A ij : Represents the elements of the perspective transformation matrix, which is used to control the perspective effect of the image;

[0066] W, H: represent the width and height of the image respectively.

[0067] Embodiment 3,

[0068] This embodiment is a further limitation of Embodiment 1. Figure 2 The figure shows a schematic diagram of a distorted image correction model based on a fast multi-scale residual network. The model is divided into a backbone network and a tributary network. The backbone network is used to restore image features, and the tributary network is used to output image classification features. The backbone network includes, from input to output, a 3×3 convolutional layer, 5 FMRB and LCAB module groups, and a model fitting and resampling unit, wherein the output end of the 3×3 convolutional layer is connected to the input end of the fifth FMRB and LCAB module group through a residual connection, and the output end of the first FMRB and LCAB module group is connected to the input end of the fourth FMRB and LCAB module group through a residual connection.

[0069] The tributary network is a derived network of the trunk network, the input end of the tributary network is connected to the output end of the second FMRB and LCAB module group in the trunk network, and the output end of the tributary network is connected to the model fitting and resampling unit in the trunk network; the tributary network includes two 3×3 convolutional layers and one fully connected layer from input to output.

[0070] In the FMRB and LCAB module groups, the FMRB module is a self-designed fast multi-scale residual block (Fast Multi-scale Residual Block), and the LCAB is a self-designed lightweight channel attention module (Lightweight Channel Attention Block). The structure of the FMRB module is as follows: Figure 3 As shown in , the data processing flow is as follows: the original input data first passes through the 1×1 convolution layer, and is divided into two data paths after output. One data path is input into the 1×5 convolution layer, and the other data path is input into the 1×3 convolution layer. The data output from the two convolution layers are added together, and the added data is divided into two paths again. One data path is input into the 1×5 convolution layer, and the other data path is input into the 1×3 convolution layer. The data output from the two convolution layers is added together again, and the obtained data is added to the original input data. The added data is input into the 1×1 convolution layer and then output. The structure of the LCAB module is as shown in Figure 4 The data processing flow is as follows: the original input data first passes through the global average pooling layer, and then enters the fully connected layer after output, and then enters the 1×1 convolutional layer after output from the fully connected layer. The data output from the 1×1 convolutional layer is processed by Sigmoid, and then the processed data is added to the original input data and output.

[0071] In the model fitting unit, resampling is used. Resampling is the process of correcting the image based on the calculated forward distortion flow. It is a key step to ensure that the intermediate representation of the network output can be effectively converted into the final high-quality corrected image. It applies the features or flow mapping predicted by the network to the original image and generates a visually coherent corrected image by adjusting pixel positions and interpolation.

[0072] The specific steps are as follows:

[0073] (1) At the beginning, each pixel in the corrected image is temporarily assumed to be at its initial position in the distorted image. This initial position is directly obtained based on the deformation parameters of the corresponding single coordinate axis with the same number.

[0074] (2) Use an iterative search method to locate the point of the original image and use an iterative formula to estimate the deviation of the current pixel.

[0075] The iteration formula is:

[0076] p(i+1)=qf(p(i))

[0077] Among them, p(i+1) represents the position of point p after update at the i+1th iteration, q represents the position of the target point in the corrected image, p(i) represents the position of point p at the i-th iteration, and f() represents the forward flow of calculation from source pixel p to the undistorted image.

[0078] The meaning of the iterative formula is the difference between the current position of the current pixel and the correct position. According to the calculated deviation, the position of the pixel is adjusted to move it to the correct position. This adjustment is done step by step, and the distance of each movement is determined by the size of the deviation. Repeat the iterative calculation, perform calculations and position adjustments multiple times. After each iteration, the position of the pixel will be closer to its correct position in the original image.

[0079] (3) Continue iterating until the position adjustment becomes very small or the preset convergence condition is reached. At this time, it is considered that the pixel has reached the most accurate position.

[0080] (4) Once all pixels have been iteratively adjusted and reached the most accurate position, the resampling process of the entire image is completed and the corrected image is generated.

[0081] In order to make the model loss function of the distorted image correction model based on the fast multi-scale residual network as simple as possible, the commonly used EPE loss and cross entropy loss functions are used to define the overall network training. It is mainly aimed at the classification network and the distortion parameter estimation network. The overall loss is defined as follows:

[0082] Among them, l paramrepresents the EPE loss function, l class Represents the cross entropy loss function, λ represents the weight of the overall loss, which is a compromise between EPE loss and cross entropy loss; Among them, (u r , v r ) represents the true value, (u, v) represents the predicted value; Where n represents the number of categories, p(x i ) represents the probability of the i-th class.

[0083] The data processing flow in the distorted image correction model based on the fast multi-scale residual network is as follows: starting from the input image, the image is uniformly cropped to a size of 512×512. First, the image is converted into 64 feature maps through a 3x3 convolution layer. Next, the network is processed in sequence by multiple fast multi-scale residual blocks (FMRB) and lightweight channel attention blocks (LCAB) to further extract image features. The FMRB module enhances the depth and feature retention capabilities of the network through the residual connection mechanism, ensuring the effective transmission of gradients in the deep network and preventing information loss. The LCAB module introduces the attention mechanism, which selectively enhances important feature channels and suppresses unimportant channels by calculating the importance weight of each channel. The network gradually increases the number of channels of the feature map from 64 to 512 at each stage, and finally compresses the high-dimensional features into low-dimensional representations through the fully connected layer (FC). Finally, the network is divided into two branches: the backbone network learns the displacement field from the distorted image to the normal image for model fitting, and the branch network learns the displacement field classification for category classification.

[0084] To further optimize and generate the final output. Overall, the network combines the residual network structure and the attention mechanism to effectively extract and enhance image features, thereby improving the accuracy of image processing and classification.

Claims

1. A method for distorted image correction based on a fast multi-scale residual network, characterized in that: The method comprises the following steps: S1. Dataset construction: by using the geometric distortion model, a paired dataset of distorted images and standard images is generated, as well as a deformation displacement field file corresponding to the number of the distorted image; S2. Construct a distorted image correction model based on a fast multi-scale residual network. The model is divided into a trunk network and a tributary network. The trunk network is used to restore image features, and the tributary network is used to output image classification features. S3, using the paired data set of the distorted image and the standard image and the deformation displacement field file to train the distorted image correction model based on the fast multi-scale residual network; S4. Process the image to be corrected using the trained distorted image correction model based on the fast multi-scale residual network to obtain a corrected image and its corresponding classification.

2. The method for distorted image correction based on a fast multi-scale residual network according to claim 1, characterized in that: The geometric distortion model includes a barrel distortion model, a pincushion distortion model, a perspective distortion model, a rotation distortion model, a shear distortion model and a waveform distortion model.

3. The method for distorted image correction based on a fast multi-scale residual network according to claim 2, characterized in that: The backbone network includes, from input to output, a 3×3 convolutional layer, 5 FMRB and LCAB module groups, and a model fitting and resampling unit, wherein the output end of the 3×3 convolutional layer is connected to the input end of the fifth FMRB and LCAB module group through a residual connection, and the output end of the first FMRB and LCAB module group is connected to the input end of the fourth FMRB and LCAB module group through a residual connection.

4. The method for distorted image correction based on a fast multi-scale residual network according to claim 3, characterized in that: The tributary network is a derived network of the trunk network, the input end of the tributary network is connected to the output end of the second FMRB and LCAB module group in the trunk network, and the output end of the tributary network is connected to the model fitting and resampling unit in the trunk network; the tributary network includes two 3×3 convolutional layers and one fully connected layer from input to output.

5. The method for distorted image correction based on a fast multi-scale residual network according to claim 3, characterized in that: The data processing flow of the FMRB module in the FMRB and LCAB module group is as follows: the original input data first passes through a 1×1 convolutional layer, and is divided into two data paths after output. One path of data is input into a 1×5 convolutional layer, and the other path of data is input into a 1×3 convolutional layer. The data output by the two convolutional layers are added together, and the added data is divided into two paths again. One path of data is input into a 1×5 convolutional layer, and the other path of data is input into a 1×3 convolutional layer. The data output by the two convolutional layers are added together again, and the obtained data is added to the original input data. The added data is input into a 1×1 convolutional layer and then output.

6. The method for distorted image correction based on a fast multi-scale residual network according to claim 3, characterized in that: The data processing flow of the LCAB module in the FMRB and LCAB module group is as follows: the original input data first passes through the global average pooling layer, and then enters the fully connected layer after output. After the output from the fully connected layer, it enters the 1×1 convolutional layer. The data output from the 1×1 convolutional layer is processed by Sigmoid, and then the processed data is added to the original input data and output.

Images