A perspective transformation method for lens distortion images based on deep learning

Abstract

The present invention relates to the field of image processing, and more specifically to a deep learning-based perspective transformation method for lens-distorted images. The method comprises the following steps: using ResNet to generate a correction image stream to correct the distorted image, and using elastic telescopic convolution to generate an interpolation region and complete the interpolation based on the distortion coefficient #imgabs0# of the distortion stream; matching feature points of the corrected image with an image at a specified angle using a SuperPoint network; and using the RANSAC algorithm to calculate a homography matrix and convert the perspective angle of the corrected image to the specified angle. The present invention can effectively correct distortion in an image and convert the corrected image to the specified perspective angle, ensuring the accuracy and consistency of the image information.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method for perspective transformation of lens deformation and distortion images based on deep learning. Background Art

[0002] Due to the characteristics of camera lenses such as wide-angle lenses and fisheye lenses, the captured images are deformed and distorted. This deformation and distortion poses a huge challenge to the accurate extraction and analysis of information in the image. Especially in some key application scenarios, such as autonomous driving, drone navigation, security monitoring, etc., we have extremely high requirements for the accuracy and consistency of image information, and the problem of lens deformation and distortion has become a technical problem that needs to be solved urgently. Traditional image correction methods usually rely on geometric models and physical parameters. These methods require precise measurement and calibration of the intrinsic and extrinsic parameters of the camera. The operation is complicated and has poor adaptability to environmental changes. In recent years, with the rapid development of deep learning technology, image processing methods based on deep learning have gradually become a research hotspot due to their outstanding performance in processing complex visual tasks. The present invention is committed to proposing a method for correcting lens deformation and distorted images and performing perspective transformation, aiming to overcome these challenges and improve image information. Summary of the Invention

[0003] This application provides a lens deformation distorted image perspective transformation method based on deep learning, which can effectively correct the distortion in the image and convert the corrected image to a specified perspective angle to ensure the accuracy and consistency of the image information.

[0004] To solve the above problems, this application provides the following solutions:

[0005] R1: Use ResNet to generate a rectified image stream to correct the distorted image, and according to the distortion coefficient of the distorted stream , generate the interpolation area and complete the interpolation through elastic scaling convolution;

[0006] R2: Match feature points between the corrected image and the image at the specified angle through the SuperPoint network;

[0007] R3: Use the RANSAC algorithm to calculate the homography matrix and transform the perspective angle of the corrected image to the specified angle.

[0008] Wherein, step R1 is specifically as follows:

[0009] R1.1 Input the distorted image into the ResNet model and output the corrected image stream of the distorted image;

[0010] R1.2 applies the rectified image flow to the distorted image to obtain the rectified image;

[0011] R1.3 Calculation of distortion coefficient of distorted flow ;

[0012] R1.4 When When the colored pixels of the image become sparse, elastic scaling convolution is used. Get the interpolation area and perform interpolation.

[0013] Furthermore, step R1.1 is specifically as follows:

[0014] The corrected image flow refers to two matrices with the same length and width as the distorted image, namely the horizontal corrected image flow matrix and the vertical corrected image flow matrix;

[0015] The transversely rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located The pixel point on the horizontal axis after correction is ;

[0016] The longitudinally rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located The pixel point on the vertical axis after correction is .

[0017] Furthermore, step R1.3 is specifically as follows:

[0018] The distortion coefficient of the distortion flow The calculation method is as follows:

[0019] R1.3.1 Defining the Error Function Using the Least Squares Method

[0020]

[0021] in is the width of the image, is the height of the image, is the distortion function;

[0022] The distortion function steps are as follows:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] The distortion function input is , the output is and ,in, and are the coordinates of the image center, and is the normalized coordinate, which converts the coordinate point into a coordinate system with the center of the image as the origin. Yes Normalized distance from the image center, is the distortion factor, which increases with the distance from the center increases with the increase of

[0029] R1.3.2 Solve the error function smallest value.

[0030] Furthermore, step R1.4 is specifically as follows:

[0031] R1.4.1 Generate a grayscale image of the rectified image;

[0032] R1.4.2 To ensure the existence of a center point, the convolution kernel size at any position is an odd number;

[0033] R1.4.3 In position The convolution kernel size The Euclidean distance from the pixel to the center of the image There is a positive correlation;

[0034] The value of the convolution kernel R1.4.4 is set as follows:

[0035] by As the radius, around the coordinate Form a circle, when the convolution kernel If the position is not within this circular range, the convolution kernel The value of the position is 0, otherwise it is greater than 0;

[0036] R1.4.5 According to the above convolution kernel settings, for each pixel in the grayscale image Perform convolution operation. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels.

[0037] R1.4.6 For each pixel in the grayscale image, determine whether its grayscale value is equal to 0. If it is equal to 0, it belongs to the colorless pixel area;

[0038] R1.4.7 Perform an AND operation on the area surrounded by colored pixels and the area surrounded by colorless pixels to obtain an interpolated area;

[0039] R1.4.8 Apply linear interpolation to the interpolation region of the rectified image to obtain the corrected image.

[0040] Compared with the existing technology, the above technical solution provides a lens distortion image perspective transformation method based on deep learning, which has the following beneficial effects:

[0041] 1. This method uses elastic scaling convolution to determine the enclosing area of ​​colored pixels, making the edges more closely aligned with the original image. Since image points become more dispersed closer to the edge, elastic scaling convolution can provide a wider receptive field to handle these situations.

[0042] 2. This method uses a circular convolution kernel to construct the receptive field, which can retain some sharp corner features of the rectified image, reduce the diffusion area, and ensure image fineness;

[0043] 3. This method uses ResNet to generate a rectified image stream and interpolate the identified interpolation area, which can effectively correct the distortion in the image;

[0044] 4. This method uses the advanced SuperPoint network and the highly robust RANSAC algorithm to perform perspective conversion, which makes the entire method more stable and feasible, and ensures the accuracy and consistency of image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of a method for perspective transformation of lens deformation and distortion images based on deep learning in an embodiment of the present invention.

[0047] Figure 2 In the embodiment of the present invention, ResNet is used to generate a corrected image stream to correct the distorted image, and the distortion coefficient of the distorted stream is used , a schematic diagram of generating the interpolation area and completing the interpolation through elastic scaling convolution.

[0048] Figure 3 This figure is a schematic diagram of using an elastically retractable convolution kernel to perform a convolution operation in an embodiment of the present invention.

[0049] Figure 4Schematic diagram of matching feature points between a corrected image and an image at a specified angle using a SuperPoint network in an embodiment of the present invention.

[0050] Figure 5 This is a schematic diagram of using the RANSAC algorithm to calculate the homography matrix and converting the perspective angle of the corrected image to a specified angle in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] An embodiment of the present application provides a method for perspective transformation of lens distortion images based on deep learning. The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or are inherent to these processes, methods, products, or devices.

[0053] like Figure 1 As shown in the embodiment of the present application, an embodiment of a lens deformation distorted image perspective transformation method based on deep learning is shown, and the specific steps are as follows:

[0054] R1. The embodiment of the present invention uses ResNet to generate a correction image stream to correct the distorted image, and according to the distortion coefficient of the distortion stream , generate the interpolation area and complete the interpolation through elastic scaling convolution, refer to Figure 2 ;

[0055] R1.1 The embodiment of the present invention inputs a distorted image into the ResNet model and outputs a corrected image stream of the distorted image. The corrected image stream refers to two matrices of the same length and width as the distorted image, namely the horizontal corrected image stream matrix and the vertical corrected image stream matrix. The horizontal corrected image stream matrix In the middle Position elements Indicates that before correction, it was originally located The pixel point on the horizontal axis after correction is , the longitudinal rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located The pixel point on the vertical axis after correction is ;

[0056] R1.2 applies the rectified image flow to the distorted image to obtain the rectified image;

[0057] R1.3 Calculation of the distortion coefficient of the distorted flow according to the embodiment of the present invention

[0058] The distortion coefficient of the distortion flow The calculation method is as follows:

[0059] R1.3.1 Defining the Error Function Using the Least Squares Method

[0060]

[0061] in is the width of the image, is the height of the image, is the distortion function;

[0062] The distortion function steps are as follows:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] The distortion function input is , the output is and ,in, and are the coordinates of the image center, and is the normalized coordinate, which converts the coordinate point into a coordinate system with the center of the image as the origin. Yes Normalized distance from the image center, is the distortion factor, which increases with the distance from the center increases with the increase of

[0069] R1.3.2 Solve the error function Minimum distortion coefficient of distortion flow Value, in some optional schemes, the maximum value method is BFGS;

[0070] R1.4 When When the colored pixels of the image become sparse, elastic convolution is used to obtain the interpolation area and perform interpolation. Figure 3

[0071] R1.4.1 Generate a grayscale image of the rectified image;

[0072] R1.4.2 To ensure the existence of a center point, the convolution kernel size of the elastic convolution at any position is an odd number;

[0073] R1.4.3 Convolution Kernel Size in the Embodiments of the Present Invention The Euclidean distance from the pixel to the center of the image There is a positive correlation, in some optional solutions, the and The relationship is:

[0074]

[0075] in is the longest Euclidean distance from a pixel in the image to the center of the image, and it is easy to get , For the scaling ratio, when The bigger, The easier it is, the bigger it is. is the minimum convolution kernel size;

[0076] R1.4.4 The convolution kernel value of the embodiment of the present invention is set as follows:

[0077] by As the radius, around the coordinate Form a circle, when the convolution kernel If the position is not within this circular range, the convolution kernel The value of the position is 0, otherwise it is greater than 0. In some optional solutions, when The position is within this circular range, and the convolution kernel The value at the position is 1;

[0078] R1.4.5 According to the above convolution kernel settings, for each pixel in the grayscale image Perform convolution operation. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels.

[0079] R1.4.6 For each pixel in the grayscale image, determine whether its grayscale value is equal to 0. If it is equal to 0, it belongs to the colorless pixel area;

[0080] R1.4.7 Perform an AND operation on the area surrounded by colored pixels and the area surrounded by colorless pixels to obtain an interpolated area;

[0081] R1.4.8 Apply linear interpolation to the interpolation region of the rectified image to obtain the corrected image.

[0082] R2: Use the SuperPoint network to match the feature points of the corrected image with the specified angle image, refer to Figure 4 .

[0083] R3: Use the RANSAC algorithm to calculate the homography matrix and convert the perspective angle of the corrected image to the specified angle, refer to Figure 5 .

[0084] The above describes the embodiments of the present invention in detail with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. After knowing the contents described in the present invention, ordinary technicians in this technical field can make several equivalent changes and substitutions without departing from the principles of the present invention. These equivalent changes and substitutions should also be regarded as falling within the scope of protection of the present invention.

Claims

1. A lens distortion image perspective transformation method based on deep learning, characterized in that: The method for perspective transformation of lens deformation-distorted images based on deep learning includes: ResNet is used to generate a correction image stream to correct the distorted image. When the distortion coefficient k of the distortion stream is greater than 0, an interpolation region is generated by elastic scaling convolution according to the distortion coefficient k of the distortion stream and the interpolation is completed. The elastic scaling convolution generates an interpolation region and completes the interpolation, wherein the elastic scaling convolution generates an interpolation region and completes the interpolation, including: converting the corrected image into a grayscale image; increasing the convolution kernel size of the elastic scaling convolution at any position. Set to an odd number to ensure that the convolution kernel has a center point; ensure the convolution kernel size of the elastic convolution The Euclidean distance d from the pixel to the center of the image x,y The process of setting the convolution kernel value of the elastic convolution is as follows: The value of is used as the radius, and a circle is obtained around the point with coordinates (x, y). When the position (i, j) in the convolution kernel is not within the range of this circle, the value at the position (i, j) of the convolution kernel is 0, otherwise it is greater than 0; according to the above convolution kernel setting, a convolution operation is performed on each pixel (x, y)∈w×h in the grayscale image. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels; determine whether the grayscale value of each pixel in the grayscale image is equal to 0. If it is equal to 0, it belongs to the colorless pixel area, otherwise it does not belong; perform an AND operation on the area surrounded by colored pixels and the colorless pixel area to obtain an interpolation area; use linear interpolation on the interpolation area of ​​the corrected image to obtain the corrected image; The SuperPoint network is used to match the feature points of the corrected image with the existing image at a specified angle; The RANSAC algorithm is used to calculate the matched feature points to obtain the homography matrix, and the homography matrix is ​​used to convert the perspective angle of the corrected image to the specified angle.

2. The method for perspective transformation of lens distortion images based on deep learning according to claim 1, characterized in that: The distorted image is corrected by using ResNet to generate a corrected image stream. The correction steps are as follows: Input the distorted image into the ResNet model and output the corrected image stream of the distorted image; Apply the rectified image flow to the distorted image to obtain the rectified image; The corrected image flow refers to two matrices with the same length and width as the distorted image, namely the horizontal corrected image flow matrix and the vertical corrected image flow matrix; The horizontally rectified image flow matrix D h The element at position (x,y) in Indicates the horizontal axis position of the pixel originally located at (x, y) before correction The longitudinal rectified image flow matrix D v The element at position (x,y) in The pixel point originally located at (x, y) before correction is located at the vertical axis position after correction.

3. The method for perspective transformation of lens distortion images based on deep learning according to claim 2, characterized in that: The calculation method of the distortion coefficient k of the distorted flow is as follows: Use the least squares method to define the error function and find the k value that minimizes the error function E(k); The error function is defined as follows: Where w is the width of the image, h is the height of the image, and distortion is the distortion function; The distortion function steps are as follows: factor=1+k·r 2 x d =x c +x n ·factor·x c ,and d =and c +y n ·factor·y c The distortion function input is x, y, k, and the output is x d and y d , where the range of the horizontal coordinate x is [1, w], the range of the vertical coordinate y is [1, h], and x c and y c is the coordinate of the center of the image, x n and y n is the normalized coordinate, which converts the coordinate point to a coordinate system with the center of the image as the origin, r is the normalized distance of the point (x, y) from the center of the image, and factor is the distortion factor, which increases as the distance r from the center increases.

Images