Deep learning-based lens deformation and distortion image perspective transformation method

Abstract

The invention relates to the field of image processing, in particular to a lens deformation distorted image perspective transformation method based on deep learning, which comprises the following steps of: generating a correction image stream by using ResNet to correct a distorted image, and generating an interpolation region and completing interpolation through elastic telescopic convolution according to a distortion coefficient # imgabs0 # of a distortion stream; carrying out feature point matching on the corrected image and an image at a specified angle through a SuperPoint network; and calculating a homography matrix by using an RANSAC (Random Sample Consensus) algorithm, and converting the perspective angle of the corrected image into a specified angle. According to the invention, distortion in the image can be effectively corrected, the corrected image is converted to a specified perspective angle, and the accuracy and consistency of image information are ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a lens deformation and distortion image perspective transformation method based on deep learning. Background Art

[0002] Due to the characteristics of camera lenses such as wide-angle lenses and fisheye lenses, the captured images are deformed and distorted. This deformation and distortion poses a huge challenge to the accurate extraction and analysis of information in the image. Especially in some key application scenarios, such as autonomous driving, drone navigation, security monitoring, etc., we have extremely high requirements for the accuracy and consistency of image information, and the problem of lens deformation and distortion has become a technical problem that needs to be solved urgently. Traditional image correction methods usually rely on geometric models and physical parameters. These methods require precise measurement and calibration of the intrinsic and extrinsic parameters of the camera, which are complex to operate and have poor adaptability to environmental changes. In recent years, with the rapid development of deep learning technology, image processing methods based on deep learning have gradually become a research hotspot due to their outstanding performance in processing complex visual tasks. The present invention is committed to proposing a method for correcting lens deformation and distorted images and performing perspective transformation, aiming to overcome these challenges to improve image information. Summary of the invention

[0003] The present application provides a lens deformation distorted image perspective transformation method based on deep learning, which can effectively correct the distortion in the image and convert the corrected image to a specified perspective angle to ensure the accuracy and consistency of the image information.

[0004] To solve the above problems, this application provides the following solutions: R1: Use ResNet to generate a rectified image stream to correct the distorted image, and use the distortion coefficient of the distorted stream , generate the interpolation area and complete the interpolation through elastic scaling convolution; R2: Match the feature points of the corrected image with the image at the specified angle through the SuperPoint network; R3: Use the RANSAC algorithm to calculate the homography matrix and transform the perspective angle of the corrected image to the specified angle.

[0005] Wherein, step R1 is specifically as follows: R1.1 Input the distorted image into the ResNet model and output the corrected image stream of the distorted image; R1.2 Apply the corrected image flow to the distorted image to obtain the corrected image; R1.3 Calculation of distortion coefficient of distorted flow ; R1.4 When When the colored pixels of the image become sparse, elastic scaling convolution is used. Get the interpolation area and perform interpolation.

[0006] Furthermore, step R1.1 is specifically as follows: The corrected image flow refers to two matrices of the same length and width as the distorted image, namely a horizontal corrected image flow matrix and a vertical corrected image flow matrix; The lateral correction image flow matrix In the middle Position elements Indicates that before correction, it was originally located at The pixel point on the horizontal axis after correction is ; The longitudinally rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located at The pixel point on the vertical axis after correction is .

[0007] Furthermore, step R1.3 is specifically as follows: The distortion coefficient of the distorted flow The calculation method is as follows: R1.3.1 Defining the error function using the least squares method in is the width of the image, is the height of the image, is the distortion function; The distortion function steps are as follows: The distortion function input is , the output is and ,in, and are the coordinates of the center of the image, and is the normalized coordinate, which transforms the coordinate point into a coordinate system with the center of the image as the origin. Yes The normalized distance from the center of the image, is the distortion factor, which increases with the distance from the center increases with the increase of R1.3.2 Solve the error function Smallest value.

[0008] Furthermore, step R1.4 is specifically as follows: R1.4.1 Generate a grayscale image of the rectified image; R1.4.2 To ensure the existence of a center point, the size of the convolution kernel at any position is an odd number; R1.4.3 In position The convolution kernel size The Euclidean distance from the pixel to the center of the image There is a positive correlation; R1.4.4 The value of the convolution kernel is set as follows: by As the radius, around the coordinate Form a circle, when the convolution kernel If the position is not within this circular range, the convolution kernel The value of the position is 0, otherwise it is greater than 0; R1.4.5 According to the above convolution kernel settings, for each pixel in the grayscale image Perform a convolution operation. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels. R1.4.6 For each pixel in the grayscale image, determine whether its grayscale value is equal to 0. If it is equal to 0, it belongs to the colorless pixel area; R1.4.7 Perform an AND operation on the area surrounded by colored pixels and the area surrounded by colorless pixels to obtain an interpolation area; R1.4.8 Apply linear interpolation to the interpolation area of ​​the corrected image to obtain the corrected image.

[0009] Compared with the prior art, the above technical solution provides a lens deformation distorted image perspective transformation method based on deep learning, which has the following beneficial effects: 1. This method uses elastic scaling convolution to determine the surrounding area of ​​colored pixels, so that the edge is more closely aligned with the original image. Since the closer to the edge, the more scattered the image points are, elastic scaling convolution can provide a wider receptive field to handle these situations; 2. This method uses a circular convolution kernel to construct the receptive field, which can retain some sharp-angle features of the corrected image, reduce the diffusion area, and ensure the fineness of the image; 3. This method uses ResNet to generate a rectified image stream and interpolate the identified interpolation area, which can effectively correct the distortion in the image; 4. This method uses the advanced SuperPoint network and the highly robust RANSAC algorithm to perform perspective conversion, which can make the entire method more stable and feasible, and ensure the accuracy and consistency of image information. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 This is a flowchart of a method for perspective transformation of a lens deformation and distorted image based on deep learning in an embodiment of the present invention.

[0012] Figure 2 In the embodiment of the present invention, ResNet is used to generate a corrected image stream to correct the distorted image, and the distortion coefficient of the distorted stream is , schematic diagram of generating the interpolation region and completing the interpolation through elastic scaling convolution.

[0013] Figure 3 This is a schematic diagram of using an elastically retractable convolution kernel to perform a convolution operation in an embodiment of the present invention.

[0014] Figure 4 It is a schematic diagram of matching feature points between a corrected image and an image at a specified angle through a SuperPoint network in an embodiment of the present invention.

[0015] Figure 5 This is a schematic diagram of using the RANSAC algorithm to calculate the homography matrix and converting the perspective angle of the corrected image to a specified angle in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The embodiment of the present application provides a perspective transformation method for lens deformation distorted images based on deep learning. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0018] like Figure 1 As shown, an embodiment of a lens deformation distorted image perspective transformation method based on deep learning in an embodiment of the present application, the specific steps are as follows: R1. The embodiment of the present invention uses ResNet to generate a corrected image stream to correct the distorted image, and according to the distortion coefficient of the distorted stream , generate the interpolation area and complete the interpolation through elastic scaling convolution, refer to Figure 2 ; R1.1 In this embodiment of the present invention, a distorted image is input into the ResNet model, and a corrected image stream of the distorted image is output. The corrected image stream refers to two matrices of the same length and width as the distorted image, namely, a horizontal corrected image stream matrix and a vertical corrected image stream matrix. In the middle Position elements Indicates that before correction, it was originally located at The pixel point on the horizontal axis after correction is , the longitudinal rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located at The pixel point on the vertical axis after correction is ; R1.2 Apply the corrected image flow to the distorted image to obtain the corrected image; R1.3 Calculation of the distortion coefficient of the distorted flow according to the embodiment of the present invention The distortion coefficient of the distorted flow The calculation method is as follows: R1.3.1 Defining the error function using the least squares method in is the width of the image, is the height of the image, is the distortion function; The distortion function steps are as follows: The distortion function input is , the output is and ,in, and are the coordinates of the center of the image, and is the normalized coordinate, which transforms the coordinate point into a coordinate system with the center of the image as the origin. Yes The normalized distance from the center of the image, is the distortion factor, which increases with the distance from the center increases with the increase of R1.3.2 Solve the error function Minimum distortion coefficient of distortion flow Value, in some optional schemes, the method for finding the maximum value is BFGS; R1.4 When When the colored pixels of the image become sparse, elastic convolution is used to obtain the interpolation area and perform interpolation. Figure 3 R1.4.1 Generate a grayscale image of the rectified image; R1.4.2 To ensure the existence of a center point, the convolution kernel size of the elastically retractable convolution at any position is an odd number; R1.4.3 Convolution kernel size of the embodiment of the present invention The Euclidean distance from the pixel to the center of the image is positively correlated, in some optional schemes, the and The relationship is:

[0019] in is the longest Euclidean distance from a pixel in the image to the center of the image, and it is easy to get , For the scaling ratio, The bigger, The easier it is, the bigger it is. is the minimum convolution kernel size; R1.4.4 The value of the convolution kernel of the embodiment of the present invention is set as follows: by As the radius, around the coordinate Form a circle, when the convolution kernel If the position is not within this circular range, the convolution kernel The value of the position is 0, otherwise it is greater than 0. In some optional schemes, when The position is within this circular range, and the convolution kernel The value at the position is 1; R1.4.5 According to the above convolution kernel settings, for each pixel in the grayscale image Perform a convolution operation. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels. R1.4.6 For each pixel in the grayscale image, determine whether its grayscale value is equal to 0. If it is equal to 0, it belongs to the colorless pixel area; R1.4.7 Perform an AND operation on the area surrounded by colored pixels and the area surrounded by colorless pixels to obtain an interpolation area; R1.4.8 Apply linear interpolation to the interpolation area of ​​the corrected image to obtain the corrected image.

[0019] R2: Use the SuperPoint network to match the feature points of the corrected image with the image at the specified angle, refer to Figure 4 .

[0020] R3: Use the RANSAC algorithm to calculate the homography matrix and convert the perspective angle of the corrected image to the specified angle, refer to Figure 5 .

[0021] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. For ordinary technicians in this technical field, after knowing the contents recorded in the present invention, they can make several equivalent changes and substitutions without departing from the principle of the present invention. These equivalent changes and substitutions should also be regarded as belonging to the protection scope of the present invention.

Claims

1. A lens deformation image perspective transformation method based on deep learning, characterized in that: The method for perspective transformation of lens deformation distorted images based on deep learning includes: Use ResNet to generate a corrected image stream to correct the distorted image, and use the distortion coefficient of the distortion stream , generate the interpolation area and complete the interpolation through elastic scaling convolution; The SuperPoint network is used to match the feature points of the corrected image with the existing image at a specified angle; The RANSAC algorithm is used to calculate the matched feature points to obtain the homography matrix, and the homography matrix is ​​used to convert the perspective angle of the corrected image to the specified angle.

2. The lens deformation distorted image perspective transformation method based on deep learning according to claim 1, characterized in that: The correction steps of using ResNet to generate a correction image stream to correct the distorted image are as follows: Input the distorted image into the ResNet model and output the corrected image stream of the distorted image; Apply the rectified image flow to the distorted image to obtain the rectified image; The corrected image flow refers to two matrices of the same length and width as the distorted image, namely a horizontal corrected image flow matrix and a vertical corrected image flow matrix; The lateral correction image flow matrix In the middle Position elements Indicates that before correction, it was originally located at The pixel point is at the horizontal axis position after correction ; The longitudinally rectified image flow matrix In the middle Position elements Indicates that before correction, it was originally located at The pixel point on the vertical axis after correction is .

3. The lens deformation distorted image perspective transformation method based on deep learning according to claim 2, characterized in that: The calculation of the distortion flow distortion coefficient Calculation method, the calculation method is as follows: Use the least squares method to define the error function and solve it so that the error function Smallest value; The error function is defined as follows: in is the width of the image, is the height of the image, is the distortion function; The distortion function steps are as follows: The distortion function input is , the output is and , where the horizontal axis The value range is [1, w ] , vertical coordinate The value range is [1, h] , and are the coordinates of the center of the image, and is the normalized coordinate, which transforms the coordinate point into a coordinate system with the center of the image as the origin. Yes The normalized distance from the center of the image, is the distortion factor, which increases with the distance from the center increases with the increase of .

4. The method for perspective transformation of lens deformation distorted images based on deep learning according to claim 3, characterized in that: The interpolation region is generated and interpolation is completed by elastic scaling convolution, as detailed below: when When , the colored pixels of the image will become sparse, and elastic scaling convolution is used to obtain the interpolation area and perform interpolation. The steps are: Convert a color rectified image into a grayscale image; The convolution kernel size of the elastic convolution at any position Set it to an odd number to ensure that the convolution kernel has a center point; Ensure the convolution kernel size of the elastic scaling convolution The Euclidean distance from the pixel to the center of the image There is a positive correlation; The process of setting the value of the convolution kernel of the elastic scaling convolution is as follows: The value of is taken as the radius, and the coordinates are The points of the convolution kernel are If the position is not within this circular range, the convolution kernel The value of the position is 0, otherwise it is greater than 0; According to the above convolution kernel settings, for each pixel in the grayscale image Perform a convolution operation. If the convolution result meets the threshold condition, the pixel belongs to the area surrounded by colored pixels. Determine whether the gray value of each pixel in the grayscale image is equal to 0. If it is equal to 0, it belongs to the colorless pixel area, otherwise it does not belong to it; Perform an AND operation on the area surrounded by colored pixels and the area surrounded by colorless pixels to obtain an interpolation area; Linear interpolation is applied to the interpolation area of ​​the rectified image to obtain the corrected image.

Images