A method for non-linear distortion correction of a fish-eye lens

Abstract

The present invention discloses a method for correcting nonlinear distortion of a fisheye lens, relating to the field of image processing technology. The method comprises: first, establishing a nonlinear spatial spherical projection model of the fisheye lens, and solving the model using differential equations to obtain the nonlinear distortion amount of the fisheye image; second, obtaining the original fisheye image, and proposing a fast pole location algorithm to extract the effective area of the image to obtain an effective area image; then, establishing a coordinate system based on the effective area image to obtain the pixel coordinate length ratio; and finally, proposing a correction algorithm based on the pixel coordinate length ratio to reconstruct the corrected image using the distortion amount. The present invention helps retain more complete detail information, significantly improves structural similarity and target recognition accuracy, and applies basic mathematics to solve practical engineering problems, bringing greater practical value to fisheye lens vision technology.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for correcting nonlinear distortion of a fisheye lens. Background Art

[0002] Fisheye cameras offer a wide field of view, several times larger than that of conventional cameras. They have direct or indirect applications in remote sensing, autonomous driving, intelligent surveillance, military aircraft guidance, space robot inspection, machine vision navigation, industrial pipeline detection, and field inspection. Therefore, fisheye lens technology has become a key branch of computer vision and computer graphics. The compression of the field of view by fisheye lenses also causes distortion in fisheye images, affecting the stability and accuracy of subsequent visual recognition systems, including image classification, object detection, segmentation, and localization. Furthermore, fisheye lens technology is of great significance in research related to stereo matching, 3D reconstruction, and autonomous robot navigation and tracking.

[0003] Currently, fisheye image distortion correction methods primarily include those based on camera calibration, projection models, and deep learning. Camera calibration methods require a relatively complex mathematical model of the lens, rely on internal camera parameters, and are not universally applicable. Projection model methods are based on several common projection models, but actual fisheye lens designs do not strictly adhere to these models, resulting in certain correction errors. Widely used and highly regarded correction methods based on longitude coordinates and longitude / latitude mapping are more accurate and faster than previous methods, but they introduce stretching distortion at the edges of the corrected image. Deep learning-based correction methods have been proposed in recent years. While they offer accurate correction results, they also involve complex distortion models and require large datasets. Eliminating the drawbacks of fisheye lens imaging distortion and leveraging its wide field of view can not only enhance the practical value of fisheye lens technology but also provide more effective solutions for the field of computer vision.

[0004] Therefore, a fisheye lens nonlinear distortion correction method is proposed to solve the problems of complex model of existing fisheye image distortion correction technology, reliance on camera internal parameters, and stretching deformation of the image edge. This is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a method for correcting nonlinear distortion of a fisheye lens, which can achieve the effect of quickly and accurately correcting fisheye images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for correcting nonlinear distortion of a fisheye lens comprises the following steps:

[0008] S1. Establish a nonlinear spatial spherical projection model of the fisheye lens, and use differential equations to solve the model to obtain the nonlinear distortion of the fisheye image;

[0009] S2, obtaining the original fisheye image, proposing a fast pole location algorithm to extract the effective area of the image, and obtaining an effective area image;

[0010] S3. Establish a coordinate system based on the effective area image to obtain the pixel coordinate length ratio;

[0011] S4. A correction algorithm based on pixel coordinate length ratio is proposed, and the correction image is reconstructed with the help of the distortion obtained in S1.

[0012] The above method optionally includes: introducing the concept of image pixel level per millimeter, quantizing the original fisheye image, and converting the pixel value of the original fisheye image and the length of the image at the resolution.

[0013] In the above method, optionally, the step of obtaining the nonlinear distortion amount in S1 includes: locally decomposing the model and approximating the curve segments with tangent segments; and combining the number of pixels per millisecond to derive the number of pixels of the bottom surface segment of the model corresponding to the number of pixels of the surface segment, reflecting the nonlinear distortion amount.

[0014] Optionally, in the above method, S2 specifically includes the following steps:

[0015] S2.1. Obtain a threshold value of the grayscale image using the Otsu algorithm, and use the threshold value to binarize the original fisheye image to obtain a binary image;

[0016] S2.2, perform morphological processing on the binary image;

[0017] S2.3. Using the image region attribute measurement function, set a polygonal bounding box and calculate the coordinates of the four vertices of the circular region in the fisheye image.

[0018] S2.4. The return value is the position and size of the minimum bounding box, which determines the boundary vertex coordinates, center point and radius of the effective area of the fisheye image to achieve effective area extraction.

[0019] Optionally, in the above method, S2.2 specifically includes: creating a disk-shaped structure element, specifying the number of line structure elements for approximating the disk shape, and performing a morphological operation of dilation on the binary image using the obtained number of structure elements.

[0020] Optionally, the above method S3 specifically includes the following steps:

[0021] S3.1. Establish a joint coordinate system of the uv pixel coordinate system and the xoy image coordinate system;

[0022] S3.2. Taking the ideal image as the starting point, calculate the distance from the coordinates of each pixel in the ideal image to the center of the circle;

[0023] S3.3. Calculate the length ratios of all ideal point coordinates and the pixel coordinates corresponding to the ideal points.

[0024] In the above method, optionally, the pixel coordinate length ratio calculation formula in S3.3 is:

[0025]

[0026] Among them, t is the pixel coordinate length ratio, x0 is the horizontal coordinate of the center point, u is the horizontal coordinate of the ideal point, and R is the image radius.

[0027] In the above method, optionally, the correction algorithm based on pixel coordinate length ratio in S4 is a method of fusing trigonometric functions with a bilinear interpolation method combined with a reverse mapping method.

[0028] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a method for correcting nonlinear distortion of a fisheye lens, which has the following beneficial effects:

[0029] (1) The present invention establishes a feasible spatial spherical model for the fisheye lens to characterize its nonlinear distortion law, and solves the model based on the geometric characteristics of calculus. It does not require the internal and external parameters of the fisheye lens and a large amount of data sets, which greatly improves the universality and accuracy of the correction algorithm.

[0030] (2) A method for extracting effective regions based on fast pole location is proposed, in which a dilation morphological operation is used, which is beneficial to dilate the bright white area in the image by adding pixels to the perceived boundary of the object in the image. When using an approximation method for the structural element, the morphological operation using disk approximation runs faster. The above improvements improve the accuracy and speed of effective region extraction.

[0031] (3) The concept of pixel coordinate length ratio is defined to replace the tangent slope at a certain point of the curve to express the growth rate of the distortion. This optimizes the correction effect when it is close to the tangent gradient, solves the problem that the slope changes faster as it approaches the x-axis, and improves the stretching deformation phenomenon at the edge of the image after correction by the traditional correction algorithm.

[0032] (4) After correction, the image detail information is retained more completely, the target recognition accuracy is significantly improved, and the overall visual effect of the image is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0034] Figure 1 This is a flow chart of a method for correcting nonlinear distortion of a fisheye lens disclosed in the present invention;

[0035] Figure 2 This is a diagram of the nonlinear spatial spherical projection model of the fisheye lens disclosed in this embodiment;

[0036] Figure 3 A partial exploded view of the spherical projection model disclosed in the present invention

[0037] Figure 4 The effective area extraction flow chart disclosed in the present invention;

[0038] Figure 5 This is a schematic diagram of the interpolation implementation of the correction algorithm disclosed in the present invention;

[0039] Figure 6 This is an example correction diagram disclosed in the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] In this application, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0042] Reference Figure 1As shown, the present invention discloses a method for correcting nonlinear distortion of a fisheye lens, comprising the following steps:

[0043] S1. Establish a nonlinear spatial spherical projection model of the fisheye lens, and use differential equations to solve the model to obtain the nonlinear distortion of the fisheye image;

[0044] S2, obtaining the original fisheye image, proposing a fast pole location algorithm to extract the effective area of the image, and obtaining an effective area image;

[0045] S3. Establish a coordinate system based on the effective area image to obtain the pixel coordinate length ratio;

[0046] S4. A correction algorithm based on pixel coordinate length ratio is proposed, and the correction image is reconstructed with the help of the distortion obtained in S1.

[0047] Furthermore, S1 specifically includes: introducing the concept of pixel level per millimeter of the image, quantizing the original fisheye image, and converting the pixel value of the original fisheye image and the length of the image at the resolution.

[0048] Specifically, by studying the optical principles of the fisheye lens, its nonlinear projection law was discovered. Based on this, the fisheye lens was simulated as a semi-sphere in space, and the image imaging plane was simulated as the bottom surface of the fisheye lens, and a nonlinear spatial spherical projection model of the fisheye lens was established.

[0049] Furthermore, S1 specifically includes: introducing the concept of pixel level per millimeter of the image, quantizing the original fisheye image, and converting the pixel value of the original fisheye image and the length of the image at the resolution.

[0050] Specifically, refer to Figure 2 and Figure 3 As shown, based on the geometric properties of calculus, the nonlinear space spherical projection model is locally decomposed. Δx is set to 1 mm. When Δx is small enough, Δx-dy is a high-order infinitesimal of Δx. Therefore, near a point P in the image, a tangent segment can be used to approximate the curve segment. Combined with the number of pixels per millimeter, the number of pixels on the line segment |Q'Q| corresponds to the number of pixels on |P'P|, which reflects the amount of nonlinear distortion and is expressed as |P'P|-|Q'Q|. The specific calculation method is as follows:

[0051]

[0052]

[0053]

[0054] |OO'|=cosα·|PP'|;

[0055] Where P and P' are two points on the lens model arc, let P'(x,y), P(x+Δx,y+Δy), and α be the angle between the radial distance from the ideal object point in space to the origin of the camera coordinate system and the radial distance from the distorted image point in the distorted image to the origin of the image coordinate system.

[0056] Further, refer to Figure 4 As shown, S2 specifically includes the following steps:

[0057] S2.1. Obtain a threshold value of the grayscale image using the Otsu algorithm, and use the threshold value to binarize the original fisheye image to obtain a binary image;

[0058] S2.2, perform morphological processing on the binary image;

[0059] S2.3. Using the image region attribute measurement function, set a polygonal bounding box and calculate the coordinates of the four vertices of the circular region in the fisheye image.

[0060] S2.4. The return value is the position and size of the minimum bounding box, which determines the boundary vertex coordinates, center point and radius of the effective area of the fisheye image to achieve effective area extraction.

[0061] Furthermore, S2.2 specifically includes: creating a disk-shaped structure element, specifying the number of line structure elements for approximating the disk shape, and performing a morphological operation of dilation on the binary image using the obtained number of structure elements.

[0062] Further, refer to Figure 5 As shown, S3 specifically includes the following steps:

[0063] S3.1. Establish a joint coordinate system of the uv pixel coordinate system and the xoy image coordinate system;

[0064] S3.2. Taking the ideal image as the starting point, calculate the distance from the coordinates of each pixel in the ideal image to the center of the circle;

[0065] S3.3. Calculate the length ratios of all ideal point coordinates and the pixel coordinates corresponding to the ideal points.

[0066] Specifically, the pixel coordinate length ratio uses the ratio of the horizontal coordinate length to the radius at a certain point on the image as a variable, replacing the tangent slope at a certain point on the spherical curve to represent the growth rate of the distortion amount.

[0067] Furthermore, the calculation formula for the pixel coordinate length ratio in S3.3 is:

[0068]

[0069] Among them, t is the pixel coordinate length ratio, x0 is the horizontal coordinate of the center point, u is the horizontal coordinate of the ideal point, and R is the image radius.

[0070] Furthermore, the correction algorithm based on pixel coordinate length ratio in S4 adopts trigonometric function fusion bilinear interpolation method combined with reverse mapping method.

[0071] Specifically, first use the calculated result of the pixel coordinate length ratio t, and with the help of the angle variable β reflecting the increase in the tangent slope, calculate the distance r from the distortion point to the center of the image circle d As well as the distortion variable σ, the angle variable β, calculate the distance r from the distortion point to the center of the image d And the expression of distortion σ is:

[0072] β=arctan(t)

[0073] r d = r·cosβ

[0074] σ=rr d ,

[0075] Among them, r is the distance from the pixel coordinate to the center of the circle o(x0,y0), and the expression is

[0076] Secondly, calculate the angle between the coordinates of the ideal point and the horizontal coordinate Angle The expression is as follows:

[0077]

[0078] Then use trigonometric functions to calculate the distance r from the distortion point to the origin d The relationship with the coordinates of the distortion point is as follows:

[0079]

[0080] Then, the coordinates of the distortion point corresponding to the ideal point can be obtained by combining the above formulas. The expression is as follows:

[0081]

[0082] Finally, the mapping formula is processed segment by segment, and the expression is as follows:

[0083]

[0084] The ideal image has been reconstructed by interpolation, and the correction of a single fisheye image is completed. At the same time, the present invention is suitable for processing a large number of fisheye images and processing video streams shot by fisheye cameras.

[0085] In one specific embodiment

[0086] In order to verify the fisheye image distortion correction effect of the present invention, fisheye images from the Internet and the public dataset VOC-360 were selected for correction experiments. Figure 6 As shown, the experimental results show that the fisheye image correction algorithm of the present invention is more accurate in extraction at the edge of the image, does not cut off the effective information of the image at the edge, and the contour details in the image are relatively complete. The present invention effectively improves the visual effect of the corrected image, solves the problem of poor image edge structure, reduces pixel loss and algorithm complexity, retains more detail information, and improves the running speed of the correction algorithm. A target detection method based on a convolutional neural network is used to identify pedestrian targets in the corrected image, and the target recognition accuracy, structural similarity and peak signal-to-noise ratio are used as evaluation indicators for the fisheye image correction effect. The accuracy of multi-scale pedestrian target recognition can reach 99%, and it also has obvious advantages in the quantitative evaluation of structural similarity and peak signal-to-noise ratio. Moreover, this method is not limited by the fisheye camera model and the original image size, has good universality, has strong practical engineering application value, and can be applied to a variety of computer vision tasks.

[0087] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0088] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for correcting nonlinear distortion of a fisheye lens, characterized in that: The following steps are involved: S1. Establish a nonlinear spatial spherical projection model of the fisheye lens, and use differential equations to solve the model to obtain the nonlinear distortion of the fisheye image; S2. Obtain the original fisheye image, propose a fast pole location algorithm to extract the effective area of the image, and obtain the effective area image, including: S2.1 Use The algorithm obtains the threshold of the grayscale image and uses the threshold to perform binarization processing on the original fisheye image to obtain a binary image; S2.2, perform morphological processing on the binary image; S2.

3. Using the image region attribute measurement function, set a polygonal bounding box and calculate the coordinates of the four vertices of the circular region in the fisheye image. S2.

4. Return the position and size of the minimum bounding box to determine the boundary vertex coordinates, center point, and radius of the effective area of the fisheye image, and realize effective area extraction. S3. Establish a coordinate system based on the effective area image and obtain the pixel coordinate length ratio. The calculation formula is: ； in, t is the pixel coordinate length ratio, x 0 is the horizontal coordinate of the center point of the circle, u is the horizontal coordinate of the ideal point, R is the image radius; S4, propose a correction algorithm based on pixel coordinate length ratio, and reconstruct the corrected image with the help of the distortion obtained in S1. The correction algorithm based on pixel coordinate length ratio adopts trigonometric function fusion bilinear interpolation method combined with reverse mapping method.

2. The method for correcting nonlinear distortion of a fisheye lens according to claim 1, wherein: S1 specifically includes: introducing the concept of pixel level per millimeter of the image, quantizing the original fisheye image, and converting the pixel value of the original fisheye image and the length of the image at the resolution.

3. The method for correcting nonlinear distortion of a fisheye lens according to claim 1, wherein: The steps of obtaining the nonlinear distortion amount in S1 include: locally decomposing the model and approximating the curve segments with tangent segments; and combining the number of pixels per millisecond to determine the number of pixels of the bottom surface segment of the model corresponding to the number of pixels of the surface segment, reflecting the nonlinear distortion amount.

4. The method for correcting nonlinear distortion of a fisheye lens according to claim 1, wherein: S2.2 specifically includes: creating a disk-shaped structure element, specifying the number of line structure elements used to approximate the disk shape, and performing a morphological operation of dilation on the binary image using the obtained number of structure elements.

5. The method for correcting nonlinear distortion of a fisheye lens according to claim 1, wherein: S3 specifically includes the following steps: S3.

1. Establish a joint coordinate system of the uv pixel coordinate system and the xoy image coordinate system; S3.

2. Taking the ideal image as the starting point, calculate the distance from the coordinates of each pixel in the ideal image to the center of the circle; S3.

3. Calculate the length ratios of all ideal point coordinates and the pixel coordinates corresponding to the ideal points.

Images