A semi-automatic fish-eye lens calibration method and system
By utilizing orthogonal parallel line pairs in natural scenes and extracting curves using neural networks, combined with principal line calculation and error function optimization, fisheye lens calibration without a calibration plate was achieved. This solves the problems of limited calibration scenarios and poor flexibility in existing technologies, and enables efficient and accurate lens calibration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-02-26
- Publication Date
- 2026-05-29
AI Technical Summary
Existing fisheye lens calibration methods rely on calibration boards, which have limitations in calibration scenarios and lack of flexibility, making it difficult to meet the calibration needs of assembled or complex environments.
A semi-automatic fisheye lens calibration method is adopted, which utilizes orthogonal parallel line pairs in natural scenes and pre-trained neural networks to extract curves. Combining principal line calculation and least squares solution, the focal length and distortion coefficient are optimized through the error function, thus achieving calibration without a calibration plate.
It enables efficient and accurate calibration of fisheye lenses in both pre-assembled and complex scenarios, improving the scenario adaptability and accuracy of calibration.
Smart Images
Figure CN122115585A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fisheye camera intrinsic parameter calibration technology, and more specifically to a semi-automatic fisheye lens calibration method and system. Background Technology
[0002] Currently, fisheye lenses, as optical lenses with an ultra-wide field of view, are widely used in various fields such as autonomous driving, robot navigation, surveillance, security, and augmented reality, thanks to their ability to capture vast scene information. In these applications, images captured by fisheye lenses inevitably suffer from severe radial distortion. This distortion causes the shape and position of objects in the image to deviate from the actual scene, directly affecting the accuracy of subsequent data processing such as image measurement, target recognition, and scene reconstruction. Therefore, accurate calibration of the fisheye lens to obtain its intrinsic parameters and distortion coefficients is a crucial prerequisite for eliminating image distortion and ensuring the reliability of subsequent applications.
[0003] In fisheye lens calibration technology, calibration methods relying on calibration boards are currently the most widely used traditional calibration schemes. These methods typically require the pre-fabrication of a standard calibration board with a specific pattern (such as a checkerboard or dot array). Images of the calibration board are captured at different poses using the fisheye lens. Then, based on the known coordinates of feature points on the calibration board and the pixel coordinates of corresponding feature points in the images, a mathematical model is established to solve for the lens parameters. For example, the Zhang Zhengyou calibration method, a classic planar calibration method, is widely used in the calibration of conventional lenses and fisheye lenses. Its core idea is to utilize the two-dimensional planar features provided by the checkerboard calibration board to solve for parameters through feature point matching from multiple viewpoint images.
[0004] However, the above-mentioned fisheye lens calibration method relying on calibration plates has many limitations in practical applications and is difficult to meet the calibration needs of some special scenarios: First, the calibration scenarios are limited. This type of method requires that the calibration environment has sufficient space to set up the calibration plate, and the calibration plate must be completely within the field of view of the lens and maintain clear imaging. For fisheye lenses that have already been assembled or application scenarios with limited space and complex environments, the conditions for setting up the calibration plate are often not met, making it difficult to carry out the calibration work; Second, the calibration flexibility is poor. When it is necessary to recalibrate fisheye lenses in different installation positions or different application scenarios, the calibration plate must be carried and set up repeatedly, which has poor adaptability.
[0005] Therefore, how to calibrate a fisheye lens without a calibration plate is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] In view of the above problems, the present invention proposes a semi-automatic fisheye lens calibration method and system that overcomes or at least partially solves the above problems.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a semi-automatic fisheye lens calibration method, specifically including the following steps: S1. Obtain multiple calibration images using a fisheye camera, wherein the calibration images include N pairs of orthogonal parallel lines; wherein N is greater than or equal to 2; S2. Extract curves from the calibration image using a pre-trained neural network; S3. Based on the calibration image after curve extraction, select two pairs of orthogonal parallel straight lines; S4. Based on the calibration image after selecting orthogonal pairs of parallel lines, calculate the orthogonal vanishing points of the parallel lines in the orthogonal pairs of parallel lines, and calculate the principal line equation based on the pairs of orthogonal vanishing points. S5. Based on the principal equation, calculate the principal point position, where the principal point position is the coordinate of the intersection of the fisheye lens optical axis and the imaging plane; S6. Based on the orthogonal parallel line pairs, select sampling points, obtain corrected image points according to the calibration model and the principal point position, establish an error function through the image points and the focal length and distortion coefficient to be optimized, optimize the curve to be optimized by minimizing the global error function, and obtain the optimized focal length and distortion coefficient.
[0008] Further, in step S2, the neural network adapts to the calibration image using a fisheye lens imaging model and calculates the corresponding offset of the convolution kernel; The input to the neural network is the captured fisheye image and the field of view of the fisheye lens, and the output of the neural network is the Bézier curve parameters and confidence level.
[0009] Furthermore, the specific process by which the neural network adapts to the calibration image using a fisheye lens imaging model and calculates the corresponding offset of the convolutional kernel is as follows: S21. Based on the fisheye lens imaging model, construct a virtual imaging sphere for the fisheye lens, calculate the incident angle of each pixel using a double-precision model, and establish the correspondence between the imaging plane and the virtual imaging sphere:
[0010] In the formula, Represents a point in three-dimensional space. Represents image points, and They represent direction and Pixel coordinates of direction r Represents the radius of the virtual imaging sphere; Image points in convolutional kernels at different locations are projected onto the virtual imaging sphere, and the sampling direction is calculated. , :
[0011] In the formula, This represents the position of the center of the virtual imaging sphere, which is the origin in the virtual imaging sphere coordinate system. n This indicates that in the imaging virtual spherical coordinate system, The direction vector, that is, the direction vector of the direction; S22, along the sampling direction on the virtual sphere. , Resample, backproject all sampled points onto the imaging plane, and calculate the relative offset between the sampled points and all points in the convolution kernel:
[0012] In the formula, This represents the discrete offset of the sampling position relative to the center of the convolution kernel. express The unit offset of resampling in the direction. express The unit offset of resampling in the direction. T Represents the transpose function; S23. Construct a fisheye image adaptation layer based on the offset. In the curve extraction network, use the adaptation layer to replace the traditional convolution. The replacement position is located in front of the feature map.
[0013] Furthermore, the specific process of step S4 is as follows: S41. Based on the selected pair of orthogonal parallel lines, calculate the intersection point position of the parallel lines in the calibration image using the endpoints of the parallel lines in the pair of orthogonal parallel lines. The intersection point position is the coordinate of the orthogonal vanishing point. S42. Obtain at least two pairs of orthogonal vanishing points through at least two pairs of orthogonal parallel lines. The secondary expression of the orthogonal vanishing point in each pair of orthogonal vanishing points is as follows: , , and ,in, and This forms a pair of orthogonal vanishing points. and This forms another pair of orthogonal vanishing points. Calculate the direction of the principal line:
[0014] Intercept of the main line:
[0015] S43. The equation of the principal line is obtained based on the direction and intercept of the principal line: .
[0016] Furthermore, in step S5, the specific process of calculating the principal point position is as follows: Estimate the homogeneous coordinates of the principal point using the least squares method.
[0017]
[0018] In the formula, , .
[0019] Furthermore, the specific process of step S6 is as follows: S61. Uniformly sample along the selected pairs of orthogonal parallel lines. N l sampling points For the sampling points The corrected image points are obtained based on the calibration model. ; S62, Select from the calibration image M The straight line, the first The set of sampling points of a straight line on an image is represented as Define the point after distortion removal Construct the error function for a single straight line:
[0020] In the formula, ,for A straight line is used to construct the global objective function:
[0021] In the formula, , Indicates the first The weight of a line in the global objective function is linearly related to the line length. Indicates the first The length of a straight line is approximately equal to the distance between its endpoints.
[0022] In the formula, Indicates the starting sampling point. This indicates the final sampling point.
[0023] Furthermore, the positions of the sampling points on the virtual imaging sphere on the tangent plane of the virtual imaging sphere are as follows:
[0024] In the formula, , This represents the angular resolution per unit pixel.
[0025] In a second aspect, embodiments of the present invention provide a semi-automatic fisheye lens calibration system; comprising: The acquisition module acquires multiple calibration images using a fisheye camera, and the calibration images include at least two sets of orthogonal pairs of parallel lines. The curve extraction module extracts curves from the calibration image captured by the fisheye lens using a pre-trained neural network. The data processing module calculates the orthogonal vanishing points of the parallel lines in the orthogonal parallel line pairs based on the calibration image after selecting pairs of orthogonal parallel lines, obtains pairs of orthogonal vanishing points, calculates the principal line equation based on the pairs of orthogonal vanishing points, and calculates the principal point position based on the principal line equation. The parameter optimization module selects sampling points based on orthogonal pairs of parallel lines, obtains corrected image points according to the calibration model and principal point positions, establishes an error function using the image points and the focal length and distortion coefficients to be optimized, and obtains the optimized focal length and distortion coefficients through the error function.
[0026] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a semi-automatic fisheye lens calibration method, which has the following beneficial effects: 1. This invention provides a feature basis for solving fisheye lens parameters by utilizing orthogonal parallel line pairs in natural scenes. At the same time, it combines a semi-automatic mode of neural network curve extraction and manual feature selection to ensure feature reliability. It merges principal line calculation, least squares solution and error function optimization to construct the parameter solution process, thus realizing efficient and accurate calibration of fisheye lenses in assembled or complex scenes.
[0027] 2. This invention utilizes a computer processor to adjust the intrinsic parameters of a fisheye camera, forming a semi-automatic fisheye lens calibration system that does not require a calibration plate, enabling fisheye cameras to have rapid and accurate calibration capabilities in unstructured environments. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0029] Figure 1 This is a flowchart of the semi-automatic fisheye lens calibration method provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the semi-automatic fisheye lens calibration method provided in this embodiment of the invention; Figure 3 This is a schematic diagram illustrating the use of neural networks for curve advance in an embodiment of the present invention; Figure 4 This is a schematic diagram of the imaging principle provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the result of manually selecting a straight line and calculating the principal line in an embodiment of the present invention; Figure 6 This is a schematic diagram of the results of solving multiple principal lines and estimating principal points by least squares in an embodiment of the present invention; Figure 7 This is a structural diagram of the semi-automatic fisheye lens calibration system provided in an embodiment of the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] This invention discloses a semi-automatic fisheye lens calibration method; such as... Figure 1 As shown, it includes the following steps: S1. Obtain multiple calibration images using a fisheye camera, wherein the calibration images include N pairs of orthogonal parallel lines; wherein N is greater than or equal to 2; S2. Extract curves from the calibration image using a pre-trained neural network; S3. Based on the calibration image after curve extraction, select two pairs of orthogonal parallel straight lines; S4. Based on the calibration image after selecting orthogonal pairs of parallel lines, calculate the orthogonal vanishing points of the parallel lines in the orthogonal pairs of parallel lines, and calculate the principal line equation based on the pairs of orthogonal vanishing points. S5. Based on the principal equation, calculate the principal point position, where the principal point position is the coordinate of the intersection of the fisheye lens optical axis and the imaging plane; S6. Based on the orthogonal parallel line pairs, select sampling points, obtain corrected image points according to the calibration model and the principal point position, establish an error function through the image points and the focal length and distortion coefficient to be optimized, optimize the curve to be optimized by minimizing the global error function, and obtain the optimized focal length and distortion coefficient.
[0032] This invention utilizes pairs of orthogonal parallel lines in natural scenes to provide a feature basis for solving fisheye lens parameters. At the same time, it combines a semi-automatic mode of neural network curve extraction and manual feature selection to ensure feature reliability. It merges principal line calculation, least squares solution and error function optimization to construct the parameter solution process, thus realizing efficient and accurate calibration of fisheye lenses in assembled or complex scenes.
[0033] The following is a detailed description of each of the above steps: In step S1, as Figure 2 As shown in the schematic diagram, the fisheye camera to be calibrated is first placed at multiple positions and angles to acquire several images. These images are then filtered based on the following criteria: they must contain at least two pairs of orthogonal parallel lines. Images that meet the filtering criteria are used as the calibration images for the fisheye lens; images that do not meet the requirements are discarded.
[0034] In this embodiment, in order to ensure that some of the acquired images can meet the screening condition of "having at least two pairs of orthogonal parallel lines", multiple positions and angles are used to cover as much scene area as possible. At the same time, the orthogonal parallel line pairs under multiple views are distributed in different imaging areas, which can cover different fields of view of the fisheye lens, avoiding the problem of inaccurate edge distortion correction caused by a single view only covering the central area.
[0035] In step S2, curve extraction is performed on the calibration image obtained from the fisheye image using a neural network; This neural network adapts to fisheye lens images using a calibration-free fisheye lens imaging model, and calculates the corresponding offset of the convolutional kernel in this way. The neural network's input consists of the captured fisheye image and the fisheye lens's field of view; the output is the detected Bézier curve parameters and the confidence score. The output detection result is as follows: Figure 3 As shown; Figure 3 In the process, the model identifies distorted lines caused by fisheye lens photography and labels the line endpoints with red spheres and the lines with blue lines. Step S2 specifically includes: S21. A fisheye lens imaging model is constructed based on the requirement of no calibration and only the field of view angle. This model includes orthographic projection, isometric projection, isometric fixed-angle projection, and stereo projection. Specifically, these four projection models represent different nonlinear mapping laws between the spatial incident angle and the image imaging radius. The orthographic projection model is suitable for imaging scenes with significant edge compression; the isometric projection model describes an ideal distribution where the incident angle and imaging distance have a linear relationship; the isometric fixed-angle projection model focuses on maintaining the consistency of the imaging area ratio; and the stereo projection model can maintain the angular characteristics of the local shape. In practical applications, this method selects one of these models as the basic mapping function based on the nominal type or imaging characteristics of the fisheye lens. This model is then used to establish the geometric correspondence between the two-dimensional image pixel coordinates and the three-dimensional incident light rays. Thus, without the need for complex parameter calibration, coordinate remapping calculation from the distorted image to the target correction plane is achieved through mathematical inverse transformation. Figure 4 As shown, the fisheye lens imaging model describes the angle of incidence. Distance from the imaging point to the imaging center Relationship,
[0036] A virtual imaging sphere for the fisheye lens is constructed based on the imaging model. The incident angle of each pixel is calculated using a dual longitude model, and the correspondence between the imaging plane and the virtual imaging sphere is established.
[0037]
[0038] In the formula, Represents a point in three-dimensional space. Represents image points, and They represent direction and Pixel coordinates of direction This represents the radius of the virtual imaging sphere.
[0039] A virtual imaging sphere is constructed for the fisheye image. Image points from convolutional kernels at different locations are projected onto the virtual imaging sphere, and the sampling direction is calculated. and .
[0040]
[0041] In the formula, This represents the position of the center of the virtual imaging sphere, which is the origin in the virtual imaging sphere coordinate system. n This indicates that in the imaging virtual spherical coordinate system, The direction vector, that is, the direction vector of the direction; S22. Resample on the virtual sphere, keeping the center position of the convolution kernel unchanged, along... and Uniform sampling is performed on the surface of the sphere. For example, for a sampling point on the virtual sphere, its position on the tangent plane of the virtual sphere is denoted as a three-dimensional point. Accordingly, the discrete offset of this sampling position relative to the center of the convolution kernel is denoted as . .
[0042]
[0043] In the formula, , This represents the angular resolution per unit pixel.
[0044] All sampling points are back-projected onto the imaging plane, and the relative offset between the sampling points and all points in the convolution kernel is calculated.
[0045]
[0046] In the formula, This represents the discrete offset of the sampling position relative to the center of the convolution kernel. express The unit offset of resampling in the direction. express The unit offset of resampling in the direction. T Represents the transpose function; S23. Construct a fisheye image adaptation layer based on the offset. In the curve extraction network, use the adaptation layer to replace the traditional convolution. The replacement position is located in front of the feature map.
[0047] In step S3, for the calibration image after curve extraction, manually select two sets of orthogonal pairs of parallel straight lines; In this embodiment, staff manually select orthogonal pairs of parallel lines using a computer, and then annotate them using a labeling tool, recording the information of the selected orthogonal pairs of parallel lines.
[0048] In step S4, for each image where lines have been selected, the orthogonal vanishing points of the parallel lines are calculated, and the principal line equation is calculated. All selected lines and their indices, along with the calculated principal line results, are shown in Figure 5. The numbers within the red boxes in the figure represent the line indices selected in that image, and the green dashed lines represent the calibrated principal lines. The specific process of S4 is as follows: S41. Based on the selected pair of parallel lines in the image, use the endpoints to calculate the intersection point of the parallel lines in the fisheye image, i.e., the coordinates of the vanishing point. S42. For each pair of orthogonal parallel lines, a pair of orthogonal vanishing points can be calculated. Two pairs of orthogonal parallel lines can yield two pairs of orthogonal vanishing points. and This indicates that the homogeneous expression for each vanishing point can be expressed as: ; S43. Using two sets of orthogonal vanishing point pairs, the direction of the main line can be calculated as follows:
[0049] The intercept of the principal line can be calculated as follows:
[0050] At this point, the homogeneous equation of the principal line is .
[0051] In step S5, based on the principal line equation calculated for each calibration image, the principal point positions are calculated using the least squares method; for example... Figure 6 As shown in the figure, the principal lines calculated from multiple calibration images and the principal point positions estimated by the least squares method are illustrated. In the figure, the white dashed lines represent the principal lines that have been calibrated before the current image, and the red pentagram in the center represents the principal point positions estimated by the least squares method. The specific process of step S5 includes: Using the multiple sets of principal direction vectors and intercepts calculated in step S4, the homogeneous coordinates of the camera principal point are estimated using the least squares method.
[0052] In the formula, , .
[0053] In step S6, an error function is established, and the focal length and distortion coefficients are optimized based on the curves in the selected image; the specific process includes: S61. Uniformly sample on the selected orthogonal straight line pairs. There are [number] sampling points. The parameters to be optimized include focal length and distortion coefficients. .in Focal length These are fourth-order distortion coefficients. To address the severe distortion caused by fisheye lenses, this invention employs the Kannala-Brandt projection model (i.e., the KB model) as the calibration benchmark. The specific computational logic of this model is as follows: First, calculate the incident angle of the sampling point relative to the camera's optical axis. This incident angle represents the angle of light rays when the spatial point enters the lens. Secondly, a polynomial adjustment function for the incident angle is constructed using fourth-order distortion coefficients. The original incident angle is then substituted into this polynomial for nonlinear transformation, thereby obtaining the distorted incident angle containing distortion information. This step simulates the degree of radial bending of light as it passes through a fisheye lens. Finally, based on the focal length parameter to be optimized, the calculated distortion incident angle is mapped to the imaging plane through projection relationship, thereby solving the coordinates of the corrected image point corresponding to the sampling point on the image.
[0054] For any sampling point Based on the KB calibration model, the corrected image points can be obtained. , and The relationship can be represented as .
[0055] S62. Selected common data in the calibration image The straight line, the first The set of sampling points of a straight line on an image is represented as Define the point after distortion removal Construct the error function for a single straight line:
[0056] In the formula, ,for A straight line is used to construct the global objective function:
[0057] In the formula, , Indicates the first The length of a straight line is approximately equal to the distance between its endpoints.
[0058] In the formula, Indicates the starting sampling point. This indicates the final sampling point.
[0059] In this embodiment, the curve in the fisheye image is essentially a distorted projection of a straight line in the actual scene. If the focal length and distortion coefficients are accurate, the straight line formed by all sampled points after parameter correction should approximate the actual straight line. Based on geometric constraints, the optimal parameters are solved in reverse by minimizing the "deviation between the corrected sampled points and the ideal straight line".
[0060] Example 1: This example is applied to an indoor unstructured scene and uses a semi-automatic fisheye lens calibration method to calibrate the intrinsic parameters of the fisheye camera. Specifically, this example mainly includes six core parts: obtaining calibration images from multiple angles and positions using the fisheye lens camera; extracting curves from the calibration images using a neural network; manually selecting two sets of orthogonal parallel line pairs; calculating the orthogonal vanishing points of all selected parallel line pairs in each calibration image and calculating the principal line equation; calculating the principal point position using the least squares method based on the principal line equations of multiple calibration images; and estimating the focal length and distortion coefficients simultaneously by constructing an error function.
[0061] Example 2 is applied to an outdoor unstructured scene, employing a semi-automatic fisheye lens calibration method to calibrate the intrinsic parameters of the fisheye camera. Specifically, this example mainly includes six core parts: obtaining calibration images from multiple angles and positions using the fisheye lens camera; extracting curves from the calibration images using a neural network; manually selecting two sets of orthogonal parallel line pairs; calculating the orthogonal vanishing points of all selected parallel line pairs in each calibration image and calculating the principal line equation; calculating the principal point position using the least squares method based on the principal line equations of multiple calibration images; and simultaneously estimating the focal length and distortion coefficient by constructing an error function.
[0062] In summary, this invention addresses the shortcomings of existing fisheye lens calibration methods that do not require a calibration plate in terms of feature reliability and scene adaptability, proposing an improved scheme based on orthogonal line features of natural scenes. This method first extracts all curve features from a neural network constructed using the fisheye image as input, then filters out straight line edge priors that meet certain conditions. A semi-automatic mode with manual assistance is then used to select pairs of orthogonal parallel lines. During feature extraction, the weights of effective geometric priors are dynamically strengthened, thereby preserving the orthogonal constraint meaning of scene lines and improving the initial reliability of parameter solving.
[0063] Furthermore, by performing the main line calculation and multi-graph least squares solution in stages, a linear back-projection error function is simultaneously constructed to optimize the focal length and distortion coefficient, significantly reducing the impact of single image feature errors and improving calibration accuracy and stability. This makes it suitable for applications where the reliance on calibration boards is limited, such as in situations with already mounted lenses or in confined spaces. This invention effectively improves the scene adaptability of fisheye lens calibration, providing an efficient and robust lens calibration solution that does not require a calibration board in complex scenarios.
[0064] Based on the same inventive concept, embodiments of the present invention also provide a semi-automatic fisheye lens calibration system, such as... Figure 7 As shown, it includes: The acquisition module acquires multiple calibration images using a fisheye camera, and the calibration images include at least two sets of orthogonal pairs of parallel lines. The curve extraction module extracts curves from the calibration image captured by the fisheye lens using a pre-trained neural network. The data processing module calculates the orthogonal vanishing points of the parallel lines in the orthogonal parallel line pairs based on the calibration image after selecting pairs of orthogonal parallel lines, obtains pairs of orthogonal vanishing points, calculates the principal line equation based on the pairs of orthogonal vanishing points, and calculates the principal point position based on the principal line equation. The parameter optimization module selects sampling points based on orthogonal pairs of parallel lines, obtains corrected image points according to the calibration model and principal point positions, establishes an error function using the image points and the focal length and distortion coefficients to be optimized, and obtains the optimized focal length and distortion coefficients through the error function.
[0065] This invention utilizes a computer processor to adjust the intrinsic parameters of a fisheye camera, forming a semi-automatic fisheye lens calibration system that does not require a calibration plate, enabling fisheye cameras to have rapid and accurate calibration capabilities in unstructured environments.
[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A semi-automatic fisheye lens calibration method, characterized in that, Includes the following steps: S1. Obtain multiple calibration images using a fisheye camera, wherein the calibration images include N pairs of orthogonal parallel lines; wherein N is greater than or equal to 2; S2. Extract curves from the calibration image using a pre-trained neural network; S3. Based on the calibration image after curve extraction, select two pairs of orthogonal parallel straight lines; S4. Based on the calibration image after selecting orthogonal pairs of parallel lines, calculate the orthogonal vanishing points of the parallel lines in the orthogonal pairs of parallel lines, and calculate the principal line equation based on the pairs of orthogonal vanishing points. S5. Based on the principal equation, calculate the principal point position, where the principal point position is the coordinate of the intersection of the fisheye lens optical axis and the imaging plane; S6. Based on the orthogonal parallel line pairs, select sampling points, obtain corrected image points according to the calibration model and the principal point position, establish an error function through the image points and the focal length and distortion coefficient to be optimized, optimize the curve to be optimized by minimizing the global error function, and obtain the optimized focal length and distortion coefficient.
2. The semi-automatic fisheye lens calibration method as described in claim 1, characterized in that, In step S2, the neural network adapts to the calibration image using a fisheye lens imaging model and calculates the corresponding offset of the convolution kernel. The input to the neural network is the captured fisheye image and the field of view of the fisheye lens, and the output of the neural network is the Bézier curve parameters and confidence level.
3. The semi-automatic fisheye lens calibration method as described in claim 2, characterized in that, The specific process by which the neural network adapts to the calibration image using a fisheye lens imaging model and calculates the corresponding offset of the convolutional kernel is as follows: S21. Based on the fisheye lens imaging model, construct a virtual imaging sphere for the fisheye lens, calculate the incident angle of each pixel using a double-precision model, and establish the correspondence between the imaging plane and the virtual imaging sphere: In the formula, Represents a point in three-dimensional space. Represents image points, and They represent direction and Pixel coordinates of direction r Represents the radius of the virtual imaging sphere; Image points in convolutional kernels at different locations are projected onto the virtual imaging sphere, and the sampling direction is calculated. , : In the formula, This represents the position of the center of the virtual imaging sphere, which is the origin in the virtual imaging sphere coordinate system. n This indicates that in the imaging virtual spherical coordinate system, The direction vector, that is, the direction vector of the direction; S22, along the sampling direction on the virtual sphere. , Resample, backproject all sampled points onto the imaging plane, and calculate the relative offset between the sampled points and all points in the convolution kernel: In the formula, This represents the discrete offset of the sampling position relative to the center of the convolution kernel. express The unit offset of resampling in the direction. express The unit offset of resampling in the direction. T Represents the transpose function; S23. Construct a fisheye image adaptation layer based on the offset. In the curve extraction network, use the adaptation layer to replace the traditional convolution. The replacement position is located in front of the feature map.
4. The semi-automatic fisheye lens calibration method as described in claim 1, characterized in that, The specific process of step S4 is as follows: S41. Based on the selected pair of orthogonal parallel lines, calculate the intersection point position of the parallel lines in the calibration image using the endpoints of the parallel lines in the pair of orthogonal parallel lines. The intersection point position is the coordinate of the orthogonal vanishing point. S42. Obtain at least two pairs of orthogonal vanishing points through at least two pairs of orthogonal parallel lines. The secondary expression of the orthogonal vanishing point in each pair of orthogonal vanishing points is as follows: , , and ,in, and This forms a pair of orthogonal vanishing points. and This forms another pair of orthogonal vanishing points. Calculate the direction of the principal line: Intercept of the main line: S43. The equation of the principal line is obtained based on the direction and intercept of the principal line: 。 5. The semi-automatic fisheye lens calibration method as described in claim 1, characterized in that, In step S5, the specific process of calculating the principal point position is as follows: Estimate the homogeneous coordinates of the principal point using the least squares method. In the formula, , .
6. The semi-automatic fisheye lens calibration method as described in claim 1, characterized in that, The specific process of step S6 is as follows: S61. Uniformly sample along the selected pairs of orthogonal parallel lines. N l sampling points For the sampling points The corrected image points are obtained based on the calibration model. ; S62, Select from the calibration image M The straight line, the first The set of sampling points of a straight line on an image is represented as Define the point after distortion removal Construct the error function for a single straight line: In the formula, ,for A straight line is used to construct the global objective function: In the formula, , Indicates the first The weight of a line in the global objective function is linearly related to the line length. Indicates the first The length of a straight line is approximately equal to the distance between its endpoints. In the formula, Indicates the starting sampling point. This indicates the final sampling point.
7. The semi-automatic fisheye lens calibration method as described in claim 3, characterized in that, The positions of the sampling points on the virtual imaging sphere on the tangent plane of the virtual imaging sphere are as follows: In the formula, , This represents the angular resolution per unit pixel.
8. A semi-automatic fisheye lens calibration system, characterized in that, include: The acquisition module acquires multiple calibration images using a fisheye camera, and the calibration images include at least two sets of orthogonal pairs of parallel lines. The curve extraction module extracts curves from the calibration image captured by the fisheye lens using a pre-trained neural network. The data processing module calculates the orthogonal vanishing points of the parallel lines in the orthogonal parallel line pairs based on the calibration image after selecting pairs of orthogonal parallel lines, obtains pairs of orthogonal vanishing points, calculates the principal line equation based on the pairs of orthogonal vanishing points, and calculates the principal point position based on the principal line equation. The parameter optimization module selects sampling points based on orthogonal pairs of parallel lines, obtains corrected image points according to the calibration model and principal point positions, establishes an error function using the image points and the focal length and distortion coefficients to be optimized, and obtains the optimized focal length and distortion coefficients through the error function.