A fisheye camera calibration and correction method

Abstract

The present invention discloses a calibration and correction method for a fisheye camera, which belongs to the field of image processing technology, and includes the following steps: photographing and obtaining a plurality of checkerboard images within the field of view of the fisheye camera; obtaining an uncorrected coordinate point set P; constructing an actual physical coordinate point set Q of the checkerboard image; performing a rotation and translation transformation on the set Q to obtain a rotation and translation transformation coordinate point set S; performing a refraction transformation on the set Q to obtain a refraction transformation coordinate point set T; calculating the residual e of the pixel coordinate points on the checkerboard image, and accumulating to obtain a total residual value E; minimizing the total residual value E to obtain the optimal refraction parameter corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to the checkerboard image; constructing an actual physical pixel rotation and translation mapping matrix; constructing an actual physical pixel refraction mapping matrix; and correcting the input distorted image through the actual physical pixel refraction mapping matrix. The present invention improves the accuracy of the application of fisheye cameras in machine vision.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a calibration and correction method for a fisheye camera. Background Art

[0002] In the field of machine vision, fisheye cameras are widely used in workpiece processing, quality inspection, equipment monitoring and other scenarios because they can provide a wide-angle field of view. However, due to the characteristics of fisheye cameras themselves, the images they obtain have obvious geometric distortion, which seriously affects the reliability and accuracy of fisheye cameras in machine vision application scenarios. Therefore, it is particularly important to calibrate fisheye cameras and use the calibration parameters to correct images.

[0003] At present, the existing image distortion correction technology usually uses the plane perspective projection constraint to map the straight line in the three-dimensional space to the two-dimensional plane, which is suitable for images with small field of view and small distortion. However, the image taken by the fisheye lens does not meet the constraint rule because it projects the observed object onto the spherical surface.

[0004] Therefore, how to provide a calibration and correction method for a fisheye camera to provide effective support for the application of fisheye cameras in machine vision is a technical problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0005] To this end, the present invention provides a fisheye camera calibration and correction method to solve the problems existing in the prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for calibrating and correcting a fisheye camera comprises the following steps:

[0008] Step S1: within the field of view of the fisheye camera, the checkerboard calibration plate is moved multiple times so that the checkerboard calibration plate fully covers the field of view, and the fisheye camera synchronously shoots to obtain multiple checkerboard images;

[0009] Step S2: Based on the multiple checkerboard images obtained in step S1, preprocess the checkerboard images, extract corner points and sort them to obtain an uncorrected coordinate point set P of the checkerboard images;

[0010] Step S3: constructing a set Q of actual physical coordinate points of the chessboard image based on the parameter information of the chessboard calibration plate input by the user;

[0011] Step S4: Based on the rotation and translation matrices corresponding to different chessboard images, the actual physical coordinate point set Q of the chessboard image is subjected to rotation and translation transformation to obtain a rotation and translation transformed coordinate point set S;

[0012] Step S5: Based on the rotation and translation transformation coordinate point set S, perform a refraction transformation on it to obtain a refraction transformation coordinate point set T;

[0013] Step S6: based on the refraction transformation coordinate point set T and the uncorrected coordinate point set P, the residual e of the pixel coordinate points on the checkerboard image is calculated, and the total residual value E is accumulated;

[0014] Step S7: repeating steps S4 to S6, using an iterative optimization method to minimize the total residual value E obtained in step S6, and obtaining the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to the checkerboard image;

[0015] Step S8: Based on the preset output rectified image, combined with the optimal rotation and translation matrices corresponding to different checkerboard images, construct an actual physical pixel rotation and translation mapping matrix;

[0016] Step S9: constructing an actual physical pixel refraction mapping matrix based on the constructed actual physical pixel rotation and translation mapping matrix and combining the optimal refraction parameters corresponding to the fisheye camera;

[0017] Step S10: Correct the input distorted image by using the pixel interpolation method through the constructed actual physical pixel refraction mapping matrix.

[0018] Furthermore, the step S2 specifically includes:

[0019] Each checkerboard image is denoised and equalized, and the checkerboard corner point recognition algorithm is used to identify the corner points of each checkerboard image. The identified checkerboard corner points are sorted from left to right and from top to bottom to obtain the uncorrected coordinate point set P, P of the checkerboard image. n represents the set of uncorrected coordinate points extracted from the nth chessboard image, P nm Represents the mth uncorrected coordinate point of the nth checkerboard image.

[0020] Furthermore, the step S3 specifically includes:

[0021] According to the number of chessboard rows b entered by the user 0 , Number of columns b 1 and the length of the chessboard side b 2 Construct the actual physical coordinate point set Q of the chessboard image, where the chessboard corner points in set Q are arranged from left to right and from top to bottom. n represents the actual physical coordinate point set corresponding to the nth chessboard image, Q nm Indicates the mth actual physical coordinate point of the nth chessboard image, Q nm The calculation formula for the pixel coordinates in the x, y, and z directions is:

[0022] ;

[0023] ;

[0024] ;

[0025] Where m=0, 1, …b 0 *b 1 -1, % is the remainder symbol of division, & is the floor symbol of division, Q nm =[q x ,q y ,q z ] T .

[0026] Furthermore, in step S4, the mth actual physical coordinate point Q of the nth chessboard image is nm After the rotation and translation transformation, the mth rotation and translation transformation coordinate point S of the nth chessboard image is obtained nm The calculation formula is:

[0027] ;

[0028] Where m=0, 1, …b 0 *b 1 -1, % is the remainder symbol of division, & is the floor symbol of division, Q nm =[q x ,q y ,q z ] T .

[0029] Furthermore, in step S4, the mth actual physical coordinate point Q of the nth chessboard image is nm After the rotation and translation transformation, the mth rotation and translation transformation coordinate point S of the nth chessboard image is obtained nm The calculation formula is:

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] Among them, c 0 、c 1 、c 2、c 3 、c 4 、c 5 is the camera refraction parameter, f 0 、f 1 、h 0 is the intermediate variable, S x , S y , S z is the rotation and translation transformation coordinate point S nm The pixel coordinates in the x, y, and z directions.

[0036] Furthermore, in step S6, the residual value e of the m-th coordinate point of the n-th chessboard image is calculated as follows:

[0037] ;

[0038] Among them, P x , P y is the pixel coordinates in the x and y directions of the m-th uncorrected coordinate point of the n-th checkerboard image, T x , T y The pixel coordinates in the x and y directions of the m-th refraction transformation coordinate point of the n-th chessboard image.

[0039] Furthermore, the step S7 specifically includes:

[0040] Use the LM algorithm to continuously iterate and update the refraction parameters and the rotation and translation matrix corresponding to each checkerboard image until the total residual value E is less than the set value t 0 Or satisfy the minimum number of iterations t 1 , after exiting the iteration, the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to each checkerboard image are obtained.

[0041] Furthermore, the step S8 specifically includes:

[0042] Create an empty matrix E with h rows and w columns according to the preset output rectified image width w and height h 0 and E 1 As the actual physical pixel rotation and translation mapping matrices in the x-direction and y-direction respectively;

[0043] For E 0 and E 1 The input point matrix is ​​U 0 =[c,r,1] T , output point matrix U 1 The calculation formula is:

[0044] ;

[0045] Among them, R0 is the optimal rotation and translation matrix corresponding to the first chessboard image, and U 1 Fill in the value of [0] into the matrix E 0 (r, c) position, U 1 Fill in the value of [1] into the matrix E 1 The (r, c) position;

[0046] Complete E 0 and E 1 All rows and columns of , complete the calculation of the actual physical pixel rotation and translation mapping matrix.

[0047] Furthermore, the step S9 specifically includes:

[0048] Create an empty matrix E 2 、E 3 As the actual physical pixel refraction mapping matrix in the x-direction and y-direction respectively;

[0049] For the matrix E 2 and E 3 The rth row and cth column of , the calculation method in step S5 is used, wherein the camera refraction parameter is the optimal refraction parameter after iteration, and the input value S is set x The value of E 0 (r, c), output value T x Fill in E 2 (r, c), set the input value S y The value of E 1 (r, c), output value T y Fill in E 3 (r, c);

[0050] Complete E 2 and E 3 All rows and columns of , complete the calculation of the actual physical pixel refraction mapping matrix.

[0051] Furthermore, the step S10 specifically includes:

[0052] For the rth row and cth column of the output rectified image, its x-direction index i x The size of the refraction mapping matrix E 2 The value of (r, c), whose y-direction index is i y The size of the refraction mapping matrix E 3 The value of (r, c);

[0053] According to the x-direction index i x and the y-direction index i y , find the pixel of the input distorted image (i y ,i x) position, and use the grayscale values ​​corresponding to the four pixels in the upper left, lower left, upper right, and lower right of the position to perform pixel linear interpolation, and use the result of the interpolation calculation as the grayscale value of the pixel in the rth row and cth column of the output corrected image;

[0054] After traversing all rows and columns of the output corrected image, the grayscale value of each position is calculated, the calculation of the output corrected image is completed, and the correction of the input distorted image is realized.

[0055] The present invention has the following advantages:

[0056] The present invention moves a checkerboard calibration plate multiple times within the field of view of a fisheye camera so that the checkerboard calibration plate fully covers the field of view, and the fisheye camera synchronously shoots to obtain a plurality of checkerboard images; based on the plurality of checkerboard images obtained in step S1, the checkerboard images are preprocessed, corner points are extracted and sorted, and an uncorrected coordinate point set P of the checkerboard images is obtained; based on parameter information of the checkerboard calibration plate input by a user, an actual physical coordinate point set Q of the checkerboard images is constructed; based on the rotation and translation matrices corresponding to different checkerboard images, the actual physical coordinate point set Q of the checkerboard images is rotationally and translationally transformed to obtain a rotationally and translationally transformed coordinate point set S; based on the rotationally and translationally transformed coordinate point set S, a refraction transformation is performed on it to obtain a refraction transformed coordinate point set T; based on the refraction transformed coordinate point set T and the uncorrected coordinate point set P, a residual e of a pixel coordinate point on the checkerboard image is calculated, and a total residual value E is accumulated to obtain.

[0057] By adopting an iterative optimization method, the total residual value E obtained is minimized to obtain the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to the checkerboard image; based on the preset output correction image, combined with the optimal rotation and translation matrices corresponding to different checkerboard images, the actual physical pixel rotation and translation mapping matrix is ​​constructed; based on the constructed actual physical pixel rotation and translation mapping matrix, combined with the optimal refraction parameters corresponding to the fisheye camera, the actual physical pixel refraction mapping matrix is ​​constructed; through the constructed actual physical pixel refraction mapping matrix, the input distorted image is corrected by the pixel interpolation method. The present invention improves the accuracy of the application of fisheye cameras in machine vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0059] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0060] Figure 1 The present invention provides a flow chart of a method for calibrating and correcting a fisheye camera. DETAILED DESCRIPTION

[0061] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] A method for calibrating and correcting a fisheye camera, such as Figure 1 As shown, the following steps are included:

[0063] Step S1: within the field of view of the fisheye camera, the checkerboard calibration plate is moved multiple times so that the checkerboard calibration plate fully covers the field of view, and the fisheye camera synchronously shoots to obtain multiple checkerboard images, specifically:

[0064] First, place the checkerboard calibration plate at the center of the fisheye camera field of view and shoot the checkerboard calibration plate to obtain the first checkerboard image I 0 Then, within the field of view of the fisheye camera, move the chessboard N times and shoot with the camera simultaneously. After N shots, ensure that all corners within the field of view are basically covered by the chessboard calibration plate, and obtain other chessboard images I n (n=0, 1…N).

[0065] Step S2: Based on the multiple checkerboard images obtained in step S1, the checkerboard images are preprocessed, corner points are extracted and sorted, and an uncorrected coordinate point set P of the checkerboard images is obtained, specifically:

[0066] The first checkerboard image and other checkerboard images are subjected to denoising and equalization processing, and the checkerboard corner point recognition algorithm is used to recognize the corner points of the first checkerboard image and other checkerboard images, and the recognized checkerboard corner points are sorted in order from left to right and from top to bottom to obtain the uncorrected coordinate point sets P, P of the checkerboard images. nrepresents the set of uncorrected coordinate points extracted from the nth chessboard image, P nm Represents the mth uncorrected coordinate point of the nth checkerboard image.

[0067] Step S3: Based on the parameter information of the chessboard calibration plate input by the user, construct the actual physical coordinate point set Q of the chessboard image, specifically:

[0068] The parameter information of the checkerboard calibration plate input by the user is the number of checkerboard rows b 0 , Number of columns b 1 and the length of the chessboard side b 2 , according to the number of chessboard rows b entered by the user 0 , Number of columns b 1 and the length of the chessboard side b 2 Construct the actual physical coordinate point set Q of the chessboard image. The chessboard corner points in set Q are arranged from left to right and from top to bottom. n represents the actual physical coordinate point set corresponding to the nth chessboard image, Q nm Indicates the mth actual physical coordinate point of the nth chessboard image, Q nm The calculation formula for the pixel coordinates in the x, y, and z directions is:

[0069] ;

[0070] ;

[0071] ;

[0072] Where m=0, 1, …b 0 *b 1 -1, % is the remainder symbol of division, & is the floor symbol of division, Q nm =[q x ,q y ,q z ] T .

[0073] Step S4: Based on the rotation and translation matrices corresponding to different chessboard images, the actual physical coordinate point set Q of the chessboard image is subjected to rotation and translation transformation to obtain the rotation and translation transformed coordinate point set S, which is specifically:

[0074] Different checkerboard images have different rotation and translation matrices. For the nth checkerboard image, its rotation and translation matrix is ​​R n , if it is the first iteration calculation, then R n The initial value of is:

[0075] ;

[0076] Among them, a 0 -a 8 The default initial value size.

[0077] For the mth actual physical coordinate point Q of the nth chessboard image nm After the rotation and translation transformation, the mth rotation and translation transformation coordinate point S of the nth chessboard image is obtained nm The calculation formula is:

[0078] ;

[0079] Among them, R n is the rotation and translation matrix corresponding to the nth chessboard image.

[0080] Step S5: Based on the rotation and translation transformation coordinate point set S, a refraction transformation is performed on it to obtain a refraction transformation coordinate point set T, which is specifically:

[0081] The refraction transformation related parameters used when performing the refraction transformation are the camera internal parameters, and the refraction transformation related parameters corresponding to different checkerboard images are the same.

[0082] The mth rotation and translation transformation coordinate point S of the nth chessboard image nm After the refraction transformation, the mth refraction transformation coordinate point T of the nth checkerboard image is obtained. nm The calculation formula for the pixel coordinates in the x and y directions is:

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] Among them, c 0 、c 1 、c 2 、c 3 、c 4 、c 5 is the camera refraction parameter. If it is the first iteration calculation, the camera refraction parameter needs to be initialized to a fixed coefficient, f 0 、f 1 、h 0 is the intermediate variable, S x , S y , S z is the rotation and translation transformation coordinate point Snm The pixel coordinates in the x, y, and z directions.

[0089] Step S6: Based on the refraction transformation coordinate point set T and the uncorrected coordinate point set P, the residual e of the pixel coordinate points on the checkerboard image is calculated, and the total residual value E is obtained by accumulation.

[0090] The calculation method of the residual value e of the mth coordinate point of the nth chessboard image is:

[0091] ;

[0092] Among them, P x , P y is the pixel coordinates in the x and y directions of the m-th uncorrected coordinate point of the n-th checkerboard image, T x , T y The pixel coordinates in the x and y directions of the m-th refraction transformation coordinate point of the n-th chessboard image.

[0093] Calculate the residual values ​​of all the checkerboard points of all checkerboard calibration plates, and accumulate them to obtain the total residual value E.

[0094] Step S7: Repeat steps S4 to S6, and use iterative optimization to minimize the total residual value E obtained in step S6, and obtain the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to the checkerboard image, specifically:

[0095] Use the LM algorithm to continuously iterate and update the refraction parameters and the rotation and translation matrix corresponding to each checkerboard image until the total residual value E is less than the set value t 0 Or satisfy the minimum number of iterations t 1 , after exiting the iteration, the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to each checkerboard image are obtained.

[0096] Step S8: Based on the preset output rectified image, combined with the optimal rotation and translation matrices corresponding to different checkerboard images, an actual physical pixel rotation and translation mapping matrix is ​​constructed, specifically:

[0097] Create an empty matrix E with h rows and w columns according to the preset output rectified image width w and height h 0 and E 1 As the actual physical pixel rotation and translation mapping matrices in the x-direction and y-direction respectively;

[0098] For E 0 and E 1 The rth row and cth column in (r is less than h, c is less than w), the input point matrix is ​​U 0 =[c,r,1] T , output point matrix U1 The calculation formula is:

[0099] ;

[0100] Among them, R 0 is the optimal rotation and translation matrix corresponding to the first chessboard image, and U 1 Fill in the value of [0] into the matrix E 0 (r, c) position, U 1 Fill in the value of [1] into the matrix E 1 The (r, c) position;

[0101] Complete E 0 and E 1 All rows and columns of , complete the calculation of the actual physical pixel rotation and translation mapping matrix.

[0102] Step S9: Based on the constructed actual physical pixel rotation and translation mapping matrix, combined with the optimal refraction parameters corresponding to the fisheye camera, an actual physical pixel refraction mapping matrix is ​​constructed, specifically:

[0103] Create an empty matrix E 2 、E 3 As the actual physical pixel refraction mapping matrix in the x-direction and y-direction respectively;

[0104] For the matrix E 2 and E 3 The rth row and cth column of the image are calculated by the calculation method in step S5, wherein the camera refraction parameter is set to the optimal refraction parameter after iteration. , set the input value S x The value of E 0 (r, c), output value T x Fill in E 2 (r, c), according to , set the input value S y The value of E 1 (r, c), output value T y Fill in E 3 (r, c);

[0105] Complete E 2 and E 3 All rows and columns of , complete the calculation of the actual physical pixel refraction mapping matrix.

[0106] Step S10: Correct the input distorted image by using the constructed actual physical pixel refraction mapping matrix and pixel interpolation method, specifically:

[0107] For the rth row and cth column of the output rectified image, its x-direction index i xThe size of the refraction mapping matrix E 2 The value of (r, c), whose y-direction index is i y The size of the refraction mapping matrix E 3 The value of (r, c);

[0108] According to the x-direction index i x and the y-direction index i y , find the pixel of the input distorted image (i y ,i x ) position, and use the grayscale values ​​corresponding to the four pixels in the upper left, lower left, upper right, and lower right of the position to perform pixel linear interpolation, and use the result of the interpolation calculation as the grayscale value of the pixel in the rth row and cth column of the output corrected image;

[0109] After traversing all rows and columns of the output corrected image, the grayscale value of each position is calculated, the calculation of the output corrected image is completed, and the correction of the input distorted image is realized.

[0110] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for calibrating and correcting a fisheye camera, characterized in that: The following steps are involved: Step S1: within the field of view of the fisheye camera, the checkerboard calibration plate is moved multiple times so that the checkerboard calibration plate fully covers the field of view, and the fisheye camera synchronously shoots to obtain multiple checkerboard images; Step S2: Based on the multiple checkerboard images obtained in step S1, preprocess the checkerboard images, extract corner points and sort them to obtain an uncorrected coordinate point set P of the checkerboard images; Step S3: constructing a set Q of actual physical coordinate points of the chessboard image based on the parameter information of the chessboard calibration plate input by the user; Step S4: Based on the rotation and translation matrices corresponding to different chessboard images, the actual physical coordinate point set Q of the chessboard image is subjected to rotation and translation transformation to obtain a rotation and translation transformed coordinate point set S; Step S5: Based on the rotation and translation transformation coordinate point set S, perform a refraction transformation on it to obtain a refraction transformation coordinate point set T; Step S6: based on the refraction transformation coordinate point set T and the uncorrected coordinate point set P, the residual e of the pixel coordinate points on the checkerboard image is calculated, and the total residual value E is accumulated; Step S7: repeating steps S4 to S6, using an iterative optimization method to minimize the total residual value E obtained in step S6, and obtaining the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to the checkerboard image; Step S8: Based on the preset output rectified image, combined with the optimal rotation and translation matrices corresponding to different checkerboard images, construct an actual physical pixel rotation and translation mapping matrix; Step S9: constructing an actual physical pixel refraction mapping matrix based on the constructed actual physical pixel rotation and translation mapping matrix and combining the optimal refraction parameters corresponding to the fisheye camera; Step S10: Correct the input distorted image by using the pixel interpolation method through the constructed actual physical pixel refraction mapping matrix.

2. The method for calibrating and correcting a fisheye camera as claimed in claim 1, characterized in that: The step S2 specifically includes: Each checkerboard image is denoised and equalized, and the checkerboard corner point recognition algorithm is used to identify the corner points of each checkerboard image. The identified checkerboard corner points are sorted from left to right and from top to bottom to obtain the uncorrected coordinate point set P, P of the checkerboard image. n represents the set of uncorrected coordinate points extracted from the nth chessboard image, P nm Represents the mth uncorrected coordinate point of the nth checkerboard image.

3. The method for calibrating and correcting a fisheye camera as claimed in claim 2, characterized in that: The step S3 specifically includes: The actual physical coordinate point set Q of the chessboard image is constructed according to the number of chessboard rows b0, the number of columns b1 and the side length b2 input by the user. The chessboard corner points in set Q are arranged from left to right and from top to bottom. n represents the actual physical coordinate point set corresponding to the nth chessboard image, Q nm Indicates the mth actual physical coordinate point of the nth chessboard image, Q nm The calculation formula for the pixel coordinates in the x, y, and z directions is: ； ； ； Where, m = 0, 1, ... b0*b1-1, % is the remainder symbol of division, & is the floor symbol of division, Q nm =[q x ,q y ,q z ] T .

4. The method for calibrating and correcting a fisheye camera as claimed in claim 3, characterized in that: In step S4, the mth actual physical coordinate point Q of the nth chessboard image is nm After the rotation and translation transformation, the mth rotation and translation transformation coordinate point S of the nth chessboard image is obtained nm The calculation formula is: ; Among them, R n is the rotation and translation matrix corresponding to the nth chessboard image.

5. The method for calibrating and correcting a fisheye camera as claimed in claim 4, characterized in that: In step S5, the mth rotation and translation transformation coordinate point S of the nth chessboard image is performed. nm After the refraction transformation, the mth refraction transformation coordinate point T of the nth checkerboard image is obtained. nm The calculation formula for the pixel coordinates in the x and y directions is: ； ； ； ； ； Among them, c0, c1, c2, c3, c4, c5 are camera refraction parameters, f0, f1, h0 are intermediate variables, S x , S y , S z is the rotation and translation transformation coordinate point S nm The pixel coordinates in the x, y, and z directions.

6. The method for calibrating and correcting a fisheye camera as claimed in claim 5, characterized in that: In step S6, the residual value e of the mth coordinate point of the nth chessboard image is calculated as follows: ； Among them, P x , P y is the pixel coordinates in the x and y directions of the m-th uncorrected coordinate point of the n-th checkerboard image, T x , T y The pixel coordinates in the x and y directions of the m-th refraction transformation coordinate point of the n-th chessboard image.

7. The method for calibrating and correcting a fisheye camera as claimed in claim 6, characterized in that: The step S7 specifically includes: The LM algorithm is used to continuously iterate and update the refraction parameters and the rotation and translation matrix corresponding to each checkerboard image until the total residual value E is less than the set value t0 or the minimum number of iterations t1 is met. After exiting the iteration, the optimal refraction parameters corresponding to the fisheye camera and the optimal rotation and translation matrix corresponding to each checkerboard image are obtained.

8. The method for calibrating and correcting a fisheye camera as claimed in claim 7, characterized in that: The step S8 specifically includes: According to the preset output rectified image width w and height h, create empty matrices E0 and E1 with h rows and w columns as the actual physical pixel rotation and translation mapping matrices in the x and y directions respectively; For the rth row and cth column in E0 and E1, the input point matrix is ​​U0=[c, r, 1] T , the calculation formula of the output point matrix U1 is: ； Where R0 is the optimal rotation and translation matrix corresponding to the first checkerboard image, the value of U1[0] is filled into the (r, c) position of the matrix E0, and the value of U1[1] is filled into the (r, c) position of the matrix E1; After traversing all rows and columns of E0 and E1, the calculation of the actual physical pixel rotation and translation mapping matrix is ​​completed.

9. The method for calibrating and correcting a fisheye camera as claimed in claim 8, characterized in that: The step S9 specifically includes: Create empty matrices E2 and E3 as actual physical pixel refraction mapping matrices in the x-direction and y-direction respectively; For the rth row and cth column of matrices E2 and E3, the calculation method in step S5 is used, where the camera refraction parameter is the optimal refraction parameter after iteration, and the input value S is set x The value is E0 (r, c), the output value is T x Fill in E2 (r, c), set the input value S y The value is E1 (r, c), the output value is T y Fill in E3 (r, c); After traversing all rows and columns of E2 and E3, the calculation of the actual physical pixel refraction mapping matrix is ​​completed.

10. The method for calibrating and correcting a fisheye camera according to claim 9, wherein: The step S10 specifically includes: For the rth row and cth column of the output rectified image, its x-direction index i x The size is the value of the refraction mapping matrix E2 (r, c), and its y-direction index i y The size of is the value of the refraction mapping matrix E3 (r, c); According to the x-direction index i x and the y-direction index i y , find the pixel of the input distorted image (i y ,i x ) position, and use the grayscale values ​​corresponding to the four pixels in the upper left, lower left, upper right, and lower right of the position to perform pixel linear interpolation, and use the result of the interpolation calculation as the grayscale value of the pixel in the rth row and cth column of the output corrected image; After traversing all rows and columns of the output corrected image, the grayscale value of each position is calculated, the calculation of the output corrected image is completed, and the correction of the input distorted image is realized.

Images