Beyond-visual-range camera adaptive fuzzy calibration algorithm
Through the adaptive fuzzy calibration algorithm, the problem of blurred image calibration when the over-visual camera is imaged at close range is solved, and the precise corner positioning of the blurred image and the adaptive checkerboard size are realized, which improves the calibration accuracy and Efficiency reduces the complexity of manual operations and ensures the accuracy and automation of calibration results.
Patent Information
- Application Number
- CN202411027824.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
There is a blur problem when imaging at close range of cameras. Traditional calibration methods are difficult to accurately detect the corner points of the checkerboard, and the selection of the checkerboard size requires human intervention, and checkerboards of different sizes need to be recalibrated.
Adaptive fuzzy calibration algorithm is used to remove the influence of blur through image preprocessing, locate the corner points of the checkerboard, use the refined method of subpixel-level corner points and directions, optimize the energy function, obtain the exact pixel coordinates of all corner points, and introduce the self-directed Adapt to the mechanism of checkerboard size calibration, intelligently obtain multiple checkerboard sizes and avoid manual intervention.
It improves the accuracy and efficiency of calibration, reduces the complexity of manual operation, shortens the calibration space range, improves the degree of automation, and ensures the accuracy and reliability of calibration results.
Smart Images

Figure CN118967830A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of over-the-horizon camera fuzzy adaptive calibration, and in particular to an over-the-horizon camera fuzzy adaptive calibration algorithm. Background Art
[0002] Over-the-horizon cameras are a type of camera. Due to the characteristics of over-the-horizon cameras, clear imaging is far away, while imaging is fuzzy at close range. Traditional calibration methods are prone to large calibration errors when facing fuzzy images, and may even fail to detect chessboard corners. The selection of chessboard size often requires human intervention, and chessboards of different sizes need to be recalibrated. In summary, in order to solve these problems, the present invention proposes an innovative method. Summary of the invention
[0003] The purpose of the present invention is to provide an adaptive fuzzy calibration algorithm for an over-the-horizon camera to solve the problem that the existing over-the-horizon camera mentioned in the above background technology has the characteristics of clear imaging at a long distance and fuzzy imaging at close range, and the traditional calibration method is prone to large calibration errors or even failure to detect chessboard corners when facing fuzzy images, and the selection of chessboard size often requires human intervention, and chessboards of different sizes need to be recalibrated.
[0004] To achieve the above object, the present invention provides the following technical solution: an over-the-horizon camera adaptive fuzzy calibration algorithm, comprising:
[0005] S1: Take photos to obtain images: Use an over-the-horizon camera (clear imaging range ≥ 17m) to take photos at a distance of 8 meters from the chessboard to obtain multiple blurred images;
[0006] S2: Image preprocessing: Use image processing algorithms to preprocess the blurred image, remove the blur effect, locate the corner points of the chessboard, and then refine the sub-pixel corner points and directions. Finally, optimize the energy function, mark all the chessboards in the image, and obtain the precise pixel coordinates of all corner points.
[0007] S3: Function solution: Substitute the precise coordinates of the checkerboard corner points in the blurred image into the OPENCV calibration function for solution, and calculate the internal parameter matrix.
[0008] Preferably, the S1 stage is based on the principle of the internal reference calibration algorithm, that is, assuming that there is a target point P in the space with a spatial position coordinate (X W , Y W , Z W), and the coordinate system where the target point is imaged in the picture is the image pixel coordinate system. In order to obtain the projection of P in the image pixel coordinate system, it is necessary to construct auxiliary coordinate systems: camera coordinate system, image physical coordinate system and image pixel coordinate system.
[0009] Preferably, the mapping relationship between the world coordinate system and the image pixel coordinate system is derived by taking the center of the lens as the origin and the straight line perpendicular to the mirror plane as the z-axis to establish a spatial rectangular coordinate system, with the x-axis and the y-axis on the lens plane, satisfying the right-hand coordinate system. At this time, the camera has a unique position in the world space, and the two coordinate systems can be related by Formula 1:
[0010]
[0011] Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, and T is the translation matrix of the camera coordinate system relative to the world coordinate system.
[0012] Preferably, the formula 1 is subjected to gain, and the gain matrix is expanded to obtain the formula 2:
[0013]
[0014] Images are stored in the computer in the form of pixel matrices. The image pixel coordinate system takes the upper left corner of the image as the origin, the u coordinate axis to the right, and the v coordinate axis downward. The pixel coordinate system is represented by (u, v).
[0015] Preferably, in Formula 2, the pixel height is Height, the bottom length is Width, the coordinates of the principal point in the pixel coordinate system are (u0, v0), and without considering image distortion, a point (x l ,y l ) in pixel coordinates can be determined by equation 3, which is:
[0016]
[0017] Preferably, the coordinates of a certain point P in the camera coordinate system are (x c ,y c ,z c ) Therefore, Formula 4 is established, and Formula 4 is:
[0018]
[0019] By transforming it, we can get formula 5: Let s = z c We get formula 6:
[0020]
[0021] Formula 6 is the mapping relationship between the camera coordinate system and the image physical coordinate system. Combining equations 1, 3, and 6, we can deduce the mapping relationship between the world coordinate system and the image pixel coordinate system, as follows:
[0022] As shown in formula 7:
[0023]
[0024] The M1 in Formula 7 is the internal parameter matrix.
[0025] Preferably, S2 is based on the principle of image blur adaptive corner point pixel coordinate recognition, that is, after obtaining the picture taken by the over-the-horizon camera, blurred image processing is performed, and then the positions of the checkerboard corner points in the blurred image are located. Since the checkerboard may be deflected in the image, two types of corner points are defined, one is the corner point parallel to the image coordinate axis, and the other is the corner point deflected by 30°. These two types of corner points are composed of four filter kernels {A, B, C, D}, and a convolution operation is performed with the input image.
[0026] Preferably, the image blur adaptive corner point pixel coordinate recognition principle calculates the similarity between each pixel point and the corner point by using the formula 8 method, where formula 8 is:
[0027]
[0028] Preferably, after the calculation of Formula 8, the approximate position of the corner point can be obtained. After the approximate position of the corner point is obtained, non-maximum suppression (NMS) is performed to obtain candidate points. After obtaining the candidate points, the sub-pixel corner point and direction refinement method in OPENCV is used to further improve the accuracy of the corner point pixel coordinates.
[0029] Preferably, the S3 stage is based on the principle of multiple checkerboard recognition in the image, that is, in order to be able to intelligently obtain multiple checkerboards in the image and avoid the tedious process of manual intervention and repeated calibration, the following method is used: define the checkerboard energy function, as shown in Formula 9:
[0030] E=α*N-β*D
[0031] Where α is a proportional coefficient used to adjust the weight of the number of line groups, N is the number of line groups, indicating the number of detected line groups, β is a proportional coefficient used to adjust the weight of the distance between line groups, and D is the average distance between line groups, indicating the average distance between line groups.
[0032] Compared with the prior art, the invention has the following beneficial effects: the adaptive blur calibration algorithm for over-the-horizon camera utilizes image processing and computer vision technology, improves the calibration algorithm and introduces an adaptive mechanism to achieve comprehensive optimization of blur image and adaptive chessboard size calibration. First, the blur image is preprocessed by an image processing algorithm to remove blur effects, locate the chessboard corner points, and then the sub-pixel corner points and directions are refined. Finally, the energy function is optimized to mark all the chessboards in the image and obtain the precise pixel coordinates of all the corner points. Secondly, the adaptive chessboard size calibration mechanism is introduced to intelligently obtain the sizes of multiple chessboards in the image according to the distribution of feature points detected in the image, thereby avoiding the cumbersome process of manual intervention and repeated calibration. The invention has the advantages of being able to improve the accuracy and efficiency of calibration. By eliminating blur effects and adaptively selecting chessboard sizes, the calibration results are more accurate and reliable. The spatial range of calibrating over-the-horizon cameras is greatly shortened. At the same time, the method of the invention can greatly reduce the complexity and workload of manual operations and improve the calibration efficiency and automation. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 This is a schematic diagram of imaging with a beyond-visual-range camera according to the present invention;
[0034] Figure 2 This is a schematic diagram of pinhole imaging of a beyond-viewing-range camera of the present invention;
[0035] Figure 3 This is a blurred picture collected by the over-the-horizon camera of the present invention;
[0036] Figure 4 This is a blurred picture collected by the over-the-horizon camera of the present invention;
[0037] Figure 5 This is a blurred picture collected by the over-the-horizon camera of the present invention;
[0038] Figure 6 It is a chessboard outline diagram of the present invention;
[0039] Figure 7 It is a chessboard outline diagram of the present invention;
[0040] Figure 8 It is a chessboard outline diagram of the present invention;
[0041] Fig. 9 It is the corner point distribution diagram of the present invention;
[0042] Fig.10 It is the corner point distribution diagram of the present invention;
[0043] Fig.11 It is the corner point distribution diagram of the present invention;
[0044] Fig.12 The statistical table of the data results of the present invention is repeated 10 times;
[0045] Fig.13 It is a schematic diagram of the main structure of the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] See also Figure 1-13 The present invention provides a technical solution: an adaptive fuzzy calibration algorithm for an over-the-horizon camera, comprising:
[0048] S1: Take photos to obtain images: Use an over-the-horizon camera (clear imaging range ≥ 17m) to take photos at a distance of 8 meters from the chessboard to obtain multiple blurred images;
[0049] S2: Image preprocessing: Use image processing algorithms to preprocess the blurred image, remove the blur effect, locate the corner points of the chessboard, and then refine the sub-pixel corner points and directions. Finally, optimize the energy function, mark all the chessboards in the image, and obtain the precise pixel coordinates of all corner points.
[0050] S3: Function solution: Substitute the precise coordinates of the checkerboard corner points in the blurred image into the OPENCV calibration function for solution, and calculate the internal parameter matrix.
[0051] The S1 stage is based on the principle of the internal reference calibration algorithm, that is, assuming that there is a target point P in the space with the spatial position coordinates in the world coordinate system (X W , Y W , Z W ), and the coordinate system where the target point is imaged in the picture is the image pixel coordinate system. In order to obtain the projection of P in the image pixel coordinate system, it is necessary to construct auxiliary coordinate systems: camera coordinate system, image physical coordinate system and image pixel coordinate system.
[0052] The mapping relationship between the world coordinate system and the image pixel coordinate system is derived as follows: a spatial rectangular coordinate system is established with the center of the lens as the origin and the straight line perpendicular to the mirror plane as the z-axis. The x-axis and the y-axis are on the lens plane, satisfying the right-hand coordinate system. At this time, the camera has a unique position in the world space, and the two coordinate systems can be connected by formula 1:
[0053]
[0054] Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, and T is the translation matrix of the camera coordinate system relative to the world coordinate system.
[0055] Formula 1 is used for gain, and the gain matrix is expanded to obtain Formula 2:
[0056]
[0057] Images are stored in the computer in the form of pixel matrices. The image pixel coordinate system takes the upper left corner of the image as the origin, the u coordinate axis to the right, and the v coordinate axis downward. The pixel coordinate system is represented by (u, v).
[0058] Formula 2: Assume that the pixel height is Height, the bottom length is Width, and the coordinates of the principal point in the pixel coordinate system are (u0, v0). Without considering image distortion, a point (x l ,y l ) in pixel coordinates can be determined by equation 3, which is
[0059] For a point P, the coordinates in the camera coordinate system are (x c ,y c ,z c ) Therefore, Formula 4 is established, and Formula 4 is:
[0060]
[0061] By transforming it, we can get formula 5: Let s = z c We get formula 6:
[0062]
[0063] Formula 6 is the mapping relationship between the camera coordinate system and the image physical coordinate system. Combining equations 1, 3, and 6, we can deduce the mapping relationship between the world coordinate system and the image pixel coordinate system, as follows:
[0064] As shown in formula 7:
[0065]
[0066] The M1 in Formula 7 is the internal parameter matrix.
[0067] S2 is based on the principle of image blur adaptive corner pixel coordinate recognition, that is, after obtaining the picture taken by the over-the-horizon camera, it performs blur image processing and then locates the corner positions of the chessboard in the blur image. Since the chessboard may be deflected in the image, two types of corner points are defined, one is the corner point parallel to the image coordinate axis, and the other is the corner point deflected by 30°. These two types of corner points are composed of four filter kernels {A, B, C, D}, which are convolved with the input image.
[0068] The principle of image blur adaptive corner point pixel coordinate recognition is to calculate the similarity between each pixel and the corner point by using the formula 8 method, where formula 8 is:
[0069]
[0070] After calculation by formula 8, the approximate position of the corner point can be obtained. After the approximate position of the corner point is obtained, non-maximum suppression (NMS) is performed to obtain candidate points. After obtaining the candidate points, the sub-pixel corner point and direction refinement method in OPENCV is used to further improve the accuracy of the corner point pixel coordinates.
[0071] The S3 stage is based on the principle of multiple checkerboard recognition in the image. In order to intelligently obtain multiple checkerboards in the image and avoid the tedious process of manual intervention and repeated calibration, the following method is used: the checkerboard energy function is defined as shown in Formula 9:
[0072] E=α*N-β*D
[0073] Where α is a proportional coefficient used to adjust the weight of the number of line groups, N is the number of line groups, indicating the number of detected line groups, β is a proportional coefficient used to adjust the weight of the distance between line groups, and D is the average distance between line groups, indicating the average distance between line groups.
[0074] The specific steps of the chessboard energy function are as follows:
[0075] Use the Canny edge detection algorithm to extract edge information in the image;
[0076] Line detection using Hough transform;
[0077] Cluster the detected straight lines and group similar straight lines into one group;
[0078] Verify each straight line group to determine whether it meets the characteristics of the checkerboard;
[0079] Check if there are enough lines in the line group;
[0080] Check whether the angles of the lines in the line group are similar (a certain angle error is allowed);
[0081] Check whether the lines in the line group intersect or are parallel to each other according to the characteristics of the chessboard;
[0082] Calculate the energy value for the group of straight lines that pass the verification.
[0083] Output result: Based on the checkerboard candidate with the highest energy value, a rectangular or polygonal box can be drawn to select the detected checkerboard area, and the image of the selected area can be extracted for further analysis or processing.
[0084] Example
[0085] The original image resolution of the over-the-horizon camera used in the experiment is 3840*2166. According to the camera imaging principle, the closer the pixel coordinates of the calibration result principal point are to the center of the image (1920,1083), the better the camera quality is.
[0086] like Figure 3-Figure 5 The following is a fuzzy picture captured by an over-the-horizon camera. The following is a fuzzy picture captured by an over-the-horizon camera. Figure 6-Figure 8 The multiple chessboard outlines are shown, and the precise pixel coordinates of the chessboard corners are substituted into the OPENCV sub-pixel refinement function to draw a corner distribution map as shown in the figure. Figure 9-11 As shown in the figure, it can be seen from the image that the algorithm of the present invention can accurately obtain the pixel coordinates of the corner points in a blurred image. The experiment is repeated 10 times and the result data is stable. The ten average values of the principal point coordinates (μ0, ν0) are (1924.276, 1082.204), the mean square error of μ0 is 0.156, and the mean square error of v0 is 0.013; the average value of the reprojection error Ret is 0.0396, and the mean square error is 0.0004455.
[0087] Experimental data show that under the blurred image background, the deviation between the principal point obtained by the method of the present invention and the theoretical principal point is very small, and the reprojection error is also much lower than the standard of 0.1 of the maximum reprojection error of available data in engineering experience. It can be seen that the method of the present invention has achieved a significant improvement in calibration accuracy, and can also perform adaptive calibration under multi-chessboard backgrounds, greatly improving efficiency and operational convenience. The algorithm of the present invention can also save test space in the automatic assembly of over-the-horizon cameras on factory production lines, while improving production efficiency.
[0088] The method of the present invention can realize camera calibration of blurred images, and improves the accuracy and efficiency of calibration. By eliminating the blur effect and adaptively selecting the chessboard size, the calibration result is more accurate and reliable. The spatial range of calibrating over-the-horizon cameras is greatly shortened. At the same time, the method of the present invention can greatly reduce the complexity and workload of manual operation, and improve the efficiency and automation of calibration.
[0089] In summary, the present invention solves the problems of fuzzy calibration and adaptive chessboard size calibration. The method improves the accuracy and efficiency of calibration and reduces the complexity of manual operation through image processing and computer vision technology.
[0090] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An adaptive fuzzy calibration algorithm for an over-the-horizon camera, characterized in that: include: S1: Take photos to obtain images: Use an over-the-horizon camera (clear imaging range ≥ 17m) to take photos at a distance of 8 meters from the chessboard to obtain multiple blurred images; S2: Image preprocessing: Use image processing algorithms to preprocess the blurred image, remove the blur effect, locate the corner points of the chessboard, and then refine the sub-pixel corner points and directions. Finally, optimize the energy function, mark all the chessboards in the image, and obtain the precise pixel coordinates of all corner points. S3: Function solution: Substitute the precise coordinates of the checkerboard corner points in the blurred image into the OPENCV calibration function for solution, and calculate the internal parameter matrix.
2. The adaptive fuzzy calibration algorithm for an over-the-horizon camera according to claim 1, characterized in that: The S1 stage is based on the principle of the internal reference calibration algorithm, that is, assuming that there is a target point P in the space with a spatial position coordinate in the world coordinate system (X W , Y W , Z W ), and the coordinate system where the target point is imaged in the picture is the image pixel coordinate system. In order to obtain the projection of P in the image pixel coordinate system, it is necessary to construct auxiliary coordinate systems: camera coordinate system, image physical coordinate system and image pixel coordinate system.
3. The adaptive fuzzy calibration algorithm for an over-the-horizon camera according to claim 2, characterized in that: The mapping relationship between the world coordinate system and the image pixel coordinate system is derived as follows: a spatial rectangular coordinate system is established with the center of the lens as the origin and the vertical mirror plane straight line as the z-axis. The x-axis and the y-axis are on the lens plane, satisfying the right-hand coordinate system. At this time, the camera has a unique position in the world space, and the two coordinate systems can be connected by formula 1: Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, and T is the translation matrix of the camera coordinate system relative to the world coordinate system.
4. The adaptive fuzzy calibration algorithm for an over-the-horizon camera according to claim 1, characterized in that: Formula 1 is used for gain, and the gain matrix is expanded to obtain Formula 2: Images are stored in the computer in the form of pixel matrices. The image pixel coordinate system takes the upper left corner of the image as the origin, the u coordinate axis to the right, and the v coordinate axis downward. The pixel coordinate system is represented by (u, v).
5. The adaptive fuzzy calibration algorithm for a beyond-horizon camera according to claim 1, characterized in that: In Formula 2, the pixel height is Height, the bottom length is Width, and the coordinates of the principal point in the pixel coordinate system are (u0, v0). Without considering image distortion, a point (x l ,y l ) in pixel coordinates can be determined by equation 3, which is 6. The adaptive fuzzy calibration algorithm for an over-the-horizon camera according to claim 1, characterized in that: The coordinates of a certain point P in the camera coordinate system are (x c ,y c ,z c ) Therefore, Formula 4 is established, and Formula 4 is: By transforming it, we can get formula 5: Let s = z c We get formula 6: Formula 6 is the mapping relationship between the camera coordinate system and the image physical coordinate system. Combining equations 1, 3, and 6, we can deduce the mapping relationship between the world coordinate system and the image pixel coordinate system, as shown in equation 7: The M1 in Formula 7 is the internal parameter matrix.
7. The adaptive fuzzy calibration algorithm for a beyond-horizon camera according to claim 1, characterized in that: The S2 is based on the principle of image blur adaptive corner point pixel coordinate recognition, that is, after obtaining the picture taken by the over-the-horizon camera, it performs blurred image processing, and then locates the corner points of the chessboard in the blurred image. Since the chessboard may be deflected in the image, two types of corner points are defined, one is the corner point parallel to the image coordinate axis, and the other is the corner point deflected by 30°. These two types of corner points are composed of four filter kernels {A, B, C, D}, and a convolution operation is performed with the input image.
8. The adaptive fuzzy calibration algorithm for a beyond-horizon camera according to claim 1, characterized in that: The image blur adaptive corner pixel coordinate recognition principle calculates the similarity between each pixel and the corner point by using the formula 8 method, where formula 8 is:
9. The adaptive fuzzy calibration algorithm for a beyond-horizon camera according to claim 1, characterized in that: After the calculation of Formula 8, the approximate position of the corner point can be obtained. After the approximate position of the corner point is obtained, non-maximum suppression (NMS) is performed to obtain candidate points. After the candidate points are obtained, the sub-pixel corner point and direction refinement method in OPENCV is used to further improve the accuracy of the corner point pixel coordinates.
10. The adaptive fuzzy calibration algorithm for a beyond-horizon camera according to claim 1, characterized in that: The S3 stage is based on the principle of multiple checkerboard recognition in the image, that is, in order to intelligently obtain multiple checkerboards in the image and avoid the tedious process of manual intervention and repeated calibration, the following method is used: define the checkerboard energy function, as shown in Formula 9: E=α*N-β*D Where α is a proportional coefficient used to adjust the weight of the number of line groups, N is the number of line groups, indicating the number of detected line groups, β is a proportional coefficient used to adjust the weight of the distance between line groups, and D is the average distance between line groups, indicating the average distance between line groups.
Citation Information
Patent Citations
Quick calibration method for robot vision system
CN106780623A
Multi-camera in-loop simulation test method and system facing panoramic vision perception
CN109407547A
Curve ACC target vehicle identification method based on multi-sensor fusion
CN110949395A
Checkerboard sub-pixel-level corner self-adaptive detection method
CN111260731A
Vehicle-mounted camera calibration method and device, equipment and storage medium
CN113658272A
