Camera external parameter calibration method based on small balls with different colors

Abstract

The invention relates to the field of computer vision, in particular to a camera external parameter calibration method based on small balls with different colors, which comprises the following steps: S1, key point detection network training: adopting a key point detection network based on deep learning, detecting key points of markers through the network, providing accurate data support for subsequent calibration calculation, and obtaining a key point detection result; the robustness and the accuracy of the calibration process are enhanced; s2, marker design and verification: selecting a small ball with a clear color feature as a main marker; s3, data set manufacturing and multi-view camera acquisition are carried out; s4, training and verifying a deep learning model; and S5, detecting, calibrating and calculating. The method does not depend on checkerboard angular points, is higher in adaptability to complex environments such as low-quality images, large visual angles and long-distance shooting, effectively reduces calibration errors, improves the calibration precision of a multi-view camera system, is not sensitive to small changes of shooting angles, and avoids error accumulation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a camera extrinsic parameter calibration method based on different color balls. BACKGROUND

[0002] Multi-view RGB camera calibration technology is an important technology in the field of computer vision. It realizes the cooperative work of multiple cameras through accurate calibration of multiple RGB cameras, so as to obtain more comprehensive and accurate image information. This technology is of great significance to improve the accuracy and efficiency of image processing, and has wide application prospects in the fields of three-dimensional reconstruction, object recognition and scene understanding. In multi-view RGB camera calibration, checkerboard is often used as a calibration board because its corner points are clear and easy to detect. By shooting multiple checkerboard pictures at different angles and positions, the intrinsic and extrinsic parameters of the camera can be calculated.

[0003] Although the existing checkerboard calibration method is widely used in camera calibration, it has many shortcomings in multi-view camera system calibration. It depends on the accuracy of corner point extraction, and large errors may occur when shooting low-quality images, large angles and long distances. It is sensitive to shooting angles, and small changes in angle may lead to error accumulation. The number of calibration points is limited and the precision is affected by the clarity of the corner points. Distortion correction depends on accurate corner point detection, and corner point error directly affects the correction result. And due to the two-dimensional limitation, it cannot fully utilize the three-dimensional space information, and the precision is insufficient in complex multi-view camera system calibration scene.

[0004] Therefore, we improve it and propose a camera extrinsic parameter calibration method based on different color balls. SUMMARY

[0005] The purpose of the present application is to solve the problems of error-prone corner point extraction, sensitive shooting angle, insufficient precision due to two-dimensional limitation, unstable corner point extraction and increased calibration result error in current camera system calibration.

[0006] In order to achieve the above-mentioned purpose of the application, the present application provides a camera extrinsic parameter calibration method based on different color balls to improve the above-mentioned problems.

[0007] The present application is as follows:

[0008] A camera extrinsic parameter calibration method based on different color balls, comprising the following steps:

[0009] S1, key point detection network training: a key point detection network based on deep learning is adopted, and the key points of the marker are detected through the network to provide accurate data support for subsequent calibration calculation;

[0010] S2, identification object design and verification: select small balls with clear color characteristics as the main identification object, and use the T-shaped geometry of the calibration rod and the spacing between the small balls as auxiliary detection identification information, develop an identification object scheme, test the identification object scheme under light, angle, and occlusion scenarios, and verify its feasibility;

[0011] S3, data set production and multi-camera acquisition: use a multi-camera array to acquire images of the identification object under various scenes and angles, generate a diversified training data set, the data set includes images under different lighting conditions, different angles, different scales, and different backgrounds, and contains the accurate position of the identification object, the coordinates of the key points, and the corresponding labels in each frame of image;

[0012] S4, deep learning model training and verification: based on a deep learning model, train an efficient target detection network to identify the key points of the identification object;

[0013] S5, detection calibration calculation: after detecting the key points of the identification object, real-time calibration is performed using multi-camera data, and the relative position and attitude between the cameras are determined through calibration calculation to accurately obtain the external relative position and attitude information between the cameras.

[0014] As a preferred technical solution of the present application, in the training process of step S4, data augmentation techniques are used to improve the generalization ability of the model, including random cropping, flipping, and rotation. After training, the detection effect is verified.

[0015] As a preferred technical solution of the present application, the detection calibration calculation includes the following steps:

[0016] A1, real-time key point detection: use the trained model for real-time detection to obtain the key point coordinates of the identification object under different camera angles at the same time, and the key point information includes the positions of the calibration rod small balls and the landmark small balls;

[0017] A2, robust optimization algorithm and error control: through multi-stage data processing and optimization process, eliminate abnormal points and improve the accuracy of calibration, and when eliminating redundant points, use spatial density clustering method to eliminate near neighbor redundant points.

[0018] As a preferred technical solution of the present application, the calibration rod in step S3 is composed of N multi-color small balls, and its three-dimensional coordinates satisfy the preset geometric constraints, such as linear arrangement spacing d:‖P k -P k-1 ‖2=d(k=2,..., N).

[0019] As a preferred technical solution of the present application, in step S4, the loss function design is:

[0020] Total loss Ltotal Including detection loss L det With key point loss L kpt：

[0021] L total =αL det +βL kpt；

[0022] Wherein L kpt Adopt smooth L1 loss:

[0023] .

[0024] As the preferred technical solution of the application, the near neighbor redundant point is removed: through the spatial density clustering method, the redundant points are identified and removed, for each three-dimensional point, the number of neighbors within the radius δ dense is calculated, and δ dense is set as a pre-set density threshold for measuring the density between points, based on spatial density clustering, the redundant point judgment condition is defined: .

[0025] As the preferred technical solution of the application, the low precision point filtering: combined with the detection confidence s and the re-projection residual e, the low quality points are dynamically removed, for each point, the detection confidence s is calculated through the matching quality of the point in the image and space, the feature similarity index, the smaller s represents the lower quality of the point, by re-projecting the point to the image plane and comparing with the actual observation point, the re-projection residual e is calculated, the larger e represents the worse fitting effect of the point to the model, with the optimization iteration process, τ s and τ e are dynamically adjusted to ensure that abnormal points can be effectively identified and removed in different scenes:

[0026] s<τ s Or e=║P obs -P proj ║2>τ e；

[0027] Wherein p proj is the projection point under the current external parameter.

[0028] As the preferred technical solution of the application, the robustness based on RANSAC: randomly select 4 non-coplanar points as the minimum sample set, use RANSAC algorithm to iteratively fit the camera external parameter, and calculate the number of inliers to evaluate the pros and cons of the model, through multiple iterations and optimization, remove noise data, and finally converge to the optimal solution, the algorithm formula: , wherein Ⅱ( . ) is an indicator function, is the residual threshold, and finally the model with the most inliers is selected.

[0029] As a preferred technical solution of the present application, the calibration quality evaluation and guidance: through the spatial distribution uniformity analysis, guide the calibration data supplement, key formula and algorithm: effective point distribution uniformity index, divide the calibration area into MxN grid, according to the Euclidean distance index to evaluate the effectiveness and uniformity of the pretreated monitoring point data, count the effective point number n ij , calculate the coefficient of variation, to reflect the discrete degree of data distribution in a single area:

[0030] .

[0031] Compared with the prior art, the beneficial effects of the present application are:

[0032] In the scheme of the present application:

[0033] 1. In order to solve the problems of existing camera system calibration, such as easy error in corner point extraction, sensitive to shooting angle, insufficient precision of two-dimensional limitation, unstable corner point extraction, and increased calibration result error, the present application is not dependent on the chessboard corner point, has stronger adaptability to complex environments such as low-quality images, large viewing angle and long-distance shooting, effectively reduces the calibration error, improves the calibration precision of multi-camera system, is not sensitive to small changes in shooting angle, avoids error accumulation, can maintain stable calibration effect under different shooting angles, and enhances the robustness of the calibration process;

[0034] 2. The present application breaks through the two-dimensional limitation and fully utilizes the three-dimensional space information, can more accurately obtain the external relative position and attitude information between cameras in the complex calibration scene of multi-camera system, optimizes the calibration effect, automatically detects key points through a deep learning model, reduces manual intervention, speeds up the calibration, improves the calibration efficiency of multi-camera system, and reduces the labor cost;

[0035] 3. The identification object scheme of the present application is feasible after scene test verification such as light, angle and shielding, can adapt to more complex calibration scenes, expands the application range of multi-camera system calibration, and based on the identification object scheme of different color balls and calibration rods, has high flexibility, can be flexibly adjusted according to different multi-camera system requirements, and is convenient for system expansion. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The principle flow chart of the camera external parameter calibration method based on different color balls provided by the present application is shown. DETAILED DESCRIPTION

[0037] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0038] Specifically, please refer to Figure 1 A camera extrinsic parameter calibration method based on different color balls, comprising the following steps:

[0039] S1, key point detection network training: a key point detection network based on deep learning is adopted, the key points of the marker are detected through the network, accurate data support is provided for subsequent calibration calculation, and the robustness and accuracy of the calibration process are enhanced;

[0040] S2, marker design and verification: small balls with clear color characteristics are selected as the main markers, and the T-shaped geometry of the calibration rod and the spacing between the small balls are used as auxiliary detection identification information, a marker scheme is formulated, the marker scheme is tested under light, angle and occlusion scenes, and the feasibility thereof is verified;

[0041] S3, data set making and multi-camera acquisition: a multi-camera array is used to acquire images of the markers under various scenes and angles, and diversified training data sets are generated, the data sets include images under different light conditions, different angles, different scales and different backgrounds, and contain accurate positions of the markers, coordinates of the key points and corresponding labels in each frame of image;

[0042] S4, deep learning model training and verification: a high-efficiency target detection network is trained based on a deep learning model to identify the key points of the markers;

[0043] S5, detection calibration calculation: after the key points of the markers are detected, real-time calibration is performed by using multi-camera data, the relative positions and postures between the cameras are determined through calibration calculation, and the extrinsic relative position and posture information between the cameras is accurately obtained.

[0044] The present application does not depend on the chessboard corner points, is more adaptable to complex environments such as low-quality images, large angles and long-distance shooting, effectively reduces the calibration error, improves the calibration precision of the multi-camera system, is not sensitive to small changes in shooting angle, avoids error accumulation, can maintain stable calibration effect under different shooting angles, and enhances the robustness of the calibration process.

[0045] Embodiment 1, please refer to Figure 1 A camera extrinsic parameter calibration method based on different color balls, comprising the following steps:

[0046] S1, key point detection network training: a key point detection network based on deep learning is adopted to detect the key points of the marker, providing accurate data support for subsequent calibration calculation to enhance the robustness and accuracy of the calibration process;

[0047] S2, marker design and verification: small balls with clear color characteristics are selected as the main markers, and the T-shaped geometry of the calibration rod and the spacing between the small balls are used as auxiliary detection information. The marker scheme is developed, and the feasibility of the marker scheme is verified through testing under different light conditions, angles, and occlusion scenarios.

[0048] S3, data set production and multi-camera acquisition: a multi-camera array is used to capture images of the markers under various scenes and angles, generating a diverse training data set. The data set includes images under different lighting conditions, different angles, different scales, and different backgrounds, and contains the accurate position of the markers in each frame of image, the coordinates of the key points, and the corresponding labels.

[0049] S4, deep learning model training and verification: based on a deep learning model, a high-efficiency target detection network is trained to identify the key points of the markers.

[0050] S5, detection and calibration calculation: after detecting the key points of the markers, real-time calibration is performed using multi-camera data to determine the relative positions and attitudes of each camera, thereby accurately obtaining the external relative position and attitude information between cameras.

[0051] The present application does not rely on the corner points of the checkerboard, and is more adaptable to complex environments such as low-quality images, large angles, and long-distance shooting, effectively reducing the calibration error and improving the calibration accuracy of the multi-camera system. It is not sensitive to small changes in shooting angle, avoids error accumulation, maintains stable calibration effect under different shooting angles, and enhances the robustness of the calibration process.

[0052] The detection and calibration calculation includes the following steps:

[0053] A1, real-time key point detection: a trained model is used for real-time detection to obtain the key point coordinates of the markers under different camera angles at the same time. The key point information includes the positions of the calibration rod small balls and the landmark small balls.

[0054] A2, robust optimization algorithm and error control: through a multi-stage data processing and optimization process, abnormal points are removed and the accuracy of the calibration is improved. When removing redundant points, a spatial density clustering method is used to remove nearby redundant points.

[0055] The calibration rod in step S3 is composed of N multi-color balls, and the three-dimensional coordinates thereof satisfy preset geometric constraints, such as linear arrangement spacing d: ‖P k -P k-1 ‖2=d (k=2,..., N).

[0056] To be compatible with illumination changes, a color similarity threshold (HSV space) is defined: △H<τ H , △S+△V<τ SV , where τ H , τ SV are experimental calibration thresholds for distinguishing the colors of adjacent balls.

[0057] In step S4, the loss function is designed:

[0058] The total loss L total includes a detection loss L det and a key point loss L kpt：

[0059] L total =αL det +βL kpt；

[0060] where L kpt adopts a smooth L1 loss:

[0061] .

[0062] In example 2, the camera parameter calibration method based on balls of different colors provided in example 1 is further optimized, specifically, near neighbor redundant points are removed: through a spatial density clustering method, redundant points are identified and removed, for each three-dimensional point, the number of neighbors within a δ dense radius in space is calculated, δ dense is set as a pre-set density threshold for measuring the density between points, based on spatial density clustering, a redundant point determination condition is defined: .

[0063] According to the actual data distribution, the δ dense threshold is dynamically adjusted to ensure the accuracy and robustness of redundant point removal, if there are other points within a δ dense radius around a certain point, only the one with the highest confidence is retained.

[0064] Low-precision point filtering: combined with detection confidence s and reprojection residual e, dynamically remove low-quality points, for each point, calculate the detection confidence s through its matching quality in the image and space, feature similarity index, the smaller s represents the lower quality of the point, by projecting the point to the image plane and comparing it with the actual observation point, calculate its reprojection residual e, the larger e represents the worse fitting effect of the point and the model, with the optimization iteration process, dynamically adjust τ s and τ e , ensure that abnormal points can be effectively identified and removed in different scenes:

[0065] s<τ s or e=║P obs -P proj ║2>τ e；

[0066] where p proj is the projection point under the current external parameter.

[0067] Robustness based on RANSAC: randomly select 4 non-coplanar points as the minimum sample set, use RANSAC algorithm to iteratively fit the camera external parameter, and calculate the number of inliers to evaluate the model, through multiple iterations and optimization, remove noise data, and finally converge to the optimal solution, algorithm formula: , where Ⅱ( . ) is the indicator function, is the residual threshold, finally select the model with the most inliers.

[0068] Calibration quality evaluation and guidance: through spatial distribution uniformity analysis, guide calibration data supplement, key formula and algorithm: effective point distribution uniformity index, divide the calibration area into M×N grid, according to the Euclidean distance index to evaluate the effectiveness and uniformity of the preprocessed monitoring point data, to ensure the accuracy and robustness of the visual calibration process. Statistics of the number of effective points n ij in each unit, calculate the coefficient of variation to reflect the distribution dispersion degree of data in a single area:

[0069] .

[0070] Sparse area detection: sparse area refers to the area with large gaps or insufficient data in the detection process, usually judged by the number of neighbors and distance distribution index, through grid division method, that is, the space is divided into grid units, the number of points in each unit is counted, and then the sparse area is identified, the sparse unit judgment condition is defined:

[0071] .

[0072] Calculation of relative positions of cameras in camera system:

[0073] Let the coordinates of a calibration sphere in the world coordinate system be P w =[X w , Y w , Z w , 1] T , and the pixel coordinates of the projection onto the i-th camera be p i =[u i , v i , 1] T , then the projection relationship is:

[0074] λ i P i =K i [R i |t i ]P w ,

[0075] where K i is the intrinsic matrix of the i-th camera;

[0076] R i , t i are the extrinsic parameters (rotation matrix and translation vector) of the i-th camera;

[0077] λ i is the depth scaling factor;

[0078] By combining the relative positions of the key points with the known positions in the three-dimensional space, the relative positions and rotation matrices between the cameras are calculated according to the above camera projection model. The relative poses between the cameras can be solved by the PnP (Perspective-n-Point) algorithm. This step will use the geometric constraints of multi-view images to solve the camera extrinsic parameters, and then obtain the spatial position of each camera.

[0079] Reprojection residual fitting: In order to improve the accuracy of calibration, a step of reprojection error optimization can be introduced. By minimizing the reprojection residual fitting, the extrinsic parameters of each camera can be optimized. Specifically, by calculating the reprojection error of each marker key point, the difference between the real image points and the calculated projection points is minimized to optimize the pose of each camera. The optimization method can be based on the nonlinear least squares method (such as Levenberg-Marquardt) to solve;

[0080] Reprojection residual, define the objective function as the sum of the squared residuals of all observed points:

[0081] ;

[0082] where π( . ) is the projection function, and the Levenberg-Marquardt algorithm is used to solve:

[0083] .

[0084] The application accurately models the spatial coordinates of the calibration rod in three-dimensional space through the relative position relationship between the multi-camera, calculates the extrinsic parameters of the camera system, and innovatively designs the calibration rod: adopts multi-color small ball markers, distinguishes adjacent small balls through different colors, combines deep learning, improves the detection and differentiation degree, and avoids the problem of false matching caused by repeated shapes. The color scheme is designed based on the principle of high contrast (such as yellow, blue, and green arranged alternately), and a color tolerance threshold is set to compatible with light change scenes,

[0085] Near neighbor redundant point removal: based on spatial distance and geometric constraint to identify over-dense points, retain the centroid position and eliminate redundant points, reduce the risk of local overfitting, low-precision small ball point filtering: through output confidence and re-projection residual evaluation, dynamically eliminate low-precision marker point data caused by imaging blur or deformation, RANSAC-based robust optimization: iteratively select matching point sets that meet multi-view geometric constraints, resist light changes, motion blur interference.

[0086] Effective point uniform distribution detection: (1) divide the calibration area into grid units, count the number of effective points in each unit, and calculate the visual distribution uniformity index; (2) visually prompt the sparse area (such as the edge or occluded area), guide the user to adjust the pose of the calibration rod or supplement the calibration data, and ensure the completeness of the spatial coverage.

[0087] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the preferred embodiments of the present application are given in the drawings, but do not limit the patent scope of the present application. The present application can be realized in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structure made by using the content of the present application specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A method for calibrating camera extrinsic parameters based on small balls of different colors, characterized in that, Includes the following steps: S1. Keypoint Detection Network Training: A deep learning-based keypoint detection network is used to detect key points of the markers, providing accurate data support for subsequent calibration calculations. S2. Marker Design and Verification: Select small balls with distinct color characteristics as the main markers, and use the T-shaped geometry of the calibration rod and the spacing between the small balls as auxiliary detection information. Develop a marker scheme and test the scheme under lighting, angle, and occlusion scenarios to verify its feasibility. S3. Dataset Creation and Multi-Camera Acquisition: Use a multi-camera array to acquire images of markers in various scenes and perspectives to generate diverse training datasets. The datasets include images under different lighting conditions, angles, scales, and backgrounds, and contain the accurate location of the markers, the coordinates of key points, and the corresponding labels in each frame. S4. Deep Learning Model Training and Validation: Based on a deep learning model, train an efficient object detection network to identify key points of markers; S5. Detection and Calibration Calculation: After detecting the key points of the marker, real-time calibration is performed using multi-camera data. The relative position and attitude between each camera are determined through calibration calculation to accurately obtain the external relative position and attitude information between each camera.

2. The camera extrinsic parameter calibration method based on different colored balls according to claim 1, characterized in that, Step S4 involves using data augmentation techniques during training to improve the model's generalization ability. These techniques include random cropping, flipping, and rotation. After training is completed, the detection results are validated.

3. The camera extrinsic parameter calibration method based on different colored balls according to claim 2, characterized in that, The calibration calculation includes the following steps: A1. Real-time key point detection: Using a trained model, real-time detection is performed to obtain the coordinates of key points of the marker under different camera views at the same time. The key point information includes the position of the calibration rod ball and the position of the landmark ball. A2. Robust Optimization Algorithm and Error Control: Through multi-stage data processing and optimization, outliers are eliminated and the calibration accuracy is improved. In the process of eliminating redundant points, spatial density clustering is used to eliminate nearest-neighbor redundant points.

4. The camera extrinsic parameter calibration method based on different colored balls according to claim 3, characterized in that, In step S3, the calibration rod consists of N multicolored spheres, whose three-dimensional coordinates satisfy preset geometric constraints, such as a linear arrangement with a spacing of d: ||P k -P k-1 ||2=d(k=2,...,N).

5. The camera extrinsic parameter calibration method based on different colored balls according to claim 4, characterized in that, Loss function design in step S4: Total loss L total Including detection loss L det With key point loss L kpt： L total =αL det +βL kpt Where L kpt Using smoothed L1 loss: 。 6. The camera extrinsic parameter calibration method based on different colored balls according to claim 5, characterized in that, Nearest Neighbor Redundancy Removal: Redundant points are identified and removed using spatial density clustering. For each 3D point, its spatial δ value is calculated. dense The number of neighbors within the radius is set to δ. dense A pre-defined density threshold is used to measure the density between points. Based on spatial density clustering, the criteria for determining redundant points are defined as follows: .

7. The camera extrinsic parameter calibration method based on different colored balls according to claim 6, characterized in that, Low-precision point filtering: Combining the detection confidence s and the reprojection residual e, low-quality points are dynamically removed. For each point, the detection confidence s is calculated based on its matching quality and feature similarity in the image and space. The smaller the s, the lower the quality of the point. The reprojection residual e is calculated by reprojecting the point onto the image plane and comparing it with the actual observed point. The larger the e, the worse the fit between the point and the model. As the optimization iteration process progresses, τ is dynamically adjusted. s and τ e This ensures that outliers can be effectively identified and removed in different scenarios. s<τ s Or e=║P obs -P proj ║2>τ e； Where p proj This is the projection point under the current extrinsic parameters.

8. The camera extrinsic parameter calibration method based on different colored balls according to claim 7, characterized in that, Robustness based on RANSAC: Four non-coplanar points are randomly selected as the minimum sample set. The RANSAC algorithm is used to iteratively fit the camera's extrinsic parameters, and the number of interior points is calculated to evaluate the model's performance. Through multiple iterations and optimizations, noisy data is removed, and the algorithm finally converges to the optimal solution. Algorithm formula: , of which II ( . ) is an indicator function. The residual threshold is used to select the model with the most interior points.

9. A camera extrinsic parameter calibration method based on different colored balls according to claim 8, characterized in that, Calibration Quality Assessment and Guidance: Spatial distribution uniformity analysis guides the supplementation of calibration data. Key formulas and algorithms include: a uniformity index for effective point distribution; dividing the calibration area into an M×N grid; evaluating the effectiveness and uniformity of preprocessed monitoring point data based on the Euclidean distance index; and counting the number of effective points n within each unit. ij The coefficient of variation is calculated to reflect the dispersion of the data distribution within a single region. 。

Images