Camera calibration method, device, equipment and storage medium

Abstract

The application belongs to the technical field of data processing, and discloses a camera calibration method, device, equipment and storage medium; the method comprises the following steps: initializing a camera model to obtain an initial camera model, and inputting a calibration board image sequence of the camera model into the initial camera model; screening the calibration board image sequence through the initial camera model to obtain a reference frame image and a residual image; obtaining an initial calibration result of the current initial camera model according to the reference frame image, screening the residual image according to the initial calibration result to obtain a candidate image set corresponding to the initial camera model; determining a target image set from the candidate image sets corresponding to multiple initial camera models according to a preset evaluation function, and obtaining a calibration result of the camera model according to the target image set; the application adaptively selects a reference image from the calibration board image sequence, evaluates the angle difference of the reference image to obtain an optimal image set, optimizes the camera model based on the optimal image set, and obtains a more accurate calibration result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a camera calibration method, apparatus, device, and storage medium. Background Technology

[0002] When calibrating a camera using a planar calibration board, the accuracy of the calibration results depends heavily on the pose of the captured image. Within a manually selected set of shooting poses, there is a high probability of degenerate poses.

[0003] Existing calibration toolkits require users to place the calibration plate in different locations to capture images, then use optimization algorithms to optimize the estimated parameters and calculate the maximum likelihood calibration parameters. Tan et al. proposed using a monitor to display the pose, but did not determine how to select the optimal shooting pose. Some work has been done in the past to guide users on how to place the camera. In photogrammetry, determining the placement of a given number of cameras offline is crucial to obtaining the most accurate 3D reconstruction results possible. Currently, the consistency of calibration results cannot be guaranteed during camera calibration.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a camera calibration method that addresses the technical problem of low calibration accuracy caused by degenerate poses in the shooting pose set affecting camera calibration when using a planar calibration plate in the prior art.

[0006] To achieve the above objectives, the present invention provides a camera calibration method, the method comprising the following steps:

[0007] The camera model is initialized to obtain an initial camera model, and the calibration board image sequence of the camera model is input into the initial camera model;

[0008] The calibration board image sequence is filtered using the initial camera model to obtain reference frame images and remaining images;

[0009] The initial calibration result of the current initial camera model is obtained based on the reference frame image, and the remaining images are filtered based on the initial calibration result to obtain the candidate image set corresponding to the initial camera model;

[0010] A target image set is determined from a set of candidate images corresponding to multiple initial camera models according to a preset evaluation function, and the calibration result of the camera model is obtained based on the target image set.

[0011] Optionally, the initialization of the camera model to obtain an initial camera model, and the input of the calibration board image sequence of the camera model to the initial camera model, includes:

[0012] Set the principal point coordinates of the camera model and initialize the distortion coefficients of the camera model to obtain an initial camera model. The principal point coordinates are determined by the image center of the camera model.

[0013] The calibration plate image sequence of the camera model is input into the initial camera model.

[0014] Optionally, the step of filtering the calibration board image sequence using the initial camera model to obtain reference frame images and remaining images includes:

[0015] Randomly select the image corresponding to the current frame from the calibration board image sequence;

[0016] The camera intrinsic parameters are obtained by using a preset algorithm based on the image corresponding to the current frame and the acquired reference frame image. The angle error is obtained based on the camera intrinsic parameters. When the angle error is less than the angle threshold, the image corresponding to the current frame is used as the reference frame image corresponding to the current frame. Then, the step of randomly selecting the image corresponding to the current frame from the calibration board image sequence is returned to execute, so as to obtain multiple reference frame images.

[0017] The images other than the multiple reference frame images in the calibration board image sequence are taken as the remaining images.

[0018] Optionally, before obtaining the camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm, the method further includes:

[0019] Detect whether a reference frame image has been acquired;

[0020] If not, then the camera intrinsic parameters are obtained based on the image corresponding to the current frame using a preset algorithm, the angle error is obtained based on the camera intrinsic parameters, and when the angle error is less than the angle threshold, the image corresponding to the current frame is used as the initial reference frame image.

[0021] If so, then the step of obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm is executed, wherein the acquired reference frame image includes at least the initial reference frame image.

[0022] Optionally, the step of obtaining the initial calibration result of the current initial camera model based on the reference frame image, and filtering the remaining images based on the initial calibration result to obtain a candidate image set corresponding to the initial camera model, includes:

[0023] The initial calibration result of the current initial camera model is obtained based on the reference frame image using a preset algorithm;

[0024] Randomly select images to be screened from the remaining images, calculate the angle error of the images to be screened based on the initial calibration results, and when the angle error of the images to be screened is greater than the angle threshold, select the images to be screened as images in the candidate image set.

[0025] Repeat the steps of randomly selecting images to be screened from the remaining images, calculating the angle error of the images to be screened based on the initial calibration result, and including the images to be screened as images in the candidate image set when the angle error of the images to be screened is greater than the angle threshold, until the angle error of each image to be screened in the remaining images is calculated to obtain the candidate image set.

[0026] Optionally, calculating the angle error of the image to be screened based on the initial calibration result includes:

[0027] Based on the initial calibration results, the horizontal focal length, vertical focal length, and principal point coordinates of the initial camera model are obtained.

[0028] The corner pixel coordinates are determined from the image to be screened, and the corner pixel coordinates are transformed according to the horizontal axis focal length and the vertical axis focal length to obtain the corner normalized coordinates;

[0029] The checkerboard direction vector is obtained based on the normalized coordinates of the corner points, and the angle between the checkerboard direction vectors is calculated based on the three-dimensional origin coordinates of the initial camera model.

[0030] The angle error of the image to be screened is obtained based on the included angle.

[0031] Optionally, the step of determining a target image set from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and obtaining the calibration result of the camera model based on the target image set, includes:

[0032] A candidate image is randomly selected from the candidate image set corresponding to any one of the multiple initial camera models;

[0033] The verification image set obtained by iterating through the candidate images is used to obtain the verification calibration result of the verification image set through a preset algorithm;

[0034] The reprojection error of the candidate image set is obtained based on the verification calibration results;

[0035] The score of the candidate image set is determined by a preset evaluation function based on the reprojection error, the verification calibration result, and the initial calibration result.

[0036] The target image set is determined from the candidate image set corresponding to the initial camera model based on the scores of multiple candidate image sets.

[0037] The reference calibration result of the initial camera model is obtained based on the target image set, and the reference calibration result is used as the calibration result of the camera model.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a camera calibration device, the camera calibration device comprising:

[0039] The image filtering module is used to initialize the camera model to obtain an initial camera model, and input the calibration plate image sequence of the camera model into multiple initial camera models;

[0040] The image filtering module is also used to filter the calibration board image sequence through the initial camera model to obtain reference frame images and remaining images;

[0041] The camera calibration module is used to obtain the initial calibration result of the current initial camera model based on the reference frame image, and to filter the remaining images based on the initial calibration result to obtain a candidate image set corresponding to the initial camera model;

[0042] The camera calibration module is further configured to determine a target image set from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and obtain the calibration result of the camera model based on the target image set.

[0043] In addition, to achieve the above objectives, the present invention also proposes a camera calibration device, the camera calibration device comprising: a memory, a processor, and a camera calibration program stored in the memory and executable on the processor, the camera calibration program being configured to implement the steps of the camera calibration method as described above.

[0044] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a camera calibration program, which, when executed by a processor, implements the steps of the camera calibration method described above.

[0045] This invention adaptively selects reference images from the calibration board image sequence, evaluates the angular differences of the reference images to obtain the optimal image set, and optimizes the camera model based on the optimal image set to obtain more accurate calibration results. This solves the technical problem that when using a planar calibration board to calibrate a camera, degenerate poses in the shooting pose set affect the camera calibration, resulting in low camera calibration accuracy. Attached Figure Description

[0046] Figure 1This is a schematic diagram of the structure of a camera calibration device in the hardware operating environment involved in the embodiments of the present invention;

[0047] Figure 2 This is a flowchart illustrating the first embodiment of the camera calibration method of the present invention;

[0048] Figure 3 This is a flowchart illustrating the second embodiment of the camera calibration method of the present invention;

[0049] Figure 4 This is a flowchart illustrating the third embodiment of the camera calibration method of the present invention;

[0050] Figure 5 This is a structural block diagram of the first embodiment of the camera calibration device of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the camera calibration device structure in the hardware operating environment involved in the embodiments of the present invention.

[0054] like Figure 1 As shown, the camera calibration device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the camera calibration device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a camera calibration program.

[0057] exist Figure 1 In the camera calibration device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the camera calibration device of the present invention can be set in the camera calibration device, and the camera calibration device calls the camera calibration program stored in the memory 1005 through the processor 1001 and executes the camera calibration method provided in the embodiment of the present invention.

[0058] This invention provides a camera calibration method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of a camera calibration method according to the present invention.

[0059] In this embodiment, the camera calibration method includes the following steps:

[0060] Step S10: Initialize the camera model to obtain an initial camera model, and input the calibration board image sequence of the camera model into the initial camera model.

[0061] Understandably, the camera model can be a model set according to parameters such as the focal length of a real camera.

[0062] Understandably, the calibration board image sequence can be an image sequence obtained by taking a chessboard calibration board image with a real camera, or it can be a continuous chessboard calibration board image obtained from a chessboard video.

[0063] It should be noted that the initial camera model can be obtained by initializing the camera model by determining the principal point coordinates of the camera model from the image center of the camera model and initializing the distortion coefficients of the camera model. Initializing the distortion coefficients of the camera model can be done by setting the distortion coefficients to 0, which will result in the initial camera model.

[0064] It is worth noting that multiple camera models are initialized to obtain multiple initialized camera models. The calibration board image sequence of the camera models is input into multiple initialized camera models in parallel to obtain multiple candidate image sets. This avoids the image data deviation of a single candidate image set, provides fault tolerance for calculating the final calibration result, and makes the final calibration result of the camera model more accurate.

[0065] It should be emphasized that initializing the camera model to obtain an initial camera model and inputting the calibration board image sequence of the camera model into the initial camera model can be used to initialize multiple camera models to obtain multiple initial camera models, and the calibration board image sequence of the camera model can be synchronously input into each initial camera model.

[0066] It should be noted that in image measurement and machine vision applications, to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations; this process of solving for the parameters is called camera calibration.

[0067] Step S20: Filter the calibration board image sequence using the initial camera model to obtain reference frame images and remaining images.

[0068] Understandably, the reference frame image is an image selected by the initial camera model from the calibration board image sequence with an angle error less than an angle threshold.

[0069] It should be noted that the angle error can be understood as the angle difference between the checkerboard corner points in the calibration board image sequence and the points in the checkerboard in the camera coordinate space when the rectangular condition is met.

[0070] Simply put, the checkerboard mapping points in the camera coordinate space should be rectangles, but the actual mapping points are irregular quadrilaterals. The angle difference between the interior angles of the quadrilateral and the interior angles of the rectangle is the angle error.

[0071] It should be understood that the reference frame image can be multiple images or just one image, and the images in the calibration board image sequence other than the reference frame image can be regarded as the remaining images.

[0072] Step S30: Obtain the initial calibration result of the current initial camera model based on the reference frame image, and filter the remaining images based on the initial calibration result to obtain the candidate image set corresponding to the initial camera model.

[0073] Understandably, the reference frame image can be an image with a smaller error selected by the angle error, and a more accurate initial calibration result can be determined based on the reference frame image.

[0074] Understandably, the initial calibration result can be the initial calibration result of the initial camera model calculated based on the reference frame image using the Zhang Zhengyou calibration method.

[0075] It should be understood that there may be images with small angle errors and images with large angle errors among the remaining images. The angle error of each image in the remaining images is calculated based on the calibration results, and the remaining images are filtered according to the angle error.

[0076] It should be noted that the remaining images are filtered based on their angle error, and those with larger angle errors are selected as images in the candidate image set.

[0077] It should be emphasized that the candidate image set may include a reference frame image and images selected from the remaining images based on the reference frame image.

[0078] Step S40: Determine the target image set from the candidate image set corresponding to multiple initial camera models according to the preset evaluation function, and obtain the calibration result of the camera model according to the target image set.

[0079] It should be understood that, based on the preset evaluation function, the evaluation value of the candidate image set corresponding to each initial camera model can be calculated. The target image set is selected from the numerous candidate image sets based on the magnitude of the evaluation value. Then, the Zhang Zhengyou calibration algorithm is used to calculate the final calibration result based on the target image set, and the final calibration result is used as the calibration result of the camera model.

[0080] It should be noted that a candidate image is randomly obtained from the candidate image set corresponding to any one of the multiple initial camera models.

[0081] The candidate images are iterated to obtain a verification image set. Iteration can be understood as obtaining sub-images by rotating, flipping, etc., based on the candidate image set. The verification image set is constructed based on the sub-images. The verification calibration result of the verification image set is obtained through a preset algorithm. It can be understood that the preset algorithm can be Zhang Zhengyou calibration method. The reprojection error of the candidate image set is obtained based on the verification calibration result.

[0082] It should be noted that the preset evaluation function can be constructed based on the reciprocal of the reprojection error and parameter estimation variance of the candidate image set. The calculation formula for the preset evaluation function can be found in the following formula:

[0083]

[0084] in, The initial calibration result of the candidate image set is represented by α, and the validation calibration result is represented by C. α and β are weighting coefficients used to control the reprojection error and the variance of parameter estimation.

[0085] It should be further explained that the score of the candidate image set is determined by a preset evaluation function based on the reprojection error, the verification calibration result, and the initial calibration result; the target image set in the candidate image set corresponding to the initial camera model is determined based on the scores of the multiple candidate image sets; the reference calibration result of the initial camera model is obtained based on the target image set, and the reference calibration result is used as the calibration result of the camera model.

[0086] It should be further explained that, in order to prevent the calibration accuracy from being affected by different initial frame selections, a multi-parallel approach is adopted to generate multiple candidate image sets from the calibration board image sequence. Then, the candidate image sets are evaluated, and the best image set is used to generate the final camera calibration result.

[0087] In practice, firstly, an initial image is selected from each candidate image set. Then, an iterative method is used to generate some images to cover areas where the current calibration results are not good. This process generates 15-20 images, which are used to obtain a calibration result C = {fx, fy, cx, cy, k1, k2} as the initial result for camera calibration. Here, cx and cy are the principal point coordinates, k1 and k2 are parameters that are generally 0, and x and fy can be understood as the calibration result. Then, considering both the reprojection error and the variance between the initial calibration result and the validation calibration result, the scoring function for evaluating each candidate image set is the reciprocal of the reprojection error and the parameter estimation variance. Based on the above evaluation function, the candidate image set with the highest score is selected as the final image set used for camera calibration.

[0088] It should be noted that multiple camera models are initialized to obtain multiple initialized camera models. The calibration board image sequence of the camera models is input into multiple initialized camera models in parallel to obtain multiple candidate image sets. This avoids the image data deviation of a single candidate image set, provides fault tolerance for calculating the final calibration result, and makes the final calibration result of the camera model more accurate.

[0089] It is worth noting that by evaluating candidate image sets corresponding to multiple initial camera models, and obtaining candidate image sets that are more conducive to optimizing the calibration results based on the evaluation results, more accurate calibration results can be calculated.

[0090] This embodiment adaptively selects reference images from the calibration board image sequence, evaluates the angular differences of the reference images to obtain the optimal image set, and optimizes the camera model based on the optimal image set to obtain more accurate calibration results. This solves the technical problem that when using a planar calibration board to calibrate the camera, degenerate poses in the shooting pose set affect the camera calibration, resulting in low camera calibration accuracy.

[0091] refer to Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of a camera calibration method according to the present invention.

[0092] Based on the first embodiment described above, the camera calibration method of this embodiment further includes, in step S20:

[0093] Step S21: Randomly select the image corresponding to the current frame from the calibration board image sequence.

[0094] It is understandable that the images in the calibration board image sequence are taken frame by frame, and each image corresponds to a frame in a continuous image sequence. The image corresponding to the current frame can be simply understood as the randomly selected frame.

[0095] It should be understood that, in order to avoid obtaining multiple identical calibration board image sequences by copying the calibration board image sequence due to the special or unrepresentative nature of the randomly selected images, the calibration board image sequences are synchronously input into the corresponding initial camera models, and then reference frame images are further obtained.

[0096] It should be noted that the selection here can be an action performed to obtain an image, and may not have any other filtering or selection meaning.

[0097] Step S22: Obtain camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm, obtain the angle error based on the camera intrinsic parameters, and when the angle error is less than the angle threshold, use the image corresponding to the current frame as the reference frame image corresponding to the current frame, and return to execute the step of randomly selecting the image corresponding to the current frame from the calibration board image sequence to obtain multiple reference frame images.

[0098] Understandably, the preset algorithm can be Zhang Zhengyou's calibration algorithm.

[0099] It should be understood that the acquired reference frame image can be an image whose angle error has been calculated to be greater than the angle threshold by a preset algorithm. The angle threshold can be a preset angle error value, such as 1°, 5°, 8°, etc., and can be adjusted according to the actual situation.

[0100] It should be understood that camera intrinsic parameters can be the camera's internal coordinate transformation parameters obtained during camera calibration.

[0101] It should be noted that selecting reference frame images from the calibration board image sequence can be done by selecting images with angle errors less than the angle threshold. The camera intrinsic parameters can be calculated using the selected reference frame images with small angle errors, which can more accurately and effectively filter the calibration board image sequence.

[0102] It should be further noted that the number of reference frame images can be 3, 4, 5, etc. This embodiment does not limit this and can be adjusted according to the actual situation.

[0103] It should be emphasized that the current randomly selected image can be determined by calculating the camera intrinsic parameters together with all current reference frame images and the current randomly selected image, and then calculating the angle error of the current randomly selected image based on the camera intrinsic parameters.

[0104] In the specific implementation, it is first determined that three reference frame images need to be selected. An image is randomly selected from the image sequence, and its intrinsic parameters are estimated and the angle error of that frame is calculated using this image alone. If the angle error is less than a threshold, this frame is used as the first reference frame image; otherwise, images are continued to be selected from the image sequence until the first reference frame image is obtained. After obtaining the first frame image, the constraints on the camera model parameters are reduced to relax the camera model, setting only the distortion coefficients to zero. Then, another image is randomly selected from the image sequence. The camera parameters are jointly estimated using the first reference frame image and the currently stacked images obtained from the image sequence, and the angle error of this single frame image is calculated using the obtained parameters. If the angle error is less than a threshold, this frame image is used as the second reference frame image; otherwise, images are continued to be selected from the calibration board image sequence. After obtaining the second frame, the constraints on the camera model parameters are further reduced to relax the camera model, i.e., no constraints are added to the camera model. Then, the camera parameters are estimated and the angle error is calculated using the first reference frame image, the second reference frame image, and the image randomly obtained from the calibration board image sequence. When the angle error is less than the threshold, the frame image is used as the third reference frame image, thus completing the selection of all reference frame images.

[0105] It should be emphasized that before obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm, it is checked whether an acquired reference frame image exists. If not, it indicates that the first reference frame image needs to be acquired now. In this case, the camera intrinsic parameters are directly calculated based on the image corresponding to the current frame using the preset algorithm, and the angle error of the image corresponding to the current frame is obtained based on the camera intrinsic parameters. When the angle error is less than the angle threshold, the image corresponding to the current frame is used as the initial reference frame image (that is, the first reference frame image among all reference frame images). If yes, the step of obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using the preset algorithm is executed. The acquired reference frame image includes at least the initial reference frame image.

[0106] Step S23: Take the images other than the multiple reference frame images in the calibration board image sequence as the remaining images.

[0107] Understandably, by calculating the angle error of images randomly obtained from the calibration board images, the calibration board image sequence is divided into two parts: one part is the selected reference frame image, and the other part is the remaining images excluding the reference frame image.

[0108] It should be emphasized that the selection of reference frame images does not calculate the angle error of each image in the calibration board image sequence. It only determines whether a random image is a reference frame image from the randomly acquired images. Once the number of reference frame images reaches the number set by the user, no more images are randomly acquired, regardless of whether there are other images with smaller angle errors.

[0109] It is worth noting that multiple initial camera models can be manually selected simultaneously to screen the candidate image set. This is to avoid the randomly selected reference frame images in this step being images with angle errors close to the angle threshold, and also to avoid the randomly selected reference frame images in this step being images with very small angle errors. This would lead to a bias in the selection of images in the subsequent candidate image set, and further result in inaccurate final camera calibration results or over-convergence.

[0110] It is worth emphasizing that in the process of camera calibration using image sequences, not only are there a large number of similar images, but also a large number of degenerate poses. These images provide only slight assistance to the camera calibration process. Therefore, we need to identify the keyframes in the image sequence that play a decisive role in camera calibration. If the currently estimated parameters can be applied well to a certain frame (i.e., the angle error of a certain frame is calculated using the current parameter estimation results; the smaller the angle error, the better the current parameters can be applied to that frame), then it can be said that the image of that frame provides slight assistance to the final camera calibration. In other words, we prefer to obtain images that do not conform to the current parameters to participate in the camera calibration, so that the camera calibration results have higher versatility.

[0111] After obtaining the initial model parameters, we can evaluate subsequent images to determine their contribution to camera calibration. First, we extract images from the image sequence. Then, we use only one frame to estimate the camera model's parameters. Next, we calculate the angle error of that frame using the parameter estimation results. If the angle error is less than a threshold, we consider the current frame a valid, high-quality image that can help the model converge. Then, we calculate the angle error of the current frame using the parameter estimation results of all currently selected frames. If the angle error is greater than a threshold, it indicates that the current frame is helpful for optimizing the model parameters, and we add that frame to the candidate image set. We iterate through this process to filter images. Once the camera parameter accuracy meets the requirements, we stop selecting keyframes.

[0112] In this embodiment, reference frame images are selected from the calibration board image sequence. The angle error of the reference frame images is less than a preset threshold. The accuracy of the initial calibration results calculated from different reference frame images is different. When the candidate image set is screened in the subsequent process, a candidate image set that is more conducive to optimizing the initial camera model calibration results is obtained. The larger the angle error of the images screened from the image set, the more conducive it is to obtaining a more accurate calibration result for the camera model.

[0113] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a camera calibration method according to the present invention.

[0114] Based on the first embodiment described above, the camera calibration method of this embodiment further includes, in step S30:

[0115] Step S31: Obtain the initial calibration result of the current initial camera model based on the reference frame image using a preset algorithm.

[0116] Understandably, the preset algorithm can be Zhang Zhengyou's calibration algorithm. The initial calibration result calculated based on the reference frame image may not be accurate enough because the reference frame images are selected from images with very small angle errors. It is inevitable that the initial calibration result calculated by the initial camera model based on the reference frame image will be one-sided.

[0117] It should be understood that the selection of reference frame images may be to better select images with poor angular deviations from the remaining images for training and optimization of the camera model's calibration results.

[0118] Step S32: Randomly select an image to be screened from the remaining images, calculate the angle error of the image to be screened based on the initial calibration result, and when the angle error of the image to be screened is greater than the angle threshold, select the image to be screened as an image in the candidate image set.

[0119] Understandably, the image to be filtered can be a randomly selected image from the remaining images. It is necessary to calculate the angle error of the image to be filtered based on the reference frame image and the image to be filtered, and then filter the image to determine whether the image to be filtered is an image in the candidate image set.

[0120] It should be understood that the reference frame image is an image selected from the calibration image sequence with an angle error less than the angle threshold, and the images in the candidate image set can be images selected from the remaining images with an angle error greater than the angle threshold.

[0121] It should be noted that the calculation of the angle error of the image to be screened based on the initial calibration result can be achieved by obtaining the horizontal and vertical focal lengths and principal point coordinates of the initial camera model based on the initial calibration result; determining the corner pixel coordinates from the image to be screened; transforming the corner pixel coordinates based on the horizontal and vertical focal lengths to obtain normalized corner coordinates; obtaining the checkerboard direction vector based on the normalized corner coordinates; calculating the angle between the checkerboard direction vectors based on the three-dimensional origin coordinates of the initial camera model; and obtaining the angle error of the image to be screened based on the angle.

[0122] In simple terms, it can be understood as using the rectangular information of the checkerboard pattern to directly project points on the image into 3D coordinates in the camera coordinate system, and then determining whether the 3D points in the camera coordinate system satisfy the rectangular condition. The angle difference between the four points of the rectangle on the checkerboard pattern and the angle between the four points of the actual rectangle can be understood as the angle error.

[0123] In practical implementation, assuming the camera model has no distortion, the pixel coordinates of the four corner points of a rectangle are: a(u1,v1), b(u2,v2), c(u3,v3), d(u4,v4). The formula for converting pixel coordinates to normalized coordinates is:

[0124]

[0125] Where (x,y) represents normalized coordinates, (u,v) represents pixel coordinates, and f x f y c represents focal length. x ,c y This represents the coordinates of the principal point. Using the above formula, the pixel coordinates of the four corner points are converted into normalized coordinates: A(x1,y1,1), B(x2,y2,1), C(x3,y3,1), D(x4,y4,1).

[0126] Assuming the origin O(0,0,0) represents the camera's 3D coordinates, the following relationship exists:

[0127]

[0128] Where t1, t2, t3, and t4 represent proportionality coefficients. This represents the coordinates of a corner point in the 3D camera coordinate system.

[0129] because Let represent the four corner points of the same chessboard grid. Then, we obtain the following homogeneous linear equation system:

[0130]

[0131] From the above equation, we can see that the degree of freedom of the solution to this system of equations is one, that is...

[0132]

[0133] Where a, b, c, and d are all known constants, which can be obtained by solving equation (15). It is not difficult to conclude that... and The angle between the two vectors is independent of t, and based on the characteristics of the checkerboard rectangle, the angle should be 90 degrees. The higher the accuracy of the camera's intrinsic parameters, the closer the angle between these two vectors will be to 90 degrees. The evaluation metrics are defined as follows:

[0134]

[0135] Here, `degress(.)` represents the angle function used to calculate the angle between two vectors, ε. error This represents the angular error.

[0136] Step S33: Repeat the steps of randomly obtaining images to be screened from the remaining images, calculating the angle error of the images to be screened based on the initial calibration result, and using the images to be screened as images in the candidate image set when the angle error of the images to be screened is greater than the angle threshold, until the angle error of each image to be screened in the remaining images is calculated to obtain the candidate image set.

[0137] Understandably, the angle error of each image in the remaining images is calculated based on the reference frame image, and images with an angle error greater than the angle threshold are used as the candidate image set.

[0138] It should be understood that, since multiple initial camera models filter the images in the calibration board image sequence, each initial camera model will obtain a corresponding candidate image set, and it is easy to imagine that the images in each candidate image set can be different.

[0139] In this embodiment, images with angle errors greater than an angle threshold are selected from the remaining calibration board image sequence using the selected reference frame images as the candidate image set. Selecting images with angle errors greater than the angle threshold can provide more degenerate pose references for camera model calibration. The calibration results of the camera model are calculated based on the images in this candidate image set, which can make the calibration results of the camera model more versatile.

[0140] Furthermore, this embodiment of the invention also proposes a storage medium storing a camera calibration program, which, when executed by a processor, implements the steps of the camera calibration method described above.

[0141] Reference Figure 5 , Figure 5This is a structural block diagram of the first embodiment of the camera calibration device of the present invention.

[0142] like Figure 5 As shown, the camera calibration device proposed in this embodiment of the invention includes:

[0143] Image filtering module 10 is used to initialize the camera model to obtain an initial camera model, and input the calibration plate image sequence of the camera model into multiple initial camera models.

[0144] The image filtering module 10 is also used to filter the calibration board image sequence through the initial camera model to obtain reference frame images and remaining images;

[0145] The camera calibration module 20 is used to obtain the initial calibration result of the current initial camera model based on the reference frame image, and to filter the remaining images based on the initial calibration result to obtain a candidate image set corresponding to the initial camera model.

[0146] The camera calibration module 20 is further configured to determine a target image set from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and obtain the calibration result of the camera model based on the target image set.

[0147] This embodiment adaptively selects reference images from the calibration board image sequence, evaluates the angular differences of the reference images to obtain the optimal image set, and optimizes the camera model based on the optimal image set to obtain more accurate calibration results. This solves the technical problem that when using a planar calibration board to calibrate the camera, degenerate poses in the shooting pose set affect the camera calibration, resulting in low camera calibration accuracy.

[0148] In one embodiment, the image filtering module 10 is further configured to set the principal point coordinates of the camera model and initialize the distortion coefficients of the camera model to obtain an initial camera model, wherein the principal point coordinates are determined by the image center of the camera model;

[0149] The calibration plate image sequence of the camera model is input into the initial camera model.

[0150] In one embodiment, the image filtering module 10 is further configured to randomly select the image corresponding to the current frame from the calibration board image sequence;

[0151] The camera intrinsic parameters are obtained by using a preset algorithm based on the image corresponding to the current frame and the acquired reference frame image. The angle error is obtained based on the camera intrinsic parameters. When the angle error is less than the angle threshold, the image corresponding to the current frame is used as the reference frame image corresponding to the current frame. Then, the step of randomly selecting the image corresponding to the current frame from the calibration board image sequence is returned to execute, so as to obtain multiple reference frame images.

[0152] The images other than the multiple reference frame images in the calibration board image sequence are taken as the remaining images.

[0153] In one embodiment, the image filtering module 10 is further configured to detect whether an acquired reference frame image exists;

[0154] If not, then the camera intrinsic parameters are obtained based on the image corresponding to the current frame using a preset algorithm, the angle error is obtained based on the camera intrinsic parameters, and when the angle error is less than the angle threshold, the image corresponding to the current frame is used as the initial reference frame image.

[0155] If so, then the step of obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm is executed, wherein the acquired reference frame image includes at least the initial reference frame image.

[0156] In one embodiment, the camera calibration module 20 is further configured to obtain the initial calibration result of the current initial camera model based on the reference frame image using a preset algorithm;

[0157] Randomly select images to be screened from the remaining images, calculate the angle error of the images to be screened based on the initial calibration results, and when the angle error of the images to be screened is greater than the angle threshold, select the images to be screened as images in the candidate image set.

[0158] Repeat the steps of randomly selecting images to be screened from the remaining images, calculating the angle error of the images to be screened based on the initial calibration result, and including the images to be screened as images in the candidate image set when the angle error of the images to be screened is greater than the angle threshold, until the angle error of each image to be screened in the remaining images is calculated to obtain the candidate image set.

[0159] In one embodiment, the camera calibration module 20 is further configured to obtain the horizontal axis focal length, vertical axis focal length, and principal point coordinates of the initial camera model based on the initial calibration result;

[0160] The corner pixel coordinates are determined from the image to be screened, and the corner pixel coordinates are transformed according to the horizontal axis focal length and the vertical axis focal length to obtain the corner normalized coordinates;

[0161] The checkerboard direction vector is obtained based on the normalized coordinates of the corner points, and the angle between the checkerboard direction vectors is calculated based on the three-dimensional origin coordinates of the initial camera model.

[0162] The angle error of the image to be screened is obtained based on the included angle.

[0163] In one embodiment, the camera calibration module 20 is further configured to randomly obtain a candidate image from the candidate image set corresponding to any one of the plurality of initial camera models;

[0164] The verification image set obtained by iterating through the candidate images is used to obtain the verification calibration result of the verification image set through a preset algorithm;

[0165] The reprojection error of the candidate image set is obtained based on the verification calibration results;

[0166] The score of the candidate image set is determined by a preset evaluation function based on the reprojection error, the verification calibration result, and the initial calibration result.

[0167] The target image set is determined from the candidate image set corresponding to the initial camera model based on the scores of multiple candidate image sets.

[0168] The reference calibration result of the initial camera model is obtained based on the target image set, and the reference calibration result is used as the calibration result of the camera model.

[0169] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0170] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0171] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0172] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0174] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A camera calibration method, characterized in that, The camera calibration method includes: The camera model is initialized to obtain an initial camera model, and the calibration board image sequence of the camera model is input into the initial camera model; The calibration board image sequence is filtered using the initial camera model to obtain reference frame images and remaining images; The initial calibration result of the current initial camera model is obtained based on the reference frame image, and the remaining images are filtered based on the initial calibration result to obtain the candidate image set corresponding to the initial camera model; A target image set is determined from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and the calibration result of the camera model is obtained based on the target image set; The step of obtaining the initial calibration result of the current initial camera model based on the reference frame image, and filtering the remaining images based on the initial calibration result to obtain a candidate image set corresponding to the initial camera model, includes: The initial calibration result of the current initial camera model is obtained based on the reference frame image using a preset algorithm; Randomly select images to be screened from the remaining images, calculate the angle error of the images to be screened based on the initial calibration results, and when the angle error of the images to be screened is greater than the angle threshold, select the images to be screened as images in the candidate image set. The process of randomly selecting images to be screened from the remaining images, calculating the angle error of the images to be screened based on the initial calibration result, and including the images to be screened as images in the candidate image set when the angle error of the images to be screened is greater than the angle threshold is repeated until the angle error of each image to be screened in the remaining images is calculated to obtain the candidate image set. The image angle error is used to characterize whether the shooting pose corresponding to the image is a degenerate pose. The candidate image set also includes a reference frame image. The step of filtering the calibration board image sequence using the initial camera model to obtain reference frame images and remaining images includes: Randomly select the image corresponding to the current frame from the calibration board image sequence; The camera intrinsic parameters are obtained by using a preset algorithm based on the image corresponding to the current frame and the acquired reference frame image. The angle error is obtained based on the camera intrinsic parameters. When the angle error is less than the angle threshold, the image corresponding to the current frame is used as the reference frame image corresponding to the current frame. Then, the step of randomly selecting the image corresponding to the current frame from the calibration board image sequence is returned to execute, so as to obtain multiple reference frame images. The images other than the multiple reference frame images in the calibration board image sequence are taken as the remaining images.

2. The camera calibration method as described in claim 1, characterized in that, The process of initializing the camera model to obtain an initial camera model and inputting the calibration board image sequence of the camera model into the initial camera model includes: Set the principal point coordinates of the camera model and initialize the distortion coefficients of the camera model to obtain an initial camera model. The principal point coordinates are determined by the image center of the camera model. The calibration plate image sequence of the camera model is input into the initial camera model.

3. The camera calibration method as described in claim 1, characterized in that, Before obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm, the method further includes: Detect whether a reference frame image has been acquired; If not, then the camera intrinsic parameters are obtained based on the image corresponding to the current frame using a preset algorithm, the angle error is obtained based on the camera intrinsic parameters, and when the angle error is less than the angle threshold, the image corresponding to the current frame is used as the initial reference frame image. If so, then the step of obtaining camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm is executed, wherein the acquired reference frame image includes at least the initial reference frame image.

4. The camera calibration method as described in claim 1, characterized in that, The step of calculating the angle error of the image to be screened based on the initial calibration result includes: Based on the initial calibration results, the horizontal focal length, vertical focal length, and principal point coordinates of the initial camera model are obtained. The corner pixel coordinates are determined from the image to be screened, and the corner pixel coordinates are transformed according to the horizontal axis focal length and the vertical axis focal length to obtain the corner normalized coordinates; The checkerboard direction vector is obtained based on the normalized coordinates of the corner points, and the angle between the checkerboard direction vectors is calculated based on the three-dimensional origin coordinates of the initial camera model. The angle error of the image to be screened is obtained based on the included angle.

5. The camera calibration method as described in claim 1, characterized in that, The step of determining a target image set from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and obtaining the calibration result of the camera model based on the target image set, includes: A candidate image is randomly selected from the candidate image set corresponding to any one of the multiple initial camera models; The verification image set obtained by iterating through the candidate images is used to obtain the verification calibration result of the verification image set through a preset algorithm; The reprojection error of the candidate image set is obtained based on the verification calibration results; The score of the candidate image set is determined by a preset evaluation function based on the reprojection error, the verification calibration result, and the initial calibration result. The target image set is determined from the candidate image set corresponding to the initial camera model based on the scores of multiple candidate image sets. The reference calibration result of the initial camera model is obtained based on the target image set, and the reference calibration result is used as the calibration result of the camera model.

6. A camera calibration device, characterized in that, The camera calibration device includes: The image filtering module is used to initialize the camera model to obtain an initial camera model, and input the calibration plate image sequence of the camera model into multiple initial camera models; The image filtering module is also used to filter the calibration board image sequence through the initial camera model to obtain reference frame images and remaining images; The camera calibration module is used to obtain the initial calibration result of the current initial camera model based on the reference frame image, and to filter the remaining images based on the initial calibration result to obtain a candidate image set corresponding to the initial camera model. The camera calibration module is further configured to determine a target image set from a candidate image set corresponding to multiple initial camera models according to a preset evaluation function, and obtain the calibration result of the camera model based on the target image set; The camera calibration module is further configured to obtain the initial calibration result of the current initial camera model based on the reference frame image using a preset algorithm; randomly select images to be screened from the remaining images; calculate the angle error of the images to be screened based on the initial calibration result; and when the angle error of the images to be screened is greater than an angle threshold, select the images to be screened as images in the candidate image set; repeat the steps of randomly selecting images to be screened from the remaining images, calculating the angle error of the images to be screened based on the initial calibration result, and selecting the images to be screened as images in the candidate image set when the angle error of the images to be screened is greater than an angle threshold, until the angle error of each image to be screened in the remaining images is calculated to obtain a candidate image set, wherein the image angle error is used to characterize whether the shooting pose corresponding to the image is a degenerate pose, and the candidate image set also includes the reference frame image; The image filtering module is further configured to randomly select the image corresponding to the current frame from the calibration board image sequence; obtain camera intrinsic parameters based on the image corresponding to the current frame and the acquired reference frame image using a preset algorithm; obtain the angle error based on the camera intrinsic parameters; and when the angle error is less than the angle threshold, use the image corresponding to the current frame as the reference frame image corresponding to the current frame, and return to execute the step of randomly selecting the image corresponding to the current frame from the calibration board image sequence to obtain multiple reference frame images; and use the other images in the calibration board image sequence besides the multiple reference frame images as the remaining images.

7. A camera calibration device, characterized in that, The device includes: a memory, a processor, and a camera calibration program stored in the memory and executable on the processor, the camera calibration program being configured to implement the camera calibration method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a camera calibration program, which, when executed by a processor, implements the camera calibration method as described in any one of claims 1 to 5.

Images