Three-dimensional calibration method and system based on adaptive multi-view optimization algorithm, and medium

Abstract

The invention provides a three-dimensional calibration method and system based on a self-adaptive multi-view optimization algorithm and a medium, and the method comprises the steps: automatically shooting images of a calibration target surface at multiple angles based on a camera, and obtaining a multi-view image; extracting feature points based on the multi-view image, and extracting positions of calibration points in the image based on the feature points; searching feature points corresponding to the same calibration point in different images based on a feature matching algorithm; analyzing the density and distribution of image feature points based on a feature matching result, evaluating the current posture of the camera, and dynamically adjusting a shooting angle according to an evaluation result; analyzing a re-projection error of the same calibration point based on a feature matching result, and dynamically adjusting internal and external parameters of the camera based on the re-projection error; the method comprises the following steps: shooting a multi-view image, performing full-view scanning on a calibration target surface, and analyzing calibrated feature matching and reprojection errors to dynamically adjust internal and external parameters of a camera, thereby realizing a high-precision, high-adaptability and high-efficiency calibration process.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of machine vision and image processing, in particular to a three-dimensional calibration method, system and medium based on an adaptive multi-view optimization algorithm. BACKGROUND

[0002] Three-dimensional calibration is a basic link in the field of machine vision, and directly determines the accuracy and reliability of subsequent three-dimensional reconstruction, spatial positioning and dimension measurement. In the prior art, a two-dimensional checkerboard is usually used in combination with single-view shooting, and a camera internal and external parameter is estimated by combining a Zhang calibration algorithm, but the method has the following disadvantages in a multi-angle, multi-view or multi-camera system:

[0003] The posture is not adjustable: the traditional calibration device has a fixed view angle and lacks flexible angle adaptability;

[0004] Image quality depends on ambient light: in a high-reflectivity or low-illumination scene, the calibration failure rate is high;

[0005] Insufficient feature point redundancy: a planar checkerboard cannot provide spatial depth information, and three-dimensional reconstruction error is large;

[0006] Frequent human intervention: manual adjustment of the calibration posture is required frequently, which is low in efficiency and prone to errors;

[0007] Cannot adapt to automation: automatic calibration processes cannot be performed on unmanned production lines.

[0008] To solve the above problems, the application provides a system integrating a hardware adjustment structure, an intelligent algorithm and a visual feedback mechanism, which can automatically complete three-dimensional high-precision calibration in a dynamic environment. SUMMARY

[0009] The purpose of the embodiments of the application is to provide a three-dimensional calibration method, system and medium based on an adaptive multi-view optimization algorithm, which can realize a high-precision, strong-adaptability and high-efficiency calibration process by shooting multi-view images, full-view angle scanning of a calibration target surface and dynamic adjustment of camera internal and external parameters based on feature matching and re-projection error analysis of the calibration.

[0010] The embodiments of the application also provide a three-dimensional calibration method based on an adaptive multi-view optimization algorithm, which comprises the following steps:

[0011] Based on observation of the calibration target surface from different positions and angles by a camera, multiple-angle images of the calibration target surface are automatically shot to obtain multi-view images;

[0012] Feature points are extracted based on the multi-view images, and the positions of calibration points in the images are extracted based on the feature points;

[0013] Based on the feature matching algorithm, the feature points corresponding to the same calibration point in different images are searched to obtain the feature matching results;

[0014] Analyze the density and distribution of image feature points based on feature matching results, evaluate the current camera posture, and dynamically adjust the shooting angle based on the evaluation results;

[0015] The reprojection error of the same calibration point is analyzed based on the feature matching results, and the camera intrinsics and extrinsic parameters are dynamically adjusted based on the reprojection error.

[0016] Optionally, in the three-dimensional calibration method based on the adaptive multi-view optimization algorithm described in the embodiment of the present application, the camera observes the calibration target surface from different positions and angles, automatically captures images of the calibration target surface at multiple angles, and obtains multi-view images, specifically including:

[0017] Initialize the camera and gimbal to obtain their initial parameters;

[0018] Configuring the camera's acquisition parameters and calibrating the gimbal's zero point and motion range based on the initial parameters, wherein the camera's parameters include focal length, aperture, and white balance;

[0019] The gimbal rotates slowly, scanning the full viewing angle range in 1° steps. Every time the gimbal rotates 1°, it collects and calibrates the target surface images in sequence.

[0020] Perform frame processing on the calibration target surface image to obtain each frame image;

[0021] Each frame of image is marked with a timestamp and stored in a cache queue to obtain a multi-view image.

[0022] Optionally, in the three-dimensional calibration method based on the adaptive multi-view optimization algorithm described in the embodiment of the present application, extracting feature points based on the multi-view image, and extracting the position of the calibration point in the image based on the feature points specifically includes:

[0023] Extract feature points of multiple visual images based on feature point extraction algorithm;

[0024] Use Harris corner point or SIFT algorithm to extract the corner points and edge intersection points of the calibration target surface in the image;

[0025] Use the descriptor of the feature points to initially match the corner points, edge intersection points and feature points to determine the calibration points;

[0026] Establish a camera projection model and calculate the three-dimensional coordinates of the calibration points;

[0027] The calculated three-dimensional coordinates of the calibration points are projected onto each image plane through the camera projection model to obtain the two-dimensional coordinates of the calibration points in the image and the position of the calibration points in the image.

[0028] Optionally, in the three-dimensional calibration method based on the adaptive multi-view optimization algorithm, the feature matching algorithm is used to search for feature points corresponding to the same calibration point in different images to obtain a feature matching result, and the method specifically comprises the following steps:

[0029] The SIFT algorithm is used to extract feature points and descriptors from each image.

[0030] Any two images are selected, and the Euclidean distance between the descriptor of each feature point in one image and the descriptors of all feature points in the other image is calculated.

[0031] A distance threshold is set, and it is determined whether the Euclidean distance is less than the distance threshold. If yes, the current feature point is selected as a matching point, and if no, the current feature point is removed.

[0032] A distance ratio threshold is set, and the distance ratio between the descriptor of each feature point and the descriptors of the nearest neighbor and the second nearest neighbor feature points is calculated by using a ratio test method.

[0033] If the distance ratio is less than the distance ratio threshold, a feature matching result is obtained.

[0034] If the distance ratio is greater than or equal to the distance ratio threshold, the matching point is extracted.

[0035] Optionally, in the three-dimensional calibration method based on the adaptive multi-view optimization algorithm, the feature matching result is used to analyze the feature point density and distribution of the image, the current pose of the camera is evaluated, and the shooting angle is dynamically adjusted according to the evaluation result.

[0036] Each image is divided into a plurality of small regions, and the number of feature points in each small region is counted.

[0037] The feature point density of each small region is obtained by dividing the number of feature points in the small region by the area of the region.

[0038] The shape of the calibration target surface in the field of view of the camera is obtained, and it is determined whether the shape is regular. If yes, the feature point distribution is obtained, the feature point density and distribution of a plurality of images are used to evaluate the current pose, and an evaluation result is obtained.

[0039] If no, the camera is rotated, tilted or translated.

[0040] The feature point density is analyzed based on the evaluation result to determine whether the feature point density is greater than or equal to a set density threshold.

[0041] If yes, the distance between the camera and the calibration target surface is increased.

[0042] If no, the distance between the camera and the calibration target surface is decreased.

[0043] Optionally, in the three-dimensional calibration method based on the adaptive multi-view optimization algorithm, the re-projection error of the same calibration point is analyzed based on the feature matching result, and the camera intrinsic parameter and the camera extrinsic parameter are dynamically adjusted based on the re-projection error. Specifically, the method comprises the following steps:

[0044] The feature point coordinates of the same calibration point in different images are obtained based on the feature matching result.

[0045] The projection relationship from the calibration point in the world coordinate system to the image pixel coordinate system is established based on the camera imaging model.

[0046] For each calibration point, the coordinates of the calibration point in the world coordinate system are projected onto the image through the current camera intrinsic parameter and the camera extrinsic parameter to obtain the coordinates of the re-projection point.

[0047] The Euclidean distance between the re-projection point and the actual image feature point coordinates obtained through the feature matching is calculated to obtain the re-projection error.

[0048] The camera intrinsic parameter and the camera extrinsic parameter are dynamically adjusted based on the re-projection error.

[0049] In a second aspect, the embodiments of the present application provide a three-dimensional calibration system based on an adaptive multi-view optimization algorithm. The system comprises a memory and a processor. The memory comprises a program of a three-dimensional calibration method based on an adaptive multi-view optimization algorithm. When the program of the three-dimensional calibration method based on an adaptive multi-view optimization algorithm is executed by the processor, the following steps are implemented:

[0050] Based on the observation of the calibration target surface from different positions and angles by the camera, multiple-angle images of the calibration target surface are automatically captured to obtain multi-view images.

[0051] Feature points are extracted based on the multi-view images, and the positions of the calibration points in the images are extracted based on the feature points.

[0052] Feature points corresponding to the same calibration point in different images are searched based on a feature matching algorithm to obtain a feature matching result.

[0053] The image feature point density and distribution are analyzed based on the feature matching result, the current pose of the camera is evaluated, and the shooting angle is dynamically adjusted according to the evaluation result.

[0054] The re-projection error of the same calibration point is analyzed based on the feature matching result, and the camera intrinsic parameter and the camera extrinsic parameter are dynamically adjusted based on the re-projection error.

[0055] Optionally, in the three-dimensional calibration system based on the adaptive multi-view optimization algorithm, based on the observation of the calibration target surface from different positions and angles by the camera, multiple-angle images of the calibration target surface are automatically captured to obtain multi-view images. Specifically, the method comprises the following steps:

[0056] Initialize the camera and the gimbal to obtain initial parameters of the camera and the gimbal;

[0057] Configure the collection parameters of the camera and the zero point and the motion range of the gimbal based on the initial parameters, wherein the parameters of the camera include focal length, aperture and white balance;

[0058] Slowly rotate the gimbal to scan the full visual angle range by 1° step, and collect the target surface images in sequence for each 1° rotation of the gimbal;

[0059] Frame process the target surface images to obtain each frame of image;

[0060] Mark the time stamp of each frame of image and store it in the cache queue to obtain the multi-view images.

[0061] Optionally, in the three-dimensional calibration system based on the adaptive multi-view optimization algorithm, the feature points are extracted based on the multi-view images, and the positions of the calibration points in the images are extracted based on the feature points, and the method specifically comprises the following steps:

[0062] Extract the feature points of the multi-view images based on the feature point extraction algorithm;

[0063] Extract the corner points and edge intersection points of the target surface in the images by using the Harris corner point or SIFT algorithm;

[0064] Determine the calibration points by using the descriptors of the feature points to initially match the corner points, the edge intersection points and the feature points;

[0065] Establish a camera projection model and calculate the three-dimensional coordinates of the calibration points;

[0066] Project the calculated three-dimensional coordinates of the calibration points onto each image plane by using the camera projection model to obtain the two-dimensional coordinates of the calibration points in the images and the positions of the calibration points in the images.

[0067] In a third aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium comprises a three-dimensional calibration method program based on an adaptive multi-view optimization algorithm, and the three-dimensional calibration method program based on the adaptive multi-view optimization algorithm is executed by a processor to implement the steps of the three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to any one of the above embodiments.

[0068] From the above, the three-dimensional calibration method, system and medium based on the adaptive multi-view optimization algorithm provided by the embodiments of the present application are known. The multi-view images are obtained by observing the calibration target surface from different positions and angles based on the camera, automatically shooting the images of multiple angles of the calibration target surface, and extracting the feature points based on the multi-view images. The positions of the calibration points in the images are extracted based on the feature points. The feature matching algorithm is used to search for the feature points corresponding to the same calibration point in different images, and the feature matching result is obtained. The image feature point density and distribution are analyzed based on the feature matching result, the current posture of the camera is evaluated, and the shooting angle is dynamically adjusted according to the evaluation result. The reprojection error of the same calibration point is analyzed based on the feature matching result, and the camera internal and external parameters are dynamically adjusted based on the reprojection error. The calibration target surface is scanned in full view by shooting the multi-view images, and the camera internal and external parameters are dynamically adjusted by analyzing the feature matching and the reprojection error of the calibration. The high-precision, strong-adaptability and high-efficiency calibration process is realized. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0070] Figure 1 The flow chart of the three-dimensional calibration method based on the adaptive multi-view optimization algorithm provided by the embodiments of the present application is shown in the figure.

[0071] Figure 2 The multi-view image acquisition method flow chart of the three-dimensional calibration method based on the adaptive multi-view optimization algorithm provided by the embodiments of the present application is shown in the figure.

[0072] Figure 3 The calibration point position analysis flow chart of the three-dimensional calibration method based on the adaptive multi-view optimization algorithm provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0073] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0074] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0075] Please refer to Figure 1 , Figure 1 is a flowchart of a three-dimensional calibration method based on an adaptive multi-view optimization algorithm in some embodiments of the present application. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm is used in a terminal device. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm comprises the following steps:

[0076] S101, based on the camera observing the calibration target surface from different positions and angles, automatically shooting images of the calibration target surface from multiple angles to obtain multi-view images;

[0077] S102, extracting feature points based on the multi-view images, and extracting the positions of the calibration points in the images based on the feature points;

[0078] S103, searching for feature points corresponding to the same calibration point in different images based on a feature matching algorithm to obtain a feature matching result;

[0079] S104, analyzing the feature point density and distribution of the images based on the feature matching result, evaluating the current pose of the camera, and dynamically adjusting the shooting angle according to the evaluation result;

[0080] S105, analyzing the re-projection error of the same calibration point based on the feature matching result, and dynamically adjusting the camera internal and external parameters based on the re-projection error.

[0081] It is necessary to note that high-precision and high-resolution industrial cameras or professional photography cameras are selected to ensure clear and accurate images. The parameters of the camera (such as focal length, aperture, and sensitivity) need to be adjusted and set according to the actual situation to meet the imaging requirements under different lighting conditions and shooting distances.

[0082] Calibration target: a planar object with specific patterns or features, such as a checkerboard target, a circular point array target, etc. Its pattern design should have features that are easy to identify and locate, so that feature points can be accurately extracted in the image. The size and accuracy of the calibration target also need to be selected according to the specific application and the field of view range of the camera.

[0083] Camera motion mechanism: used to control the position and angle of the camera in space. It can be a mechanical arm, a gimbal, a track, etc. Through precise motor drive and control system, it realizes flexible adjustment of the camera in different positions and angles. The motion mechanism should have high positioning accuracy and repeatability to ensure that the position and attitude of the camera are accurate and reliable during each shooting.

[0084] According to the image feature point density and distribution, the current attitude is evaluated, and the evaluation formula is as follows:

[0085]

[0086] θ (Pitch) represents the rotation angle of the device or camera around the horizontal axis (X-axis), i.e. the degree of up-down tilt.

[0087] θ = 0°: the camera is horizontal and facing forward.

[0088] θ > 0°: the camera looks up.

[0089] θ < 0°: the camera looks down.

[0090] φ (Yaw) represents the rotation angle of the device or camera around the vertical axis (Y-axis), i.e. the degree of left-right turning.

[0091] φ = 0°: the camera is facing straight ahead.

[0092] φ > 0°: the camera turns left.

[0093] φ < 0°: the camera turns right.

[0094] s is a scale factor, which represents the depth information or scale factor when the three-dimensional world point is projected to the image plane.

[0095] p ij represents the predicted projection position of point Xi in the jth image after transformation by the camera internal and external parameters, i represents the number of calibration points (i.e. the number of three-dimensional points), and j represents the number of image frames (i.e. the point in which image).

[0096] M represents the number of image regions or feature points for statistics or calculation;

[0097] represents the gray value intensity or pixel value information of the image at the pixel point p ij .

[0098] The gimbal automatically adjusts the angles θ and φ to maximize the information entropy S, forming an automatic angle finding mechanism.

[0099] Please refer to Figure 2 , Figure 2 is a multi-view image acquisition method flowchart of a three-dimensional calibration method based on an adaptive multi-view optimization algorithm in some embodiments of the present application. According to the embodiment of the present application, the camera automatically captures images of the calibration target surface at multiple angles based on observing the calibration target surface from different positions and angles, and obtains multi-view images, specifically including:

[0100] S201, initializing and setting the camera and the gimbal to obtain initial parameters of the camera and the gimbal;

[0101] S202, configuring the acquisition parameters of the camera and the zero point and motion range of the calibration gimbal based on the initial parameters, the parameters of the camera including focal length, aperture and white balance;

[0102] S203, the gimbal slowly rotates to scan the full angle range at 1° step, and the calibration target surface image is collected in sequence every 1° rotation of the gimbal;

[0103] S204, frame processing is performed on the calibration target surface image to obtain each frame image;

[0104] S205, each frame image is labeled with a time stamp and stored in a cache queue to obtain multi-view images.

[0105] Please refer to Figure 3 , Figure 3 is a calibration point position analysis flowchart of a three-dimensional calibration method based on an adaptive multi-view optimization algorithm in some embodiments of the present application. According to the embodiment of the present application, the feature points are extracted based on the multi-view images, and the positions of the calibration points in the images are extracted based on the feature points, specifically including:

[0106] S301, extracting feature points of the multi-view images based on a feature point extraction algorithm;

[0107] S302, using Harris corner point or SIFT algorithm to extract corner points and edge intersection points of the calibration target surface in the image;

[0108] S303, using the descriptor of the feature points to initially match the corner points, edge intersection points and feature points to determine the calibration points;

[0109] S304, establishing a camera projection model, and calculating the three-dimensional coordinates of the calibration points;

[0110] S305, projecting the calculated three-dimensional coordinates of the calibration points onto each image plane through the camera projection model to obtain the two-dimensional coordinates of the calibration points in the images, and obtaining the positions of the calibration points in the images.

[0111] It should be noted that, according to the imaging principle of the camera, a mathematical model of the camera is established, such as a pinhole camera model. The model describes how a point in three-dimensional space is projected onto a two-dimensional image plane through the internal parameters (such as focal length, principal point coordinates, etc.) and external parameters (such as rotation and translation relationship) of the camera.

[0112] Solving camera parameters: using the matched feature points, the internal and external parameters of the camera are solved through a series of mathematical calculations. For example, an optimization algorithm such as least squares method is used to minimize the projection error, so as to obtain the optimal camera parameter estimation.

[0113] Calculating the three-dimensional coordinates of the calibration points: on the basis of the known camera parameters, combined with the image information under multiple viewing angles, the coordinates of the calibration points in three-dimensional space can be calculated through triangulation and other methods. Specifically, for each calibration point, find its corresponding feature points in images under different viewing angles, establish an equation set according to the projection model of the camera and the known camera parameters, and solve the three-dimensional coordinates of the calibration point.

[0114] Projecting to the image plane to determine the position: projecting the calculated three-dimensional coordinates of the calibration points onto each image plane through the projection model of the camera to obtain the two-dimensional coordinates of the calibration points in the images, i.e. determining the positions of the calibration points in the images.

[0115] According to the embodiment of the present application, the feature matching algorithm is used to search for feature points corresponding to the same calibration point in different images to obtain the feature matching result, which specifically includes:

[0116] Extracting feature points and descriptors from each image based on the SIFT algorithm;

[0117] Selecting any two images, calculating the Euclidean distance between the descriptor of each feature point in one image and the descriptors of all feature points in the other image;

[0118] Setting a distance threshold, judging whether the Euclidean distance is less than the distance threshold, if yes, selecting the current feature point as the matching point, if no, eliminating the current feature point;

[0119] Setting a distance ratio threshold, and using the ratio test method to calculate the distance ratio of each feature point to the nearest neighbor and the second nearest neighbor feature point descriptors;

[0120] If the distance ratio is less than the distance ratio threshold, the feature matching result is obtained;

[0121] If the distance ratio is greater than or equal to the distance ratio threshold, the matching point is extracted.

[0122] It should be noted that the SIFT algorithm is first used to extract feature points and their descriptors for each image. SIFT feature points are scale-invariant, rotation-invariant, and partially affine-invariant. Its descriptor is a 128-dimensional vector that can well represent the local characteristics of the feature points.

[0123] Initial matching: For any two images, the Euclidean distance between each feature point descriptor in one image and all feature point descriptors in the other image is calculated, and the feature point with the closest distance is selected as the initial matching point. To improve matching accuracy, a distance threshold can be set, and only matching points with a distance less than the threshold are retained.

[0124] Ratio test: To further remove mismatched points, a ratio test is used. For each feature point, the ratio of its distance to the nearest neighbor and the next nearest neighbor feature point descriptors is calculated. If the ratio is less than a preset threshold (usually around 0.7), the matching point is considered reliable; otherwise, it is removed.

[0125] According to an embodiment of the present invention, the density and distribution of image feature points are analyzed based on feature matching results, the current posture of the camera is evaluated, and the shooting angle is dynamically adjusted according to the evaluation results, specifically including:

[0126] Divide each image into several small areas and count the number of feature points in each small area;

[0127] Divide the number of feature points in a small area by the area of ​​the area to obtain the feature point density of the small area;

[0128] Obtain the shape of the calibration target surface in the camera's field of view and analyze whether the shape is regular. If it is regular, obtain the distribution of feature points. Based on the density and distribution of feature points in multiple images, perform current posture evaluation and obtain the evaluation results.

[0129] If it is irregular, rotate, tilt or translate the camera;

[0130] Analyze whether the density of feature points is greater than or equal to the set density threshold based on the evaluation results;

[0131] If it is greater than or equal to the set density threshold, the distance between the camera and the calibration target surface is increased;

[0132] If it is less than the set density threshold, the distance between the camera and the calibration target surface is reduced.

[0133] It should be noted that if the density of feature points in the image is too low, it may mean that the camera is too far from the calibration target surface, or the camera's field of view does not fully cover the target surface, in which case the camera pose may be suboptimal. Conversely, if the density of feature points is too high and overlap occurs, it may mean that the camera is too close to the target surface. In addition, if the feature points are concentrated on one side of the image, it may indicate that the camera is tilted, resulting in an asymmetric position of the target surface in the image.

[0134] Relationship between distribution uniformity and pose: Uneven distribution of feature points may indicate a problem with the camera pose. For example, the calibration target surface should present a regular shape in the camera's field of view. If the distribution of feature points deviates significantly from the expected uniformity, it may be that the camera has been rotated, tilted, or translated, causing the target surface to be distorted in the image.

[0135] Multi-view pose estimation: This method evaluates the density and distribution of feature points across multiple images. If similar feature point distribution anomalies are observed across multiple images, such as a sparse distribution of feature points in a particular direction, it can be more certain that there is a problem with the camera's pose in that direction. By comparing the changing patterns of feature points across different viewpoints, the camera's pose trend can be inferred.

[0136] Dynamically adjust the shooting angle based on the evaluation results.

[0137] Distance adjustment: If the density of feature points is too low, the distance between the camera and the calibration target surface can be appropriately reduced; if the density is too high, the distance can be increased. Distance adjustment can be achieved by controlling the track or robotic arm on which the camera is mounted.

[0138] Angle rotation: When feature points are unevenly distributed and exhibit certain rotational characteristics, rotate the camera to adjust the posture. For example, if the feature points are concentrated in the upper left corner of the image, it means that the camera may have been rotated clockwise by a certain angle. In this case, rotate the camera counterclockwise to restore the calibration target surface to a more symmetrical position in the image.

[0139] According to an embodiment of the present invention, the reprojection error of the same calibration point is analyzed based on the feature matching results, and the camera intrinsic and extrinsic parameters are dynamically adjusted based on the reprojection error, specifically including:

[0140] Based on the feature matching results, the coordinates of the feature points corresponding to the same calibration point in different images are obtained;

[0141] Based on the camera imaging model, the projection relationship between the calibration point in the world coordinate system and the image pixel coordinate system is established;

[0142] For each calibration point, the coordinates of the calibration point in the world coordinate system are projected onto the image using the current camera intrinsic and extrinsic parameters to obtain the coordinates of the reprojected point;

[0143] Calculate the Euclidean distance between the reprojection point and the actual image feature point coordinates obtained by feature matching to obtain the reprojection error;

[0144] Dynamically adjust camera intrinsics and extrinsic parameters based on reprojection error.

[0145] It should be noted that the reprojection error calculation formula is as follows:

[0146]

[0147] Represents the observed feature point p ij estimated or predicted value of;

[0148] K represents the camera intrinsic parameter matrix;

[0149] R represents the rotation matrix in the camera extrinsic parameters;

[0150] t represents the translation vector in the camera extrinsic parameters;

[0151] E(K,R,t) represents the reprojection error objective function;

[0152] p ij The predicted projection position of point Xi in the jth image after the camera's intrinsic and extrinsic parameters are transformed. This predicted point is calculated using a camera model (e.g., a pinhole model) and the current camera parameters (intrinsic parameter matrix K, rotation matrix R, translation vector t). i represents the number of the calibration point (i.e., the 3D point number). j represents the image frame number (i.e., the image in which the point is located).

[0153] m represents the region index number of the image in the vertical direction (row direction).

[0154] N represents the region index number of the image in the horizontal direction (column direction).

[0155] The internal parameter K and the external parameter of each frame are optimized through LM iteration until the error converges or the iteration is terminated.

[0156] Internal parameter adjustment: Adjusting camera intrinsic parameters typically involves optimizing parameters such as focal length and principal point coordinates. For example, if reprojection error is found to be large in certain areas of the image, this may indicate an inaccurate focal length setting and require adjustment. During the optimization process, the intrinsic parameters are updated according to a specific step size based on the gradient of the reprojection error. Furthermore, to ensure the physical significance and numerical stability of the intrinsic parameters, it may be necessary to restrict the parameter's value range.

[0157] Extrinsic parameter adjustment: Extrinsic parameter adjustment primarily involves optimizing the rotation matrix R and translation vector t. If the reprojection error indicates a deviation in the camera pose, such as an inaccurate camera rotation angle or an error in the translation position, the extrinsic parameters need to be adjusted accordingly. An optimization algorithm updates the extrinsic parameter matrix based on the partial derivatives of the reprojection error with respect to the rotation and translation parameters, gradually adjusting the camera pose to a state that minimizes the reprojection error.

[0158] In a second aspect, an embodiment of the present application provides a three-dimensional calibration system based on an adaptive multi-view optimization algorithm, the system comprising: a memory and a processor, the memory including a program for a three-dimensional calibration method based on an adaptive multi-view optimization algorithm, and the program for the three-dimensional calibration method based on an adaptive multi-view optimization algorithm, when executed by the processor, implementing the following steps:

[0159] Based on the camera observing the calibration target surface from different positions and angles, the images of the calibration target surface at multiple angles are automatically captured to obtain multi-view images;

[0160] Extract feature points based on multi-view images, and extract the positions of calibration points in the images based on the feature points;

[0161] Based on the feature matching algorithm, the feature points corresponding to the same calibration point in different images are searched to obtain the feature matching results;

[0162] Analyze the density and distribution of image feature points based on feature matching results, evaluate the current camera posture, and dynamically adjust the shooting angle based on the evaluation results;

[0163] The reprojection error of the same calibration point is analyzed based on the feature matching results, and the camera intrinsics and extrinsic parameters are dynamically adjusted based on the reprojection error.

[0164] It should be noted that the camera should be a high-precision, high-resolution industrial camera or professional photography camera to ensure clear and accurate images. Camera parameters (such as focal length, aperture, and sensitivity) need to be adjusted and set according to actual conditions to meet imaging requirements under different lighting conditions and shooting distances.

[0165] Calibration target: A planar object with a specific pattern or features, such as a checkerboard pattern or circular dot array. The pattern should be easily identifiable and located to accurately extract feature points from the image. The size and accuracy of the calibration target also need to be selected based on the specific application and the camera's field of view.

[0166] Camera motion mechanism: This controls the camera's position and angle in space. This can be a robotic arm, gimbal, or track. Through precise motor drive and control systems, the camera can be flexibly adjusted to different positions and angles. The motion mechanism should possess high positioning accuracy and repeatability to ensure accurate and reliable camera position and orientation for every shot.

[0167] According to an embodiment of the present invention, based on observing the calibration target surface from different positions and angles by a camera, images of the calibration target surface at multiple angles are automatically captured to obtain multi-view images, specifically including:

[0168] Initialize the camera and gimbal to obtain their initial parameters;

[0169] Configure the camera's acquisition parameters and calibrate the gimbal's zero point and range of motion based on the initial parameters. The camera's parameters include focal length, aperture, and white balance.

[0170] The gimbal rotates slowly, scanning the full viewing angle range in 1° steps. Every time the gimbal rotates 1°, it collects and calibrates the target surface images in sequence.

[0171] Perform frame processing on the calibration target surface image to obtain each frame image;

[0172] Each frame of image is marked with a timestamp and stored in a cache queue to obtain a multi-view image.

[0173] According to an embodiment of the present invention, extracting feature points based on multi-view images, and extracting positions of calibration points in the images based on the feature points, specifically includes:

[0174] Extract feature points of multiple visual images based on feature point extraction algorithm;

[0175] Use Harris corner point or SIFT algorithm to extract the corner points and edge intersection points of the calibration target surface in the image;

[0176] Use the descriptor of the feature points to initially match the corner points, edge intersection points and feature points to determine the calibration points;

[0177] Establish a camera projection model and calculate the three-dimensional coordinates of the calibration points;

[0178] The calculated three-dimensional coordinates of the calibration points are projected onto each image plane through the camera projection model to obtain the two-dimensional coordinates of the calibration points in the image and the position of the calibration points in the image.

[0179] It should be noted that based on the imaging principle of a camera, a mathematical model of the camera, such as the pinhole camera model, is established. This model describes how a point in three-dimensional space is projected onto a two-dimensional image plane using the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as rotation and translation relationships).

[0180] Solving camera parameters: using the matched feature points, the internal and external parameters of the camera are solved through a series of mathematical calculations. For example, an optimization algorithm such as least squares method is used to minimize the projection error, so as to obtain the optimal camera parameter estimation.

[0181] Calculating the three-dimensional coordinates of the calibration points: based on the known camera parameters, combined with the image information under multiple viewing angles, the coordinates of the calibration points in the three-dimensional space can be calculated through triangulation and other methods. Specifically, for each calibration point, find its corresponding feature points in the images under different viewing angles, establish an equation set according to the camera projection model and the known camera parameters, and solve the three-dimensional coordinates of the calibration point.

[0182] Projecting to the image plane to determine the position: the three-dimensional coordinates of the calibration points calculated are projected to each image plane through the camera projection model to obtain their two-dimensional coordinates in the image, i.e. the position of the calibration points in the image is determined.

[0183] According to the embodiment of the present application, the feature matching algorithm is used to search for the feature points corresponding to the same calibration point in different images to obtain the feature matching result, which specifically includes:

[0184] Extracting feature points and descriptors based on the SIFT algorithm for each image;

[0185] Selecting any two images, calculating the Euclidean distance between the descriptor of each feature point in one image and the descriptors of all feature points in the other image;

[0186] Setting a distance threshold, judging whether the Euclidean distance is less than the distance threshold, if yes, selecting the current feature point as the matching point, if no, eliminating the current feature point;

[0187] Setting a distance ratio threshold, using the ratio test method to calculate the distance ratio of each feature point and the nearest neighbor and the second nearest neighbor feature point descriptors;

[0188] If the distance ratio is less than the distance ratio threshold, the feature matching result is obtained;

[0189] If the distance ratio is greater than or equal to the distance ratio threshold, the matching point is extracted.

[0190] It should be noted that first, the SIFT algorithm is used to extract feature points and their descriptors for each image. The SIFT feature points have scale invariance, rotation invariance and partial affine invariance, and their descriptors are 128-dimensional vectors, which can well represent the local features of the feature points.

[0191] Initial matching: For any two images, the Euclidean distance between each feature point descriptor in one image and all feature point descriptors in the other image is calculated, and the feature point with the closest distance is selected as the initial matching point. To improve matching accuracy, a distance threshold can be set, and only matching points with a distance less than the threshold are retained.

[0192] Ratio test: To further remove mismatched points, a ratio test is used. For each feature point, the ratio of its distance to the nearest neighbor and the next nearest neighbor feature point descriptors is calculated. If the ratio is less than a preset threshold (usually around 0.7), the matching point is considered reliable; otherwise, it is removed.

[0193] According to an embodiment of the present invention, the density and distribution of image feature points are analyzed based on feature matching results, the current posture of the camera is evaluated, and the shooting angle is dynamically adjusted according to the evaluation results, specifically including:

[0194] Divide each image into several small areas and count the number of feature points in each small area;

[0195] Divide the number of feature points in a small area by the area of ​​the area to obtain the feature point density of the small area;

[0196] Obtain the shape of the calibration target surface in the camera's field of view and analyze whether the shape is regular. If it is regular, obtain the distribution of feature points. Based on the density and distribution of feature points in multiple images, perform current posture evaluation and obtain the evaluation results.

[0197] If it is irregular, rotate, tilt or translate the camera;

[0198] Analyze whether the density of feature points is greater than or equal to the set density threshold based on the evaluation results;

[0199] If it is greater than or equal to the set density threshold, the distance between the camera and the calibration target surface is increased;

[0200] If it is less than the set density threshold, the distance between the camera and the calibration target surface is reduced.

[0201] It should be noted that if the density of feature points in the image is too low, it may mean that the camera is too far from the calibration target surface, or the camera's field of view does not fully cover the target surface, in which case the camera pose may be suboptimal. Conversely, if the density of feature points is too high and overlap occurs, it may mean that the camera is too close to the target surface. In addition, if the feature points are concentrated on one side of the image, it may indicate that the camera is tilted, resulting in an asymmetric position of the target surface in the image.

[0202] Uniformity and pose relationship: uneven distribution of feature points may imply that the camera pose is problematic. For example, the calibration target surface should present a regular shape in the camera field of view. If the feature point distribution deviates significantly from the expected uniform state, it may be that the camera has rotated, tilted or translated, causing the target surface to be deformed in the image.

[0203] Multi-view based pose evaluation: the feature point density and distribution of multiple images are evaluated comprehensively. If similar feature point distribution abnormalities exist in multiple images, such as sparse feature points in a certain direction, it can be more certain that the camera pose in that direction is problematic. By comparing the change rules of feature points under different angles, the trend of camera pose change can also be inferred.

[0204] According to the evaluation results, the shooting angle is dynamically adjusted.

[0205] Distance adjustment: if the evaluation finds that the feature point density is too low, the distance between the camera and the calibration target surface is appropriately reduced; if the density is too high, the distance is increased. The distance adjustment can be achieved by controlling the track or mechanical arm on which the camera is installed.

[0206] Angle rotation: when the feature point distribution is uneven and presents a certain rotation feature, the camera is rotated to adjust the pose. For example, if the feature points are concentrated in the upper left corner of the image, it means that the camera may have rotated clockwise by a certain angle, so the camera is rotated counterclockwise to restore the calibration target surface to a more symmetrical position in the image.

[0207] According to the embodiments of the present application, the reprojection error of the same calibration point is analyzed based on the feature matching result, and the camera intrinsic and extrinsic parameters are dynamically adjusted based on the reprojection error, specifically including:

[0208] Based on the feature matching result, the feature point coordinates of the same calibration point in different images are obtained;

[0209] Based on the camera imaging model, the projection relationship from the calibration point in the world coordinate system to the image pixel coordinate system is established;

[0210] For each calibration point, the coordinates of the calibration point in the world coordinate system are projected onto the image through the current camera intrinsic and extrinsic parameters to obtain the coordinates of the reprojection point;

[0211] The Euclidean distance between the reprojection point and the actual image feature point coordinates obtained through feature matching is calculated to obtain the reprojection error;

[0212] The camera intrinsic and extrinsic parameters are dynamically adjusted based on the reprojection error.

[0213] It should be noted that the reprojection error calculation formula is as follows:

[0214]

[0215] The internal parameter K and the external parameter of each frame are optimized through LM iteration until the error converges or the iteration is terminated.

[0216] Internal parameter adjustment: Adjusting camera intrinsic parameters typically involves optimizing parameters such as focal length and principal point coordinates. For example, if reprojection error is found to be large in certain areas of the image, this may indicate an inaccurate focal length setting and require adjustment. During the optimization process, the intrinsic parameters are updated according to a specific step size based on the gradient of the reprojection error. Furthermore, to ensure the physical significance and numerical stability of the intrinsic parameters, it may be necessary to restrict the parameter's value range.

[0217] Extrinsic parameter adjustment: Extrinsic parameter adjustment primarily involves optimizing the rotation matrix R and translation vector t. If the reprojection error indicates a deviation in the camera pose, such as an inaccurate camera rotation angle or an error in the translation position, the extrinsic parameters need to be adjusted accordingly. An optimization algorithm updates the extrinsic parameter matrix based on the partial derivatives of the reprojection error with respect to the rotation and translation parameters, gradually adjusting the camera pose to a state that minimizes the reprojection error.

[0218] The third aspect of the present invention provides a computer-readable storage medium, which includes a three-dimensional calibration method program based on an adaptive multi-view optimization algorithm. When the three-dimensional calibration method program based on an adaptive multi-view optimization algorithm is executed by a processor, the steps of the three-dimensional calibration method based on an adaptive multi-view optimization algorithm as described above are implemented.

[0219] The present invention discloses a three-dimensional calibration method, system and medium based on an adaptive multi-view optimization algorithm. The method comprises the following steps: observing a calibration target surface from different positions and angles based on a camera, automatically capturing images of the calibration target surface at multiple angles, and obtaining multi-view images; extracting feature points based on the multi-view images, and extracting the positions of calibration points in the images based on the feature points; searching for feature points corresponding to the same calibration point in different images based on a feature matching algorithm, and obtaining feature matching results; analyzing the density and distribution of image feature points based on the feature matching results, evaluating the current posture of the camera, and dynamically adjusting the shooting angle according to the evaluation results; analyzing the reprojection error of the same calibration point based on the feature matching results, and dynamically adjusting the camera's internal and external parameters based on the reprojection error; scanning the calibration target surface at all angles by capturing multi-view images, and dynamically adjusting the camera's internal and external parameters based on the analysis of the calibration feature matching and reprojection error, thereby achieving a high-precision, highly adaptable and efficient calibration process.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0221] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0222] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0223] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0224] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the existing technology, can be embodied in the form of a software product. The software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. A three-dimensional calibration method based on an adaptive multi-view optimization algorithm, characterized in that: include: Based on the camera observing the calibration target surface from different positions and angles, the images of the calibration target surface at multiple angles are automatically captured to obtain multi-view images; Extract feature points based on multi-view images, and extract the positions of calibration points in the images based on the feature points; Based on the feature matching algorithm, the feature points corresponding to the same calibration point in different images are searched to obtain the feature matching results; Analyze the density and distribution of image feature points based on feature matching results, evaluate the current camera posture, and dynamically adjust the shooting angle based on the evaluation results; The reprojection error of the same calibration point is analyzed based on the feature matching results, and the camera intrinsics and extrinsic parameters are dynamically adjusted based on the reprojection error.

2. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to claim 1, characterized in that: Based on the camera observing the calibration target surface from different positions and angles, the camera automatically captures images of the calibration target surface at multiple angles to obtain multi-view images, specifically including: Initialize the camera and gimbal to obtain their initial parameters; Configuring the camera's acquisition parameters and calibrating the gimbal's zero point and motion range based on the initial parameters, wherein the camera's parameters include focal length, aperture, and white balance; The gimbal rotates slowly, scanning the full viewing angle range in 1° steps. Every time the gimbal rotates 1°, it collects and calibrates the target surface images in sequence. Perform frame processing on the calibration target surface image to obtain each frame image; Each frame of image is marked with a timestamp and stored in a cache queue to obtain a multi-view image.

3. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to claim 2, characterized in that: Extract feature points based on multi-view images, and extract the positions of calibration points in the images based on the feature points, specifically including: Extract feature points of multiple visual images based on feature point extraction algorithm; Use Harris corner point or SIFT algorithm to extract the corner points and edge intersection points of the calibration target surface in the image; Use the descriptor of the feature points to initially match the corner points, edge intersection points and feature points to determine the calibration points; Establish a camera projection model and calculate the three-dimensional coordinates of the calibration points; The calculated three-dimensional coordinates of the calibration points are projected onto each image plane through the camera projection model to obtain the two-dimensional coordinates of the calibration points in the image and the position of the calibration points in the image.

4. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to claim 3, characterized in that: Based on the feature matching algorithm, feature points corresponding to the same calibration point in different images are searched to obtain feature matching results, which include: Extract feature points and descriptors from each image based on the SIFT algorithm; Select any two images and calculate the Euclidean distance between each feature point descriptor in one image and all feature point descriptors in the other image; Set the distance threshold to determine whether the Euclidean distance is less than the distance threshold. If it is less than the distance threshold, the current feature point is selected as the matching point. If it is greater than the distance threshold, the current feature point is removed. Set the distance ratio threshold and use the ratio test method to calculate the distance ratio of each feature point to the nearest neighbor and next nearest neighbor feature point descriptors; If the distance ratio is less than the distance ratio threshold, the feature matching result is obtained; If the distance ratio is greater than or equal to the distance ratio threshold, the matching point is extracted.

5. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to claim 4 is characterized in that: Based on the feature matching results, the density and distribution of image feature points are analyzed, the current camera posture is evaluated, and the shooting angle is dynamically adjusted according to the evaluation results. Specifically, the following steps are performed: Divide each image into several small areas and count the number of feature points in each small area; Divide the number of feature points in a small area by the area of ​​the area to obtain the feature point density of the small area; Obtain the shape of the calibration target surface in the camera's field of view and analyze whether the shape is regular. If it is regular, obtain the distribution of feature points. Based on the density and distribution of feature points in multiple images, perform current posture evaluation and obtain the evaluation results. If it is irregular, rotate, tilt or translate the camera; Analyze whether the density of feature points is greater than or equal to the set density threshold based on the evaluation results; If it is greater than or equal to the set density threshold, the distance between the camera and the calibration target surface is increased; If it is less than the set density threshold, the distance between the camera and the calibration target surface is reduced.

6. The three-dimensional calibration method based on the adaptive multi-view optimization algorithm according to claim 5, characterized in that: Analyze the reprojection error of the same calibration point based on the feature matching results, and dynamically adjust the camera intrinsics and extrinsic parameters based on the reprojection error. Specifically, Based on the feature matching results, the coordinates of the feature points corresponding to the same calibration point in different images are obtained; Based on the camera imaging model, the projection relationship between the calibration point in the world coordinate system and the image pixel coordinate system is established; For each calibration point, the coordinates of the calibration point in the world coordinate system are projected onto the image using the current camera intrinsic and extrinsic parameters to obtain the coordinates of the reprojected point; Calculate the Euclidean distance between the reprojection point and the actual image feature point coordinates obtained by feature matching to obtain the reprojection error; Dynamically adjust camera intrinsics and extrinsic parameters based on reprojection error.

7. A three-dimensional calibration system based on an adaptive multi-view optimization algorithm, characterized in that: The system includes: a memory and a processor, wherein the memory includes a program of a three-dimensional calibration method based on an adaptive multi-view optimization algorithm, and when the program of the three-dimensional calibration method based on the adaptive multi-view optimization algorithm is executed by the processor, the following steps are implemented: Based on the camera observing the calibration target surface from different positions and angles, the images of the calibration target surface at multiple angles are automatically captured to obtain multi-view images; Extract feature points based on multi-view images, and extract the positions of calibration points in the images based on the feature points; Based on the feature matching algorithm, the feature points corresponding to the same calibration point in different images are searched to obtain the feature matching results; Analyze the density and distribution of image feature points based on feature matching results, evaluate the current camera posture, and dynamically adjust the shooting angle based on the evaluation results; The reprojection error of the same calibration point is analyzed based on the feature matching results, and the camera intrinsics and extrinsic parameters are dynamically adjusted based on the reprojection error.

8. The three-dimensional calibration system based on the adaptive multi-view optimization algorithm according to claim 7, characterized in that: Based on the camera observing the calibration target surface from different positions and angles, the camera automatically captures images of the calibration target surface at multiple angles to obtain multi-view images, specifically including: Initialize the camera and gimbal to obtain their initial parameters; Configuring the camera's acquisition parameters and calibrating the gimbal's zero point and motion range based on the initial parameters, wherein the camera's parameters include focal length, aperture, and white balance; The gimbal rotates slowly, scanning the full viewing angle range in 1° steps. Every time the gimbal rotates 1°, it collects and calibrates the target surface images in sequence. Perform frame processing on the calibration target surface image to obtain each frame image; Each frame of image is marked with a timestamp and stored in a cache queue to obtain a multi-view image.

9. The three-dimensional calibration system based on the adaptive multi-view optimization algorithm according to claim 8, characterized in that: Extract feature points based on multi-view images, and extract the positions of calibration points in the images based on the feature points, specifically including: Extract feature points of multiple visual images based on feature point extraction algorithm; Use Harris corner point or SIFT algorithm to extract the corner points and edge intersection points of the calibration target surface in the image; Use the descriptor of the feature points to initially match the corner points, edge intersection points and feature points to determine the calibration points; Establish a camera projection model and calculate the three-dimensional coordinates of the calibration points; The calculated three-dimensional coordinates of the calibration points are projected onto each image plane through the camera projection model to obtain the two-dimensional coordinates of the calibration points in the image and the position of the calibration points in the image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a three-dimensional calibration method program based on an adaptive multi-view optimization algorithm. When the three-dimensional calibration method program based on an adaptive multi-view optimization algorithm is executed by a processor, the steps of the three-dimensional calibration method based on an adaptive multi-view optimization algorithm as described in any one of claims 1 to 6 are implemented.

Images