Automatic camera calibration method and device based on truncated pyramid trajectory

Abstract

The invention provides a truncated pyramid trajectory-based camera automatic calibration method and device, and the method comprises the steps: constructing a truncated pyramid geometric model, generating a plurality of point locations with different rotation angles by taking the tower center of the truncated pyramid geometric model as a reference point, forming a robot motion point location set through combining with truncated pyramid boundary points, and carrying out the automatic calibration of a camera through planning the motion path of a robot. And sequentially arriving at each point location and staying for a short time to collect images, calibrating internal and external parameters of the camera by using the image data, and optimizing through an optimization algorithm. According to the scheme, the calibration precision of a large-view-angle camera or a distortion camera is enhanced through the design of a truncated pyramid structure and a rotating point position, the method can adapt to complex environments such as illumination variation and partial shielding, an optimization algorithm is introduced, the dependence of the calibration process on initial parameters is greatly reduced, the local optimum problem is avoided, and the calibration precision is improved. And the calibration robustness and accuracy are improved. Meanwhile, the robot is used for automatically controlling relative movement of the camera and the calibration object, and the automation degree and efficiency of camera calibration are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a camera automatic calibration method based on a truncated pyramid trajectory, a camera automatic calibration device based on a truncated pyramid trajectory, an electronic device, and a computer-readable medium. Background Art

[0002] With the rapid development of computer vision technology, camera calibration, as a key technology in computer vision systems, has been widely studied and applied. The purpose of camera calibration is to determine the camera's intrinsic and extrinsic parameters, thereby establishing an accurate mapping relationship between image coordinates and actual three-dimensional space coordinates. This is particularly important for applications such as stereo vision measurement, augmented reality, and robot navigation. Traditional camera calibration methods rely primarily on manual operation, typically by photographing a calibration plate with a known geometric shape (such as a checkerboard or dot array).

[0003] Existing camera calibration methods primarily rely on classic planar calibration plates or three-dimensional calibration objects. While they can obtain relatively accurate camera intrinsic and extrinsic parameters, they still suffer from the following drawbacks: Complex and inefficient operations: Traditional camera calibration methods typically require precision-made calibration plates or three-dimensional calibration objects and require multiple image acquisitions at different poses and angles. This cumbersome and time-consuming process makes it difficult to meet the requirements of automated calibration. Poor applicability: Most methods are only applicable to specific scenes or structures and are highly dependent on external environmental factors, such as lighting conditions and the integrity of the calibration plate, which can affect calibration accuracy and stability. Sensitivity to initial parameters: Some algorithms are highly sensitive to the selection of initial parameters during optimization, easily falling into local optima, resulting in inaccurate calibration results and affecting calibration robustness. Limited accuracy: The accuracy of traditional calibration methods is significantly reduced when the camera has a large field of view or severe distortion, especially when applied to long-range or large-scale scenes. Low automation: Most calibration methods still rely on manual intervention and operation, making efficient automation difficult and unsuitable for large-scale or continuous calibration tasks. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a camera automatic calibration method based on a truncated pyramid trajectory and a corresponding camera automatic calibration device based on a truncated pyramid trajectory, an electronic device and a computer-readable medium to overcome the above problems or at least partially solve the above problems.

[0005] The present invention discloses a camera automatic calibration method based on a truncated pyramid trajectory, the method comprising:

[0006] Construct a truncated pyramid geometric model;

[0007] Taking the center of the truncated pyramid geometric model as a reference point, multiple points with different preset rotation angles are generated;

[0008] Multiple points with different preset rotation angles, together with the boundary points of the truncated pyramid geometric model, constitute a robot motion point set;

[0009] Planning a motion path of the robot according to the robot motion point set, and controlling the robot to move to each point of the robot motion point set according to the planned motion path and to stay briefly to collect image data;

[0010] The image data is used to calibrate the internal and external parameters of the camera, and optimization is performed based on an optimization algorithm to obtain the internal and external parameters of the camera.

[0011] Optionally, constructing a truncated pyramid geometric model includes:

[0012] Setting geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface;

[0013] Calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters;

[0014] Determine the three-dimensional coordinates of the upper plane boundary point and the bottom surface boundary point based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary point and the geometric center point of the square, and the distance between the bottom surface boundary point and the geometric center point of the square;

[0015] A truncated pyramid geometric model is generated based on the three-dimensional coordinates of the tower center position, the upper plane boundary points and the bottom surface boundary points.

[0016] Optionally, a plurality of points with different preset rotation angles are generated with the center position of the truncated pyramid geometric model as a reference point, including:

[0017] Taking the center of the truncated pyramid as the reference point, apply independent rotations of positive and negative preset angles on each direction axis in three-dimensional space to generate points rotated along a single direction axis.

[0018] Taking the center of the truncated pyramid geometric model as the reference point, at least two direction axes are arranged and combined in three-dimensional space, and then positive and negative preset rotation angles are applied for combined rotation to generate a point position where at least two direction axes are combined and rotated.

[0019] Optionally, the image data is used to calibrate camera intrinsic and extrinsic parameters, and optimization is performed based on an optimization algorithm to obtain camera intrinsic and extrinsic parameters, including:

[0020] Based on the collected image data, the Zhang Zhengyou calibration method is used to perform initial calibration of the camera's internal and external parameters to obtain the camera's initial internal and external parameters;

[0021] An optimization model with minimization of projection error as the objective function is constructed. The initial intrinsic and extrinsic parameters of the camera are used as the initial optimization values. A genetic algorithm is used for global optimization to obtain the globally optimized intrinsic and extrinsic parameters of the camera. A nonlinear least squares optimization algorithm is then used to locally optimize the globally optimized intrinsic and extrinsic parameters of the camera until the projection error is less than a preset threshold, and the locally optimized intrinsic and extrinsic parameters of the camera are output.

[0022] Optionally, the motion path of the robot includes a sequence from bottom to top or from top to bottom based on the set of motion points of the robot.

[0023] The present invention also discloses a camera automatic calibration device based on a truncated pyramid trajectory, the device comprising:

[0024] A truncated pyramid building module, used to build a truncated pyramid geometric model;

[0025] A rotation point generation module is used to generate multiple points with different preset rotation angles using the center position of the truncated pyramid geometric model as a reference point;

[0026] A motion point set composition module is used for multiple points with different preset rotation angles, which together with the boundary points of the truncated pyramid geometric model constitute the robot motion point set;

[0027] a calibration image data acquisition module, configured to plan a motion path of the robot according to the robot motion point set, and control the robot to move to each point of the robot motion point set along the planned motion path and to briefly stop there to collect image data;

[0028] The camera internal and external parameter calibration and optimization module is used to calibrate the camera internal and external parameters using the image data and optimize based on an optimization algorithm to obtain the camera internal and external parameters.

[0029] Optionally, the truncated pyramid building block includes:

[0030] The geometric parameter setting submodule is used to set the geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface;

[0031] A parameter calculation submodule is used to calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters;

[0032] A submodule for determining the three-dimensional coordinates of boundary points is used to determine the three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points based on geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary points and the geometric center point of the square, and the distance between the bottom surface boundary points and the geometric center point of the square;

[0033] The truncated pyramid generation submodule is used to generate a truncated pyramid geometric model based on the three-dimensional coordinates of the pyramid center position, the upper plane boundary points and the bottom surface boundary points.

[0034] Optionally, the rotation point generation module includes:

[0035] The single-axis rotation point generation submodule is used to generate points rotated along a single axis by applying independent rotations of positive and negative preset angles on each axis in three-dimensional space, taking the center of the truncated pyramid geometric model as the reference point.

[0036] The multi-axis combined rotation point generation submodule is used to use the center position of the truncated pyramid geometric model as a reference point, arrange and combine at least two direction axes in three-dimensional space, and then apply positive and negative preset rotation angles for combined rotation to generate points of at least two direction axes combined rotation.

[0037] Optionally, the camera internal and external parameter calibration and optimization module includes:

[0038] An initial calibration submodule, configured to perform initial calibration of the camera's internal and external parameters using the Zhang Zhengyou calibration method based on the collected image data to obtain the camera's initial internal and external parameters;

[0039] The calibration optimization submodule is used to construct an optimization model with minimization of projection error as the objective function, use the initial intrinsic and extrinsic parameters of the camera as the initial optimization values, use a genetic algorithm to perform global optimization, obtain the globally optimized intrinsic and extrinsic parameters of the camera, and use a nonlinear least squares optimization algorithm to locally optimize the globally optimized intrinsic and extrinsic parameters of the camera until the projection error is less than a preset threshold, and output the locally optimized intrinsic and extrinsic parameters of the camera.

[0040] Optionally, the motion path of the robot includes a sequence from bottom to top or from top to bottom based on the set of motion points of the robot.

[0041] The present invention also discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0042] The memory is used to store computer programs;

[0043] The processor is configured to implement the camera automatic calibration method based on truncated pyramid trajectory as described in the present invention when executing the program stored in the memory.

[0044] The present invention also discloses one or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to perform the camera automatic calibration method based on truncated pyramid trajectories as described in the present invention.

[0045] The present invention includes the following advantages:

[0046] The present invention discloses a camera automatic calibration method based on a truncated pyramid trajectory. First, a truncated pyramid geometric model is constructed, and multiple points with different rotation angles are generated using the center of the pyramid as a reference point. The robot motion point set is formed by combining the boundary points of the truncated pyramid. The robot's motion path is planned, and the robot reaches each point in turn and stops briefly to collect images. Finally, the image data is used to calibrate the internal and external parameters of the camera, and the parameters are optimized using an optimization algorithm. This solution enhances the calibration accuracy of wide-angle cameras or cameras with distortion through the design of the truncated pyramid structure and the rotation points. It performs better in long-distance or large-scale scenes, and can adapt to complex environments such as lighting changes and partial occlusion. The optimization algorithm is introduced to greatly reduce the dependence of the calibration process on the initial parameters, avoid local optimal problems, and improve the robustness and accuracy of the calibration. At the same time, the robot is used to automatically control the relative motion of the camera and the calibration object, reducing manual intervention and improving the automation and efficiency of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the steps of a camera automatic calibration method based on a truncated pyramid trajectory provided by an embodiment of the present invention;

[0048] Figure 2 is an example diagram of a truncated pyramid geometric model provided by an embodiment of the present invention;

[0049] Figure 3 This is a structural block diagram of a camera automatic calibration device based on a truncated pyramid trajectory provided by an embodiment of the present invention;

[0050] Figure 4 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0051] Figure 5 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] This method designs a specific truncated pyramid trajectory structure, using its center as a reference point to generate multiple points at different rotation angles. This method, combined with the boundary points of the truncated pyramid, forms a robot motion point set. The robot's motion path is then planned, and the camera is guided to follow the path to each point, stopping briefly to capture images. Finally, this image data is used to calibrate the camera's internal and external parameters, which are then optimized using an optimization algorithm. This method achieves automatic calibration of the camera's internal and external parameters. By constructing three-dimensional spatial feature points of the truncated pyramid trajectory and combining it with an efficient optimization algorithm, this method improves calibration accuracy and robustness, making it particularly suitable for camera calibration scenarios with wide viewing angles or those with distortion.

[0054] Reference Figure 1 , shows a flowchart of a camera automatic calibration method based on a truncated pyramid trajectory provided in an embodiment of the present invention, which may specifically include the following steps:

[0055] Step 101, constructing a truncated pyramid geometric model;

[0056] Reference Figure 2 An example diagram of a truncated pyramid. In the present invention, the truncated pyramid structure is selected as the method for automatic camera calibration based on the following reasons: 1. The geometric characteristics of the truncated pyramid structure are highly stable. 2. Multi-view calibration support. Since the upper and lower bases of the truncated pyramid are similar in shape but different in size, and there is a certain height difference, the camera can still effectively detect key feature points when observing from different angles (especially looking down and sideways). Therefore, it is particularly suitable for multi-view camera calibration. 3. Depth information acquisition. The three-dimensional structure of the truncated pyramid contributes to the extraction of depth information and three-dimensional reconstruction. By taking images at different angles multiple times, the spatial relationship between the camera and the calibration object can be calculated more accurately. 4. Easy to model and mathematically describe. 5. Applicable to multiple calibration methods. It can be used for both eye-in-hand and eye-to-hand calibration methods, because the feature points of the truncated pyramid can be stably detected regardless of whether the camera moves or the object moves.

[0057] In one embodiment of the present invention, constructing a truncated pyramid geometric model includes:

[0058] Setting geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface;

[0059] Calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters;

[0060] Determine the three-dimensional coordinates of the upper plane boundary point and the bottom surface boundary point based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary point and the geometric center point of the square, and the distance between the bottom surface boundary point and the geometric center point of the square;

[0061] A truncated pyramid geometric model is generated based on the three-dimensional coordinates of the tower center position, the upper plane boundary points and the bottom surface boundary points.

[0062] In this embodiment, the truncated pyramid geometric model is constructed in the following manner:

[0063] 1. Set the geometric parameters of the truncated pyramid:

[0064] Tower center position O;

[0065] The distance from the tower center to the upper plane is h1;

[0066] The distance h2 from the center of the tower to the bottom;

[0067] The side length a of the upper plane;

[0068] The side length of the base is 2a.

[0069] 2. Calculate the plane height:

[0070] Upper plane height: Z1=Z0+h1.

[0071] Bottom height: Z2=Z0-h2.

[0072] 3. Calculate the distance between the boundary point and the center:

[0073] The distance between the boundary point in the upper plane and the geometric center of the square is:

[0074]

[0075] The distance between the bottom boundary point and the geometric center of the square is L2:

[0076]

[0077] 4. Determine the three-dimensional coordinates of the boundary points:

[0078] Upper plane boundary points:

[0079]

[0080] Bottom boundary points:

[0081]

[0082] Step 102: using the center of the truncated pyramid geometric model as a reference point, generating a plurality of points with different preset rotation angles;

[0083] In one embodiment of the present invention, a plurality of points with different preset rotation angles are generated with the center position of the truncated pyramid geometric model as a reference point, including:

[0084] Taking the center of the truncated pyramid as the reference point, apply independent rotations of positive and negative preset angles on each direction axis in three-dimensional space to generate points rotated along a single direction axis.

[0085] Taking the center of the truncated pyramid geometric model as the reference point, at least two direction axes are arranged and combined in three-dimensional space, and then positive and negative preset rotation angles are applied for combined rotation to generate a point position where at least two direction axes are combined and rotated.

[0086] During the camera hand-eye calibration process, image data obtained solely by translation is often insufficient to accurately calibrate the camera's internal and external parameters. Therefore, to enhance the accuracy and robustness of the calibration, this paper introduces multiple rotation angles at specific points for calibration. All rotation angles are generated with the tower center position O as the reference point. Assume that the initial pose of the tower center position is:

[0087] P O =(X0,Y0,Z0,RX0,RY0,RZ0)

[0088] Among them, (X0, Y0, Z0) is the three-dimensional coordinate of the tower center, and the following three parameters are the rotation angles of the XYZ Euler angle.

[0089] To better cover all directions in three-dimensional space, different rotation angles can be introduced along the X, Y, and Z axes. Assuming a 25° rotation angle for the optimal calibration plate, the rotation is described using the external XYZ Euler angles used by Fanuc robots. Rotations of plus or minus 25° can be applied along each axis, and combined with rotations along other axes to generate more points.

[0090] The points generated according to the rotation angle are as follows:

[0091] P9=(X0, Y0, Z0, RX0+25°, RY0, RZ0)

[0092] P 10 =(X0, Y0, Z0, RX0, RY0+25°, RZ0)

[0093] P 11 =(X0, Y0, Z0, RX0, RY0, RZ0+25°)

[0094] P 12 =(X0, Y0, Z0, RX0-25°, RY0, RZ0)

[0095] P13 =(X0, Y0, Z0, RX0, RY0-25°, RZ0)

[0096] P 14 =(X0, Y0, Z0, RX0, RY0, RZ0-25°)

[0097] P 15 =(X0, Y0, Z0, RX0+25°, RY0+25°, RZ0)

[0098] Step 103: A plurality of points with different preset rotation angles and the boundary points of the truncated pyramid geometric model together constitute a robot motion point set;

[0099] Step 104: planning a motion path of the robot according to the robot motion point set, and controlling the robot to move to each point in the robot motion point set according to the planned motion path and to briefly stop to collect image data;

[0100] In one embodiment of the present invention, the motion path of the robot includes a motion point set of the robot in a sequence from bottom to top or from top to bottom.

[0101] In this embodiment, the preset point set consists of two parts:

[0102] 1. Boundary points of the truncated pyramid geometric model (covering a wider spatial range).

[0103] 2. Multiple rotation angle points at the tower center (enhancing the diversity and accuracy of calibration).

[0104] Assume that the point set is: P = {P1, P2, ..., Pm}

[0105] Each point Pi contains pose information: Pi = (xi,yi,zi,EulerXi,EulerYi,EulerZi)

[0106] Steps of the path planning method:

[0107] First, group the points in the preset point set by type: Group 1: Boundary points of the truncated pyramid trajectory. Group 2: Rotation angle points at the pyramid center. Then, arrange them in bottom-to-top or top-to-bottom order (for example, starting from the bottom point and gradually moving to the top plane point).

[0108] The robot is controlled to move to each point in sequence along the planned trajectory, and stays at each point Pi for a short time (e.g., 1-2 seconds) to ensure that the camera captures a sufficiently clear image for image acquisition and data recording.

[0109] Step 105 , calibrating the camera's intrinsic and extrinsic parameters using the image data, and optimizing based on an optimization algorithm to obtain the camera's intrinsic and extrinsic parameters.

[0110] In one embodiment of the present invention, the image data is used to calibrate camera intrinsic and extrinsic parameters, and optimization is performed based on an optimization algorithm to obtain camera intrinsic and extrinsic parameters, including:

[0111] Based on the collected image data, the Zhang Zhengyou calibration method is used to perform initial calibration of the camera's internal and external parameters to obtain the camera's initial internal and external parameters;

[0112] An optimization model with minimization of projection error as the objective function is constructed. The initial intrinsic and extrinsic parameters of the camera are used as the initial optimization values. A genetic algorithm is used for global optimization to obtain the globally optimized intrinsic and extrinsic parameters of the camera. A nonlinear least squares optimization algorithm is then used to locally optimize the globally optimized intrinsic and extrinsic parameters of the camera until the projection error is less than a preset threshold, and the locally optimized intrinsic and extrinsic parameters of the camera are output.

[0113] Global optimization algorithms—genetic algorithms (GAs). These algorithms are based on natural selection and genetic mechanisms. They gradually optimize the objective function through population initialization, selection, crossover, and mutation. GAs can avoid falling into local optima, but convergence is slow.

[0114] 1. Encoding: Use real number encoding to represent the camera's internal and external parameters as chromosome genes:

[0115] Chromosome=[fx,fy,cx,cy,k1,k2,r1,r2,r3,tx,ty,tz]

[0116] Where: fx, fy: focal length parameters. cx, cy: principal point coordinates. k1, k2: distortion coefficients. r1, r2: rotation vectors. tx, ty,: translation vectors.

[0117] 2. Population initialization: Randomly generate the initial population, and the population size is generally set to 50-100 individuals.

[0118] 3. Fitness Function: Use projection error as fitness function:

[0119] Where: pi: actual feature point coordinates. pi^: predicted feature point coordinates. N: total number of feature points.

[0120] 4. Genetic Operations: Selection: Uses roulette wheel selection or tournament selection to select outstanding individuals based on their fitness. Crossover: Uses real-valued crossover (such as arithmetic crossover) to generate offspring individuals. Mutation: Randomly perturbs the gene values in chromosomes to increase population diversity.

[0121] 5. Iteration: Update the population and gradually approach the optimal solution through several generations of evolution (e.g., 100 generations). Set termination conditions, such as the number of generations or error convergence.

[0122] Nonlinear least squares optimization (Levenberg-Marquardt algorithm). Genetic algorithms can quickly find the optimal global solution, but due to computational precision limitations, the final solution may not be refined enough. Therefore, based on the optimal solution obtained through global optimization, the Levenberg-Marquardt algorithm is further used for local refinement optimization.

[0123] Optimization goal:

[0124] This algorithm combines gradient descent and Newton's method, resulting in fast convergence and high accuracy. It can also quickly optimize solutions with good initial parameters in a local area.

[0125] In order to solve the problems of inaccurate parameter initialization, local optimal solution and insufficient calibration accuracy in camera calibration, the present invention adopts a method combining global optimization (genetic algorithm, GA) + local optimization (nonlinear least squares optimization algorithm, LM) to achieve a balance between accuracy and stability.

[0126] Optimization process: Global optimization phase: Genetic algorithm is used to generate initial parameter solutions. Local optimization phase: Based on the genetic algorithm results, LM algorithm is used for further optimization. Result evaluation: Projection error is calculated to ensure that the error is within a reasonable range (e.g., less than 0.5 pixels).

[0127] This embodiment uses a genetic algorithm with strong global search capabilities to perform global search through population evolution, which can effectively avoid falling into the local optimal solution. It also uses the LM algorithm to perform local fine-tuning when the initial value is good, and can quickly converge to the exact solution, thereby achieving a balance between accuracy and stability through two stages.

[0128] In summary, the proposed automated calibration method, which integrates software with a camera vision system and combines it with advanced calibration algorithms, efficiently achieves robot hand-eye calibration without requiring a complex calibration process. This technology automatically captures the feature point information required for calibration and is applicable to different industrial robot brands. It offers excellent compatibility and ease of use, and is widely applicable in industrial scenarios such as precision assembly and automated polishing.

[0129] Specifically, it has the following advantages:

[0130] 1. High calibration efficiency and high degree of automation:

[0131] This method automatically generates the robot end-point pose points required for camera calibration using a pre-planned truncated pyramid trajectory. The robot automatically moves along the trajectory and collects image data, significantly reducing manual intervention and improving calibration efficiency and automation.

[0132] 2. High precision and strong applicability:

[0133] By constructing a truncated pyramid geometry model and designing a rational motion path, high calibration accuracy can be achieved even in wide viewing angles or camera scenes with significant distortion. Furthermore, the invention exhibits strong adaptability, with good fault tolerance and stability to conditions such as illumination variations and partial occlusion of the calibration plate.

[0134] 3. Strong robustness and low parameter dependence:

[0135] The method of the present invention greatly reduces the dependence on initial parameters by introducing multi-angle rotation points and optimization algorithms, avoids the local optimal problem caused by improper selection of initial parameters in traditional methods, and thus improves the robustness and accuracy of calibration.

[0136] 4. Simple implementation and good scalability:

[0137] The truncated pyramid trajectory planning method of the present invention features a simple structure and convenient computation. The generated point set is not only suitable for hand-eye calibration but can also be extended to other vision calibration scenarios. Furthermore, this method has good scalability, and calibration accuracy can be further improved or applied to more complex application scenarios by adjusting geometric parameters or optimizing algorithms.

[0138] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0139] Reference Figure 3 , shows a structural block diagram of a camera automatic calibration device based on truncated pyramid trajectory provided in an embodiment of the present invention, which may specifically include the following modules:

[0140] A truncated pyramid construction module 301 is used to construct a truncated pyramid geometric model;

[0141] A rotation point generation module 302 is configured to generate a plurality of points with different preset rotation angles using the center of the truncated pyramid geometric model as a reference point;

[0142] A motion point set forming module 303 is used for forming a robot motion point set together with the boundary points of the truncated pyramid geometric model using a plurality of points with different preset rotation angles;

[0143] The calibration image data acquisition module 304 is used to plan the robot's motion path according to the robot motion point set, and control the robot to move to each point of the robot motion point set according to the planned motion path and briefly stop to collect image data;

[0144] The camera intrinsic and extrinsic parameter calibration and optimization module 305 is configured to calibrate the camera intrinsic and extrinsic parameters using the image data, and perform optimization based on an optimization algorithm to obtain the camera intrinsic and extrinsic parameters.

[0145] Optionally, the truncated pyramid building block includes:

[0146] The geometric parameter setting submodule is used to set the geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface;

[0147] A parameter calculation submodule is used to calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters;

[0148] A submodule for determining the three-dimensional coordinates of boundary points is used to determine the three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points based on geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary points and the geometric center point of the square, and the distance between the bottom surface boundary points and the geometric center point of the square;

[0149] The truncated pyramid generation submodule is used to generate a truncated pyramid geometric model based on the three-dimensional coordinates of the pyramid center position, the upper plane boundary points and the bottom surface boundary points.

[0150] Optionally, the rotation point generation module includes:

[0151] The single-axis rotation point generation submodule is used to generate points rotated along a single axis by applying independent rotations of positive and negative preset angles on each axis in three-dimensional space, taking the center of the truncated pyramid geometric model as the reference point.

[0152] The multi-axis combined rotation point generation submodule is used to use the center position of the truncated pyramid geometric model as a reference point, arrange and combine at least two direction axes in three-dimensional space, and then apply positive and negative preset rotation angles for combined rotation to generate points of at least two direction axes combined rotation.

[0153] Optionally, the camera internal and external parameter calibration and optimization module includes:

[0154] An initial calibration submodule, configured to perform initial calibration of the camera's internal and external parameters using the Zhang Zhengyou calibration method based on the collected image data to obtain the camera's initial internal and external parameters;

[0155] The calibration optimization submodule is used to construct an optimization model with minimization of projection error as the objective function, use the initial intrinsic and extrinsic parameters of the camera as the initial optimization values, use a genetic algorithm to perform global optimization, obtain the globally optimized intrinsic and extrinsic parameters of the camera, and use a nonlinear least squares optimization algorithm to locally optimize the globally optimized intrinsic and extrinsic parameters of the camera until the projection error is less than a preset threshold, and output the locally optimized intrinsic and extrinsic parameters of the camera.

[0156] Optionally, the motion path of the robot includes a sequence from bottom to top or from top to bottom based on the set of motion points of the robot.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 4As shown, it includes a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0159] Memory 403, used for storing computer programs;

[0160] The processor 401 is configured to implement the camera automatic calibration method based on truncated pyramid trajectories as described in the above embodiment when executing the program stored in the memory 403 .

[0161] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0162] The communication interface is used for communication between the above terminal and other devices.

[0163] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0164] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0165] like Figure 5As shown, in another embodiment provided by the present invention, a computer-readable storage medium 501 is further provided, in which instructions are stored. When the computer-readable storage medium 501 is run on a computer, the computer executes the camera automatic calibration method based on the truncated pyramid trajectory described in the above embodiment.

[0166] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the automatic camera calibration method based on truncated pyramid trajectories described in the above embodiment.

[0167] In the above embodiments, all or part of the embodiments can be implemented through software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0168] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0169] Each embodiment in this specification is described in a related manner. Similar portions between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, refer to the description of the method embodiments.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A camera automatic calibration method based on truncated pyramid trajectory, characterized in that: The method comprises: Construct a truncated pyramid geometric model; Taking the center of the truncated pyramid geometric model as a reference point, multiple points with different preset rotation angles are generated; Multiple points with different preset rotation angles, together with the boundary points of the truncated pyramid geometric model, constitute a robot motion point set; Planning a motion path of the robot according to the robot motion point set, and controlling the robot to move to each point of the robot motion point set according to the planned motion path and to stay briefly to collect image data; The image data is used to calibrate the internal and external parameters of the camera, and optimization is performed based on an optimization algorithm to obtain the internal and external parameters of the camera.

2. The method according to claim 1, characterized in that Construct a truncated pyramid geometry model, including: Setting geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface; Calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters; Determine the three-dimensional coordinates of the upper plane boundary point and the bottom surface boundary point based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary point and the geometric center point of the square, and the distance between the bottom surface boundary point and the geometric center point of the square; A truncated pyramid geometric model is generated based on the three-dimensional coordinates of the tower center position, the upper plane boundary points and the bottom surface boundary points.

3. The method according to claim 1, characterized in that Using the center of the truncated pyramid as the reference point, multiple points with different preset rotation angles are generated, including: Taking the center of the truncated pyramid as the reference point, apply independent rotations of positive and negative preset angles on each direction axis in three-dimensional space to generate points rotated along a single direction axis. Taking the center of the truncated pyramid geometric model as the reference point, at least two direction axes are arranged and combined in three-dimensional space, and then positive and negative preset rotation angles are applied for combined rotation to generate a point position where at least two direction axes are combined and rotated.

4. The method according to claim 1, wherein The image data is used to calibrate the camera's internal and external parameters, and optimization is performed based on an optimization algorithm to obtain the camera's internal and external parameters, including: Based on the collected image data, the Zhang Zhengyou calibration method is used to perform initial calibration of the camera's internal and external parameters to obtain the camera's initial internal and external parameters; An optimization model with minimization of projection error as the objective function is constructed. The initial intrinsic and extrinsic parameters of the camera are used as the initial optimization values. A genetic algorithm is used for global optimization to obtain the globally optimized intrinsic and extrinsic parameters of the camera. A nonlinear least squares optimization algorithm is then used to locally optimize the globally optimized intrinsic and extrinsic parameters of the camera until the projection error is less than a preset threshold, and the locally optimized intrinsic and extrinsic parameters of the camera are output.

5. The method according to claim 1, wherein The motion path of the robot includes a motion point set based on the robot in a bottom-to-top or top-to-bottom order.

6. A camera automatic calibration device based on truncated pyramid trajectory, characterized in that: The device comprises: A truncated pyramid building module, used to build a truncated pyramid geometric model; A rotation point generation module is used to generate multiple points with different preset rotation angles using the center position of the truncated pyramid geometric model as a reference point; A motion point set composition module is used for multiple points with different preset rotation angles, which together with the boundary points of the truncated pyramid geometric model constitute the robot motion point set; a calibration image data acquisition module, configured to plan a motion path of the robot according to the robot motion point set, and control the robot to move to each point of the robot motion point set along the planned motion path and to briefly stop there to collect image data; The camera internal and external parameter calibration and optimization module is used to calibrate the camera internal and external parameters using the image data and optimize based on an optimization algorithm to obtain the camera internal and external parameters.

7. The device according to claim 6, characterized in that The truncated pyramid building blocks include: The geometric parameter setting submodule is used to set the geometric parameters of the truncated pyramid geometric model; the geometric parameters include the position of the pyramid center, the distance from the pyramid center to the upper plane, the distance from the pyramid center to the bottom surface, the side length of the upper plane, and the side length of the bottom surface; A parameter calculation submodule is used to calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters; A submodule for determining the three-dimensional coordinates of boundary points is used to determine the three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points based on geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary points and the geometric center point of the square, and the distance between the bottom surface boundary points and the geometric center point of the square; The truncated pyramid generation submodule is used to generate a truncated pyramid geometric model based on the three-dimensional coordinates of the pyramid center position, the upper plane boundary points and the bottom surface boundary points.

8. The device according to claim 6, characterized in that The rotation point generation module includes: The single-axis rotation point generation submodule is used to generate points rotated along a single axis by applying independent rotations of positive and negative preset angles on each axis in three-dimensional space, taking the center of the truncated pyramid geometric model as the reference point. The multi-axis combined rotation point generation submodule is used to use the center position of the truncated pyramid geometric model as a reference point, arrange and combine at least two direction axes in three-dimensional space, and then apply positive and negative preset rotation angles for combined rotation to generate points of at least two direction axes combined rotation.

9. An electronic device, characterized in that: comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is configured to implement the camera automatic calibration method based on truncated pyramid trajectories according to any one of claims 1 to 5 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to perform the camera automatic calibration method based on truncated pyramid trajectories according to any one of claims 1 to 5.

Images