A camera automatic calibration method and device based on truncated pyramid trajectory

Abstract

The application provides a camera automatic calibration method and device based on a truncated pyramid track, comprising: constructing a truncated pyramid geometric model, generating a plurality of point positions with different rotation angles with the tower core as a reference point, combining the truncated pyramid boundary points to form a robot motion point position set, and reaching each point position in turn and collecting images by planning the robot motion path and short-term staying, using the image data for camera internal and external parameter calibration, and optimizing by an optimization algorithm. The scheme enhances the calibration accuracy of large-angle cameras or distorted cameras through the design of the truncated pyramid structure and the rotating point position, and can adapt to complex environments such as light changes and partial occlusions, introduces an optimization algorithm, greatly reduces the dependence of the calibration process on the initial parameters, avoids the local optimal problem, and improves the robustness and accuracy of the calibration. At the same time, the relative motion of the camera and the calibration object is automatically controlled by the robot, which improves the automation degree and efficiency of the camera calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a camera automatic calibration method based on truncated pyramid trajectory, a camera automatic calibration device based on truncated pyramid trajectory, an electronic device and a computer readable medium. BACKGROUND

[0002] With the rapid development of computer vision technology, camera calibration as a key technology in computer vision system has been widely studied and applied. The purpose of camera calibration is to determine the intrinsic and extrinsic parameters of the camera, so as to establish an accurate mapping relationship between image coordinates and actual three-dimensional space coordinates. This is particularly important for applications such as stereo vision measurement, augmented reality, robot navigation, etc. Traditional camera calibration methods mainly rely on manual operation, usually by shooting known geometric shape calibration board (such as chessboard or circular dot array) to calibrate.

[0003] This existing camera calibration method mainly relies on classic planar calibration board or three-dimensional calibration object for calibration, although it can obtain relatively accurate camera intrinsic and extrinsic parameters, but still has the following shortcomings: complex operation and low efficiency: traditional camera calibration method usually needs precise calibration board or three-dimensional calibration object, and needs to collect images at different postures and angles for many times, the operation process is tedious and time-consuming, it is difficult to meet the demand of automatic calibration. Poor applicability: most methods are only suitable for specific scenes or structures, and have high dependence on external environment, such as lighting conditions, integrity of calibration board, etc., which will affect the calibration accuracy and stability. Sensitive to initial parameters: some algorithms are very sensitive to the selection of initial parameters in the optimization process, which is easy to fall into local optimum, resulting in inaccurate calibration results, affecting the robustness of calibration. Limited accuracy: when the camera has a large viewing angle or has serious distortion, the accuracy of traditional calibration method will be significantly reduced, especially when applied in long distance or large range scene. Low degree of automation: most calibration methods still rely on manual intervention and manual operation, which is difficult to realize efficient automation, and is not conducive to the development of large-scale or continuous calibration tasks. SUMMARY

[0004] In view of the above problems, the present application is proposed in order to provide a camera automatic calibration method based on truncated pyramid trajectory and corresponding camera automatic calibration device based on truncated pyramid trajectory, electronic device and computer readable medium, which can overcome the above problems or at least partially solve the above problems.

[0005] The present application discloses a camera automatic calibration method based on truncated pyramid trajectory, which comprises:

[0006] constructing a truncated pyramid geometric model;

[0007] generate a plurality of point positions with different preset rotation angles with the tower core position of the truncated pyramid geometric model as a reference point;

[0008] The plurality of point positions with different preset rotation angles, together with the boundary points of the truncated pyramid geometric model, constitute a robot motion point set;

[0009] According to the robot motion point set, a motion path of the robot is planned, and the robot is controlled to move to each point position of the robot motion point set and stay for a short time to collect image data according to the planned motion path;

[0010] The image data is used for camera internal and external parameter calibration, and optimization is performed based on an optimization algorithm to obtain camera internal and external parameters.

[0011] Optionally, the truncated pyramid geometric model is constructed, including:

[0012] The geometric parameters of the truncated pyramid geometric model are set; the geometric parameters include the tower core position, the distance from the tower core to the upper plane, the distance from the tower core to the bottom plane, the side length of the upper plane, and the side length of the bottom plane;

[0013] Based on the geometric parameters, the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point are calculated;

[0014] Based on the geometric parameters, the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point, the three-dimensional coordinates of the upper plane boundary point and the bottom plane boundary point are determined;

[0015] Based on the tower core position, the three-dimensional coordinates of the upper plane boundary point and the bottom plane boundary point, the truncated pyramid geometric model is generated.

[0016] Optionally, a plurality of point positions with different preset rotation angles are generated with the tower core position of the truncated pyramid geometric model as a reference point, including:

[0017] With the tower core position of the truncated pyramid geometric model as a reference point, independent rotation of positive and negative preset rotation angles is applied on each direction axis in three-dimensional space to generate a single direction axis rotation point;

[0018] With the tower core position of the truncated pyramid geometric model as a reference point, at least two direction axes are arranged and combined in three-dimensional space, and then positive and negative preset rotation angles are applied for combined rotation to generate at least two direction axis combined rotation points.

[0019] Optionally, the image data is used for camera internal and external parameter calibration, and optimization is performed based on an optimization algorithm to obtain camera internal and external parameters, including:

[0020] Based on the collected image data, the initial camera internal and external parameters are calibrated by using Zhang Zhengyou calibration method, and the initial camera internal and external parameters are obtained;

[0021] An optimization model with the minimum projection error as the objective function is constructed, the initial camera internal and external parameters are taken as the initial optimization values, the global optimization is performed by using genetic algorithm, the globally optimized camera internal and external parameters are obtained, and the locally optimized camera internal and external parameters are obtained by using the nonlinear least square optimization algorithm until the projection error is less than the preset threshold value, and the locally optimized camera internal and external parameters are output.

[0022] Optionally, the motion path of the robot comprises a sequence from bottom to top or from top to bottom based on the robot motion point set.

[0023] The application further discloses a camera automatic calibration device based on a truncated pyramid trajectory, which comprises:

[0024] A truncated pyramid construction module is configured to construct a truncated pyramid geometric model.

[0025] A rotation point generation module is configured to generate a plurality of points with different preset rotation angles with the tower center position of the truncated pyramid geometric model as a reference point.

[0026] A motion point set composition module is configured to jointly form a robot motion point set with the plurality of points with different preset rotation angles and the boundary points of the truncated pyramid geometric model.

[0027] A calibration image data acquisition module is configured to plan a motion path of a robot according to the robot motion point set, and control the robot to move to each point of the robot motion point set and stay for a short time to acquire image data according to the planned motion path.

[0028] A camera internal and external parameter calibration optimization module is configured to calibrate camera internal and external parameters by using the image data, and optimize the camera internal and external parameters based on an optimization algorithm.

[0029] Optionally, the truncated pyramid construction module comprises:

[0030] A geometric parameter setting submodule is configured to set geometric parameters of the truncated pyramid geometric model, and the geometric parameters comprise a tower center position, a distance from the tower center to an upper plane, a distance from the tower center to a bottom plane, a side length of the upper plane and a side length of the bottom plane.

[0031] A parameter calculation submodule is configured to calculate an upper plane height, a bottom plane height, a distance between an upper plane boundary point and a square geometric center point and a distance between a bottom plane boundary point and the square geometric center point based on the geometric parameters.

[0032] The boundary point three-dimensional coordinate determination submodule is configured to determine three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary points and the square geometric center point, and the distance between the bottom surface boundary points and the square geometric center point.

[0033] The truncated pyramid generation submodule is configured to generate a truncated pyramid geometric model based on the tower core position and the three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points.

[0034] Optionally, the rotation point generation module comprises:

[0035] The single-axis rotation point generation submodule is configured to generate a single-axis rotation point by taking the tower core position of the truncated pyramid geometric model as a reference point and applying independent rotation of a positive or negative preset rotation angle in each direction axis in the three-dimensional space.

[0036] The multi-axis combined rotation point generation submodule is configured to generate a combined rotation point of at least two direction axes by taking the tower core position of the truncated pyramid geometric model as a reference point, arranging and combining at least two direction axes in the three-dimensional space, and applying combined rotation of a positive or negative preset rotation angle.

[0037] Optionally, the camera internal and external parameter calibration optimization module comprises:

[0038] The initial calibration submodule is configured to perform initial calibration of the camera internal and external parameters by using Zhang Zhengyou calibration method based on the collected image data, and obtain initial camera internal and external parameters.

[0039] The calibration optimization submodule is configured to construct an optimization model with a projection error minimization as an objective function, take the initial camera internal and external parameters as optimization initial values, perform global optimization by using a genetic algorithm, obtain globally optimized camera internal and external parameters, and perform local optimization on the globally optimized camera internal and external parameters by using a nonlinear least square optimization algorithm until the projection error is less than a preset threshold, and output the locally optimized camera internal and external parameters.

[0040] Optionally, the motion path of the robot comprises a sequence from bottom to top or from top to bottom based on the robot motion point set.

[0041] The application further discloses an electronic device, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0042] The memory is used for storing a computer program.

[0043] The processor is used for realizing the camera automatic calibration method based on truncated pyramid trajectory as described in the application when executing the program stored on the memory.

[0044] The application further discloses one or more computer readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the camera automatic calibration method based on truncated pyramid trajectory as described in the application.

[0045] The application comprises the following advantages:

[0046] The camera automatic calibration method based on truncated pyramid trajectory comprises the following steps: constructing a truncated pyramid geometric model, generating a plurality of point positions with different rotation angles with the truncated pyramid as a reference point, combining the truncated pyramid boundary points to form a robot motion point set, planning a robot motion path, sequentially reaching each point position and collecting images for a short time, calibrating the camera internal and external parameters by using the image data, and optimizing the parameters by using an optimization algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a step flow chart of a camera automatic calibration method based on a truncated pyramid trajectory provided by an embodiment of the application;

[0048] Figure 2 is an example diagram of a truncated pyramid geometric model provided by an embodiment of the application;

[0049] Figure 3 is a structural block diagram of a camera automatic calibration device based on a truncated pyramid trajectory provided by an embodiment of the application;

[0050] Figure 4 is a block diagram of an electronic device provided by an embodiment of the application;

[0051] Figure 5 is a schematic diagram of a computer readable medium provided by an embodiment of the application. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0053] The present application generates a plurality of point positions with different rotation angles by taking the tower center as the reference point, and forms a robot motion point set by combining the truncated pyramid boundary points, and guides the camera to reach each point position in turn and stay for a short time to collect images along the motion path, and finally uses the image data to calibrate the internal and external parameters of the camera, and optimizes the parameters through an optimization algorithm, so as to realize the automatic calibration of the internal and external parameters of the camera. The method improves the calibration accuracy and robustness by constructing the three-dimensional space feature points of the truncated pyramid track and combining the efficient optimization algorithm, and is especially suitable for camera calibration scenes with large viewing angles or distortion.

[0054] Reference Figure 1 , a step flow chart of a camera automatic calibration method based on a truncated pyramid track provided in an embodiment of the present application is shown, which can specifically include the following steps:

[0055] Step 101, constructing a truncated pyramid geometric model;

[0056] Reference Figure 2 The truncated pyramid example diagram, in the present application, the truncated pyramid structure is selected as the camera automatic calibration method, which is based on the following reasons: 1. The truncated pyramid structure has strong geometric stability. 2. Multi-view calibration support. Since the upper and lower bottom surfaces of the truncated pyramid are similar in shape but different in size, and there is a certain height difference, the camera can still effectively detect the key feature points when observed from different angles (especially from the top and side). Therefore, it is especially suitable for multi-view camera calibration. 3. Depth information acquisition. The three-dimensional structure of the truncated pyramid helps to extract depth information and three-dimensional reconstruction. By taking multiple images from different angles, the spatial relationship between the camera and the calibration object can be more accurately calculated. 4. Easy to model and mathematically describe. 5. Suitable for various calibration methods. It can be used for Eye-in-Hand and Eye-to-Hand calibration methods, because no matter whether the camera moves or the object moves, the feature points of the truncated pyramid can be stably detected.

[0057] In an embodiment of the present application, the truncated pyramid geometric model is constructed, which includes:

[0058] Setting the geometric parameters of the truncated pyramid geometric model; the geometric parameters include the tower center position, the distance from the tower center to the upper plane, the distance from the tower center to the bottom plane, the side length of the upper plane, and the side length of the bottom plane;

[0059] Calculating the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters;

[0060] Based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom surface boundary point and the square geometric center point, the three-dimensional coordinates of the upper plane boundary point and the bottom surface boundary point are determined.

[0061] Based on the three-dimensional coordinates of the tower center position, the upper plane boundary point and the bottom surface boundary point, a truncated pyramid geometric model is generated.

[0062] In this embodiment, the truncated pyramid geometric model is constructed in the following way:

[0063] 1. Set the geometric parameters of the truncated pyramid:

[0064] Tower center position O;

[0065] Distance from tower center to upper plane h1;

[0066] Distance from tower center to bottom surface h2;

[0067] Side length of upper plane a;

[0068] Side length of bottom surface 2a.

[0069] 2. Calculate the plane height:

[0070] Upper plane height: Z1 = Z0 + h1.

[0071] Bottom surface height: Z2 = Z0 - h2.

[0072] 3. Calculate the distance between the boundary point and the center:

[0073] The distance between the boundary point in the upper plane and the square geometric center point is:

[0074]

[0075] The distance between the boundary point in the bottom surface and the square geometric center point is L2:

[0076]

[0077] 4. Determine the three-dimensional coordinates of the boundary points:

[0078] Upper plane boundary point:

[0079]

[0080] Bottom surface boundary point:

[0081]

[0082] Step 102, taking the tower center position of the truncated pyramid geometric model as the reference point, generate a plurality of point positions with different preset rotation angles;

[0083] In an embodiment of the present application, a plurality of point positions with different preset rotation angles are generated with the tower core position of the truncated pyramid geometric model as the reference point, including:

[0084] A single directional axis rotation point position is generated by applying independent rotation of positive and negative preset rotation angles on each directional axis in three-dimensional space with the tower core position of the truncated pyramid geometric model as the reference point.

[0085] At least two directional axis combination rotation point positions are generated by applying positive and negative preset rotation angles for combination rotation after arranging and combining at least two directional axes in three-dimensional space with the tower core position of the truncated pyramid geometric model as the reference point.

[0086] In the camera hand-eye calibration process, the image data obtained by only translational motion is often insufficient to accurately calibrate the internal and external parameters of the camera. Therefore, in order to enhance the accuracy and robustness of the calibration, the present application introduces a plurality of rotation angles for calibration at specific point positions. All rotation angles are generated with the tower core position O as the reference point. It is assumed that the initial pose of the tower core position is:

[0087] P O =(X0, Y0, Z0, RX0, RY0, RZ0)

[0088] Where (X0, Y0, Z0) is the three-dimensional coordinate of the tower core, and the last three parameters are the rotation angles of X-Y-Z Euler angles.

[0089] In order to better cover each direction in three-dimensional space, different rotation angles can be introduced on the X, Y, and Z axes respectively. It is assumed that the optimal calibration board rotation angle is 25°, and the X-Y-Z Euler angles are used to describe the rotation adopted by the Fanuc robot. Positive and negative 25° rotation can be performed on each axis, and combination rotation with other axes can be combined to generate more point positions.

[0090] The point positions generated according to the rotation angles are as follows:

[0091] P9=(X0, Y0, Z0, RX0+25°, RY0, RZ0)

[0092] P 10 =(X0, Y0, Z0, RX0, RY0+25°, RZ0)

[0093] P 11 =(X0, Y0, Z0, RX0, RY0, RZ0+25°)

[0094] P 12 =(X0, Y0, Z0, RX0-25°, RY0, RZ0)

[0095] P13 = (X0, Y0, Z0, RX0, RY0-25°, RZ0)

[0096] P 14 = (X0, Y0, Z0, RX0, RY0+25°, RZ0)

[0097] P 15 = (X0, Y0, Z0, RX0+25°, RY0+25°, RZ0)

[0098] Step 103, a plurality of points with different preset rotation angles and boundary points of the truncated pyramid geometric model jointly constitute a robot motion point set;

[0099] Step 104, according to the robot motion point set, a motion path of the robot is planned, and the robot is controlled to move to each point of the robot motion point set according to the planned motion path and stay for a short time to collect image data;

[0100] In an embodiment of the present application, the motion path of the robot includes an order from bottom to top or from top to bottom based on the robot motion point set.

[0101] In this embodiment, the preset point set is composed of two parts:

[0102] 1. Boundary points of the truncated pyramid geometric model (covering a wider spatial range).

[0103] 2. Multi-rotation angle points of the tower core position (enhancing the diversity and accuracy of calibration).

[0104] Suppose the point set is: P = {P1, P2, …, Pm}

[0105] Each point Pi contains pose information: Pi = (xi, yi, zi, EulerXi, EulerYi, EulerZi)

[0106] Steps of the path planning method:

[0107] First, group the points in the preset point set according to the type of the points: the first group: boundary points of the truncated pyramid trajectory. The second group: rotation angle points of the tower core position. Then arrange them in order from bottom to top or from top to bottom (for example, start from the bottom point and gradually move to the upper plane point).

[0108] Control the robot to move to each point according to the planned trajectory, and stay for a short time (for example, 1-2 seconds) at each point Pi to ensure that the camera collects clear enough images, and perform image acquisition and data recording.

[0109] In step 105, camera internal and external parameters are calibrated using the image data, and optimization is performed based on an optimization algorithm to obtain the camera internal and external parameters.

[0110] In an embodiment of the present application, camera internal and external parameters are calibrated using the image data, and optimization is performed based on an optimization algorithm to obtain the camera internal and external parameters, including:

[0111] Based on the collected image data, Zhang Zhengyou calibration method is used to perform initial calibration of the camera internal and external parameters to obtain initial camera internal and external parameters.

[0112] An optimization model with the minimum projection error as the objective function is constructed, the initial camera internal and external parameters are used as the initial optimization values, a genetic algorithm is used for global optimization to obtain the globally optimized camera internal and external parameters, and a nonlinear least squares optimization algorithm is used for local optimization of the globally optimized camera internal and external parameters until the projection error is less than a preset threshold, and the locally optimized camera internal and external parameters are output.

[0113] Global optimization algorithm---genetic algorithm (Genetic Algorithm, GA). Global optimization algorithm based on natural selection and genetic mechanism. The objective function is gradually optimized through population initialization, selection, crossover, mutation and other operations. Genetic algorithm can avoid falling into local optimum, but the convergence speed is slow.

[0114] 1. Encoding: Real number encoding is used, and the camera internal and external parameters are represented as chromosome genes:

[0115] Chromosome=[fx,fy,cx,cy,k1,k2,r1,r2,r3,tx,ty,tz]

[0116] Where: fx, fy: focal length parameters. cx, cy: principal point coordinates. k1, k2: distortion coefficients. r1, r2: rotation vectors. tx, ty: translation vectors.

[0117] 2. Population initialization (Initialization): Randomly generate an initial population, and the population size is generally set to 50-100 individuals.

[0118] 3. Fitness function (Fitness Function): Projection error is used as the fitness function:

[0119] Where: pi: actual feature point coordinates. pi^: predicted feature point coordinates. N: total number of feature points.

[0120] 4. Genetic manipulation: Selection: Use tournament selection or roulette wheel selection to select good individuals based on fitness values. Crossover: Use real number crossover (e.g. arithmetic crossover) to generate offspring individuals. Mutation: Randomly perturb gene values in chromosomes to increase population diversity.

[0121] 5. Iteration: Update the population and gradually approach the optimal solution through several generations (e.g. 100 generations). Set termination conditions such as generation limit or error convergence.

[0122] Nonlinear least squares optimization (Levenberg-Marquardt algorithm). Genetic algorithms can quickly find a globally optimal solution, but due to computational precision limitations, the final solution may not be fine enough. Therefore, based on the relatively optimal solution obtained by global optimization, further local fine optimization is performed using the Levenberg-Marquardt algorithm.

[0123] Optimization objectives:

[0124] The algorithm features: combining gradient descent and Newton method, with fast convergence speed and high precision. For solutions with good initial parameters, it can quickly optimize in a local range.

[0125] In order to solve the problems of inaccurate parameter initialization, local optimal solution and insufficient calibration accuracy in camera calibration, the present application adopts a combination of global optimization (genetic algorithm, GA) and local optimization (nonlinear least squares optimization algorithm, LM) to achieve a balance between accuracy and stability.

[0126] Optimization process: Global optimization stage: use genetic algorithm to generate initial parameter solution. Local optimization stage: based on the results of genetic algorithm, use LM algorithm for further optimization. Result evaluation: calculate the projection error to ensure that the error is within a reasonable range (e.g. less than 0.5 pixels).

[0127] This embodiment uses genetic algorithm with strong global search capability to perform global search through population evolution, which can effectively avoid falling into local optimal solution, and through LM algorithm, local fine adjustment is performed under the condition of good initial value, which can quickly converge to accurate solution, so as to realize the balance between accuracy and stability through two stages.

[0128] In summary, the automatic calibration method of the camera vision system by integrating software proposed in the application, combined with advanced calibration algorithm, can efficiently realize robot hand-eye calibration without complex calibration process. Such technology can automatically capture the feature point information required in the calibration process, is suitable for different types of industrial robot brands, and performs well in compatibility and operation convenience, and can be widely applied to industrial scenes such as precision assembly and automatic polishing.

[0129] Specifically, the following advantages are achieved:

[0130] 1. High calibration efficiency and high automation degree:

[0131] The application automatically generates a set of robot end pose points required for camera calibration through a pre-planned truncated pyramid trajectory path. The robot automatically moves along the trajectory and collects image data, greatly reducing manual intervention and improving calibration efficiency and automation degree.

[0132] 2. High precision and strong applicability:

[0133] By constructing a truncated pyramid geometric model and designing a reasonable motion path, high calibration precision can still be achieved in large-angle or large-distortion camera scenes. At the same time, the application has strong adaptability and good fault tolerance and stability to changes in light, partial occlusion of the calibration board, etc.

[0134] 3. Strong robustness and small parameter dependence:

[0135] The method of the application greatly reduces the dependence on initial parameters by introducing multi-angle rotating points and optimization algorithms, avoids the local optimal problem caused by improper selection of initial parameters in traditional methods, and thus improves the robustness and accuracy of calibration.

[0136] 4. Simple implementation and good scalability:

[0137] The truncated pyramid trajectory planning method of the application has the characteristics of simple structure and convenient calculation. The generated point set is not only suitable for hand-eye calibration, but also can be applied to other vision calibration scenes. At the same time, the method has good scalability, and can further improve the calibration accuracy or be applied to more complex application scenarios by adjusting the geometric parameters or optimization algorithms.

[0138] It should be noted that for the method embodiments, for the sake of simple description, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily the necessary actions of the embodiments of the present application.

[0139] Referring to Figure 3 , a structural block diagram of a camera automatic calibration device based on a truncated pyramid trajectory provided in the embodiments of the present application is shown, which can specifically include the following modules:

[0140] The truncated pyramid construction module 301 is configured to construct a truncated pyramid geometric model.

[0141] The rotation point generation module 302 is configured to generate a plurality of point positions with different preset rotation angles with the tower center position of the truncated pyramid geometric model as a reference point.

[0142] The motion point set composition module 303 is configured to jointly form a robot motion point set with the plurality of point positions with different preset rotation angles and the boundary points of the truncated pyramid geometric model.

[0143] The calibration image data acquisition module 304 is configured to plan a motion path of the robot according to the robot motion point set, and control the robot to move to each point position of the robot motion point set and stay for a short time according to the planned motion path to acquire image data.

[0144] The camera internal and external parameter calibration optimization module 305 is configured to perform camera internal and external parameter calibration using the image data, and perform optimization based on an optimization algorithm to obtain camera internal and external parameters.

[0145] Optionally, the truncated pyramid construction module includes:

[0146] The geometric parameter setting submodule is configured to set geometric parameters of the truncated pyramid geometric model; the geometric parameters include the tower center position, the distance from the tower center to the upper plane, the distance from the tower center to the bottom plane, the side length of the upper plane, and the side length of the bottom plane.

[0147] The parameter calculation submodule is configured to calculate the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters.

[0148] The boundary point three-dimensional coordinate determination submodule is configured to determine three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points based on the geometric parameters, the upper plane height, the bottom surface height, the distance between the upper plane boundary points and the square geometric center point, and the distance between the bottom surface boundary points and the square geometric center point.

[0149] The truncated pyramid generation submodule is configured to generate a truncated pyramid geometric model based on the tower center position and the three-dimensional coordinates of the upper plane boundary points and the bottom surface boundary points.

[0150] Optionally, the rotation point generation module comprises:

[0151] The single-axis rotation point generation submodule is configured to generate a single-axis rotation point by applying independent rotation of a positive or negative preset rotation angle in each direction axis in the three-dimensional space with the tower center position of the truncated pyramid geometric model as a reference point.

[0152] The multi-axis combined rotation point generation submodule is configured to generate a combined rotation point of at least two direction axes by applying combined rotation of a positive or negative preset rotation angle after arranging and combining the at least two direction axes in the three-dimensional space with the tower center position of the truncated pyramid geometric model as a reference point.

[0153] Optionally, the camera internal and external parameter calibration optimization module comprises:

[0154] The initial calibration submodule is configured to perform initial calibration of the camera internal and external parameters by using Zhang Zhengyou calibration method based on the collected image data, to obtain initial camera internal and external parameters.

[0155] The calibration optimization submodule is configured to construct an optimization model with the minimization of the projection error as an objective function, to take the initial camera internal and external parameters as optimization initial values, to perform global optimization by using a genetic algorithm, to obtain globally optimized camera internal and external parameters, and to perform local optimization on the globally optimized camera internal and external parameters by using a nonlinear least squares optimization algorithm until the projection error is less than a preset threshold, and to output the locally optimized camera internal and external parameters.

[0156] Optionally, the motion path of the robot comprises a sequence from bottom to top or from top to bottom based on the robot motion point set.

[0157] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts refer to the part of the method embodiment.

[0158] In addition, the embodiment of the present application also provides an electronic device, such as Figure 4As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0159] Memory 403 is used to store computer programs;

[0160] When the processor 401 executes the program stored in the memory 403, it implements the automatic camera calibration method based on truncated pyramid trajectory as described in the above embodiments.

[0161] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0162] The communication interface is used for communication between the aforementioned terminal and other devices.

[0163] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0164] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0165] like Figure 5As shown, in yet another embodiment provided by the present application, a computer readable storage medium 501 is also provided, in which instructions are stored, which, when executed on a computer, cause the computer to perform the camera automatic calibration method based on the truncated pyramid trajectory as described in the above embodiments.

[0166] In yet another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when executed on a computer, cause the computer to perform the camera automatic calibration method based on the truncated pyramid trajectory as described in the above embodiments.

[0167] In the above embodiments, the implementation can be wholly or partially achieved by software, hardware, firmware, or any combination thereof. When implemented by software, the implementation can be wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, Solid State Disk (SSD)), etc.

[0168] It should be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0169] Each of the embodiments in the specification is described in a relevant manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0170] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A camera automatic calibration method based on truncated pyramid trajectory, characterized in that, The method comprises: constructing a truncated pyramid geometric model; generating a plurality of point positions with different preset rotation angles with the tower core position of the truncated pyramid geometric model as a reference point; the plurality of point positions with different preset rotation angles and the boundary points of the truncated pyramid geometric model jointly constitute a robot motion point set; according to the robot motion point set, a motion path of the robot is planned, and the robot is controlled to move to each point position of the robot motion point set according to the planned motion path and stay for a short time to collect image data; camera internal and external parameters are calibrated using the image data, and optimization is performed based on an optimization algorithm to obtain the camera internal and external parameters; generating a plurality of point positions with different preset rotation angles with the tower core position of the truncated pyramid geometric model as a reference point, comprising: applying independent rotation of positive and negative preset rotation angles on each direction axis in the three-dimensional space respectively to generate a single direction axis rotation point position with the tower core position of the truncated pyramid geometric model as a reference point; after arranging and combining at least two direction axes in the three-dimensional space, applying positive and negative preset rotation angles for combined rotation to generate at least two direction axis combined rotation point positions with the tower core position of the truncated pyramid geometric model as a reference point; calibrating camera internal and external parameters using the image data, and optimizing based on an optimization algorithm to obtain the camera internal and external parameters, comprising: based on the collected image data, using Zhang Zhengyou calibration method to perform initial calibration of camera internal and external parameters to obtain camera initial internal and external parameters; constructing an optimization model with the minimum projection error as the objective function, taking the camera initial internal and external parameters as the optimization initial value, performing global optimization using a genetic algorithm to obtain globally optimized camera internal and external parameters, and performing local optimization on the globally optimized camera internal and external parameters using a nonlinear least squares optimization algorithm until the projection error is less than a preset threshold, and outputting the locally optimized camera internal and external parameters.

2. The method of claim 1, wherein, constructing a truncated pyramid geometric model, comprising: setting geometric parameters of the truncated pyramid geometric model; the geometric parameters include the tower core position, the distance from the tower core to the upper plane, the distance from the tower core to the bottom plane, the side length of the upper plane, and the side length of the bottom plane; calculating the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point based on the geometric parameters; determining the three-dimensional coordinates of the upper plane boundary points and the bottom plane boundary points based on the geometric parameters, the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point; generating the truncated pyramid geometric model based on the tower core position, the three-dimensional coordinates of the upper plane boundary points, and the three-dimensional coordinates of the bottom plane boundary points.

3. The method of claim 1, wherein, The motion path of the robot comprises a sequence from bottom to top or from top to bottom based on the robot motion point set.

4. A camera auto-calibration apparatus based on truncated pyramid trajectory, characterized in that, The device comprises: a truncated pyramid construction module for constructing a truncated pyramid geometric model; a rotation point position generation module for generating a plurality of point positions with different preset rotation angles with the tower core position of the truncated pyramid geometric model as a reference point; The motion point set composition module is configured to jointly form a robot motion point set with boundary points of a truncated pyramid geometric model by using a plurality of points with different preset rotation angles. The calibration image data acquisition module is configured to plan a motion path of the robot according to the robot motion point set, and control the robot to move to each point of the robot motion point set and stay for a short time to acquire image data according to the planned motion path. The camera internal and external parameter calibration and optimization module is configured to calibrate camera internal and external parameters by using the image data, and optimize the camera internal and external parameters based on an optimization algorithm. The rotation point generation module includes: The single-axis rotation point generation submodule is configured to generate a single-axis rotation point by applying independent rotation of a positive or negative preset rotation angle in each direction axis in a three-dimensional space with the tower center position of the truncated pyramid geometric model as a reference point. The multi-axis combined rotation point generation submodule is configured to generate a combined rotation point of at least two direction axes by applying combined rotation of a positive or negative preset rotation angle after arranging and combining at least two direction axes in a three-dimensional space with the tower center position of the truncated pyramid geometric model as a reference point. The camera internal and external parameter calibration and optimization module includes: The initial calibration submodule is configured to calibrate initial camera internal and external parameters by using Zhang Zhengyou calibration method based on the acquired image data. The calibration optimization submodule is configured to construct an optimization model with a projection error minimization as an objective function, use the initial camera internal and external parameters as optimization initial values, perform global optimization by using a genetic algorithm to obtain globally optimized camera internal and external parameters, and perform local optimization on the globally optimized camera internal and external parameters by using a nonlinear least squares optimization algorithm until the projection error is less than a preset threshold, and output the locally optimized camera internal and external parameters.

5. The apparatus of claim 4, wherein, The truncated pyramid construction module includes: The geometric parameter setting submodule is configured to set geometric parameters of the truncated pyramid geometric model, including a tower center position, a distance from the tower center to an upper plane, a distance from the tower center to a bottom plane, a side length of the upper plane, and a side length of the bottom plane. The parameter calculation submodule is configured to calculate an upper plane height, a bottom plane height, a distance between an upper plane boundary point and a square geometric center point, and a distance between a bottom plane boundary point and the square geometric center point based on the geometric parameters. The boundary point three-dimensional coordinate determination submodule is configured to determine three-dimensional coordinates of the upper plane boundary point and the bottom plane boundary point based on the geometric parameters, the upper plane height, the bottom plane height, the distance between the upper plane boundary point and the square geometric center point, and the distance between the bottom plane boundary point and the square geometric center point. The truncated pyramid generation submodule is configured to generate the truncated pyramid geometric model based on the tower center position and the three-dimensional coordinates of the upper plane boundary point and the bottom plane boundary point.

6. An electronic device, comprising: The device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to implement the truncated-pyramid trajectory based camera auto-calibration method according to any one of claims 1-3 when executing the program stored in the memory.

7. One or more computer-readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the truncated-pyramid trajectory based camera auto-calibration method according to any one of claims 1-3.

Images