Transformation matrix determination method and device, equipment and storage medium

Abstract

The invention discloses a conversion matrix determination method and device, equipment and a storage medium. According to the technical scheme, the calibration object is identified through the camera, the first coordinates of the at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates of the at least nine feature points on the calibration object in the world coordinate system under each sampling position are determined, the first coordinates and the second coordinates are converted into the coordinates under the camera coordinate system, and the equation is constructed; according to the first coordinate, the second coordinate, a conversion matrix from a world coordinate system to a tail end coordinate system of the calibration object and a conversion matrix from a pixel coordinate system to a camera coordinate system, solving an equation of an equation, and determining a conversion matrix from the tail end coordinate system to the camera coordinate system, an inertial unit or a mechanical arm joint encoder is not needed to measure the pose of the tail end mechanism of the calibration object, and hand-eye calibration of a low-cost open-loop control system without a pose measuring element is achieved.

Description

Technical Field

[0001] The present invention relates to the field of hand-eye calibration technology, and in particular to a method, device, equipment and storage medium for determining a conversion matrix. Background Art

[0002] Hand-Eye Calibration is a key technology in robot vision systems. It is used to determine the relative position and posture relationship between the camera and the robot (camera extrinsics). It is often used in robotic arm systems and pan-tilt systems to calculate the relative position of the camera and the end-effector to achieve trajectory planning of the mechanism.

[0003] Generally speaking, the pose transformation from the camera coordinate system to the calibration object coordinate system is obtained by algorithms such as PnP (Perspective-n-Point), while the transformation from the robot end coordinate system to the ground coordinate system is obtained by the inertial measurement unit or the angle measurement element installed on the robotic arm joint.

[0004] Existing hand-eye calibration methods all share the requirement for sensors other than cameras to measure the pose of the robot's end-to-ground system. Otherwise, only the theoretical pose of the end-effector under open-loop control can be obtained. In many low-cost robotic control systems, the mechanical structure itself relies solely on open-loop control, resulting in significant errors between the theoretical and actual poses. Introducing additional sensors for measurement increases product cost and reduces competitiveness, creating a dilemma between external parameter measurement accuracy and price. Summary of the Invention

[0005] The present invention provides a method, device, equipment and storage medium for determining a conversion matrix, so as to realize hand-eye calibration of a low-cost open-loop control system without requiring measurement elements.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining a conversion matrix, the method comprising:

[0007] Determine a first conversion matrix and a second conversion matrix, wherein the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system;

[0008] Controlling the calibration object to rotate to at least two sampling positions, and determining the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system at each sampling position;

[0009] A transformation matrix from the calibration object end coordinate system to the camera coordinate system is determined according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate.

[0010] In a second aspect, an embodiment of the present invention further provides a hand-eye calibration device, the device comprising:

[0011] An auxiliary matrix determination module is used to determine a first conversion matrix and a second conversion matrix, wherein the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system;

[0012] A sampling coordinate determination module is used to control the calibration object to rotate to at least two sampling positions, and determine the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system at each sampling position;

[0013] The conversion matrix determination module is used to determine the conversion matrix from the calibration object end coordinate system to the camera coordinate system based on the first conversion matrix, the second conversion matrix, the first coordinate and the second coordinate.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for determining a transformation matrix as described in any one of the embodiments of the present invention is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for determining a transformation matrix as described in any one of the embodiments of the present invention.

[0016] The technical solution of the embodiment of the present invention uses a camera to identify the calibration object, and determines the conversion matrix from the end coordinate system to the camera coordinate system based on the first coordinate and the second coordinate collected in different coordinate systems, the conversion matrix from the world coordinate system to the end coordinate system of the calibration object, and the conversion matrix from the pixel coordinate system to the camera coordinate system. The present invention does not require an inertial unit or a robotic arm joint encoder to measure the posture of the end mechanism of the calibration object, and realizes hand-eye calibration of a low-cost open-loop control system without posture measurement elements.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a flow chart of a method for determining a conversion matrix provided in the first embodiment of the present invention;

[0020] Figure 2 This is a schematic structural diagram of a calibration object retaining one rotational degree of freedom provided in the first embodiment of the present invention;

[0021] Figure 3 1 is a schematic diagram of the structure of a hand-eye calibration system provided in Example 1 of the present invention;

[0022] Figure 4 is a flow chart of a method for determining a conversion matrix provided in the second embodiment of the present invention;

[0023] Figure 5 This is a flowchart of optimizing the transformation matrix from the terminal coordinate system to the camera coordinate system provided by the second embodiment of the present invention;

[0024] Figure 6 1 is a schematic structural diagram of a hand-eye calibration device provided in Embodiment 3 of the present invention;

[0025] Figure 7 The figure is a schematic structural diagram of an electronic device for implementing the method for determining a conversion matrix according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] Figure 1 A flow chart of a method for determining a conversion matrix is provided for the first embodiment of the present invention. This embodiment is applicable to hand-eye calibration. The method can be performed by a device for determining a conversion matrix. The device for determining a conversion matrix can be implemented in the form of hardware and / or software. The device for determining a conversion matrix can be configured in any electronic device with network communication and computing capabilities. Figure 1 As shown, the method includes:

[0030] S110: Determine a first conversion matrix and a second conversion matrix.

[0031] In this embodiment, the first transformation matrix is the transformation matrix from the world coordinate system to the end coordinate system of the calibration object, and the second transformation matrix is the transformation matrix from the pixel coordinate system to the camera coordinate system. The calibration object can be a robotic arm or a gimbal with one rotational degree of freedom.

[0032] It should be noted that for a general six-degree-of-freedom robotic arm or gimbal, this method requires fixing five degrees of freedom of the robotic arm or gimbal on which the camera is mounted, leaving only one degree of freedom of rotation. The degree of freedom fixation can be achieved through the control of the robotic arm itself or structural constraints.

[0033] join Figure 2 The figure shows a schematic diagram of a calibration object with one rotational degree of freedom. The calibration object rotates around an endpoint at a first end, and the camera is positioned at a second end. It should be noted that the calibration object rotates around the axis of the endpoint at the first end, which is vertical and fixed.

[0034] Furthermore, it should be noted that the end coordinate system of this embodiment is constructed with the endpoint of the first end of the calibration object as the origin, and the plane where the calibration object rotates is the XOY plane. The end coordinate system rotates with the rotation of the calibration object, but the position of the axis connecting the calibration object relative to the world coordinate system remains unchanged. The Z of the end coordinate system is g Axis vertical and X g With Y g In the same horizontal plane (due to issues such as control accuracy, the calibration object itself may not be horizontal).

[0035] The camera coordinate system of this embodiment is constructed with the equivalent optical center of the camera model as the origin and the camera normalized imaging plane as the XOY plane.

[0036] The world coordinate system of this embodiment is constructed with any point on the ground as the origin and the plane where the ground is located as the XOY plane.

[0037] The pixel coordinate system of this embodiment is a two-dimensional coordinate system composed of pixel points of the calibration object on the camera imaging plane.

[0038] It should be noted that this method calibrates the pose transformation matrix between the end coordinate system and the camera coordinate system.

[0039] The calibration process of the pose transformation matrix of the end coordinate system and the camera coordinate system in this method is achieved through a pre-set hand-eye calibration system, see Figure 3 The figure shows a schematic diagram of the structure of a hand-eye calibration system. g Y g Z g The coordinate system is the end coordinate system, X c Y c Z c The coordinate system is the camera coordinate system, X b Y b Z b The coordinate system is the world coordinate system. In this hand-eye calibration system, the origin of the world coordinate system and the end coordinate system Z g axis aligned, its Z b Axis and end coordinate system Z g The axes are collinear to facilitate the subsequent calculation of the transformation matrix from the end coordinate system to the camera coordinate system and improve the calculation efficiency.

[0040] The first transformation matrix is the transformation matrix from the world coordinate system to the end coordinate system. It is related to the rotation angle of the calibration object (the rotation angle of the calibration object relative to the starting position) and the distance from the rotation axis to the ground (the distance between the origin of the end coordinate system and the origin of the ground coordinate system). It can be expressed as follows:

[0041]

[0042] Among them, T b2g is the transformation matrix from the world coordinate system to the end coordinate system, θ is the rotation angle, Z b2g It is the distance between the origin of the end coordinate system and the origin of the ground coordinate system.

[0043] The second transformation matrix is the transformation matrix from the pixel coordinate system to the camera coordinate system, i.e., the camera intrinsic parameters, which are related to the camera focal length, principal point, and pixel scale factor. In the camera imaging model, the camera intrinsic parameters are usually represented by a 3×3 matrix, which is as follows:

[0044]

[0045] Among them, f x is the product of the pixel scale in the horizontal direction and the focal length, f y is the product of the pixel size in the vertical direction and the focal length, (C x , C y ) is the coordinate of the camera principal point, which is the intersection of the camera optical axis and the image sensor plane.

[0046] In practical applications, camera intrinsic parameters can be directly obtained from camera calibration information. Through the camera intrinsic parameter matrix, points in three-dimensional space can be projected onto the two-dimensional image plane to achieve the conversion from the world coordinate system to the pixel coordinate system.

[0047] S120 , controlling the calibration object to rotate to at least two sampling positions, and determining first coordinates of at least nine feature points on the calibration object in the world pixel coordinate system and second coordinates in the world coordinate system at each sampling position.

[0048] In this embodiment, the first coordinate is the coordinate of the feature point on the calibration object at each sampling position in the pixel coordinate system, and the second coordinate is the coordinate of the feature point on the calibration object in the world coordinate system. Any point on the grid can be used to represent the location of the feature point, which can be the upper left corner, upper right corner, lower left corner, lower right corner, or center point of the grid.

[0049] A calibration object is an object with known geometric and physical characteristics, and can be a checkerboard calibration plate, a graphic calibration plate, a 3D calibration plate, or a random point calibration plate. The calibration object provides the camera with a set of feature points with a clear correspondence between 3D spatial positions and 2D image positions, enabling accurate 3D reconstruction and pose estimation in various subsequent computer vision tasks.

[0050] The calibration object in this embodiment can be a checkerboard calibration plate, which is more suitable for the hand-eye calibration process in this embodiment. In an image, the checkerboard's corner features are distinct. Simple image processing algorithms, such as corner detection algorithms (e.g., Harris corner detection), can accurately extract the coordinates of the corners. This enables rapid and stable acquisition of a large amount of feature point information during the calibration process, providing a rich data foundation for camera calibration. Furthermore, the camera calibration method based on the checkerboard calibration plate is supported by comprehensive mathematical theory. By establishing the transformation relationship between the checkerboard corners in the world coordinate system, the camera coordinate system, and the image coordinate system, and utilizing the perspective projection principle and camera imaging model, calculation formulas for the camera's intrinsic and extrinsic parameters and distortion parameters can be derived. Furthermore, the checkerboard calibration plate is relatively simple to manufacture, requiring only printing or drawing a black and white checkerboard pattern on a flat surface. Compared to high-precision three-dimensional calibration objects or calibration plates made of special materials, the checkerboard calibration plate is significantly less expensive, facilitating widespread application and promotion.

[0051] Furthermore, in this embodiment, the grid points on the corners of the calibration plate can be placed at the origin of the world coordinate system, which is located at the intersection of the rotation axis of the calibration object and the ground. This process helps clarify the transformation relationship and eliminates the need for complex coordinate transformation calculations. It also reduces variables and simplifies the form of the equations when establishing the coordinate system transformation equations and solving parameters, reducing computational complexity and improving computational efficiency and accuracy. It also reduces error accumulation and improves calibration accuracy.

[0052] Furthermore, the origin position of the world coordinate system can be implemented by the following steps.

[0053] The calibration object is controlled to rotate a certain angle each time, and the coordinates of the calibration plate grid points in the pixel coordinate system are identified through an algorithm until the calibration object rotates to a certain angle and enough data is collected.

[0054] Determine the position of the origin of the world coordinate system in the pixel coordinate system. Using the nearest neighbor greedy algorithm, match each calibration grid point in the data obtained in the previous step between frames. Plot the position of each point in each frame in a single image to form a trajectory that approximates a circular arc. Ideally, all trajectories are concentric. Use the least squares method to fit this common center, which is the projection of the world coordinate system in the pixel coordinate system.

[0055] Place the top left grid point of the calibration plate at the origin of the world coordinate system. This process can be achieved by using a camera as a sensor and a robotic arm to form a closed-loop system. The rotation axis of the calibration object is translated horizontally so that the projection of the top left grid point of the calibration plate in the pixel coordinate system coincides with the projection of the world coordinate system origin in the pixel coordinate system.

[0056] In practical applications, if the calibration object (robotic arm) itself is inconvenient to perform this control process, this process can also be simulated in the field of view of the camera lens by moving the calibration plate so that its upper left point coincides with the origin of the world coordinate system.

[0057] Furthermore, the calibration object is controlled to rotate to at least two sampling positions, and the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system are determined at each sampling position.

[0058] It should be noted that since the upper left grid point of the calibration plate coincides with the origin of the world coordinate system, and the size of each grid on the calibration plate is known, the coordinates of each grid on the calibration plate in the world coordinate system are known information. Therefore, the second coordinate of each point on the calibration plate in the world coordinate system at each sampling position can be determined. In this embodiment, the feature points on the calibration object can be selected from grid points or corner points on the calibration plate.

[0059] During the data acquisition process, the calibration object (robotic arm mechanism) is continuously rotated to identify the calibration grid points and record their coordinates in the pixel coordinate system. This process generally involves image acquisition, image preprocessing, grid point identification, and coordinate recording. The pixel coordinate system usually takes the upper left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis.

[0060] Specifically, according to pre-set rules, the calibration object is controlled to rotate to different postures (sampling positions) so that the camera installed on the calibration object can capture images containing the calibration plate from multiple angles. Then, the image is processed through the image preprocessing step to make the grid features of the calibration plate on the image more clear and prominent. Furthermore, the corner point positions of the calibration plate are determined by the corner point detection algorithm. After determining the corner point positions on the calibration plate, these corner points need to be matched and sorted to determine their relative positions on the calibration plate. Matching and sorting can be performed based on information such as the distance and angle between the corner points to ensure that each corner point corresponds to a specific grid point on the calibration plate.

[0061] Furthermore, after identifying the feature points on the marking board corresponding to each corner point, the first coordinates of at least nine feature points in the pixel coordinate system are extracted from the image. Any point on the grid can be used to represent the location of the feature point, including the top left corner, top right corner, bottom left corner, bottom right corner, or center point of the grid. Furthermore, the first and second coordinates of the nine feature points are matched for subsequent hand-eye calibration calculations.

[0062] It should be noted that the number of unknowns in the present invention is eighteen. In the subsequent steps, one characteristic point on the calibration object can construct two equations, so the first coordinates and second coordinates of at least nine characteristic points are required.

[0063] S130 , determining a transformation matrix from the calibration object end coordinate system to the camera coordinate system according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate.

[0064] In this embodiment, the conversion matrix from the calibration object end coordinate system to the camera coordinate system includes a rotation matrix from the calibration object end coordinate system to the camera coordinate system and a translation matrix from the calibration object end coordinate system to the camera coordinate system. The matrix form of the conversion matrix from the calibration object end coordinate system to the camera coordinate system is expressed as follows:

[0065]

[0066] Among them, R g2c is the rotation matrix from the end coordinate system to the camera coordinate system, t g2c is the translation matrix from the end coordinate system to the camera coordinate system.

[0067] Furthermore, through the first transformation matrix and the second transformation matrix, the first coordinates of the pixel coordinate system and the second coordinates of the world coordinate system of the grid points on the calibration plate can be converted into camera coordinates in the camera coordinate system. Then, based on the consistent characteristics of the camera coordinates after the transformation of the grid points on the calibration plate, a set of equations is constructed to determine the transformation matrix from the end coordinate system of the calibration object to the camera coordinate system.

[0068] As an optional but non-limiting implementation, determining a transformation matrix from the calibration object end coordinate system to the camera coordinate system based on the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate includes:

[0069] Determine a conversion matrix from the calibration object end coordinate system to the camera coordinate system based on the coordinate system transformation relationship according to the first conversion matrix, the second conversion matrix, the first coordinate, and the second coordinate;

[0070] The coordinate system transformation relationship is expressed by the following formula:

[0071]

[0072] Where F is the second transformation matrix, is the first coordinate of the i-th point on the calibration object in the pixel coordinate system when the calibration object is rotated to the sampling position, is the second coordinate of the i-th point on the calibration object in the world coordinate system when the calibration object is rotated to the sampling position, T b2g is the first transformation matrix, T g2c is the transformation matrix from the end coordinate system to the camera coordinate system.

[0073] In this embodiment, the first coordinate of the i-th point on the calibration object in the pixel coordinate system can be converted to the corresponding camera coordinate in the camera coordinate system through the conversion matrix between the pixel coordinate system and the camera coordinate system. The second coordinate of the i-th point on the calibration object in the world coordinate system can be converted to the corresponding camera coordinate in the camera coordinate system through the conversion matrix between the world coordinate system and the end coordinate system and the conversion matrix between the end coordinate system and the camera coordinate system. Furthermore, based on the consistency of the converted camera coordinates of the same point on the calibration object at the same sampling position, an equation can be established. Since the first coordinate, the second coordinate, the first conversion matrix, and the second conversion matrix are known data, the conversion matrix from the end coordinate system of the calibration object to the camera coordinate system can be determined based on the equation.

[0074] As an optional but non-limiting implementation, determining a transformation matrix from the calibration object end coordinate system to the camera coordinate system based on the coordinate system transformation relationship according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate includes the following steps A1-A2:

[0075] Step A1: Determine the rotation angle corresponding to the sampling position according to the first coordinate and the second coordinate corresponding to the same sampling position, the first transformation matrix and the second transformation matrix, and the coordinate system transformation relationship.

[0076] Step A2: Calculate the conversion matrix from the coordinate system of the end point of the calibration object to the camera coordinate system according to the corresponding rotation angle at each sampling position.

[0077] In this embodiment, the first coordinate of the i-th point on the calibration object in the pixel coordinate system is converted to the coordinate of the camera coordinate system when the calibration object is rotated to the sampling position through the conversion matrix between the pixel coordinate system and the camera coordinate system, which can be expressed as follows:

[0078]

[0079] In this embodiment, the second coordinate of the i-th point on the calibration object in the world coordinate system is converted to the coordinate in the camera coordinate system when the calibration object is rotated to the sampling position through the conversion matrix between the world coordinate system and the end coordinate system and the conversion matrix between the end coordinate system and the camera coordinate system, which can be expressed as follows:

[0080]

[0081] According to the same sampling position, the camera coordinates of the same point on the calibration object after transformation are consistent, so the equation can be obtained:

[0082]

[0083] set up Simplifying the above formula, we get:

[0084]

[0085] In the acquisition of the calibration object rotated to the jth sampling position (pose), since the calibration object remains unchanged at the jth sampling position, the rotation angle θ of all points on the calibration object from the world coordinate system to the end coordinate system is the same value, denoted as θ j , where j = 1, 2, ..., m. In a camera-acquired image containing a calibration plate, assume that there are enough calibration plate grid points to solve the following overdetermined equation:

[0086]

[0087] AX1=0;

[0088] Furthermore, we can use SVD (Singular Value Decomposition) to decompose and obtain a non-zero least squares solution, that is, take the last column of the matrix of the decomposition result (which is also the column with the smallest eigenvalue) as the solution of X1, and then obtain

[0089]

[0090] Because Z b2g >0, so:

[0091]

[0092] Therefore, the rotation matrix [r 13 r 23 r 33 ] T Furthermore, we can classify and discuss the two values of a and find the other parameters of the rotation matrix respectively:

[0093]

[0094] Perform SVD decomposition on the rotation matrix:

[0095]

[0096] Comparing the determinant of the orthogonal matrix Σ in the two cases, the value of a corresponding to the case closer to 1 is the final estimated value of a, and then the rotation angle θ corresponding to the jth sampling position can be determined based on the estimated value of a j estimated value.

[0097] Furthermore, after determining the rotation angle corresponding to each sampling position, all θ i can be considered as a known quantity. Redefine Rearranging the equations, we can get:

[0098]

[0099] BX2=0;

[0100] According to the same method as above, SVD decomposition can be used to obtain a non-zero least squares solution, that is, the last column of the decomposition result matrix is taken as the solution of X2. In this way, the translation matrix of the transformation matrix from the end coordinate system to the camera coordinate system and the first two columns of the rotation matrix can be obtained. For the third column vector of the rotation matrix, the present invention uses a method similar to the above intra-frame estimation to first calculate Z b2g The estimated value of , and then find the third column vector of the rotation matrix. Further, the obtained rotation matrix is orthogonalized and decomposed using SVD to obtain:

[0101]

[0102] Thus, the transformation matrix from the calibration object end coordinate system to the camera coordinate system has been obtained.

[0103] It should be noted that the rough estimate of the transformation matrix from the calibration object's end coordinate system to the camera coordinate system obtained above, based on the least squares theory, may deviate from the true value. An optimization algorithm can be used to further optimize the transformation matrix from the end coordinate system to the camera coordinate system. The optimization process for the transformation matrix from the end coordinate system to the camera coordinate system is described in detail in the following examples.

[0104] The technical solution of the embodiment of the present invention uses a camera to identify the calibration object, and determines the conversion matrix from the end coordinate system to the camera coordinate system based on the first coordinate and the second coordinate collected in different coordinate systems, the conversion matrix from the world coordinate system to the end coordinate system of the calibration object, and the conversion matrix from the pixel coordinate system to the camera coordinate system. The present invention does not require an inertial unit or a robotic arm joint encoder to measure the posture of the end mechanism of the calibration object, and realizes hand-eye calibration of a low-cost open-loop control system without posture measurement elements.

[0105] Example 2

[0106] Figure 4 This is a flow chart of a method for determining a conversion matrix provided in the second embodiment of the present invention. This embodiment of the present invention is further specified based on the above embodiment. This embodiment is applicable to hand-eye calibration. The method can be executed by a device for determining a conversion matrix. The device for determining a conversion matrix can be implemented in the form of hardware and / or software. The device for determining a conversion matrix can be configured in any electronic device with network communication and computing capabilities. Figure 4 As shown, the method includes:

[0107] S210. Determine a first conversion matrix and a second conversion matrix, where the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system.

[0108] S220 , controlling the calibration object to rotate to at least two sampling positions, and determining first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and second coordinates in the world coordinate system at each sampling position.

[0109] S230 : Determine a transformation matrix from the calibration object end coordinate system to the camera coordinate system according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate.

[0110] It should be noted that after determining the rough estimate of the conversion matrix from the calibration object's terminal coordinate system to the camera coordinate system, there may be a certain error between it and the actual value. Furthermore, this embodiment is divided into three parts to optimize the conversion matrix from the terminal coordinate system to the camera coordinate system.

[0111] S240: Determine the first pixel coordinates and the second pixel coordinates of the recognition point in the pixel coordinate system.

[0112] In this embodiment, the identification point can be any point on the calibration object, the first pixel coordinate is the coordinate of the identification point in the pixel coordinate system, and the second pixel coordinate is the coordinate of the identification point in the world coordinate system, which is the coordinate transformed according to the transformation matrix from the end coordinate system to the camera coordinate system.

[0113] In this embodiment, a checkerboard calibration plate can be selected as the calibration object, and the upper left corner grid point of the calibration plate is placed at the origin of the world coordinate system, and the upper left corner grid point of the calibration plate is used as the identification point to reduce conversion and calculation errors. It should be noted that if the lower left corner grid point of the calibration plate is placed at the origin of the world coordinate system, the lower left corner grid point of the calibration plate is used as the identification point to ensure that the position of the identification point on the calibration plate and the origin of the world coordinate system are consistent.

[0114] Furthermore, the grid point at the upper left corner of the calibration plate (identification point) is identified in the camera image, and the first pixel coordinate of the identification point in the pixel coordinate system is determined.

[0115] At the same time, since the position of the grid point in the upper left corner of the calibration plate is also the position of the origin of the world coordinate system, the origin of the world coordinate system can be converted to the second pixel coordinate in the pixel coordinate system based on the conversion matrix from the world coordinate system to the end coordinate system, the rough estimate of the conversion matrix from the end coordinate system to the camera coordinate system, and the conversion matrix between the camera coordinate system and the pixel coordinate system.

[0116] S250: Determine a translation error value of a transformation matrix from a terminal coordinate system to a camera coordinate system according to the first pixel coordinate and the second pixel coordinate.

[0117] In this embodiment, the translation error value is the error value of the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system. The translation error value may be the Euclidean distance between the first pixel coordinate and the second pixel coordinate of the identification point. Thus, a translation error function may be determined based on the Euclidean distance between the first pixel coordinate and the second pixel coordinate, and the translation error value may be further minimized using the translation error function.

[0118] S260 . Optimize and update the transformation matrix from the terminal coordinate system to the camera coordinate system according to the translation error value using a quasi-Newton method until the translation error value is less than a first threshold.

[0119] In this embodiment, the quasi-Newton method is an iterative algorithm for solving unconstrained optimization problems and can be used to minimize the error function. The first threshold is used to minimize the translation error value and can be preset.

[0120] In the quasi-Newton optimization problem, the goal is to find a parameter vector x that minimizes the objective function f(x) (e.g., mean squared error, cross-entropy error, or other error functions). The quasi-Newton method iteratively updates the value of the parameter vector x, gradually approaching the point where f(x) is minimized. The principle is to use the first-order derivative (gradient) of the objective function to construct an approximate inverse matrix of the Hessian matrix (second-order derivative matrix). This approximate matrix then determines the search direction and updates the parameter vector.

[0121] In this embodiment, the translation error function can be used as the objective function. Through the quasi-Newton method, the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the translation error value determined by the translation error function until the translation error value is less than the first threshold value, so that the Euclidean distance between the first pixel coordinate and the second pixel coordinate of the identification point is minimized, thereby completing the optimization process of the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system.

[0122] It should be noted that, in addition to the optimization process of the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system, this embodiment also includes the optimization process of the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system. Steps S270-S290 are the optimization process of the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system.

[0123] S270: Determine a roll compensation angle, a pitch compensation angle, a yaw compensation angle, a roll angle error value, a pitch angle error value, and a yaw angle error value of the conversion matrix.

[0124] S280. Using the quasi-Newton method, alternately optimize the roll compensation angle according to the roll angle error value, optimize the pitch compensation angle according to the pitch angle error value, and optimize the yaw compensation angle according to the yaw angle error value, until the roll angle error value, the pitch angle error value, and the yaw angle error value are all less than a second threshold.

[0125] S290: Update the transformation matrix from the terminal coordinate system to the camera coordinate system according to the optimized roll compensation angle, pitch compensation angle, and yaw compensation angle.

[0126] In this embodiment, the roll compensation angle is the Y value of the transformation matrix from the terminal coordinate system to the camera coordinate system in the terminal coordinate system. g The angle that the axis needs to be compensated. The pitch compensation angle is the X-axis of the transformation matrix from the end coordinate system to the camera coordinate system in the end coordinate system. g The angle that the axis needs to be compensated for. The yaw compensation angle is the Z of the transformation matrix from the end coordinate system to the camera coordinate system in the end coordinate system. gThe angles of the axes that need to be compensated. By adjusting these three compensation angles, the camera coordinate system can be aligned with the end coordinate system to ensure that the target point of the visual positioning can be accurately mapped to the motion space of the calibration object.

[0127] The second threshold is a preset error value to minimize the roll angle error value, the pitch angle error value, and the yaw angle error value, and can be flexibly set.

[0128] It should be noted that the optimization process of the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system is divided into three sub-processes, corresponding to the optimization of the roll degree of freedom, the pitch degree of freedom, and the yaw degree of freedom, respectively.

[0129] In this embodiment, the optimization process of the rolling degree of freedom may be to identify the grid points of the calibration plate and determine the coordinates of each grid point of the calibration plate in the pixel coordinate system.

[0130] Furthermore, the direction of the calibration plate rows and columns in the pixel coordinate system is determined. Specifically, assuming that the calibration plate grid has r rows and c columns, the two-dimensional coordinates of the recognition result of the kth row and lth column grid point in the pixel coordinate system are p k,l , make a difference on the grid points of each row to get the overall row direction vector n row and the row direction vector angle for:

[0131]

[0132] Similarly, the column direction vector n col and the column direction vector angle for:

[0133]

[0134] Furthermore, the roll angle error function is constructed as:

[0135]

[0136] Furthermore, the rolling degree of freedom is optimized, and the transformation matrix from the end coordinate system to the camera coordinate system is set in the Y of the end coordinate system. g The roll compensation angle that the axis needs to compensate is Δθ roll , then the rotation matrix after compensation is:

[0137]

[0138] Furthermore, using the quasi-Newton method, Δθ roll To optimize the variables, minimize the roll angle error value e roll Or until the roll angle error value e rollis less than the second threshold, and the roll degree of freedom part of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the optimized roll compensation angle, that is, the transformation matrix from the terminal coordinate system to the camera coordinate system is updated.

[0139] In this embodiment, the optimization process of the pitch degree of freedom is similar to the optimization process of the roll degree of freedom, wherein the pitch angle error function is:

[0140]

[0141] Furthermore, the pitch degree of freedom is optimized, and the transformation matrix from the end coordinate system to the camera coordinate system is set in the Y of the end coordinate system. g The pitch compensation angle that the axis needs to compensate is Δθ pitch , then the rotation matrix after compensation is:

[0142]

[0143] Furthermore, using the quasi-Newton method, Δθ pitch Optimize the optimization variables to minimize the pitch angle error value e pitch Or until the pitch angle error value e pitch is less than a second threshold, and the pitch degree of freedom part of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the optimized pitch compensation angle, that is, the transformation matrix from the terminal coordinate system to the camera coordinate system is updated.

[0144] In this embodiment, the optimization process of the yaw degree of freedom can be to place the upper left corner grid point of the calibration plate at the end coordinate system Y g On the vertical projection of the negative semi-axis on the ground, determine the coordinates of the upper left corner grid point (identification point) of the calibration plate in the pixel coordinate system [x pp y pp ] T , let the transformation matrix from the end coordinate system to the camera coordinate system be in the Z of the end coordinate system g The yaw compensation angle of the axis is Δθ yaw , then the rotation matrix after compensation is:

[0145]

[0146] Among them, Rodrigues is the Rodrigues transformation from quaternion to rotation matrix. Let the coordinates of the recognition point after reverse calculation from the coordinates in the pixel coordinate system to the world coordinate system be [x pb y pb z pb ] T , calculated as follows:

[0147]

[0148] The yaw angle error function is:

[0149] e yaw =[arctan2(y bp ,x bp )] 2 ;

[0150] Among them, e yaw is the yaw angle error value, arctan2(x,y) represents the argument of the point (x,y).

[0151] Furthermore, using the quasi-Newton method, θ yaw Optimize the variables to minimize the yaw angle error value e yaw Or until the yaw angle error value e yaw is less than a second threshold, and the yaw degree of freedom part of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the optimized yaw compensation angle, that is, the transformation matrix from the terminal coordinate system to the camera coordinate system is updated.

[0152] It should be noted that the optimization processes of the above three degrees of freedom are coupled and need to be performed in turn until the roll angle error value, the pitch angle error value, and the yaw angle error value are all less than the second threshold value, and then the optimization process of the three degrees of freedom can be terminated.

[0153] This embodiment optimizes the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system through the roll compensation angle, the pitch compensation angle, and the yaw compensation angle, thereby reducing the coordinate transformation error caused by installation error or mechanical deformation.

[0154] It should be noted that, in addition to the optimization process of the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system, and the optimization process of the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system, this embodiment also includes the optimization process of the distance between the origin of the world coordinate system and the origin of the terminal coordinate system in the transformation matrix from the terminal coordinate system to the camera coordinate system. Steps S2100-S2140 are the optimization process of the distance between the origin of the world coordinate system and the origin of the terminal coordinate system in the transformation matrix from the terminal coordinate system to the camera coordinate system.

[0155] S2100: Determine pixel coordinates of at least five identification points on the calibration object in a pixel coordinate system.

[0156] S2110 , determining the world coordinates of the recognition point according to the pixel coordinates of the recognition point and the distance between the origin of the world coordinate system and the origin of the terminal coordinate system.

[0157] S2120 . Determine an error function of a distance between the origin of the world coordinate system and the origin of the terminal coordinate system according to the pixel coordinates of the identification point and the world coordinates of the identification point.

[0158] S2130. Determine an origin error value based on an error function of the distance between the origin of the world coordinate system and the origin of the end coordinate system.

[0159] S2140 . Optimize and update the distance between the origin of the world coordinate system and the origin of the end coordinate system according to the origin error value using the quasi-Newton method until the origin error value is less than a third threshold.

[0160] In this embodiment, the pixel coordinates of the identification point are the coordinates of the identification point in the pixel coordinate system, and the world coordinates of the identification point are the coordinates of the pixel coordinates of the identification point in the world coordinate system after being transformed by the distance between the origin of the world coordinate system and the origin of the terminal coordinate system.

[0161] The calibration object can be a checkerboard calibration plate, and the identification point can be a grid point on the calibration object. The grid point coordinates (identification point pixel coordinates) in the kth row and lth column on the calibration plate are: Suppose its homogenized coordinates expanded to four dimensions are

[0162] Assume the world coordinates of the identification point are The homogenized coordinates expanded to four dimensions are Then, based on the distance between the origin of the world coordinate system and the origin of the end coordinate system, the pixel coordinates of all identification points can be converted into the world coordinates of the identification points, which can be expressed as:

[0163]

[0164] Among them, Z b2g The distance between the origin of the world coordinate system and the origin of the end coordinate system.

[0165] Furthermore, based on the pixel coordinates of the recognition point and the world coordinates of the recognition point, the error function of the distance between the origin of the world coordinate system and the origin of the end coordinate system is constructed as follows:

[0166]

[0167] Among them, e z is the origin error value, and norm{v} calculates the second norm of v.

[0168] Furthermore, using the quasi-Newton method, according to the origin error value e z , the distance Z between the origin of the world coordinate system and the origin of the end coordinate system b2g Optimize and update until the origin error value e z is less than the third threshold.

[0169] The distance Z between the origin of the world coordinate system and the origin of the end coordinate system b2g Optimize the optimization variables to minimize the origin error value ez Or until the origin error value e z Less than the third threshold. Because this parameter affects the accuracy of other parameter estimates, the distance between the world coordinate system origin and the terminal coordinate system origin in the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated based on the optimized distance between the world coordinate system origin and the terminal coordinate system origin. This is to update the transformation matrix from the terminal coordinate system to the camera coordinate system.

[0170] See also Figure 5 The figure shows a flowchart for optimizing the transformation matrix from the terminal coordinate system to the camera coordinate system. Through iteration, the translation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized, the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system is optimized, and the distance between the origin of the world coordinate system and the origin of the terminal coordinate system is optimized until the error values of these three processes are all less than a preset threshold. The optimization process of the rotation matrix of the transformation matrix from the terminal coordinate system to the camera coordinate system includes iteratively optimizing the roll degree of freedom, the pitch degree of freedom, and the yaw degree of freedom until the roll angle error value, the pitch angle error value, and the yaw angle error value are all less than a preset threshold. Furthermore, the optimization process of the transformation matrix from the terminal coordinate system to the camera coordinate system is completed through the above iterations.

[0171] This embodiment completes the detailed calibration process of the transformation matrix from the end coordinate system to the camera coordinate system in the hand-eye calibration task by optimizing the translation matrix parameters, the rotation matrix parameters, and the additional distance between the origin of the world coordinate system and the origin of the end coordinate system, thereby improving the calibration accuracy and robustness during the application of the calibration object.

[0172] It should be noted that this method eliminates the end-user position measurement required for traditional hand-eye calibration, eliminating the need for additional high-precision measurement equipment and saving calibration costs. Secondly, the accuracy of the transformation matrix from the end-user coordinate system to the camera coordinate system is gradually improved from coarse to fine calibration, reducing errors and improving calibration accuracy.

[0173] The technical solution of the embodiment of the present invention uses a camera to identify the calibration object, and determines the conversion matrix from the end coordinate system to the camera coordinate system based on the first coordinate and the second coordinate collected in different coordinate systems, the conversion matrix from the world coordinate system to the end coordinate system of the calibration object, and the conversion matrix from the pixel coordinate system to the camera coordinate system. The present invention does not require an inertial unit or a robotic arm joint encoder to measure the posture of the end mechanism of the calibration object, thereby realizing hand-eye calibration of a low-cost open-loop control system without posture measurement elements. Furthermore, the present invention optimizes the translation matrix and the rotation matrix of the conversion matrix from the end coordinate system to the camera coordinate system, as well as the distance between the origin of the world coordinate system and the origin of the end coordinate system, thereby improving the accuracy of the conversion matrix from the end coordinate system to the camera coordinate system, reducing the conversion error of the target point coordinates in the hand-eye calibration task, and improving the calibration accuracy.

[0174] Example 3

[0175] Figure 6 This is a schematic diagram of the structure of a hand-eye calibration device provided in the third embodiment of the present invention. This embodiment is applicable to hand-eye calibration situations. The hand-eye calibration device can be implemented in the form of hardware and / or software. The hand-eye calibration device can be configured in any electronic device with network communication and computing capabilities. Figure 6 As shown, the device includes:

[0176] An auxiliary matrix determination module 310 is configured to determine a first conversion matrix and a second conversion matrix, wherein the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system;

[0177] A sampling coordinate determination module 320 is configured to control the calibration object to rotate to at least two sampling positions and determine the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system at each sampling position;

[0178] The transformation matrix determination module 330 is used to determine the transformation matrix from the calibration object end coordinate system to the camera coordinate system according to the first transformation matrix, the second transformation matrix, the first coordinate and the second coordinate.

[0179] As an optional but non-limiting implementation, the calibration object rotates around an endpoint of a first end, and the camera is disposed at a second end of the calibration object;

[0180] The terminal coordinate system is constructed with the endpoint of the first end of the calibration object being rotated as the origin and the plane on which the calibration object is rotated being the XOY plane;

[0181] The camera coordinate system is constructed with the equivalent optical center of the camera model as the origin and the camera normalized imaging plane as the XOY plane;

[0182] The world coordinate system is constructed with any point on the ground as the origin and the plane where the ground lies as the XOY plane;

[0183] The pixel coordinate system is a two-dimensional coordinate system composed of pixel points of the calibration object on the camera imaging plane.

[0184] As an optional but non-limiting implementation, determining a transformation matrix from the calibration object end coordinate system to the camera coordinate system based on the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate includes:

[0185] Determine a conversion matrix from the calibration object end coordinate system to the camera coordinate system based on the coordinate system transformation relationship according to the first conversion matrix, the second conversion matrix, the first coordinate, and the second coordinate;

[0186] The coordinate system transformation relationship is expressed by the following formula:

[0187]

[0188] Where F is the second transformation matrix, is the first coordinate of the i-th point on the calibration object in the pixel coordinate system when the calibration object is rotated to the sampling position, is the second coordinate of the i-th point on the calibration object in the world coordinate system when the calibration object is rotated to the sampling position, T b2g is the first transformation matrix, T g2c is the transformation matrix from the end coordinate system to the camera coordinate system.

[0189] As an optional but non-limiting implementation, determining a transformation matrix from the calibration object end coordinate system to the camera coordinate system based on the coordinate system transformation relationship according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate includes:

[0190] Determine the rotation angle corresponding to the sampling position according to the first coordinate and the second coordinate corresponding to the same sampling position, the first transformation matrix and the second transformation matrix, and the coordinate system transformation relationship;

[0191] According to the corresponding rotation angle at each sampling position, the transformation matrix from the calibration object end coordinate system to the camera coordinate system is solved.

[0192] As an optional but non-limiting implementation method, after determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, it includes:

[0193] Determine the first pixel coordinates and the second pixel coordinates of the identification point in the pixel coordinate system; the first pixel coordinates are the coordinates of the identification point in the pixel coordinate system, and the second pixel coordinates are the coordinates of the identification point in the world coordinate system, which are the coordinates transformed according to the transformation matrix from the terminal coordinate system to the camera coordinate system; the identification point is any point on the calibration object;

[0194] Determine a translation error value of a transformation matrix from a terminal coordinate system to a camera coordinate system according to the first pixel coordinate and the second pixel coordinate;

[0195] The transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the translation error value by using the quasi-Newton method until the translation error value is less than a first threshold.

[0196] As an optional but non-limiting implementation method, after determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, it includes:

[0197] Determining a roll compensation angle, a pitch compensation angle, a yaw compensation angle, a roll angle error value, a pitch angle error value, and a yaw angle error value of the conversion matrix;

[0198] By using a quasi-Newton method, the roll compensation angle is optimized according to the roll angle error value, the pitch compensation angle is optimized according to the pitch angle error value, and the yaw compensation angle is optimized according to the yaw angle error value, until the roll angle error value, the pitch angle error value, and the yaw angle error value are all less than a second threshold value;

[0199] According to the optimized roll compensation angle, pitch compensation angle, and yaw compensation angle, the transformation matrix from the terminal coordinate system to the camera coordinate system is updated.

[0200] As an optional but non-limiting implementation method, after determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, it includes:

[0201] Determine pixel coordinates of at least five identification points on the calibration object in a pixel coordinate system;

[0202] Determine the world coordinates of the identification point based on the pixel coordinates of the identification point and the distance between the origin of the world coordinate system and the origin of the terminal coordinate system;

[0203] Determine an error function of the distance between the origin of the world coordinate system and the origin of the terminal coordinate system based on the pixel coordinates of the identification point and the world coordinates of the identification point;

[0204] Determine the origin error value based on the error function of the distance between the origin of the world coordinate system and the origin of the end coordinate system;

[0205] The distance between the origin of the world coordinate system and the origin of the end coordinate system is optimized and updated according to the origin error value through the quasi-Newton method until the origin error value is less than a third threshold.

[0206] The technical solution of the embodiment of the present invention uses a camera to identify the calibration object, and determines the conversion matrix from the end coordinate system to the camera coordinate system based on the first coordinate and the second coordinate collected in different coordinate systems, the conversion matrix from the world coordinate system to the end coordinate system of the calibration object, and the conversion matrix from the pixel coordinate system to the camera coordinate system. The present invention does not require an inertial unit or a robotic arm joint encoder to measure the posture of the end mechanism of the calibration object, and realizes hand-eye calibration of a low-cost open-loop control system without posture measurement elements.

[0207] The hand-eye calibration device provided in the embodiment of the present invention can execute the method for determining the transformation matrix provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0208] Example 4

[0209] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0210] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0211] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0212] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for determining the transformation matrix.

[0213] In some embodiments, the method for determining the conversion matrix can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for determining the conversion matrix described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the method for determining the conversion matrix in any other appropriate manner (e.g., by means of firmware).

[0214] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0215] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0216] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0217] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0219] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0220] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0221] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0222] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for determining a conversion matrix, characterized in that: include: Determine a first conversion matrix and a second conversion matrix, wherein the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system; Controlling the calibration object to rotate to at least two sampling positions, and determining the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system at each sampling position; A transformation matrix from the calibration object end coordinate system to the camera coordinate system is determined according to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate.

2. The method according to claim 1, characterized in that The calibration object rotates around an endpoint of a first end, and the camera is arranged at a second end of the calibration object; The terminal coordinate system is constructed with the endpoint of the first end of the calibration object being rotated as the origin and the plane on which the calibration object is rotated being the XOY plane; The camera coordinate system is constructed with the equivalent optical center of the camera model as the origin and the camera normalized imaging as the XOY plane; The world coordinate system is constructed with any point on the ground as the origin and the plane where the ground lies as the XOY plane; The pixel coordinate system is a two-dimensional coordinate system composed of pixel points of the calibration object on the camera imaging plane.

3. The method according to claim 1, characterized in that Determining a conversion matrix from the calibration object end coordinate system to the camera coordinate system according to the first conversion matrix, the second conversion matrix, the first coordinate, and the second coordinate includes: Determine a conversion matrix from the calibration object end coordinate system to the camera coordinate system based on the coordinate system transformation relationship according to the first conversion matrix, the second conversion matrix, the first coordinate, and the second coordinate; The coordinate system transformation relationship is expressed by the following formula: Where F is the second transformation matrix, is the first coordinate of the i-th point on the calibration object in the pixel coordinate system when the calibration object is rotated to the sampling position, is the second coordinate of the i-th point on the calibration object in the world coordinate system when the calibration object is rotated to the sampling position, T b2g is the first transformation matrix, T g2c is the transformation matrix from the end coordinate system to the camera coordinate system.

4. The method according to claim 3, characterized in that According to the first transformation matrix, the second transformation matrix, the first coordinate, and the second coordinate, based on the coordinate system transformation relationship, a transformation matrix from the calibration object end coordinate system to the camera coordinate system is determined, including: Determine the rotation angle corresponding to the sampling position according to the first coordinate and the second coordinate corresponding to the same sampling position, the first transformation matrix and the second transformation matrix, and the coordinate system transformation relationship; According to the corresponding rotation angle at each sampling position, the transformation matrix from the calibration object end coordinate system to the camera coordinate system is solved.

5. The method according to claim 1, wherein After determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, including: Determine the first pixel coordinates and the second pixel coordinates of the identification point in the pixel coordinate system; the first pixel coordinates are the coordinates of the identification point in the pixel coordinate system, and the second pixel coordinates are the coordinates of the identification point in the world coordinate system, which are the coordinates transformed according to the transformation matrix from the terminal coordinate system to the camera coordinate system; the identification point is any point on the calibration object; Determine a translation error value of a transformation matrix from a terminal coordinate system to a camera coordinate system according to the first pixel coordinate and the second pixel coordinate; The transformation matrix from the terminal coordinate system to the camera coordinate system is optimized and updated according to the translation error value by using the quasi-Newton method until the translation error value is less than a first threshold.

6. The method according to claim 1, characterized in that After determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, including: Determining a roll compensation angle, a pitch compensation angle, a yaw compensation angle, a roll angle error value, a pitch angle error value, and a yaw angle error value of the conversion matrix; By using a quasi-Newton method, the roll compensation angle is optimized according to the roll angle error value, the pitch compensation angle is optimized according to the pitch angle error value, and the yaw compensation angle is optimized according to the yaw angle error value, until the roll angle error value, the pitch angle error value, and the yaw angle error value are all less than a second threshold value; According to the optimized roll compensation angle, pitch compensation angle, and yaw compensation angle, the transformation matrix from the terminal coordinate system to the camera coordinate system is updated.

7. The method according to claim 1, characterized in that After determining the transformation matrix from the calibration object end coordinate system to the camera coordinate system, including: Determine pixel coordinates of at least five identification points on the calibration object in a pixel coordinate system; Determine the world coordinates of the identification point based on the pixel coordinates of the identification point and the distance between the origin of the world coordinate system and the origin of the terminal coordinate system; Determine an error function of the distance between the origin of the world coordinate system and the origin of the terminal coordinate system based on the pixel coordinates of the identification point and the world coordinates of the identification point; Determine the origin error value based on the error function of the distance between the origin of the world coordinate system and the origin of the end coordinate system; The distance between the origin of the world coordinate system and the origin of the end coordinate system is optimized and updated according to the origin error value through the quasi-Newton method until the origin error value is less than a third threshold.

8. A device for determining a conversion matrix, characterized in that: include: An auxiliary matrix determination module is used to determine a first conversion matrix and a second conversion matrix, wherein the first conversion matrix is a conversion matrix from a world coordinate system to an end coordinate system of the calibration object, and the second conversion matrix is a conversion matrix from a pixel coordinate system to a camera coordinate system; A sampling coordinate determination module is used to control the calibration object to rotate to at least two sampling positions, and determine the first coordinates of at least nine feature points on the calibration object in the pixel coordinate system and the second coordinates in the world coordinate system at each sampling position; The conversion matrix determination module is used to determine the conversion matrix from the calibration object end coordinate system to the camera coordinate system based on the first conversion matrix, the second conversion matrix, the first coordinate and the second coordinate.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for determining the conversion matrix according to any one of claims 1 to 7 is implemented.

10. A storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, the computer executable instructions are used to perform the method for determining the conversion matrix according to any one of claims 1 to 7.

Images