Vision-based kinematic calibration method for industrial robots in large workspaces

Abstract

A vision-based kinematic calibration method for industrial robots in large workspaces solves the problem that existing vision-based robot kinematic calibration methods can only improve the absolute positioning accuracy of robots within a limited space due to field of view limitations, and belongs to the field of robot kinematic calibration technology. The present invention includes: laying out ArUco markers in the workspace according to the configuration of the industrial robot; using a monocular vision system to capture ArUco marker images according to a set shooting path to generate a fully covered ArUco map; using a monocular vision system to capture ArUco marker images in different robot postures, constructing an absolute position model of the robot end, and obtaining a measured posture; calculating the error matrix between the end nominal posture and the measured posture based on the robot joint angles and nominal values ​​of the kinematic parameters; establishing a posture error model, and obtaining the robot kinematic parameter error based on the error matrix.

Description

Technical Field

[0001] The invention relates to a vision-based kinematic calibration method for an industrial robot in a large-scale workspace, and belongs to the technical field of robot kinematic calibration. Background Art

[0002] With the demand for high-speed, high-precision, and large-load industrial robots in advanced manufacturing, the absolute positioning accuracy of robots is becoming increasingly demanding. Research on improving the absolute positioning accuracy of robots through calibration technology has become a hot topic. Vision-based robot kinematic calibration technology is favored due to its low cost and ease of operation. Currently, robot kinematic calibration mainly relies on specific markers or feature points (such as checkerboards and precision spheres) as calibration references. The paper "Kinematic identification of industrial robot using end-effector mounted monocular camera bypassing measurement of 3-D pose[J]. IEEE / ASME Transactions on Mechatronics 27.1(2021):383-394" introduces a kinematic identification method for industrial robots. This method uses a monocular camera mounted on the end effector, eliminating the need for 3D pose measurement and directly using a 2D image of a checkerboard calibration plate, simplifying the process to a single-stage estimation. However, due to the camera's field of view, the robot calibration space is limited. In the paper "Anovel vision-based calibration framework for industrial robotic manipulators[J].Robotics and Computer-Integrated Manufacturing 73(2022):102248.", a novel vision calibration framework is proposed, which uses a single camera fixed on the outside and ArUco markers at the end of the robot to calibrate the industrial robotic arm. However, this method cannot take into account both the camera field of view and the measurement distance, and its flexibility in application in large-scale workspaces is limited.

[0003] In summary, the current vision-based robot kinematic calibration methods still have a common problem: the camera field of view limits the robot's range of movement during the calibration process, causing the improvement of the robot's absolute positioning accuracy to be mainly concentrated in the restricted measurement area. Therefore, when the robot is used in a large workspace, its absolute positioning accuracy is difficult to meet the application requirements of high precision in the entire space. Summary of the Invention

[0004] In order to solve the problem that the existing vision-based robot kinematic calibration method can only improve the absolute positioning accuracy of the robot in a limited space due to the limitation of the field of view, the present invention provides a vision-based industrial robot kinematic calibration method in a large workspace.

[0005] The present invention provides a vision-based kinematic calibration method for an industrial robot in a large-scale workspace, comprising:

[0006] ArUco markers are placed in the workspace according to the industrial robot configuration.

[0007] Use a monocular vision system to capture ArUco marker images according to the set shooting path, and use image stitching technology to generate a fully covered ArUco map from the ArUco marker images;

[0008] The robot uses a monocular vision system to capture ArUco marker images in different postures, identifies the position of the robot end according to the fully covered ArUco map, and constructs an absolute position model of the robot end;

[0009] The nominal end pose is calculated based on the robot joint angles and nominal kinematic parameters, and the end measured pose is obtained using the robot end absolute position model, and the error matrix between the end nominal pose and the measured pose is obtained;

[0010] A posture error model is established, and the robot kinematic parameter error is obtained based on the error matrix.

[0011] Preferably, according to the configuration of the industrial robot, the method for laying out ArUco markers in the workspace includes:

[0012] Establish the objective function F(O) of the layout plan:

[0013] F(O)=α·V(T)+β·U(D)+γ·P(R)-δ·C(S)

[0014] Among them, O represents the layout of the mark,

[0015] V(T) represents the visibility parameter of the ArUco marker, and α represents the weight of V(T);

[0016] U(D) represents the uniformity parameter of ArUco marker placement, and β represents the weight of U(D);

[0017] P(R) represents the average positioning accuracy of the robot at all positions within its motion range, and γ represents the weight of P(R);

[0018] C(S) represents the layout cost, and δ represents the weight of C(S);

[0019] A genetic algorithm is used to optimize the layout scheme O of ArUco markers to maximize the objective function F(O). During the optimization process, the position, number and direction of the markers are continuously updated iteratively until a layout scheme that maximizes F(O) is found.

[0020] As a preference,

[0021] Where D represents the label density set of each inspection point in the workspace, σ 2 (D) is the variance of D.

[0022] As a preference,

[0023] Among them, v i The visibility of the i-th ArUco marker depends on the angle and distance between the ArUco marker and the camera. i Represents the visible area of ​​the i-th ArUco marker in the camera field of view, A total represents the total area of ​​the camera's field of view, and n represents the total number of ArUco markers.

[0024] As a preference,

[0025] Where m is the total number of positioning operations, d j is the positioning error in the jth operation, d max is the maximum allowed error distance.

[0026] Preferably, the method of capturing ArUco tagged images using a monocular vision system according to a set shooting path and generating a fully covered ArUco map from the ArUco tagged images using image stitching technology includes:

[0027] According to the set closed-loop shooting path, capture ArUco mark images in sequence to ensure that the same ArUco mark exists in adjacent ArUco mark images;

[0028] Determine the position of each ArUco marker and use the LM algorithm to minimize the reprojection error of the target's ArUco marker corner points to obtain the initial ArUco map;

[0029] The objective function of the reprojection error is

[0030]

[0031] and

[0032]

[0033] Among them, Ψ(δ,γ t ,γi c j ) represents the projection of the corner point from the three-dimensional space coordinates to the pixel coordinates, δ represents the camera intrinsic parameter matrix, γ t represents the external parameters of the camera, γ i Represents the transformation matrix from the marker coordinate system to the world coordinate system, c j represents the three-dimensional coordinates of the marked corner points, represents the pixel coordinates of the marked corner points, K is the total number of images in the closed-loop shooting path, and x k is the pose of the kth image, z k is the pose transformation from the kth image to the k+1th image, is the operator between places.

[0034] As a preferred method, the absolute position model of the robot end is constructed as follows:

[0035] Based on the fully covered ArUco map and the ArUco marker images captured by the robot using a monocular vision system at different poses, the coordinate transformation relationship between the ArUco image coordinate system and the camera coordinate system is established.

[0036] The hand-eye calibration model is used to calculate the relative pose relationship between the robot end coordinate system and the camera coordinate system, thereby determining the absolute position model of the robot end.

[0037] Preferably, a posture error model is established, and the error matrix is ​​processed by Python programming to obtain the robot kinematic parameter error.

[0038] The beneficial effects of the present invention are:

[0039] By adopting a monocular vision system and ArUco marker map, the present invention not only significantly reduces the application cost of high-precision calibration technology, but also simplifies the calibration process, making the system installation and operation simpler and faster. This feature makes the present invention particularly suitable for production environments that require frequent calibration or adjustment, saving users a lot of time and economic costs. The present invention effectively solves the problem that traditional visual calibration methods are limited by the camera's field of view, and can achieve higher-precision robot positioning in a larger workspace. This improvement not only improves production efficiency, but also improves the control quality, and is particularly suitable for application scenarios with extremely high requirements for positioning accuracy. The calibration method of the present invention is applicable to various industrial robot models and brands and has good versatility. The flexible design of the ArUco marker map allows the calibration system to be adjusted according to actual needs and has good scalability. At the same time, the enhanced information and redundancy provided by the method ensure that a high calibration accuracy can be maintained even when some markers are blocked or damaged, thereby enhancing the robustness of the system in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a specific implementation of the present invention;

[0041] Figure 2 Schematic diagram of the robot kinematic model of the present invention;

[0042] Figure 3 A schematic diagram of establishing an ArUco map of the present invention;

[0043] Figure 4 Schematic diagram of the robot kinematic calibration platform of the present invention.

[0044] Figure 5 This is a schematic diagram of regional verification of the robot positioning accuracy of the present invention.

[0045] Figure 6 This is a comparison chart of the positioning accuracy of the robot before and after calibration in a specific example of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0049] The present embodiment provides a vision-based kinematic calibration method for an industrial robot in a large workspace, comprising the following steps:

[0050] Step 1: ArUco markers are placed in the workspace according to the industrial robot configuration and the layout is optimized.

[0051] Step 2: Use a monocular vision system to capture ArUco marker images according to the set shooting path, and use image stitching technology to generate a fully covered ArUco map from the ArUco marker images;

[0052] Step 3: The robot uses a monocular vision system to capture ArUco marker images in different postures, identifies the end position of the robot based on the fully covered ArUco map, and constructs an absolute position model of the end position of the robot;

[0053] Step 4: Calculate the nominal end pose based on the robot joint angles and nominal kinematic parameter values, and use the robot end absolute position model to obtain the end measured pose, and obtain the error matrix between the nominal end pose and the measured pose;

[0054] Step 5: Establish a posture error model and obtain the robot kinematic parameter error based on the error matrix.

[0055] This implementation, by employing a monocular vision system and ArUco-marked maps, not only significantly reduces the application cost of high-precision calibration technology but also simplifies the calibration process, making system installation and operation much simpler and faster. This makes the invention particularly suitable for production environments that require frequent calibration or adjustments, saving users significant time and money.

[0056] After this implementation, the layout of on-site equipment (industrial robots, workbenches, processing equipment, etc.) is determined, and the appropriate industrial camera is selected based on the working distance and measurement accuracy. The installation position and angle of the camera and the robot end are carefully designed to optimize the quality of image capture;

[0057] Place ArUco markers with unique IDs at appropriate locations within the robot's workspace and use image stitching technology to generate an ArUco map (calibration benchmark) that fully covers the workspace. During the capture process, at least one complete ArUco marker is completely within the camera's field of view.

[0058] The robot poses are selected to cover the robot's entire working range, taking into account the robot's kinematic characteristics and the vision system's capture capabilities. The robot moves at predetermined pose points, collecting data on the six joint angles of the industrial robot at each pose. Simultaneously, an industrial camera mounted on the end of the robot captures images containing ArUco markers. The image data and joint angle data for each pose point are recorded and stored for subsequent data analysis and processing.

[0059] The layout of the ArUco markers in step 1 is as follows:

[0060] First, within the workspace of the industrial robot, the layout of the ArUco markers is optimized based on the specific configuration and operational requirements of the robot. In this embodiment, the layout of the markers follows the following principles:

[0061] (1) Marker placement uniformity: By rationally planning the position and spacing of markers, we ensure that the markers cover the entire workspace and maintain similar marker density at different locations. This process is achieved by optimizing the uniformity of the ArUco marker placement U(D), where

[0062]

[0063] D represents the set of label densities of each inspection point in the workspace, σ 2 (D) is the variance of D;

[0064] (2) ArUco marker visibility: When choosing the placement of the markers, consider the camera parameters and field of view to avoid placing ArUco markers in areas blocked by the robot or other objects. The total visibility of the ArUco markers, V(T), is calculated using the following formula to ensure that the camera can accurately detect and locate each marker.

[0065]

[0066] (3) Robot pose variation: To increase the robot’s pose range, ArUco markers are placed in a variety of positions, including non-planar positions. This not only increases the pose range but also improves the robustness and stability of the calibration, making it particularly suitable for complex workspaces and multi-pose operations.

[0067] Taking into account the visibility, uniformity and positioning accuracy of the markers, the optimization objective function is defined:

[0068] F(O)=α·V(T)+β·U(D)+γ·P(R)-δ·C(S)

[0069] Among them, F(O) is the optimization objective function, O represents the layout of the mark. α represents the weight of V(T), β represents the weight of U(D), γ represents the weight of P(R), and P(R) represents the average positioning accuracy of the robot at all positions within its range of motion. Calculate, where m is the total number of positioning operations, d j is the positioning error in the jth operation, d max is the maximum allowable error distance. C(S) represents the layout cost, which is calculated by the formula Calculate, where c k is the placement cost of the kth token, and δ represents the weight of C(S).

[0070] A genetic algorithm is used to optimize the ArUco marker placement scheme O to maximize the objective function F(O). During the optimization process, the position, number, and orientation of the markers are iteratively updated until a placement scheme that maximizes F(O) is found.

[0071] The creation of the ArUco map in step 2 is as follows:

[0072] (1) The robot captures ArUco markers in different poses to ensure that the same marker exists in adjacent images. In robot pose 1, when the robot detects markers 1 and 3 in the image at the same time, the pose relationship between the camera and marker 1 can be established. And the posture relationship between markers 3 In this way, we can derive the pose transformation matrix between marker 1 and marker 3 in pose 1 In pose 2, when markers 3 and 4 are detected simultaneously, the pose transformation matrix between them can be determined In addition, since there is a common marker 3, the pose transformation matrix between marker 4 and marker 1 can be calculated Similarly, by using the common markers in the images captured at each location, we can determine the relative position relationship of any two ArUco markers in space. By specifying one of the ArUco coordinate systems as the reference world coordinate system, we can determine the relative positions of all ArUco markers to the reference coordinate system.

[0073] (2) Due to insufficient lighting, rapid camera movement, low resolution, and poor focus, the pixel coordinates of the QR code corners become inaccurate, resulting in errors in the calculated position matrix. In short, observation noise causes simple projection relationships to be very inaccurate, and the cumulative error is large. Therefore, it is necessary to determine the precise location of each ArUco under the condition of inaccurate camera observation data. This is an optimization problem to minimize the reprojection error:

[0074]

[0075] Among them, Ψ(δ,γ t ,γ i c j ) represents the projection of the corner point from the three-dimensional space coordinates to the pixel coordinates, δ represents the camera intrinsic parameter matrix, γ t represents the external parameters of the camera, γ i Represents the transformation matrix from the marker coordinate system to the world coordinate system, c j represents the three-dimensional coordinates of the marked corner points, represents the pixel coordinates of the marked corner points, K is the total number of images in the closed-loop shooting path,

[0076] (3) At the same time, considering that the increase in the number of image stitching will lead to the accumulation of errors, in order to alleviate this problem, a closed-loop image acquisition route is set and the overall error is optimized, and the closed-loop constraint formula is introduced:

[0077]

[0078] where K is the total number of images in the closed loop, x k is the pose of the kth image, z k is the pose transformation from the kth image to the k+1th image, is the operator between places.

[0079] In step 3, the robot uses a monocular vision system to capture ArUco marker images in different postures.

[0080] The specific steps for building the absolute position model of the robot end in step 3 are:

[0081] (1) The measurement system consists of multiple ArUco markers in space, an industrial robot, and an industrial camera installed at the end of the industrial robot. The camera captures and estimates the pose of the ArUco marker through image recognition technology, and establishes the coordinate transformation relationship between the ArUco code image coordinate system and the camera coordinate system. Among them, the PnP algorithm is used to determine the 6-DOF pose of the camera relative to the calibration plate. In this algorithm, the 3D to 2D correspondence relationship of the four corner points indicated by the ArUco marker is described by the following formula:

[0082]

[0083] (2) Using the hand-eye calibration model, the relative position relationship between the KUKA kr500 robot end coordinate system and the Daheng industrial camera coordinate system is calculated, thereby determining the absolute position model of the robot end. The hand-eye matrix obtained by calibration in this step of this embodiment is:

[0084]

[0085] The error matrix of the measured pose is obtained in step 4, specifically:

[0086] (1) Establish a kinematic model based on the theoretical DH parameters of industrial robots and calculate the nominal position of the robot end in each posture;

[0087] (2) Using the absolute position model of the robot end to obtain the end measurement pose, the error matrix between the end nominal pose and the measured pose is obtained;

[0088] In step 5, a posture error model is established, and the error matrix is ​​processed by Python programming to obtain the robot kinematic parameter error.

[0089] To verify the effectiveness and superiority of the present invention, the ArUco map-based calibration method was compared with a traditional checkerboard-based calibration method. The industrial robot's operating space was further divided into nine regions, with 10 randomly set verification points in each region. By comparing the positioning accuracy of the two methods in different regions and analyzing the error distribution, the advantages of the present invention were further demonstrated.

[0090] The identified robot kinematic parameter errors are shown in the table:

[0091] Table 1 Kinematic parameter error results

[0092]

[0093]

[0094] The processed data was analyzed to evaluate the effectiveness of the proposed method in improving the robot's absolute positioning accuracy. Particular attention was paid to the distribution of errors in different regions and the comparison results with traditional methods, thereby fully verifying the effectiveness and application value of the proposed method.

[0095] In summary, the present invention successfully proposes and implements an efficient method for kinematic calibration of industrial robots in large-scale workspaces using ArUco maps and monocular vision systems. By innovatively combining the widespread deployment of ArUco markers, advanced image stitching technology, precise image processing and marker recognition, as well as advanced absolute position model construction and kinematic parameter calibration technology, the present invention significantly improves the positioning accuracy and operational efficiency of industrial robots in large-scale work areas. The system is not only easy to operate and cost-effective, but also has good adaptability and scalability, and can meet the needs of various high-precision manipulation tasks. Through the implementation of the present invention, it can provide strong technical support for various industrial applications, especially in the fields of aerospace, automobile manufacturing and precision machining, and has broad application prospects and significant economic and social value.

[0096] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A vision-based kinematic calibration method for industrial robots in large workspaces, characterized by: The method comprises: ArUco markers are placed in the workspace according to the industrial robot configuration. Use a monocular vision system to capture ArUco marker images according to the set shooting path, and use image stitching technology to generate a fully covered ArUco map from the ArUco marker images; The robot uses a monocular vision system to capture ArUco marker images in different postures, identifies the position of the robot end according to the fully covered ArUco map, and constructs an absolute position model of the robot end; The nominal end pose is calculated based on the robot joint angles and nominal kinematic parameters, and the end measured pose is obtained using the robot end absolute position model, and the error matrix between the end nominal pose and the measured pose is obtained; Establish a posture error model and obtain the robot kinematic parameter error based on the error matrix; The method of capturing ArUco tagged images using a monocular vision system according to a set shooting path and generating a fully covered ArUco map from the ArUco tagged images using image stitching technology includes the following: According to the set closed-loop shooting path, capture ArUco mark images in sequence to ensure that the same ArUco mark exists in adjacent ArUco mark images; Determine the position of each ArUco marker and use the LM algorithm to minimize the reprojection error of the target's ArUco marker corner points to obtain the initial ArUco map; The objective function of the reprojection error is and Among them, Ψ(δ,γ t ,γ i c j ) represents the projection of the corner point from the three-dimensional space coordinates to the pixel coordinates, δ represents the camera intrinsic parameter matrix, γ t represents the external parameters of the camera, γ i Represents the transformation matrix from the marker coordinate system to the world coordinate system, c j represents the three-dimensional coordinates of the marked corner points, represents the pixel coordinates of the marked corner points, K is the total number of images in the closed-loop shooting path, and x k is the pose of the kth image, z k is the pose transformation from the kth image to the k+1th image, and ⊕ is the operator between positions.

2. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 1 is characterized in that: Depending on the industrial robot configuration, methods for laying out ArUco markers within the workspace include: Establish the objective function F(O) of the layout plan: F(O)=α·V(T)+β·U(D)+γ·P(R)-δ·C(S) Among them, O represents the layout of the mark, V(T) represents the visibility parameter of the ArUco marker, and α represents the weight of V(T); U(D) represents the uniformity parameter of ArUco marker placement, and β represents the weight of U(D); P(R) represents the average positioning accuracy of the robot at all positions within its motion range, and γ represents the weight of P(R); C(S) represents the layout cost, and δ represents the weight of C(S); A genetic algorithm is used to optimize the layout scheme O of ArUco markers to maximize the objective function F(O). During the optimization process, the position, number and direction of the markers are continuously updated iteratively until a layout scheme that maximizes F(O) is found.

3. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 2, characterized in that: Where D represents the label density set of each inspection point in the workspace, σ 2 (D) is the variance of D.

4. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 2, characterized in that: Among them, v i The visibility of the i-th ArUco marker depends on the angle and distance between the ArUco marker and the camera. i Represents the visible area of ​​the i-th ArUco marker in the camera field of view, A total represents the total area of ​​the camera's field of view, and n represents the total number of ArUco markers.

5. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 2, characterized in that: Where m is the total number of positioning operations, d j is the positioning error in the jth operation, d max is the maximum allowed error distance.

6. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 1, characterized in that: Method for constructing the absolute position model of the robot end: Based on the fully covered ArUco map and the ArUco marker images captured by the robot using a monocular vision system at different poses, the coordinate transformation relationship between the ArUco image coordinate system and the camera coordinate system is established. The hand-eye calibration model is used to calculate the relative pose relationship between the robot end coordinate system and the camera coordinate system, thereby determining the absolute position model of the robot end.

7. The vision-based kinematic calibration method for industrial robots in large-scale workspaces according to claim 1, characterized in that: A posture error model is established, and the error matrix is ​​processed by Python programming to obtain the robot kinematic parameter error.

8. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by a processor, the vision-based kinematic calibration method for an industrial robot in a large-size workspace is implemented as claimed in any one of claims 1 to 7.

9. A vision-based kinematic calibration device for an industrial robot in a large workspace, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the vision-based kinematic calibration method for an industrial robot in a large-size workspace as described in any one of claims 1 to 7.

Images