Tool changing robot large field of view non-overlapping stereo camera external parameter two-step calibration method and system based on pose transformation

Abstract

The method and system for two-step calibration of large field of view non-overlapping stereo camera external parameters of tool changing robot based on pose transformation relates to the field of automatic tool changing of tunnel boring machine, and is mainly used for solving the problems of high cost and high difficulty in operation caused by in-situ calibration of stereo camera external parameters at the measuring position at the back of the cutterhead. The method sequentially acquires a first calibration image group and a second calibration image group; labels the target points in all calibration images through a global mapper; in the first calibration stage, the non-linear optimization of stereo camera external parameters is performed by using non-overlapping field of view simulation of symmetrical transfer error; in the second calibration stage, the pose transformation matrix of the left and right cameras between the two calibration stages is solved, and the final stereo camera external parameters are obtained. The method has the advantages of avoiding the problems of high cost and high difficulty in operation caused by in-situ calibration of external parameters at the measuring position at the back of the cutterhead.

Description

Technical Field

[0001] This invention relates to the field of automatic tool changer technology for tunnel boring machines, and in particular to a two-step calibration method and system for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool changer robot based on pose transformation. Background Technology

[0002] Cutting rollers are the primary rock-breaking tools in tunnel boring machines (TBMs), crushing rock through rolling. During tunnel construction, cutting roller wear and failure are frequent problems. The cutting roller inspection and replacement area is located on the back of the cutterhead, the forefront of the TBM, and is typically a collaborative task performed by multiple people. Throughout the tunnel construction cycle, frequent inspections and numerous cutting roller replacements are necessary. However, the harsh on-site construction environment makes manual operation inconvenient, inefficient, and poses significant safety hazards. Against this backdrop, cutting roller replacement robots have emerged as an effective solution to replace manual labor. Among these, cutting roller wear measurement based on vision measurement technology is a prerequisite for automated robot cutting roller replacement, while stereo camera extrinsic parameter calibration based on the on-site construction environment is the primary step in the vision measurement task.

[0003] Typically, the target should be placed at the in-situ measurement position behind the cutterhead to complete the external parameter calibration of the stereo camera. For large-diameter tunnel boring machines, if external parameter calibration is carried out at the in-situ measurement position, large targets that meet the requirements of a large field of view are required to improve calibration accuracy, which is costly to manufacture. In addition, the in-situ measurement position has a complex spatial structure and harsh environment, making target transportation difficult and manual operation risky. Therefore, using a small target in a narrow and safe area behind the cutterhead to complete off-situ calibration has become a practical and effective solution. However, completing off-situ calibration in a narrow and safe area presents the problems of a large field of view and non-overlapping field of view, and also needs to consider the operability and convenience of calibration. Few scholars have proposed specific solutions for this special application scenario. To achieve accurate measurement of cutter wear in tool-changing robots, it is of great significance to propose a large field of view non-overlapping stereo camera external parameter calibration technology. Summary of the Invention

[0004] The technical problem to be solved by this invention is to propose a two-step calibration method and system for the external parameters of a large field of view non-overlapping stereo camera for a tool-changing robot based on pose transformation. This method can solve the problems of large field of view and non-overlapping field of view in the non-in-situ calibration process of the external parameters of the stereo camera for a tool-changing robot. Flexible calibration can be completed using a single small target, laying the foundation for the hobbing wear measurement technology of tool-changing robots.

[0005] To achieve the above objectives, the present invention adopts the following technical solution;

[0006] A two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation includes the following steps:

[0007] S11: Complete the calibration image acquisition, and obtain the first calibration image group and the second calibration image group in sequence;

[0008] S12: Set a target point in the target as the origin O of the common world coordinate system of the large field-of-view non-overlapping stereo camera. w The global mapper is used to perform label mapping on all target points in all calibration images, and target points for extrinsic parameter calibration are extracted.

[0009] S13: In the first calibration stage, solve for the initial values ​​of the stereo camera's extrinsic parameters and simulate the symmetric transfer error E based on the non-overlapping fields of view of the left and right cameras. lij E rij Nonlinear optimization was performed to obtain the stereo camera extrinsic parameters for the first calibration stage.

[0010] S14: In the second calibration stage, solve for the pose transformation matrices of the left and right cameras between the two calibration stages. And obtain the final external parameters of the stereo camera.

[0011] Preferably, the target point coordinates of the first calibration image group and the second calibration image group in step S11 correspond one-to-one in terms of physical size.

[0012] Preferably, in step S11, the target pose of the second calibration image group is the same as the target pose of the last calibration image of the first calibration image group, which is the target pose FA.

[0013] Preferably, the global mapper in step S12 includes a left global mapper and a right global mapper, which are used independently for target point label mapping of the left camera calibration image and the right camera calibration image, respectively.

[0014] Preferably, the process of using a global mapper to perform label mapping on all target points in the calibration image in step S12 includes the following steps:

[0015] S51: Identify the reference circle and segmentation circle of the global mapper, perform self-localization based on the reference circle, and sort the segmentation circles;

[0016] S52: Use the reference circle and the segmentation circle to segment the calibration image into image blocks;

[0017] S53: Assign all target points to their respective image blocks to obtain the row and column labels of each target point;

[0018] S54: Using the origin O of the common world coordinate system w Using row and column labels as a reference, the world coordinates of the target point are solved to complete the label mapping.

[0019] Preferably, in step S13, the non-overlapping field of view simulation symmetric transfer error E of the left and right cameras is... lij E rij This can be expressed as the following relation:

[0020]

[0021] In the formula, i is the calibration image number, j is the target point label, and s rl s lr As a scale factor, These are the intrinsic parameters of the left and right cameras, respectively, during the first calibration phase. To calibrate the world coordinates of target points in the images for the left and right cameras. These are the ideal pixel coordinates of the target point on the left and right cameras, respectively. The pixel coordinates are the symmetrical transfer points of the simulated target points on the right and left cameras, respectively.

[0022] Preferably, the pose transformation matrix in step S14 This can be expressed as the following relation:

[0023]

[0024] In the formula, These represent the transformation matrices of the target attitude FA relative to the left and right cameras, respectively, during the first calibration phase. These represent the transformation matrices of the target attitude FA relative to the left and right cameras during the second calibration phase, respectively.

[0025] Furthermore, the stereo camera extrinsic parameters in step S14 This can be expressed as the following relation:

[0026]

[0027] This invention also provides a two-step calibration system for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, comprising:

[0028] The image acquisition module is used to acquire the first calibration image group and the second calibration image group;

[0029] The image processing module is used to solve for the pixel coordinates of target points in the first calibration image group and the second calibration image group;

[0030] The label mapping module is used to perform label mapping on target points in the first calibration image group and the second calibration image group;

[0031] The first calibration module is used to execute the first calibration algorithm and output the stereo camera extrinsic parameters for the first calibration stage.

[0032] The second calibration module executes the second calibration algorithm and outputs the final stereo camera extrinsic parameters.

[0033] The beneficial effects of this invention are as follows: By employing a two-step calibration method for the extrinsic parameters of a large field-of-view, non-overlapping stereo camera for a tool-changing robot based on pose transformation, the non-in-situ calibration of extrinsic parameters can be completed in a narrow, safe area behind the cutterhead using a small target. This solves the calibration technical challenges caused by the large field of view and non-overlapping field of view, and avoids the prominent problems of high target cost, high operational difficulty, and high safety risks associated with in-situ calibration on the back of the cutterhead. This invention lays an important foundation for the accurate measurement of cutter wear in the vision system of a tunnel boring machine tool-changing robot. Attached Figure Description

[0034] Figure 1 This is a flowchart of the two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, according to the present invention.

[0035] Figure 2 This is a schematic diagram of the global mapper structure based on the phase-shifted circular grating array target image of the present invention;

[0036] Figure 3 This is a flowchart of the label mapping process of the present invention;

[0037] Figure 4 This is a schematic diagram of the two-step calibration system for the external parameters of the large field-of-view non-overlapping stereo camera of the tool-changing robot of the present invention;

[0038] In the diagram: 1 Image acquisition module, 2 Image processing module, 3 Label mapping module, 4 First calibration module, 5 Second calibration module. Detailed Implementation

[0039] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described technical solutions are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] Reference Figure 1 A two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation is presented below.

[0041] S11: Complete the calibration image acquisition, obtaining the first and second calibration image groups sequentially. A single 20-inch small LCD display target is used to display images of different targets. The stereo camera lens is focused on a non-in-situ calibration position to acquire a clear checkerboard image, forming the first calibration image group. Next, keeping the LCD display target and stereo camera in a fixed orientation, the lens is focused on the in-situ measurement position on the back of the cutter head to acquire a blurred phase-shifting circular grating array image, forming the second calibration image group.

[0042] S12: Set a target point in the target as the origin O of the common world coordinate system of the large field-of-view non-overlapping stereo camera. w A global mapper is used to label all target points in all calibration images. For all calibration images acquired by the left camera, pre-selected and identical continuous target points are extracted for stereo camera extrinsic parameter calibration; for the right camera, the same number of continuous target points as the left camera are pre-selected, and the selected target points are extracted in all calibration images for stereo camera extrinsic parameter calibration.

[0043] S13: In the first calibration stage, solve for the initial values ​​of the stereo camera's extrinsic parameters, and simulate the symmetric transfer error E using the non-overlapping fields of view of the left and right cameras in the first calibration image group. lij E rij Minimizing the sum of the parameters is the optimization objective. An objective function is constructed and nonlinear optimization is performed to obtain the stereo camera extrinsic parameters in the first calibration stage.

[0044] S14: In the second calibration stage, based on the last image of the first calibration image group and the second calibration image group, solve for the pose transformation matrices of the left and right cameras between the two calibration stages. And obtain the final external parameters of the stereo camera.

[0045] The target point coordinates of the first calibration image group and the second calibration image group in step S11 above correspond one-to-one in terms of physical size.

[0046] In step S11 above, the target pose of the second calibration image group is the same as the target pose of the last calibration image of the first calibration image group, which is the target pose FA.

[0047] Reference Figure 2The global mapper in step S12 above includes a left global mapper and a right global mapper, which work independently and can be used for target point label mapping of the left camera calibration image and the right camera calibration image, respectively. The reference circle is white on the outside and black on the inside; the segmentation circle is black on the outside and white on the inside. The reference circles of the left global mapper are LLC1 and LLC2, and the segmentation circles are LSC1, LSC2, LSC3, LSC4, LSC5, and LSC6; the reference circles of the right global mapper are RLC1 and RLC2, and the segmentation circles are RSC1, RSC2, RSC3, RSC4, RSC5, and RSC6. The global mapper is also applicable to checkerboard target images.

[0048] Reference Figure 3 The process of using a global mapper to perform label mapping on all target points in the calibration image in step S12 above includes the following steps:

[0049] S51: Obtain the grayscale histograms of all circles in the calibration image using a shape detection algorithm. For the second calibration image group, phase-shifted circles also exist in the calibration images. The grayscale distribution of the phase-shifted circles is non-unimodal, while the reference circle and segmentation circle of the global mapper are both unimodal. The reference circle and segmentation circle of the global mapper are identified based on the difference in grayscale distribution and the grayscale mean. Self-localization is performed based on the reference circle of the global mapper, and the segmentation circles are sorted.

[0050] S52: Construct 4 dividing lines using the geometric center points of the reference circle and the dividing circle to divide the calibration image into 5 image blocks.

[0051] S53: Assign all target points to their respective image blocks to obtain row labels for each target point. Using the line connecting the geometric center points of the reference circle as the baseline, obtain the corresponding column labels based on the distance of the target point relative to the baseline.

[0052] S54: Using the origin O of the common world coordinate system w Based on the known physical distance between target points, the world coordinates of the target points are solved using row and column labels to complete the label mapping.

[0053] In step S13 above, the non-overlapping field of view simulation symmetry transfer error E of the left and right cameras lij E rij The solution for j is as follows: using the target point in the left camera image as the simulated target point of the right camera, the simulated projection point is obtained on the image plane of the right camera, and then the non-overlapping field of view simulation symmetry transfer error E of the left camera is solved. lij Using the target point in the right camera image as the simulated target point for the left camera, a simulated projection point is obtained on the left camera image plane. Then, the simulation symmetric transfer error E of the non-overlapping field of view of the right camera is solved. rij E lijE rij This can be expressed as the following relation:

[0054]

[0055] In the formula, i is the calibration image number, j is the target point label, and s rl s lr As a scale factor, These are the intrinsic parameters of the left and right cameras, respectively, during the first calibration phase. To calibrate the world coordinates of target points in the images for the left and right cameras. These are the ideal pixel coordinates of the target point on the left and right cameras, respectively. The pixel coordinates are the symmetrical transfer points of the simulated target points on the right and left cameras, respectively.

[0056] The pose transformation matrix in step S14 above This can be expressed as the following relation:

[0057]

[0058] In the formula, These represent the transformation matrices of the target attitude FA relative to the left and right cameras, respectively, during the first calibration phase. These represent the transformation matrices of the target attitude FA relative to the left and right cameras during the second calibration phase, respectively.

[0059] Furthermore, the stereo camera extrinsic parameters in step S14 above... This can be expressed as the following relation:

[0060]

[0061] Reference Figure 4 A two-step calibration system for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation is provided, comprising an image acquisition module 1, an image processing module 2, a label mapping module 3, a first calibration module 4, and a second calibration module 5.

[0062] The image acquisition module 1 described above is used to acquire different target images displayed on the liquid crystal display target, and sequentially obtain the first calibration image group and the second calibration image group.

[0063] The image processing module 2 described above is used to solve for the pixel coordinates of target points in the first calibration image group and the second calibration image group. The target points in the first calibration image group are the corner points of the checkerboard pattern, and the pixel coordinates of the corner points are extracted by a corner detection algorithm. The target points in the second calibration image group are the geometric center points of the phase-shifted circle, and the geometric center points are solved by extracting the truncated phase and then using a least-squares ellipse fitting algorithm.

[0064] The label mapping module 3 described above is used to perform label mapping on target points in the first calibration image group and the second calibration image group, and output the world coordinates of the target points pre-selected for extrinsic parameter calibration;

[0065] The first calibration module 4 is used to execute the first calibration algorithm and output the stereo camera extrinsic parameters in the first calibration stage.

[0066] The second calibration module 5 is used to execute the second calibration algorithm and output the final stereo camera extrinsic parameters.

[0067] The explanation of individual letters in the parameters (formulas) of this patent is as follows;

[0068] (1) In the lower right corner, the letter l represents the left camera, r represents the right camera, c represents the camera, and w represents the world coordinate system.

[0069] (2) The numbers 1 and 2 in the upper right corner indicate the first calibration stage and the second calibration stage respectively, and 12 indicates the interval between the two calibration stages.

[0070] (3) The order of the letter combinations in the lower right corner subheadings: rl and lr represent the right camera relative to the left camera and the left camera relative to the right camera, respectively. The order of the remaining letter combinations has no special meaning. For example: This represents the extrinsic parameters of the stereo camera. Typically, the extrinsic parameters of a stereo camera are the extrinsic parameters of the right camera relative to the left camera. Therefore, in this invention, we use... The upper right corner indicates the external parameters of the stereo camera, and the subscript 2 indicates, according to (2), that the external parameters of the stereo camera are the parameters of the second calibration stage.

[0071] This invention enables non-in-situ calibration of the extrinsic parameters of a large-field-of-view, non-overlapping stereo camera for a tool-changing robot within a confined, safe area behind the cutterhead of a tunnel boring machine (TBM). This avoids the significant problems associated with in-situ calibration at the measurement location on the back of the cutterhead, such as high target cost, operational difficulty, and high safety risks. This lays a crucial foundation for accurate measurement of cutter wear in the vision system of a TBM tool-changing robot. This invention possesses excellent feasibility and economic efficiency and can be widely applied to various types of TBMs.

[0072] The foregoing has set forth many specific details to provide a thorough understanding of the invention. However, without departing from the spirit and scope of the invention, those skilled in the art can still directly determine or derive many other variations or modifications consistent with the principles of the invention based on the disclosure. Therefore, the invention is not limited to the specific embodiments disclosed above, and the scope of the invention should be understood and recognized as covering all such variations or modifications.

Claims

1. A two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, characterized in that: Includes the following steps: S11: Complete the calibration image acquisition, and obtain the first calibration image group and the second calibration image group in sequence; A single small liquid crystal display target is used to display images of different targets; the lens of the stereo camera is focused on a non-in-situ calibration position to acquire clear checkerboard images, forming the first calibration image group; then, while keeping the posture of the liquid crystal display target and the stereo camera fixed, the lens is focused on the in-situ measurement position on the back of the cutter head to acquire blurred phase-shifting circular grating array images, forming the second calibration image group. S12: Set a target point in the target as the origin of the common world coordinate system of the large field-of-view non-overlapping stereo camera. The global mapper is used to perform label mapping on all target points in all calibration images. For all calibration images acquired by the left camera, pre-selected and identical continuous target points are extracted for stereo camera extrinsic parameter calibration. For the right camera, the same number of continuous target points as the left camera are pre-selected, and the selected target points are extracted in all calibration images for stereo camera extrinsic parameter calibration. S13: In the first calibration stage, solve for the initial values ​​of the stereo camera's extrinsic parameters, and simulate the symmetric transfer error using the non-overlapping fields of view of the left and right cameras in the first calibration image set. , Minimizing the sum of the parameters is the optimization objective. An objective function is constructed and nonlinear optimization is performed to obtain the stereo camera extrinsic parameters in the first calibration stage. ; S14: In the second calibration stage, based on the last image of the first calibration image group and the second calibration image group, solve for the pose transformation matrices of the left and right cameras between the two calibration stages. , And obtain the final stereo camera extrinsic parameters. .

2. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The target point coordinates of the first calibration image group and the second calibration image group mentioned in step S11 correspond one-to-one in terms of physical size.

3. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: In step S11, the target pose of the second calibration image group is the same as the target pose of the last calibration image of the first calibration image group, which is the target pose FA.

4. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The global mapper mentioned in step S12 includes a left global mapper and a right global mapper, which are used independently for target point label mapping of the left camera calibration image and the right camera calibration image, respectively.

5. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The process of using a global mapper to perform label mapping on all target points in the calibration image in step S12 includes the following steps: S51: Identify the reference circle and the segmentation circle of the global mapper, perform self-localization based on the reference circle, and sort the segmentation circles; S52: Use the reference circle and the segmentation circle to segment the calibration image into image blocks; S53: All target points are assigned to their respective image blocks to obtain row and column labels for each target point; S54: Using the origin of the common world coordinate system as the reference point Using row and column labels as a reference, the world coordinates of the target point are solved to complete the label mapping.

6. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The non-overlapping field of view simulation symmetric transfer error of the left and right cameras mentioned in step S13 , This can be expressed as the following relation: In the formula, i To calibrate image numbering, j Label the target point. , As a scale factor, , These are the intrinsic parameters of the left and right cameras, respectively, during the first calibration phase. Calibrate the world coordinates of target points in the images for the left and right cameras; , These are the ideal pixel coordinates of the target point on the left and right cameras, respectively. , The pixel coordinates are the symmetrical transfer points of the simulated target points on the right and left cameras, respectively.

7. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The pose transformation matrix mentioned in step S14 , This can be expressed as the following relation: In the formula, , These represent the transformation matrices of the target attitude FA relative to the left and right cameras during the first calibration phase, respectively. , These represent the transformation matrices of the target attitude FA relative to the left and right cameras during the second calibration phase, respectively.

8. The two-step calibration method for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: The stereo camera extrinsic parameters described in step S14 This can be expressed as the following relation: 。 9. The system for two-step calibration of extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, as described in claim 1, is characterized in that: include: The image acquisition module is used to acquire the first calibration image group and the second calibration image group; The image processing module is used to solve for the pixel coordinates of the target point in the first calibration image group and the second calibration image group; The label mapping module is used to perform label mapping on target points in the first calibration image group and the second calibration image group; The first calibration module is used to execute the first calibration algorithm and output the stereo camera extrinsic parameters of the first calibration stage. ; The second calibration module is used to execute the second calibration algorithm and output the final stereo camera extrinsic parameters. .

Images