Tool changing robot large-view-field non-overlapping stereo camera external parameter two-step calibration method and system based on pose transformation

Abstract

The invention discloses a two-step calibration method and system for external parameters of a large-view-field non-overlapping stereo camera of a tool changing robot based on pose transformation, and relates to the technical field of automatic tool changing of tunnel boring machines. The objective of the invention is mainly to solve the problems of high construction cost and high operation difficulty of a target led out by in-situ calibration of external parameters of a stereo camera at a measurement position at the back of a cutter head at present. The method comprises the following steps: acquiring a first calibration image group and a second calibration image group in sequence; label mapping is carried out on target points in all calibration images through a global mapper; in the first calibration stage, non-overlapping view field simulation symmetric transfer errors are utilized to carry out nonlinear optimization on external parameters of the stereo camera; and in the second calibration stage, solving the pose transformation matrix of the left camera and the right camera between the two calibration stages to obtain the final external parameters of the stereo camera. The method has the advantages that the problems of high target cost, high operation difficulty and the like caused by external parameter in-situ calibration at the measurement position of the back of the cutter head are solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of automatic tool changing of a tunnel boring machine, and in particular to a two-step calibration method and system for the outer parameters of a large-view non-overlapping stereo camera of a tool changing robot based on pose transformation. BACKGROUND

[0002] A rolling cutter is a main rock breaking tool of a tunnel boring machine, and rock breaking is realized through rolling. In the process of tunnel construction, the problem of rolling cutter wear and failure occurs frequently. The rolling cutter detection and replacement area is located at the back of the cutter head, which is the most forward position of the tunnel boring machine, and the task is usually completed by multiple people. During the construction period of the tunnel project, the rolling cutter needs to be frequently detected and replaced in large quantities, and the harsh construction environment on site brings many inconveniences to manual operation, and the efficiency is low and there are great safety hazards. Under this background, the tool changing robot becomes an effective solution to replace manual operation. Among them, the rolling cutter wear measurement based on visual measurement technology is a prerequisite for automatic tool changing of the robot, and the stereo camera outer parameter calibration based on the construction environment on site is the first step of the visual measurement task.

[0003] Under normal circumstances, the target should be placed at the original measurement position at the back of the cutter head to complete the stereo camera outer parameter calibration. For a large-diameter tunnel boring machine, if the outer parameter calibration is carried out at the original measurement position, in order to improve the calibration accuracy, a large target meeting the requirements of large view field needs to be used, which has high manufacturing cost. In addition, the space structure of the original measurement position is complex, the environment is harsh, the target transportation is difficult, and the manual operation risk is high. Therefore, using a small target in the narrow safety area behind the cutter head to complete the non-original calibration has become a practical and effective solution. However, the non-original calibration in the narrow safety area has the problems of large view field and non-overlapping view field, and the operability and convenience of calibration also need to be considered, and few scholars have proposed targeted solutions for this special application scenario. In order to realize accurate rolling cutter wear measurement of the tool changing robot, it is of great significance to propose a large-view non-overlapping stereo camera outer parameter calibration technology. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a two-step calibration method and system for the outer parameters of a large-view non-overlapping stereo camera of a tool changing robot based on pose transformation, which can solve the problems of large view field and non-overlapping view field in the non-original calibration process of the stereo camera outer parameters of the tool changing robot, and a single small target can be used for flexible calibration, thereby laying a foundation for the rolling cutter wear measurement technology of the tool changing robot.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] The two-step calibration method for the outer parameters of a large-view non-overlapping stereo camera of a tool changing robot based on pose transformation comprises the following steps:

[0007] S11: complete the calibration image acquisition, in turn, to obtain the first calibration image group and the second calibration image group;

[0008] S12: set a target point in the target as the common world coordinate system origin O of the large field of view non-overlapping stereo camera w , use the global mapper to label map all target points in all calibration images, and extract the target points for external parameter calibration;

[0009] S13: in the first calibration stage, solve the stereo camera external parameter initial value, based on the non-overlapping field of view of the left and right cameras Simulate the symmetric transfer error E lij , E rij Nonlinear optimization is performed to obtain the stereo camera external parameter in the first calibration stage

[0010] S14: in the second calibration stage, respectively solve the pose transformation matrix of the left and right cameras between the two calibration stages, and obtain the final stereo camera external parameter

[0011] Preferably, the target point coordinates of the first calibration image group and the second calibration image group in step S11 correspond one by one in physical size.

[0012] Preferably, the target posture of the second calibration image group in step S11 is the same as the target posture of the last calibration image of the first calibration image group, both of which are target posture FA.

[0013] Preferably, the global mapper in step S12 includes a left global mapper and a right global mapper, which are respectively used for target point label mapping of left camera calibration images and right camera calibration images.

[0014] Preferably, the process of using the global mapper to label map all target points in the calibration image in step S12 includes the following steps:

[0015] S51: identify the reference circle and the segmentation circle of the global mapper, based on the reference circle to develop self-positioning, and sort the segmentation circle;

[0016] S52: use the reference circle and the segmentation circle to divide the calibration image into image blocks;

[0017] S53: classify all target points into their respective image blocks to obtain the row label and the column label of each target point;

[0018] S54: take the common world coordinate system origin O w as the reference, use the row label and the column label to solve the world coordinates of the target points, and complete the label mapping.

[0019] Preferably, in step S13, the non-overlapping field of view simulation symmetric transfer error E of the left and right cameras is... lij E rij This can be expressed as the following relation:

[0020]

[0021] In the formula, i is the calibration image number, j is the target point label, and s rl s lr As a scale factor, These are the intrinsic parameters of the left and right cameras, respectively, during the first calibration phase. To calibrate the world coordinates of target points in the images for the left and right cameras. These are the ideal pixel coordinates of the target point on the left and right cameras, respectively. The pixel coordinates are the symmetrical transfer points of the simulated target points on the right and left cameras, respectively.

[0022] Preferably, the pose transformation matrix in step S14 This can be expressed as the following relation:

[0023]

[0024] In the formula, These represent the transformation matrices of the target attitude FA relative to the left and right cameras, respectively, during the first calibration phase. These represent the transformation matrices of the target attitude FA relative to the left and right cameras during the second calibration phase, respectively.

[0025] Furthermore, the stereo camera extrinsic parameters in step S14 This can be expressed as the following relation:

[0026]

[0027] This invention also provides a two-step calibration system for the extrinsic parameters of a large field-of-view non-overlapping stereo camera for a tool-changing robot based on pose transformation, comprising:

[0028] The image acquisition module is used to acquire the first calibration image group and the second calibration image group;

[0029] The image processing module is used to solve for the pixel coordinates of target points in the first calibration image group and the second calibration image group;

[0030] The label mapping module is used to perform label mapping on target points in the first calibration image group and the second calibration image group;

[0031] The first calibration module is used to execute the first calibration algorithm and output the stereo camera extrinsic parameters for the first calibration stage.

[0032] The second calibration module is used for executing a second calibration algorithm to output final stereo camera external parameters

[0033] The present application has the beneficial effects: through the two-step calibration method of the large field of view non-overlapping stereo camera external parameters of the tool changing robot based on pose transformation, the non-in-situ external parameter calibration can be completed in the narrow safety area behind the cutter head by using a small target, the calibration technical problems caused by the large field of view and the non-overlapping field of view are solved, and the problems of high target cost, high operation difficulty, high safety risk and other prominent problems caused by the in-situ external parameter calibration in the back of the cutter head are avoided. The present application lays an important foundation for the precise measurement of cutter wear of the tunnel boring machine tool changing robot vision system. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The flow chart of the two-step calibration method of the large field of view non-overlapping stereo camera external parameters of the tool changing robot based on pose transformation of the present application;

[0035] Figure 2 The global mapper structure schematic diagram based on the phase shift circular grating array target image of the present application;

[0036] Figure 3 The label mapping flow chart of the present application;

[0037] Figure 4 The structure schematic diagram of the two-step calibration system of the large field of view non-overlapping stereo camera external parameters of the tool changing robot of the present application;

[0038] In the figure: 1 image acquisition module, 2 image processing module, 3 label mapping module, 4 first calibration module, 5 second calibration module. DETAILED DESCRIPTION

[0039] The technical solutions of the present application will be described clearly and completely below by combining the drawings of the present application. Obviously, the described technical solutions are only part of the embodiments of the present application, not all the embodiments; based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] Reference Figure 1 A two-step calibration method of the large field of view non-overlapping stereo camera external parameters of the tool changing robot based on pose transformation, the specific operation steps are as follows:

[0041] S11: complete the calibration image acquisition, and obtain the first calibration image group and the second calibration image group in turn. A single 20-inch small liquid crystal display target is used to display different target images. The lens of the stereo camera is focused on the non-in-situ calibration position to acquire clear checkerboard images, which constitute the first calibration image group. Then, the posture of the liquid crystal display target and the stereo camera is kept fixed, the lens is focused on the in-situ measurement position at the back of the cutter head to acquire blurred phase-shifted circular grating array images, which constitute the second calibration image group.

[0042] S12: set a target point in the target as the common world coordinate system origin O of the large field of view non-overlapping stereo camera w , and label map all target points in all calibration images by using the global mapper. For all calibration images acquired by the left camera, the same pre-selected continuous target points are extracted for stereo camera extrinsic parameter calibration; for the right camera, the same number of pre-selected continuous target points as the left camera are extracted in all calibration images for stereo camera extrinsic parameter calibration.

[0043] S13: in the first calibration stage, solve the initial value of the stereo camera extrinsic parameter, minimize the sum of the simulated symmetrical transfer errors E lij , E rij of the left and right cameras in the first calibration image group as the optimization target, construct the objective function and perform nonlinear optimization to obtain the stereo camera extrinsic parameter

[0044] S14: in the second calibration stage, based on the last image of the first calibration image group and the second calibration image group, solve the pose transformation matrix of the left and right cameras between the two calibration stages respectively, and obtain the final stereo camera extrinsic parameter

[0045] The target point coordinates of the first calibration image group and the second calibration image group in the above step S11 correspond to each other in physical size.

[0046] The target posture of the second calibration image group in the above step S11 is the same as the target posture of the last calibration image of the first calibration image group, which is the target posture FA.

[0047] Refer to Figure 2The global mapper in step S12 includes a left global mapper and a right global mapper, which work independently and are respectively used for target point label mapping of the left camera calibration image and the right camera calibration image. The outer part of the reference circle is white and the inner part is black; the outer part of the split circle is black and the inner part is white. The reference circles of the left global mapper are LLC1 and LLC2, and the split circles are LSC1, LSC2, LSC3, LSC4, LSC5 and LSC6; the reference circles of the right global mapper are RLC1 and RLC2, and the split circles are RSC1, RSC2, RSC3, RSC4, RSC5 and RSC6. The global mapper is also applicable to the chessboard target image.

[0048] With reference to Figure 3 The process of label mapping of all target points in the calibration image by using the global mapper in step S12 includes the following steps:

[0049] S51: Obtain the gray level histogram of all circles in the calibration image by using a shape detection algorithm. For the second calibration image group, there is also a phase shift circle in the calibration image. The gray level distribution of the phase shift circle is non-unimodal, while the reference circles and the split circles of the global mapper are unimodal. The reference circles and the split circles of the global mapper are identified based on the gray level distribution difference and the gray level mean value. The self-positioning is developed based on the reference circles of the global mapper, and the split circles are sorted.

[0050] S52: Construct four split lines by using the geometric center points of the reference circles and the split circles, and divide the calibration image into five image blocks.

[0051] S53: Classify all target points into the image blocks to which the target points belong, and obtain the row labels of the target points. Take the line connecting the geometric center points of the reference circles as the reference line, and obtain the corresponding column labels according to the distance of the target points relative to the reference line.

[0052] S54: Take the common world coordinate system origin O w as the reference, use the row labels and the column labels to solve the world coordinates of the target points, and complete the label mapping.

[0053] The solving method of the non-overlapping field of view simulation symmetric transfer error E lij , E rij of the left and right cameras in step S13 is as follows: take the target points in the left camera image as the simulation target points of the right camera, obtain the simulation projection points on the right camera image plane, and then solve the non-overlapping field of view simulation symmetric transfer error E lij of the left camera; take the target points in the right camera image as the simulation target points of the left camera, obtain the simulation projection points on the left camera image plane, and then solve the non-overlapping field of view simulation symmetric transfer error E rij of the right camera. E lij、E rij The pose transformation matrix in step S14 can be expressed as the following relationship:

[0054]

[0055] In the formula, i is the calibration image number, j is the target point label, s rl 、s lr is a scale factor, and are the left and right camera intrinsic parameters of the first calibration stage, respectively, are the world coordinates of the target points in the left and right camera calibration images. and are the ideal pixel coordinates of the target points in the left and right cameras, respectively, and are the symmetric transfer point pixel coordinates of the simulated target points in the left and right cameras of the right and left cameras, respectively.

[0056] The pose transformation matrix in step S14 can be expressed as the following relationship:

[0057]

[0058] In the formula, and represent the transformation matrix of the target pose FA relative to the left and right cameras in the first calibration stage, respectively; and represent the transformation matrix of the target pose FA relative to the left and right cameras in the second calibration stage, respectively.

[0059] Further, the stereo camera extrinsic parameters in step S14 can be expressed as the following relationship:

[0060] Referring to

[0061] , a large field of view non-overlapping stereo camera extrinsic parameter two-step calibration system for a tool changing robot based on pose transformation includes an image acquisition module 1, an image processing module 2, a label mapping module 3, a first calibration module 4, and a second calibration module 5. Figure 4 The image acquisition module 1 is used to acquire different target images displayed by the liquid crystal display target, and sequentially obtains a first calibration image group and a second calibration image group.

[0062] The image processing module 2 is used to solve the target point pixel coordinates in the first calibration image group and the second calibration image group. The target points of the first calibration image group are the corner points of the chessboard, and the corner point pixel coordinates are extracted by a corner point detection algorithm; the target points of the second calibration image group are the geometric center points of the phase shift circles, and the geometric center points are solved by extracting the truncated phase and then using a least squares ellipse fitting algorithm.

[0063]

[0064] ​​The label mapping module 3 is used for label mapping of target points in the first and second sets of calibration images, and outputs world coordinates of the target points preselected for the external parameter calibration;

[0065] The first calibration module 4 is used for executing a first calibration algorithm, and outputs the stereo camera external parameters in the first calibration stage

[0066] The second calibration module 5 is used for executing a second calibration algorithm, and outputs the final stereo camera external parameters

[0067] The following is a description of the individual letters in the patent parameters (formulae) in the patent:

[0068] (1) The letter l in the subscript at the lower right corner represents the left camera, the letter r represents the right camera, the letter c represents the camera, and the letter w represents the world coordinate system.

[0069] (2) The numbers 1 and 2 in the subscript at the upper right corner represent the first calibration stage and the second calibration stage, respectively, and 12 represents the two calibration stages.

[0070] (3) The letter combination order in the subscript at the lower right corner, rl and lr represent the right camera relative to the left camera and the left camera relative to the right camera, respectively, and the remaining letter combination orders have no special meaning. For example: represents the stereo camera external parameters. Generally, the stereo camera external parameters are all external parameters of the right camera relative to the left camera. Therefore, in the present application, represents the stereo camera external parameters, and the upper right corner subscript 2 indicates that the stereo camera external parameters are the parameters in the second calibration stage according to (2).

[0071] The present application can realize the large field of view non-overlapping stereo camera external parameter non-in-situ calibration of the cutter robot in the narrow safety area behind the cutterhead of the tunnel boring machine, avoids the high cost of the target, the high difficulty of the operation, the high safety risk and other prominent problems caused by the in-situ calibration of the external parameters at the measuring position at the back of the cutterhead, and lays an important foundation for the precise measurement of the cutter wear of the tunnel boring machine cutter robot vision system. The present application has good realizability and economy, and can be widely applied to various tunnel boring machines.

[0072] The above description sets forth a large number of specific details in order to facilitate a full understanding of the present application, but those skilled in the art can still determine or deduce many other variants or modifications in accordance with the disclosure of the present application, which conform to the principles of the present application without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed above, and the scope of the present application should be understood and recognized as covering all these variants or modifications.

Claims

1. A two-step calibration method for the extrinsic parameters of a large field of view non-overlapping stereo camera based on pose transformation for a tool changing robot, characterized in that: The method comprises the following steps: S11: complete calibration image acquisition, and obtain a first calibration image group and a second calibration image group in sequence; A single small liquid crystal display target is used to display different target images; the lens of the stereo camera is focused on a non-in-situ calibration position to collect clear checkerboard images, thereby constituting the first calibration image group; then, the posture of the liquid crystal display target and the stereo camera is kept fixed, the lens is focused on an in-situ measurement position at the back of the cutter head to collect blurred phase-shifted circular grating array images, thereby constituting the second calibration image group; S12: set a certain target point in the target as the common world coordinate system origin O of the large field of view non-overlapping stereo camera w , all target points in all calibration images are labeled and mapped by using a global mapper; for all calibration images acquired by the left camera, the same pre-selected continuous target points are extracted for stereo camera external parameter calibration; for the right camera, the same number of pre-selected continuous target points as the left camera are extracted in all calibration images for stereo camera external parameter calibration; S13: In the first calibration stage, the initial value of the stereo camera external parameter is solved to minimize the simulated symmetrical transfer error E of the non-overlapping field of view of the left and right cameras in the first calibration image set lij , the sum of E rij is minimized as the optimization goal, the objective function is constructed and nonlinear optimization is performed to obtain the stereo camera external parameter of the first calibration stage S14: In the second calibration stage, based on the last image of the first calibration image group and the second calibration image group, the pose transformation matrix of the left and right cameras between the two calibration stages is solved respectively and the final stereo camera external parameters are obtained 2. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The target point coordinates of the first calibration image group and the second calibration image group in step S11 correspond to each other in physical size.

3. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The target posture of the second calibration image group in step S11 is the same as the target posture of the last calibration image of the first calibration image group, and both are target posture FA.

4. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The global mapper in step S12 comprises a left global mapper and a right global mapper, and is respectively used for target point label mapping of left camera calibration images and right camera calibration images.

5. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The process of label mapping of all target points in the calibration image by using the global mapper in step S12 comprises the following steps: S51: identify the reference circle and the segmentation circle of the global mapper, perform self-positioning based on the reference circle, and sort the segmentation circle; S52: perform image block segmentation on the calibration image by using the reference circle and the segmentation circle; S53: classify the all target points into respective image blocks to obtain the row label and the column label of each target point; S54: With the common world coordinate system origin O w As a reference, the target point world coordinates are solved using the row and column labels, and label mapping is completed.

6. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The non-overlapping field of view of the left and right cameras described in step S13 simulates a symmetrical shift error E lij , E rij may be expressed as the following relationship: where i is the index of the calibration image, j is the index of the target point, s rl , s lr is the scale factor, are the left and right camera intrinsic parameters of the first calibration stage, respectively, are the world coordinates of the target points in the left and right camera calibration images, respectively; are the ideal pixel coordinates of the target points in the left and right cameras, respectively, are the symmetric transfer point pixel coordinates of the simulated target points in the left and right cameras, respectively.

7. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The pose transformation matrix described in step S14 may be expressed as the following relation: In the formula, respectively represent the transformation matrix of the target attitude FA relative to the left and right cameras in the first calibration stage; respectively represent the transformation matrix of the target attitude FA relative to the left and right cameras in the second calibration stage.

8. The pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameter two-step calibration method according to claim 1, characterized in that: The stereo camera extrinsic parameters described in step S14 is expressed as the following relation:

9. A system for two-step calibration of pose transformation based tool changer robot large field of view non-overlapping stereo camera extrinsic parameters, characterized in that: The method comprises the following steps: An image acquisition module is configured to acquire the first calibration image group and the second calibration image group; An image processing module is configured to solve the target point pixel coordinates in the first calibration image group and the second calibration image group; A label mapping module is configured to perform label mapping on the target points in the first calibration image group and the second calibration image group. a first calibration module configured to execute a first calibration algorithm to output the extrinsic parameters of the stereo camera in the first calibration stage a second calibration module configured to execute a second calibration algorithm to output the final stereo camera extrinsic parameters

Images