Mobile mechanical arm hand-eye calibration method and device and computer equipment

Abstract

The invention relates to a mobile mechanical arm hand-eye calibration method and device and computer equipment. The method comprises the steps that under the condition that a movable chassis of a movable mechanical arm moves to a preset target position, the mechanical arm tail end of the movable mechanical arm is controlled to move into the view field range of an external camera, and a calibration object is fixed to the mechanical arm tail end; acquiring first pose data of the tail end of the mechanical arm relative to the mobile mechanical arm base and second pose data of the calibration object in a camera coordinate system of the external camera, and performing hand-eye calibration on the mobile mechanical arm base and the external camera according to the first pose data, the second pose data and a preset pose transformation relation, the preset pose transformation relation represents the transformation relation between the coordinate system of the calibration object and the coordinate system of the tail end of the mechanical arm. By adopting the method, the hand-eye calibration precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hand-eye calibration, and in particular to a mobile manipulator hand-eye calibration method and device, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Hand-eye calibration is a core technology of a robot vision guidance system, and its precision directly determines the spatial consistency between the perception and execution of the robot. In high-precision application scenarios such as industrial automation, precision assembly and medical robots, the precision of hand-eye calibration is increasingly required.

[0003] The existing hand-eye calibration process usually only performs an initial hand-eye calibration before a mobile manipulator is put into application, and relies on the initial hand-eye calibration result in subsequent operation. However, in the continuous operation of the mobile manipulator, pose disturbances such as positioning errors, camera inherent errors and manipulator errors are prone to occur, which causes the hand-eye calibration result to deviate, and especially the superposition effect of multiple errors seriously affects the hand-eye calibration precision. SUMMARY

[0004] Therefore, it is necessary to provide a mobile manipulator hand-eye calibration method and device, a computer device, a computer readable storage medium and a computer program product capable of improving the hand-eye calibration precision.

[0005] In a first aspect, the present application provides a mobile manipulator hand-eye calibration method, comprising:

[0006] In the case that the mobile chassis of the mobile manipulator moves to a preset target position, the end of the manipulator of the mobile manipulator is controlled to move into the field of view of an external camera, and the end of the manipulator is fixed with a calibration object;

[0007] First pose data of the end of the manipulator relative to the base of the mobile manipulator and second pose data of the calibration object in the camera coordinate system of the external camera are acquired;

[0008] Hand-eye calibration is performed on the base of the mobile manipulator and the external camera according to the first pose data, the second pose data and a preset pose transformation relationship to obtain a hand-eye calibration result, and the preset pose transformation relationship represents the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator.

[0009] In one of the embodiments, before the mobile chassis moves to the preset position, the method further comprises:

[0010] Obtain a plurality of sets of calibration data of the end of the robot arm in different poses, wherein each set of calibration data comprises third pose data of the end of the robot arm relative to the mobile robot arm base and fourth pose data of a calibration object of the end of the robot arm in a camera coordinate system of an external camera, when the mobile chassis is in a preset initial position;

[0011] Perform hand-eye calibration on the mobile robot arm base and the external camera based on the plurality of third pose data and the plurality of fourth pose data, to determine an initial hand-eye calibration relationship and a transformation relationship between a coordinate system of the calibration object and a coordinate system of the end of the robot arm.

[0012] The initial hand-eye calibration relationship represents a transformation relationship between a base coordinate system of the mobile robot arm and a camera coordinate system of the external camera.

[0013] In one embodiment, the performing hand-eye calibration on the mobile robot arm base and the external camera based on the plurality of third pose data and the plurality of fourth pose data, to determine an initial hand-eye calibration relationship and a transformation relationship between a coordinate system of the calibration object and a coordinate system of the end of the robot arm comprises:

[0014] For each set of calibration data, based on the third pose data, determine a first homogeneous transformation matrix between the coordinate system of the end of the robot arm and the base coordinate system of the mobile robot arm, and based on the fourth pose data, determine a second homogeneous transformation matrix between the coordinate system of the calibration object and the camera coordinate system of the external camera.

[0015] Based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, fit a preset calibration transformation equation to determine an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the end of the robot arm.

[0016] The calibration transformation equation represents a correlation relationship between the first homogeneous transformation matrix, the first homogeneous transformation matrix, the initial hand-eye calibration matrix, and the third homogeneous transformation matrix, the initial hand-eye calibration relationship comprises the initial hand-eye calibration matrix, and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the robot arm comprises the third homogeneous transformation matrix.

[0017] In one embodiment, the fitting a preset calibration transformation equation based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices to determine an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the end of the robot arm comprises:

[0018] Based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, a preset calibration transformation equation is fitted by using a least square method, to determine an initial hand-eye calibration matrix and a third homogeneous transformation matrix between a coordinate system of the calibration object and a coordinate system of the robot end.

[0019] In one of the embodiments, the second pose data of the calibration object in the camera coordinate system of the external camera is obtained, including:

[0020] The image data of the calibration object captured by the external camera is obtained.

[0021] The circular features are extracted from the image data.

[0022] Based on the circular features and preset camera parameters, the second pose data of the calibration object in the camera coordinate system of the external camera is determined.

[0023] In a second aspect, the present application further provides a mobile robot hand-eye calibration device, including:

[0024] An image acquisition module is configured to control a robot end of the mobile robot to move into a field of view of an external camera when a mobile chassis of the mobile robot moves to a preset target position, the robot end being fixed with a calibration object.

[0025] A data acquisition module is configured to obtain first pose data of the robot end relative to a mobile robot base and second pose data of the calibration object in a camera coordinate system of the external camera.

[0026] A hand-eye calibration module is configured to perform hand-eye calibration on the mobile robot base and the external camera according to the first pose data, the second pose data and a preset pose transformation relationship, to obtain a hand-eye calibration result, the preset pose transformation relationship representing a transformation relationship between a coordinate system of the calibration object and a coordinate system of the robot end.

[0027] In a third aspect, the present application further provides a computer device including a memory and a processor, the memory storing a computer program, and the processor implementing steps in the above mobile robot hand-eye calibration method embodiments when executing the computer program.

[0028] In a fourth aspect, the present application further provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement steps in the above mobile robot hand-eye calibration method embodiments.

[0029] In a fifth aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the mobile manipulator hand-eye calibration method embodiment described above.

[0030] The mobile manipulator hand-eye calibration method, device, computer equipment, computer readable storage medium and computer program product described above first controls the manipulator end of the mobile manipulator to move into the field of view of the external camera when the mobile chassis of the mobile manipulator moves to the preset target position, and the manipulator end is fixed with a calibration object, so that the external camera can capture the image of the calibration object. Then, the first pose data of the manipulator end relative to the mobile manipulator base and the second pose data of the calibration object in the camera coordinate system of the external camera are obtained. Finally, the mobile manipulator base and the external camera are hand-eye calibrated according to the first pose data, the second pose data and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the manipulator end, and a hand-eye calibration result is obtained. In this way, the hand-eye calibration can be performed once by the above-mentioned method every time the mobile chassis moves to the target position, and the hand-eye calibration process is relatively simple and generally does not excessively affect the execution of the task of the mobile manipulator. Compared with the traditional scheme which only relies on the initial hand-eye calibration result, the hand-eye calibration deviation caused by positioning errors, camera inherent errors, manipulator errors and other factors and superposition effects can be effectively reduced, the relative pose between the mobile manipulator and the external camera is accurately calibrated, and the hand-eye calibration precision is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0032] Figure 1 An application environment diagram of the mobile manipulator hand-eye calibration method in an embodiment;

[0033] Figure 2 A flowchart of the mobile manipulator hand-eye calibration method in an embodiment;

[0034] Figure 3 A schematic diagram of hand-eye calibration in an embodiment;

[0035] Figure 4 A flowchart of the mobile manipulator hand-eye calibration method in another embodiment;

[0036] Figure 5A flowchart of an initial hand-eye calibration step in one embodiment;

[0037] Figure 6 A flowchart of acquiring pose data of a calibration object in one embodiment;

[0038] Figure 7 A flowchart of a mobile manipulator hand-eye calibration method in one detailed embodiment;

[0039] Figure 8 A structural block diagram of a mobile manipulator hand-eye calibration device in one embodiment;

[0040] Figure 9 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0042] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" used in the present application and any variations thereof are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0043] The mobile manipulator hand-eye calibration method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the controller 104 of the mobile manipulator through a network. The data storage system can store data required to be processed by the controller 104.

[0044] Specifically, the mobile chassis of the mobile manipulator can be provided with a position sensor. When the mobile chassis moves to a preset target position, for example, moves to the incoming position, the controller 104 of the mobile chassis can control the end of the arm of the mobile manipulator to move into the field of view of the external camera. The external camera can be fixed at a specific position, and the end of the arm is fixed with a calibration object. At this time, the teach pendant of the mobile manipulator can automatically upload the first pose data of the end of the arm relative to the base of the mobile manipulator collected to the terminal 102, and the external camera can automatically upload the second pose data of the calibration object in the camera coordinate system of the external camera collected to the terminal 102. The operator can send the above data to the controller 104 through the terminal 102, and the terminal 102 can also upload the data at a fixed time or in real time. The teach pendant or the external camera can also directly upload the collected data to the controller 104. Further, the controller 104 performs hand-eye calibration on the base of the mobile manipulator and the external camera according to the first pose data, the second pose data, and a preset pose transformation relationship to obtain a hand-eye calibration result. The preset pose transformation relationship represents the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the arm.

[0045] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude flying vehicles, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The controller 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0046] In an exemplary embodiment, as shown in Figure 2 , a mobile manipulator hand-eye calibration method is provided. The method is applied to the controller 104 in Figure 2 for illustration, including the following steps:

[0047] S200, when the mobile chassis of the mobile manipulator moves to a preset target position, the end of the arm of the mobile manipulator is controlled to move into the field of view of the external camera.

[0048] The mobile manipulator is an automated mechanical device, including a mobile chassis, a manipulator and an end effector. The manipulator can be installed on the mobile chassis, so that the end effector can realize the desired operation through the coordinated movement of the chassis and the manipulator. Generally, the mobile chassis can be a mobile robot. Among them, the end of the manipulator is fixed with a calibration object. The preset target position is a specific spatial position that the mobile chassis needs to move to according to the actual application requirements. Taking the task of moving the manipulator as an example, the target position can be the incoming material position. The manipulator is a mechanism with multiple movable joints, which can realize the precise positioning and attitude adjustment of the end in space through the movement of the joints, and the end is fixed with a calibration object. The end of the manipulator: the outermost part of the manipulator performing operation, which is directly connected or contacted with external objects. The external camera is an image acquisition device independent of the mobile chassis and the manipulator, that is, the "eye" in the hand-eye calibration, which is used to collect image information of the calibration object to determine its position in the camera coordinate system. The calibration object is an object with specific geometric shape or pattern characteristics, such as a calibration board with black and white stripes, which can be fixed at the end of the manipulator.

[0049] Exemplarily, when the mobile chassis moves to the preset target position according to the preset path or control instruction, the controller can control the movement of each joint of the manipulator to adjust the position and attitude of the end of the manipulator, so that the calibration object fixed at the end of the manipulator enters the field of view of the external camera. Therefore, the external camera can shoot the calibration object, so as to obtain the pose data of the calibration object in the camera coordinate system of the external camera in the subsequent process, thereby establishing the spatial relationship among the mobile chassis, the manipulator and the external camera.

[0050] S400, acquiring first pose data of the end of the manipulator relative to the base of the mobile manipulator, and second pose data of the calibration object in the camera coordinate system of the external camera.

[0051] Among them, the pose data is used to describe the position and attitude of the object in space, wherein the position is usually represented by three-dimensional coordinates, and the attitude can be represented by Euler angles or quaternions. The base coordinate system of the mobile manipulator is established on the fixed coordinate system on the mobile chassis, which is used as a reference for describing the pose of the end of the manipulator. The second pose data of the calibration object in the camera coordinate system of the external camera refers to the three-dimensional spatial position coordinates and attitude of the calibration object in the camera coordinate system of the external camera, which can be extracted from the calibration object image shot by the external camera through image processing technology. The camera coordinate system is a coordinate system established with the external camera as the center, which is used to describe the spatial position of the object in the field of view of the camera.

[0052] Exemplarily, the first pose data of the robot end relative to the mobile robot base can be directly read from the teach pendant of the mobile robot. Meanwhile, in combination with the image of the calibration object captured by the external camera and the camera parameters of the external camera itself, the second pose data of the calibration object in the camera coordinate system can be determined as the key information for subsequent hand-eye calibration.

[0053] S600, according to the first pose data, the second pose data and the preset pose transformation relationship, performing hand-eye calibration on the mobile robot base and the external camera to obtain a hand-eye calibration result.

[0054] The hand-eye calibration is used to determine the relative position and attitude relationship between the robot and the external camera. By establishing the conversion relationship between the robot coordinate system, the camera coordinate system and the calibration object coordinate system, the unification between the visual information and the robot motion control can be realized, so that the robot can perform accurate operation based on visual feedback. The "eye" refers to the camera, and the "hand" refers to the robot. In the embodiment, the hand-eye calibration on the mobile robot base and the external camera is essentially considered as "eye outside hand" calibration, which means that the camera is independent of the robot and is fixed in the external environment (such as a workbench support). The purpose of the eye outside hand calibration is to solve the transformation matrix of the camera coordinate system of the external camera relative to the base coordinate system of the mobile robot.

[0055] The preset pose transformation relationship represents the transformation relationship between the coordinate system of the calibration object and the coordinate system of the robot end. This relationship can be represented in the form of mathematical expressions such as homogeneous transformation matrix. The hand-eye calibration result is the relative position and attitude relationship between the coordinate system of the robot end and the camera coordinate system of the external camera, which can be presented in the form of transformation matrix.

[0056] After obtaining the first pose data of the robot end relative to the mobile robot base, the second pose data of the calibration object in the camera coordinate system of the external camera, and the known preset pose transformation relationship, the relative position and attitude relationship between the coordinate system of the robot end and the camera coordinate system of the external camera can be calculated by using mathematical models and algorithms, such as hand-eye calibration algorithm based on least square method, so as to complete the hand-eye calibration process and obtain the hand-eye calibration result.

[0057] Exemplarily, in order to facilitate the understanding of the transformation relationship between the coordinate systems, reference can be made to the calibration schematic diagram shown in the accompanying drawings. Figure 3 After the mobile chassis reaches the incoming position, the controller controls the robot end to move the calibration object into the camera field of view. The robot end pose Then, the PnP algorithm can be used to calculate and record the relative transformation matrix of the robot end relative to the camera coordinate system of the calibration object In combination with the change matrix of the coordinate system of the calibration object relative to the coordinate system of the end of the robot arm The transformation matrix of the camera coordinate system of the external camera relative to the base coordinate system of the mobile robot arm can be quickly calculated That is, the hand-eye calibration result is obtained, and the calculation formula is shown in formula (1):

[0058] (1)

[0059] The hand-eye calibration method of the mobile robot arm first controls the end of the robot arm to move into the field of view of the external camera when the mobile chassis of the mobile robot arm moves to the preset target position, and the end of the robot arm is fixed with a calibration object, so that the external camera can capture an image of the calibration object. Then, first pose data of the end of the robot arm relative to the base of the mobile robot arm and second pose data of the calibration object in the camera coordinate system of the external camera are obtained. Finally, the mobile robot base and the external camera are hand-eye calibrated according to the first pose data, the second pose data, and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the robot arm, to obtain a hand-eye calibration result. In this way, the hand-eye calibration can be performed once by the above-mentioned method every time the mobile chassis moves to the target position, and the hand-eye calibration process is relatively simple and generally does not excessively affect the execution of the task of the mobile robot arm. Compared with the traditional scheme which only relies on the initial hand-eye calibration result, the hand-eye calibration deviation caused by positioning errors, camera inherent errors, robot arm errors, and other factors and superposition effects can be effectively reduced, the relative pose between the mobile robot arm and the external camera is accurately calibrated, and the hand-eye calibration precision is effectively improved.

[0060] In one exemplary embodiment, as shown in Figure 4 Before S200, the method further includes:

[0061] S110, when the mobile chassis is at a preset initial position, a plurality of sets of calibration data of the end of the robot arm at different poses are obtained, each set of calibration data including third pose data of the end of the robot arm relative to the base of the mobile robot arm and fourth pose data of the calibration object at the end of the robot arm in the camera coordinate system of the external camera.

[0062] S120, based on the plurality of third pose data and the plurality of fourth pose data, the mobile robot base and the external camera are hand-eye calibrated to determine an initial hand-eye calibration relationship and a transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the robot arm.

[0063] The embodiment is an instruction for how to determine a preset pose transformation relationship (i.e., a transformation relationship between a coordinate system of a calibration object and a coordinate system of an end of a robot arm) and how to determine an initial hand-eye calibration relationship when a mobile robot arm is at a starting working position. Each set of calibration data includes initial pose data (i.e., third pose data) of the end of the robot arm relative to a base of the mobile robot arm and initial pose data (i.e., fourth pose data) of the calibration object at the end of the robot arm in a camera coordinate system of an external camera. The initial hand-eye calibration relationship represents a transformation relationship between a base coordinate system of the mobile robot arm and the camera coordinate system of the external camera. The preset initial position is a preset starting working position of the mobile robot arm, at which the end of the robot arm and the external camera can be calibrated once, for example, by analyzing and calculating a plurality of third pose data and fourth pose data to obtain the transformation relationship between the base coordinate system of the mobile robot arm and the camera coordinate system of the external camera, i.e., the initial hand-eye calibration relationship. However, the initial hand-eye calibration relationship is often no longer accurate due to positioning errors in the movement of the mobile chassis and other reasons, and therefore needs to be calibrated in real time. The calibration object is fixed at the end of the robot arm, and therefore the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the robot arm has relatively small errors in the movement of the mobile chassis and can be used for hand-eye calibration.

[0064] Specifically, when the mobile chassis is at the preset initial position, the controller controls the joints of the robot arm to move to different poses. At each pose, the third pose data of the end of the robot arm relative to the base of the mobile robot arm is collected by using the teach pendant, and at the same time, the fourth pose data of the calibration object in the camera coordinate system is obtained by processing the image of the calibration object captured by the external camera in combination with the inherent camera parameters of the external camera, thereby obtaining a plurality of sets of calibration data, which provides a basis for subsequent hand-eye calibration. Further, a mathematical model and an algorithm, such as a hand-eye calibration algorithm based on the least square method, can be used to solve the corresponding equation set, i.e., the transformation relationship between the base coordinate system and the coordinate system of the end of the robot arm and the transformation relationship between the coordinate system of the calibration object and the camera coordinate system, to obtain the transformation relationship between the coordinate system of the end of the robot arm and the coordinate system of the calibration object and the transformation relationship between the base coordinate and the camera coordinate system (i.e., the initial hand-eye calibration relationship) by the least square method.

[0065] In the embodiment, the plurality of sets of calibration data in different poses can comprehensively cover various states of the end of the robot arm in space, reduce the error influence caused by a single data, and more accurately calculate the transformation relationship between the coordinate system of the end of the robot arm and the coordinate system of the calibration object through comprehensive analysis and calculation of the plurality of sets of data, so as to improve the accuracy and reliability of the subsequent real-time hand-eye calibration result.

[0066] In one example embodiment, as shown in Figure 5 S120 comprises:

[0067] S121, for each set of calibration data, determining a first homogeneous transformation matrix between the coordinate system of the robot arm end and the base coordinate system of the mobile robot arm based on the third pose data, and determining a second homogeneous transformation matrix between the coordinate system of the calibration object and the camera coordinate system of the external camera based on the fourth pose data.

[0068] S122, fitting a preset calibration transformation equation based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, determining an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the robot arm end.

[0069] The calibration transformation equation represents the correlation between the first homogeneous transformation matrix, the first homogeneous transformation matrix, the initial hand-eye calibration matrix, and the third homogeneous transformation matrix. The initial hand-eye calibration relationship includes the initial hand-eye calibration matrix, and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the robot arm end includes the third homogeneous transformation matrix. The calibration transformation equation is a mathematical equation established based on the transferability of the coordinate system transformation, and is used to correlate the first homogeneous transformation matrix, the second homogeneous transformation matrix, the initial hand-eye calibration matrix and the third homogeneous transformation matrix. Its expression can be represented as formula (2):

[0070] (2)

[0071] In formula (2), represents the initial pose data of the robot arm end, and the PnP algorithm is used to calculate the second homogeneous transformation matrix of the calibration object of the robot arm end relative to the camera coordinate system , and the third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the robot arm end , and the initial hand-eye calibration matrix of the camera coordinate system relative to the base coordinate system of the mobile robot arm that has been solved can be calculated by combining formula (2).

[0072] With the above embodiment, for each set of calibration data obtained, based on the third pose data (the pose of the robot end relative to the mobile robot base) therein, the pose data is converted into a first homogeneous transformation matrix through forward kinematics calculation of the robot, which fully characterizes the spatial conversion relationship between the robot end coordinate system and the base coordinate system of the mobile robot. At the same time, based on the fourth pose data (the pose of the calibration object in the camera coordinate system), a second homogeneous transformation matrix is constructed to describe the conversion relationship between the calibration object coordinate system and the camera coordinate system. Further, the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices calculated above are substituted into the above equation (2), and a least squares method, a nonlinear optimization algorithm, or other fitting method can be used to solve the unknown quantities in equation (2), i.e., the initial hand-eye calibration matrix and the third transformation matrix. Specifically, a set of over-determined equations can be constructed using multiple sets of matrix data, and the error on both sides of the equation is minimized through an optimization algorithm, so that the calculated matrix satisfies the conversion relationship corresponding to all calibration data, and finally determines the unique initial hand-eye calibration matrix and the third transformation matrix.

[0073] It can be understood that in the subsequent process of real-time movement of the mobile chassis, the hand-eye calibration can be performed again, or the initial hand-eye calibration matrix can be directly calibrated to obtain a new hand-eye calibration matrix, thereby further improving the efficiency of hand-eye calibration.

[0074] In this embodiment, the abstract pose data can be converted into a matrix form that can be directly used for coordinate system operation through the homogeneous transformation matrix, realizing the quantitative description of the spatial relationship in different coordinate systems. By solving the unknown matrix through fitting the calibration transformation equation, a global optimal solution based on multiple sets of data is realized, the accidental error caused by single data is reduced, and the accuracy of the initial hand-eye calibration matrix and the third transformation matrix is significantly improved, thereby providing a basis for real-time calibration of the hand-eye calibration relationship of the mobile chassis in real-time operation, and significantly improving the accuracy of the hand-eye calibration process.

[0075] In one exemplary embodiment, S122 further includes: based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, fitting the preset calibration transformation equation using a least squares method to determine the initial hand-eye calibration matrix and the third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the robot end that minimizes the fitting error.

[0076] In the foregoing embodiment, the least square method is used to solve the unknown initial hand-eye calibration matrix and the third homogeneous transformation matrix in the calibration transformation equation, so that the plurality of first homogeneous transformation matrices and the second homogeneous transformation matrices are substituted into the calibration transformation equation, and the fitting error is minimized. The fitting error refers to the deviation between the actual measurement data and the theoretical model obtained by fitting. In this embodiment, after the calculated initial hand-eye calibration matrix and the third homogeneous transformation matrix are substituted into the calibration transformation equation, the difference between the matrices on both sides of the equation can be quantified by the sum of squares of the element deviations.

[0077] In this embodiment, the least square method reduces the dominant influence of single data error on the result by synthesizing the deviations of multiple sets of calibration data, minimizes the fitting error to improve the fitting quality, makes the solved initial hand-eye calibration matrix and the third homogeneous transformation matrix closer to the real coordinate system transformation relationship, and improves the accuracy of the subsequent hand-eye calibration process.

[0078] In one exemplary embodiment, as shown in Figure 6 S400 includes:

[0079] S410, acquiring image data of the calibration object collected by the external camera.

[0080] S420, extracting the circular feature from the image data, and determining the second pose data of the calibration object in the camera coordinate system of the external camera based on the circular feature and the preset camera parameters.

[0081] The surface of the calibration object can be pre-designed with a circular pattern, such as a white circle on a black background or a concentric circular ring, which is a visual mark for positioning the calibration object. The camera parameters include intrinsic and extrinsic parameters, which are used for coordinate system transformation. The pose data of the calibration object in this embodiment can refer to the geometric center on the calibration object, such as the pose data of the center of the circular pattern in the camera coordinate system, which can be calculated in combination with the image data and the camera parameters.

[0082] Specifically, when the mobile chassis is in a preset initial position and the mechanical arm moves the calibration object to different poses, the external camera is controlled to continuously shoot or trigger to shoot the calibration object at a preset frequency, and multiple frames of image data containing complete features of the calibration object are acquired. Further, an edge detection algorithm or other image processing algorithm can be used to process the image data to extract the circular feature in the image data, and then based on the preset camera parameters, the pose data of the center of the circular feature in the camera coordinate system is calculated, i.e., the second pose data of the calibration object in the camera coordinate system is determined.

[0083] In this embodiment, the circular feature has rotational invariance, which is more easily identified accurately in a complex pose than a square feature, and the fitting process of the circular feature is more simple and fast, which is suitable for real-time calibration scenarios and is conducive to improving the efficiency and accuracy of the hand-eye calibration process. In order to make the mobile manipulator hand-eye calibration method provided by the present application more clearly, a detailed embodiment and the accompanying drawings will be combined below to make a more detailed description. Figure 7 The detailed embodiment includes the following steps:

[0084] S701, in the case that the mobile chassis of the mobile manipulator is at a preset initial position, a plurality of sets of calibration data of the manipulator end of the mobile manipulator in different poses are obtained, each set of calibration data comprising third pose data of the manipulator end relative to the base coordinate system of the mobile manipulator and fourth pose data of a calibration object of the manipulator end in a camera coordinate system of an external camera.

[0085] S702, for each set of calibration data, a first homogeneous transformation matrix between the coordinate system of the manipulator end and the base coordinate system of the mobile manipulator is determined based on the third pose data, and a second homogeneous transformation matrix between the coordinate system of the calibration object and the camera coordinate system of the external camera is determined based on the fourth pose data.

[0086] S703, based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, a preset calibration transformation equation is fitted using a least squares method to determine an initial hand-eye calibration matrix that minimizes the fitting error and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the manipulator end.

[0087] S704, in the case that the mobile chassis of the mobile manipulator moves to a preset target position, the manipulator end of the mobile manipulator is controlled to move into the field of view of the external camera, and the calibration object is fixed to the manipulator end.

[0088] S705, first pose data of the manipulator end relative to the base coordinate system of the mobile manipulator and second pose data of the calibration object in the camera coordinate system of the external camera are obtained.

[0089] S706, the mobile manipulator base and the external camera are hand-eye calibrated according to the first pose data, the second pose data and the third homogeneous transformation matrix to obtain a hand-eye calibration result.

[0090] It should be understood that although each step in the flowchart involved in the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0091] Based on the same inventive concept, the embodiments of the present application also provide a mobile manipulator hand-eye calibration device for implementing the above-mentioned mobile manipulator hand-eye calibration method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more mobile manipulator hand-eye calibration device embodiments provided below can refer to the limitations of the mobile manipulator hand-eye calibration method in the above text, which will not be repeated here.

[0092] In one exemplary embodiment, as shown in Figure 8 A mobile manipulator hand-eye calibration device 800 is provided, comprising: an image acquisition module 810, a data acquisition module 820, and a hand-eye calibration module 830, wherein:

[0093] The image acquisition module 810 is configured to control the end of the manipulator of the mobile manipulator to move into the field of view of the external camera when the mobile chassis of the mobile manipulator moves to the preset target position, and the end of the manipulator is fixed with a calibration object.

[0094] The data acquisition module 820 is configured to acquire first pose data of the end of the manipulator relative to the base of the mobile manipulator, and second pose data of the calibration object in the camera coordinate system of the external camera.

[0095] The hand-eye calibration module 830 is configured to perform hand-eye calibration on the base of the mobile manipulator and the external camera according to the first pose data, the second pose data, and a preset pose transformation relationship to obtain a hand-eye calibration result, and the preset pose transformation relationship represents a transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator.

[0096] In an example embodiment, the mobile manipulator hand-eye calibration apparatus 800 is further configured to, when the mobile chassis is at a preset initial position, acquire a plurality of sets of calibration data of the end of the manipulator at different poses, each set of calibration data comprising third pose data of the end of the manipulator relative to the base of the mobile manipulator, and fourth pose data of a calibration object of the end of the manipulator in a camera coordinate system of the external camera, perform hand-eye calibration of the base of the mobile manipulator and the external camera based on the plurality of third pose data and the plurality of fourth pose data to determine an initial hand-eye calibration relationship and a transformation relationship between a coordinate system of the calibration object and a coordinate system of the end of the manipulator, the initial hand-eye calibration relationship representing a transformation relationship between a base coordinate system of the mobile manipulator and a camera coordinate system of the external camera.

[0097] In an example embodiment, the mobile manipulator hand-eye calibration apparatus 800 is further configured to, for each set of calibration data, determine, based on the third pose data, a first homogeneous transformation matrix between the coordinate system of the end of the manipulator and the base coordinate system of the mobile manipulator, and determine, based on the fourth pose data, a second homogeneous transformation matrix between the coordinate system of the calibration object and the camera coordinate system of the external camera, fit a preset calibration transformation equation based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices to determine an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the end of the manipulator, wherein the calibration transformation equation represents a correlation relationship between the first homogeneous transformation matrix, the first homogeneous transformation matrix, the initial hand-eye calibration matrix, and the third homogeneous transformation matrix, the initial hand-eye calibration relationship comprises the initial hand-eye calibration matrix, and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator comprises the third homogeneous transformation matrix.

[0098] In an example embodiment, the mobile manipulator hand-eye calibration apparatus 800 is further configured to, based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices, fit the preset calibration transformation equation using a least squares method to determine the initial hand-eye calibration matrix and the third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the end of the manipulator that minimize a fitting error.

[0099] In an example embodiment, the data acquisition module 820 is further configured to acquire image data of the calibration object captured by the external camera, extract a circular feature from the image data, and determine the second pose data of the calibration object in the camera coordinate system of the external camera based on the circular feature and preset camera parameters.

[0100] The various modules in the mobile manipulator hand-eye calibration device can be implemented by software, hardware, or a combination thereof. The various modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the various modules.

[0101] In an exemplary embodiment, a computer device, which can be a server, has an internal structure diagram as shown in Figure 9 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store pose data of an end of a manipulator relative to a base of a mobile manipulator and other data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a mobile manipulator hand-eye calibration method.

[0102] Those skilled in the art can understand that Figure 9 The structure shown in the above description is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In an exemplary embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in any one of the above mobile manipulator hand-eye calibration method embodiments.

[0104] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in any one of the above mobile manipulator hand-eye calibration method embodiments.

[0105] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any one of the above mobile robot hand-eye calibration method embodiments.

[0106] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0108] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0109] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A mobile manipulator hand-eye calibration method, comprising: The method comprises: In the case that the mobile chassis of the mobile manipulator moves to a preset target position, moving the end of the manipulator to the field of view of the external camera, the end of the manipulator being fixed with a calibration object; Obtaining first pose data of the end of the manipulator relative to the base of the mobile manipulator, and second pose data of the calibration object in the camera coordinate system of the external camera; According to the first pose data, the second pose data, and a preset pose transformation relationship, performing hand-eye calibration on the base of the mobile manipulator and the external camera to obtain a hand-eye calibration result, the preset pose transformation relationship representing the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator.

2. The method of claim 1, wherein, Before the mobile chassis moves to the preset position, the method further comprises: In the case that the mobile chassis is at a preset initial position, obtaining multiple sets of calibration data of the end of the manipulator at different poses, wherein each set of calibration data comprises third pose data of the end of the manipulator relative to the base of the mobile manipulator, and fourth pose data of the calibration object of the end of the manipulator in the camera coordinate system of the external camera; Based on multiple third pose data and multiple fourth pose data, performing hand-eye calibration on the base of the mobile manipulator and the external camera to determine an initial hand-eye calibration relationship and a transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator; The initial hand-eye calibration relationship represents the transformation relationship between the base coordinate system of the mobile manipulator and the camera coordinate system of the external camera.

3. The method of claim 2, wherein, The hand-eye calibration based on multiple third pose data and multiple fourth pose data to determine an initial hand-eye calibration relationship and a transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator comprises: For each set of calibration data, based on the third pose data, determining a first homogeneous transformation matrix between the coordinate system of the end of the manipulator and the base coordinate system of the mobile manipulator, and based on the fourth pose data, determining a second homogeneous transformation matrix between the coordinate system of the calibration object and the camera coordinate system of the external camera; Based on multiple first homogeneous transformation matrices and multiple second homogeneous transformation matrices, fitting a preset calibration transformation equation to determine an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the end of the manipulator; The calibration transformation equation represents the correlation between the first homogeneous transformation matrix, the first homogeneous transformation matrix, the initial hand-eye calibration matrix, and the third homogeneous transformation matrix, the initial hand-eye calibration relationship comprises the initial hand-eye calibration matrix, and the transformation relationship between the coordinate system of the calibration object and the coordinate system of the end of the manipulator comprises the third homogeneous transformation matrix.

4. The method of claim 3, wherein, The fitting of the preset calibration transformation equation based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices determines an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the robot arm end. The fitting of the preset calibration transformation equation based on the plurality of first homogeneous transformation matrices and the plurality of second homogeneous transformation matrices determines an initial hand-eye calibration matrix and a third homogeneous transformation matrix between the coordinate system of the calibration object and the coordinate system of the robot arm end.

5. The method according to any one of claims 1 to 4, characterized in that, The second pose data of the calibration object in the camera coordinate system of the external camera is obtained, including: The image data of the calibration object collected by the external camera is obtained. The circular feature is extracted from the image data. The second pose data of the calibration object in the camera coordinate system of the external camera is determined based on the circular feature and the preset camera parameters.

6. A mobile manipulator hand-eye calibration apparatus, comprising: The device comprises: The image acquisition module is configured to control the robot arm end of the mobile robot arm to move into the field of view of the external camera when the moving chassis of the mobile robot arm moves to the preset target position, and the robot arm end is fixed with a calibration object. The data acquisition module is configured to obtain first pose data of the robot arm end relative to the base of the mobile robot arm and second pose data of the calibration object in the camera coordinate system of the external camera. The hand-eye calibration module is configured to perform hand-eye calibration on the base of the mobile robot arm and the external camera according to the first pose data, the second pose data, and a preset pose transformation relationship to obtain a hand-eye calibration result, and the preset pose transformation relationship represents a transformation relationship between the coordinate system of the calibration object and the coordinate system of the robot arm end.

7. The apparatus of claim 6, wherein, The device is further configured to obtain a plurality of sets of calibration data of the robot arm end at different poses when the moving chassis is at a preset initial position, wherein each set of calibration data comprises third pose data of the robot arm end relative to the base coordinate of the mobile robot arm and fourth pose data of the calibration object of the robot arm end in the camera coordinate system of the external camera, and the hand-eye calibration module is configured to perform hand-eye calibration on the base of the mobile robot arm and the external camera based on the plurality of third pose data and the plurality of fourth pose data to determine an initial hand-eye calibration relationship and a transformation relationship between the coordinate system of the calibration object and the coordinate system of the robot arm end, and the initial hand-eye calibration relationship represents a transformation relationship between the base coordinate system of the mobile robot arm and the camera coordinate system of the external camera.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images