Calibration method and device applied to manipulator end effector and three-dimensional camera, terminal and medium

Abstract

The invention discloses a calibration method and device applied to a manipulator end effector and a three-dimensional camera, a terminal and a medium. The method comprises the steps that the coarse calibration relation between the manipulator end effector and the three-dimensional camera is determined; determining a data acquisition point based on the visual field condition of the three-dimensional camera; on the basis of the rough calibration relation, the space pose of the manipulator end effector running to the data acquisition point is calculated; based on the spatial pose, determining a conversion relation between a coordinate system corresponding to a three-dimensional camera and a coordinate system corresponding to a manipulator end effector; and obtaining a target calibration result based on the data acquisition point, the coarse calibration relationship and the conversion relationship. The invention aims to carry out high-precision calibration operation on the manipulator end effector and the three-dimensional camera so as to meet use scenes with high-precision positioning requirements.

Description

Technical Field

[0001] The present application relates to the technical field of positioning of a manipulator end effector, and in particular to a calibration method, device, terminal and medium applied to a manipulator end effector and a three-dimensional camera. Background Art

[0002] At present, the end effector of industrial manipulators has the advantages of good processing consistency and high efficiency and is widely used in the field of production and manufacturing. By adding visual sensors, the scope of use of the end effector of the manipulator can be further improved, and the level of intelligent production can be improved. If you want to accurately locate and measure in the field of high precision, the accuracy of calibration is the most critical link, which will directly affect the manufacturing effect of the end effector of the industrial manipulator.

[0003] However, the current positioning and calibration technology for the end effector of the manipulator mostly uses the 3D camera external parameter calibration method and the checkerboard calibration plate for calibration. It usually requires the use of image information in the 3D camera for calibration. This process requires manual teaching to collect data for each teaching posture. The calibration accuracy is often not high enough. In addition, the accuracy consistency may not be guaranteed due to improper data collection. For scenes that require high-precision positioning and measurement, further adjustments are required to meet the project accuracy requirements. Summary of the invention

[0004] The main purpose of this application is to provide a calibration method, device, terminal and medium for a manipulator end effector and a three-dimensional camera, aiming to perform high-precision calibration operations on the manipulator end effector and the three-dimensional camera to meet the usage scenarios with high-precision positioning requirements.

[0005] To achieve the above objectives, the present application provides a calibration method for a manipulator end effector and a three-dimensional camera, the method comprising:

[0006] Based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector, a rough calibration relationship between the manipulator end effector and the 3D camera is determined;

[0007] Based on the field of view of the three-dimensional camera, determining the data collection point, wherein the data collection point is used to capture different posture changes of the end effector of the manipulator;

[0008] Based on the rough calibration relationship, calculating the spatial pose of the manipulator end effector when it runs to the data collection point;

[0009] Based on the spatial posture, determining a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the manipulator end effector;

[0010] Based on the data collection points, the rough calibration relationship and the conversion relationship, a target calibration result is calculated, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0011] Specifically, after the target calibration result is calculated based on the data collection points, the rough calibration relationship and the conversion relationship, the method further includes:

[0012] Acquire point cloud data of a preset calibration block, wherein the preset calibration block is located within the field of view of the three-dimensional camera;

[0013] The point operation is performed by controlling the end effector of the manipulator to move above the preset calibration block, and the current point coordinates of the end effector of the manipulator are recorded;

[0014] Based on the point cloud data, the current point coordinates and a preset formula, the target calibration result is updated to obtain an updated target calibration result.

[0015] Specifically, determining a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector includes:

[0016] Determining a rotation matrix according to the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector;

[0017] Determining an offset vector based on a deviation of a distance between the manipulator end effector and the plane of the three-dimensional camera on a preset coordinate axis;

[0018] Based on the rotation matrix and the offset vector, a coarse calibration matrix is ​​determined, wherein the coarse calibration matrix is ​​used to characterize a coarse calibration relationship between a manipulator end effector and a three-dimensional camera.

[0019] Specifically, the data collection points include a first data collection point, a second data collection point and a third data collection point;

[0020] The first data collection point is located at a first arc, wherein the center of the first arc is the center point of the field of view of the three-dimensional camera, and the radius of the first arc is the first standard viewing distance of the three-dimensional camera;

[0021] The second data collection point is located at a second arc, wherein the center of the second arc is the center point of the field of view of the three-dimensional camera, and the radius of the second arc is the second standard viewing distance of the three-dimensional camera;

[0022] The third data collection point is located at a third arc, wherein the center of the third arc is the center point of the field of view of the three-dimensional camera, and the radius of the third arc is the third standard viewing distance of the three-dimensional camera.

[0023] Specifically, the calculating, based on the rough calibration relationship, the spatial pose of the manipulator end effector running to the data collection point includes:

[0024] The rough calibration matrix is ​​inversely solved by coordinate transformation to obtain the spatial pose.

[0025] Specifically, determining the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector based on the spatial posture includes:

[0026] According to the principle of spatial posture consistency, the spatial posture is converted between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector to obtain a conversion relationship formula, wherein the conversion relationship formula is used to characterize the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector.

[0027] Specifically, the target calibration result is calculated based on the data collection point, the rough calibration relationship and the conversion relationship, including:

[0028] Substituting the data collection points into the conversion relationship formula to obtain a linear equation system;

[0029] Based on the conversion relationship formula, an iterative process is performed with the coarse calibration matrix as an iterative initial value to obtain an iterative result;

[0030] Repeat the steps of performing iterative processing based on the conversion relationship formula and taking the coarse calibration matrix as the iteration initial value to obtain the steps corresponding to the iterative result, until the calculation error corresponding to the iterative result is less than a preset threshold, and determine the iterative result as the target calibration result.

[0031] To achieve the above-mentioned purpose, the present application also provides a calibration device for a manipulator end effector and a three-dimensional camera, the device comprising:

[0032] The first unit is used to determine a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector;

[0033] The second unit is used to determine data collection points based on the field of view of the three-dimensional camera, wherein the data collection points are used to capture different posture changes of the end effector of the manipulator;

[0034] A third unit is used to calculate the spatial posture of the manipulator end effector running to the data collection point based on the rough calibration relationship;

[0035] A fourth unit is used to determine a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the end effector of the manipulator based on the spatial posture;

[0036] The fifth unit is used to calculate a target calibration result based on the data collection points, the rough calibration relationship and the conversion relationship, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0037] To achieve the above-mentioned purpose, the present application also provides a terminal, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any one of the methods provided in the present application.

[0038] To achieve the above objectives, the present application also provides a medium, wherein the medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in any one of the methods provided in the present application.

[0039] The present application provides a calibration method, device, terminal and medium for a manipulator end effector and a three-dimensional camera. The method can first determine a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector; determine a data acquisition point based on the field of view of the three-dimensional camera, wherein the data acquisition point is used to capture different posture changes of the manipulator end effector; then, based on the rough calibration relationship, calculate the spatial posture of the manipulator end effector when it runs to the data acquisition point; then, based on the spatial posture, determine the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector; finally, based on the data acquisition points, the rough calibration relationship and the conversion relationship, calculate a target calibration result suitable for a high-precision positioning requirement scenario, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0040] This application can perform high-precision calibration operations on the manipulator end effector and the three-dimensional camera to meet usage scenarios with high-precision positioning requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a process flow provided for an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the distribution of data collection points provided in the embodiment of the present application;

[0043] Figure 3 A schematic diagram of the arrangement of preset calibration blocks provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of the device provided in the embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0047] In the current positioning and calibration technology of the manipulator end effector, the calibration method of the 3D camera external parameter is mostly used, and the calibration is usually performed with the help of the image information in the 3D camera. This process requires manual teaching to collect data in each teaching posture, and the calibration accuracy is often not high enough. In addition, the accuracy consistency may not be guaranteed due to improper data collection. For scenes that require high-precision positioning and measurement, further adjustments are required to meet the project accuracy requirements.

[0048] Therefore, the embodiments of the present application provide a calibration method, device, terminal and medium applied to a manipulator end effector and a three-dimensional camera to solve practical technical problems.

[0049] In some embodiments, the device may be specifically integrated into an electronic device, and the electronic device may be a terminal, a server, or other device.

[0050] In some embodiments, the server may also be implemented in the form of a terminal.

[0051] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0052] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0053] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0054] The present application embodiment provides a calibration method for a robot end effector and a three-dimensional camera, such as Figure 1 As shown, the specific process of the method can be as follows:

[0055] S110 , determining a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector.

[0056] In some embodiments, obtaining a coarse calibration relationship is applicable to the first installation of the manipulator end. The coarse calibration relationship is estimated based on the installation direction of the camera and the installation deviation of the manipulator end effector. Usually when the product is shipped, the installation method is usually fixed in the same usage scenario. Therefore, it is only necessary to estimate the coarse filter calibration relationship of the installation position for the first time, and then further correct the coarse calibration relationship.

[0057] In some embodiments, the determining of the rough calibration relationship between the manipulator end effector and the 3D camera based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector includes the following steps S111 to S113:

[0058] S111 . Determine a rotation matrix according to the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector.

[0059] In some embodiments, the rotation matrix is ​​determined according to the camera installation orientation and the coordinate axis direction of the manipulator end effector.

[0060] S112: Determine an offset vector based on a deviation of a distance between the manipulator end effector and the plane of the three-dimensional camera on a preset coordinate axis.

[0061] In some embodiments, the deviation of the distance between the manipulator end effector and the three-dimensional camera plane in the preset coordinate axis x, y, z direction constitutes the offset vector of the rough calibration relationship

[0062] S113. Determine a coarse calibration matrix based on the rotation matrix and the offset vector, wherein the coarse calibration matrix is ​​used to characterize a coarse calibration relationship between the manipulator end effector and the three-dimensional camera.

[0063] In some embodiments, based on the rotation matrix and the offset vector Determine the coarse calibration matrix The coarse calibration matrix is ​​a 4×4 orthogonal matrix.

[0064] S120. Determine data collection points based on the field of view of the three-dimensional camera, wherein the data collection points are used to capture different posture changes of the end effector of the manipulator.

[0065] In some embodiments, the data collection points include a first data collection point, a second data collection point, and a third data collection point;

[0066] The first data collection point is located at a first arc, wherein the center of the first arc is the center point of the field of view of the three-dimensional camera, and the radius of the first arc is the first standard viewing distance of the three-dimensional camera;

[0067] The second data collection point is located at a second arc, wherein the center of the second arc is the center point of the field of view of the three-dimensional camera, and the radius of the second arc is the second standard viewing distance of the three-dimensional camera;

[0068] The third data collection point is located at a third arc, wherein the center of the third arc is the center point of the field of view of the three-dimensional camera, and the radius of the third arc is the third standard viewing distance of the three-dimensional camera.

[0069] Specifically, the center point of the field of view is taken as the center of the sphere, the standard viewing distance of the 3D camera is taken as the radius, and 500 mm is taken as an example, that is, the first standard viewing distance. The upper part of the center point of the field of view is selected as the reference, that is, the z direction of the preset coordinate axis, and the central angles of the circle are taken from -10° to 10° from the x and y directions of the preset coordinate axis, respectively, such as Figure 2 As shown, 9 data collection points are taken, namely, the first data collection points; the radius is reduced to 450 mm, namely, the second standard sight distance, and 9 data collection points are taken in the same way with the center angle from -10° to 10°, namely, the second data collection points; the radius is increased to 550 mm, namely, the third standard sight distance, and 9 data collection points are taken in the same way with the center angle from -10° to 10°, namely, the third data collection points, thereby obtaining a total of 27 data collection points.

[0070] S130: Based on the rough calibration relationship, calculate the spatial pose of the manipulator end effector when it runs to the data collection point.

[0071] In some embodiments, the calculating of the spatial pose of the manipulator end effector running to the data collection point based on the rough calibration relationship includes the following specific implementation process:

[0072] The rough calibration matrix is ​​inversely solved by coordinate transformation to obtain the spatial pose.

[0073] Specifically, the rough calibration relationship and the coordinate transformation method are used to perform the reverse solution to calculate the spatial pose of the industrial manipulator end effector that needs to move to the data collection point. According to the calculated spatial pose, the manipulator end effector is controlled to automatically run to each collection pose, and the data of the three-dimensional camera is collected. The spatial pose of the current manipulator end effector is recorded, and the three-dimensional point cloud data corresponding to the spatial pose of the manipulator end effector is stored. Because the rank of the matrix T is 4, the inverse matrix of T must exist, so the formula can be transformed to obtain the pose calculation method of the data collection point.

[0074] S140. Based on the spatial posture, determine the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector.

[0075] In some embodiments, the determining of the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector based on the spatial posture includes the following specific implementation process:

[0076] According to the principle of spatial posture consistency, the spatial posture is converted between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector to obtain a conversion relationship formula, wherein the conversion relationship formula is used to characterize the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector.

[0077] Specifically, the following conversion relationship formula is obtained:

[0078]

[0079] in, is the transformation matrix in the target coordinate system, which is used to represent the position and posture in the coordinate system corresponding to the end effector of the manipulator, that is, the base coordinate system; is the transformation matrix of the robot end effector (TCP point) relative to the base coordinate system, which is used to characterize the position and posture of the robot end effector; is a coarse calibration matrix, which is used to characterize the transformation relationship between the 3D camera and the robot end effector; It is the internal parameter matrix of the 3D camera itself, which represents the transformation of the camera coordinate system relative to its own installation position.

[0080] S150. Calculate a target calibration result based on the data acquisition points, the rough calibration relationship, and the conversion relationship, wherein the target calibration result is used to characterize a calibration relationship between the manipulator end effector and the three-dimensional camera.

[0081] In some embodiments, the target calibration result is calculated based on the data collection points, the rough calibration relationship and the conversion relationship, including the steps S151 to S153 as shown below:

[0082] S151, substituting the data collection points into the conversion relationship formula to obtain a linear equation group.

[0083] S152: Based on the conversion relationship formula, perform iterative processing with the coarse calibration matrix as an iteration initial value to obtain an iterative result.

[0084] S153, repeating the steps of performing iterative processing based on the conversion relationship formula and taking the coarse calibration matrix as the iteration initial value to obtain the iterative result, until the calculation error corresponding to the iterative result is less than a preset threshold, and determining the iterative result as the target calibration result.

[0085] Specifically, from the above, we can see that the variables that need to be iteratively optimized are:

[0086]

[0087] Among them, R calib is a 3×3 matrix, t calib is a 3×1 vector.

[0088] Specifically, the coarse calibration matrix is ​​used As the initial value of the iteration, the following formula is used for iteration:

[0089] X (n) =X (n-1) +ΔX (n-1)

[0090] Among them, X (n-1) is the calibration parameter vector at the n-1th iteration, ΔX (n-1) is the correction amount of this iteration, that is, the calculation error, which is determined by the first-order derivative (gradient) and second-order derivative (Hessian matrix) of the objective function; X (n) is the calibration parameter vector at the nth iteration, which contains all unknown variables in the rough calibration matrix.

[0091] In some embodiments, after the target calibration result is calculated based on the data collection points, the rough calibration relationship and the conversion relationship, the method further includes steps T1 to T3 as shown below:

[0092] T1. Obtain point cloud data of a preset calibration block, wherein the preset calibration block is located within the field of view of the three-dimensional camera.

[0093] In some embodiments, the preset calibration block is a cone with equal height and diameter. The cone diameter should be between 0.02 and 0.05 times the field of view of the 3D camera. The preset calibration blocks should be evenly distributed below the field of view of the 3D camera to be calibrated. Figure 3 As shown, the number of preset calibration blocks is selected between 6 and 20, and the 3D camera data acquisition interval includes the surface range of all preset calibration blocks.

[0094] In some embodiments, preset calibration blocks may be placed and distributed below the field of view of the three-dimensional camera.

[0095] T2. Perform point operation by controlling the manipulator end effector to move above the preset calibration block, and record the current point coordinates of the manipulator end effector.

[0096] T3. Based on the point cloud data, the current point coordinates and a preset formula, the target calibration result is updated to obtain an updated target calibration result.

[0097] In some embodiments, a preset calibration block (such as a chessboard or dot array) of known geometric dimensions is placed within the field of view of the 3D camera. The 3D camera is used to collect point cloud data of the calibration block to ensure that the feature points of the calibration block can be accurately captured. The end effector of the manipulator is manually or automatically controlled to move to a specific position above the calibration block to perform point operations. The current coordinates (x, y, z) of the robot are recorded at this time.

[0098] In some embodiments, the calibration tip coordinate formula is obtained by calculating the teaching point position and point cloud data of the manipulator end effector, as shown below:

[0099]

[0100] Among them, R base ,t n are the rotation matrix and translation vector of the preset calibration block in the base coordinate system, The transformation matrix of the calibration tip in the target coordinate system (world coordinate system or base coordinate system); is the transformation matrix of the robot end effector relative to the base coordinate system; is the calibration matrix at the n-1th iteration; It is the internal parameter matrix of the 3D camera itself, indicating the transformation of the coordinate system corresponding to the 3D camera relative to its own installation position.

[0101] Using the deviation reduction strategy, refer to the following formula:

[0102]

[0103] Specifically, ( ) is the coordinate value of the preset calibration block in the base coordinate system at the n-1th iteration, ( ) is the coordinate value of the preset calibration block in the base coordinate system at the nth iteration. In order to evaluate the accuracy of the current calibration relationship, Δx is defined n-1 , Δy n-1 and Δz n-1 Three deviation vectors that characterize the position difference between the current iteration and the previous iteration.

[0104] In order to measure the overall error, a deviation metric exp is defined to represent the square root of the sum of the squares of the deviations in the three directions of the preset coordinate axis, as shown in the following formula:

[0105]

[0106] To further optimize the calibration relationship, the target calibration result is corrected using the expression shown below, that is, after each iteration, the translation vector t in the coarse calibration matrix is ​​updated according to the deviation vector calib When the deviation metric exp is less than the preset deviation metric threshold and reaches the correction stop condition, the correction stops and the updated target calibration result T is obtained. calib .

[0107] In summary, the present application provides a calibration method for a manipulator end effector and a three-dimensional camera, which performs high-precision calibration operations on the manipulator end effector and the three-dimensional camera to meet usage scenarios with high-precision positioning requirements.

[0108] In order to better implement the above method, the embodiment of the present application also provides a calibration device for the end effector of the manipulator and the three-dimensional camera, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0109] For example, in this embodiment, the method of the embodiment of the present application will be described in detail by taking the calibration device for the manipulator end effector and the three-dimensional camera being specifically integrated in the terminal as an example.

[0110] For example, Figure 4As shown, the calibration device 400 applied to the manipulator end effector and the three-dimensional camera may include a first unit 401, a second unit 402, a third unit 403, a fourth unit 404 and a fifth unit 405, and the device includes:

[0111] The first unit 401 is used to determine a rough calibration relationship between the manipulator end effector and the 3D camera based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector;

[0112] The second unit 402 is used to determine data collection points based on the field of view of the three-dimensional camera, wherein the data collection points are used to capture different posture changes of the end effector of the manipulator;

[0113] The third unit 403 is used to calculate the spatial posture of the manipulator end effector running to the data collection point based on the rough calibration relationship;

[0114] The fourth unit 404 is used to determine the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector based on the spatial posture;

[0115] The fifth unit 405 is used to calculate a target calibration result based on the data collection points, the rough calibration relationship and the conversion relationship, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0116] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0117] It can be seen from the above that the embodiments of the present application can perform high-precision calibration operations on the manipulator end effector and the three-dimensional camera to meet the usage scenarios with high-precision positioning requirements.

[0118] The embodiment of the present application also provides an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0119] In some embodiments, the product processing device can also be integrated into multiple electronic devices. For example, the product processing device can be integrated into multiple servers, and the calibration method of the present application for the manipulator end effector and the three-dimensional camera can be implemented by multiple servers.

[0120] In this embodiment, the electronic device of this embodiment is a terminal as an example for detailed description, for example, Figure 5 As shown, it shows a schematic diagram of the structure of the terminal 500 involved in the embodiment of the present application, specifically:

[0121] The terminal 500 may include one or more processors 501 of processing cores, one or more storage media 502, a power supply 503, an input module 504, and a communication module 505. Those skilled in the art will appreciate that Figure 5 The structure of the terminal 500 shown in the figure does not constitute a limitation on the terminal 500, and the terminal 500 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0122] The processor 501 is the control center of the terminal 500. It uses various interfaces and lines to connect various parts of the entire terminal 500. It executes various functions of the terminal 500 and processes data by running or executing software programs and / or modules stored in the memory 502, and calling data stored in the memory 502, so as to monitor the terminal 500 as a whole. In some embodiments, the processor 501 may include one or more processing cores; in some embodiments, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501.

[0123] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal 500, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0124] The terminal 500 also includes a power supply 503 for supplying power to various components. In some embodiments, the power supply 503 can be logically connected to the processor 501 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 503 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0125] The terminal 500 may further include an input module 504, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0126] The terminal 500 may also include a communication module 505. In some embodiments, the communication module 505 may include a wireless module. The terminal 500 may perform short-range wireless transmission through the wireless module of the communication module 505, thereby providing the user with wireless broadband Internet access. For example, the communication module 505 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0127] Although not shown, the terminal 500 may also include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 501 in the terminal 500 will load the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 will run the application programs stored in the memory 502, thereby realizing various functions, as follows:

[0128] Based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector, a rough calibration relationship between the manipulator end effector and the 3D camera is determined;

[0129] Based on the field of view of the three-dimensional camera, determining the data collection point, wherein the data collection point is used to capture different posture changes of the end effector of the manipulator;

[0130] Based on the rough calibration relationship, calculating the spatial pose of the manipulator end effector when it runs to the data collection point;

[0131] Based on the spatial posture, determining a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the manipulator end effector;

[0132] Based on the data collection points, the rough calibration relationship and the conversion relationship, a target calibration result is calculated, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0133] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0134] It can be seen from the above that the embodiments of the present application can perform high-precision calibration operations on the manipulator end effector and the three-dimensional camera to meet the usage scenarios with high-precision positioning requirements.

[0135] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a medium and loaded and executed by a processor.

[0136] To this end, an embodiment of the present application provides a medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps of any one of the calibration methods for a manipulator end effector and a three-dimensional camera provided in the embodiment of the present application. For example, the instructions can execute the following steps:

[0137] Based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector, a rough calibration relationship between the manipulator end effector and the 3D camera is determined;

[0138] Based on the field of view of the three-dimensional camera, determining the data collection point, wherein the data collection point is used to capture different posture changes of the end effector of the manipulator;

[0139] Based on the rough calibration relationship, calculating the spatial pose of the manipulator end effector when it runs to the data collection point;

[0140] Based on the spatial posture, determining a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the manipulator end effector;

[0141] Based on the data collection points, the rough calibration relationship and the conversion relationship, a target calibration result is calculated, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

[0142] The medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0143] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, the computer instruction is stored in a medium. A processor of a computer device reads the computer instruction from the medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations provided in the above embodiments.

[0144] Since the instructions stored in the medium can execute the steps of any calibration method applied to the manipulator end effector and the three-dimensional camera provided in the embodiments of the present application, the beneficial effects that can be achieved by any calibration method applied to the manipulator end effector and the three-dimensional camera provided in the embodiments of the present application can be achieved. Please see the previous embodiments for details and will not be repeated here.

[0145] The above is a detailed introduction to the calibration method, device, terminal and medium for the end effector of a manipulator and a three-dimensional camera provided in the embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A calibration method for a manipulator end effector and a three-dimensional camera, characterized in that: The method comprises: Based on the installation direction of the 3D camera and the installation orientation of the manipulator end effector, a rough calibration relationship between the manipulator end effector and the 3D camera is determined; Based on the field of view of the three-dimensional camera, determining the data collection point, wherein the data collection point is used to capture different posture changes of the end effector of the manipulator; Based on the rough calibration relationship, calculating the spatial pose of the manipulator end effector when it runs to the data collection point; Based on the spatial posture, determining a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the manipulator end effector; Based on the data collection points, the rough calibration relationship and the conversion relationship, a target calibration result is calculated, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

2. The method according to claim 1, characterized in that After the target calibration result is calculated based on the data collection points, the rough calibration relationship and the conversion relationship, the method further includes: Acquire point cloud data of a preset calibration block, wherein the preset calibration block is located within the field of view of the three-dimensional camera; The point operation is performed by controlling the end effector of the manipulator to move above the preset calibration block, and the current point coordinates of the end effector of the manipulator are recorded; Based on the point cloud data, the current point coordinates and a preset formula, the target calibration result is updated to obtain an updated target calibration result.

3. The method according to claim 1, characterized in that The method of determining a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector comprises: Determining a rotation matrix according to the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector; Determining an offset vector based on a deviation of a distance between the manipulator end effector and the plane of the three-dimensional camera on a preset coordinate axis; Based on the rotation matrix and the offset vector, a coarse calibration matrix is ​​determined, wherein the coarse calibration matrix is ​​used to characterize a coarse calibration relationship between a manipulator end effector and a three-dimensional camera.

4. The method according to claim 1, characterized in that The data collection points include a first data collection point, a second data collection point and a third data collection point; The first data collection point is located at a first arc, wherein the center of the first arc is the center point of the field of view of the three-dimensional camera, and the radius of the first arc is the first standard viewing distance of the three-dimensional camera; The second data collection point is located at a second arc, wherein the center of the second arc is the center point of the field of view of the three-dimensional camera, and the radius of the second arc is the second standard viewing distance of the three-dimensional camera; The third data collection point is located at a third arc, wherein the center of the third arc is the center point of the field of view of the three-dimensional camera, and the radius of the third arc is the third standard viewing distance of the three-dimensional camera.

5. The method according to claim 3, characterized in that The step of calculating the spatial pose of the manipulator end effector when it moves to the data collection point based on the rough calibration relationship includes: The rough calibration matrix is ​​inversely solved by coordinate transformation to obtain the spatial pose.

6. The method according to claim 3, characterized in that The step of determining the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector based on the spatial posture includes: According to the principle of spatial posture consistency, the spatial posture is converted between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector to obtain a conversion relationship formula, wherein the conversion relationship formula is used to characterize the conversion relationship between the coordinate system corresponding to the three-dimensional camera and the coordinate system corresponding to the manipulator end effector.

7. The method according to claim 6, characterized in that The calculating and obtaining the target calibration result based on the data collection points, the rough calibration relationship and the conversion relationship includes: Substituting the data collection points into the conversion relationship formula to obtain a linear equation system; Based on the conversion relationship formula, an iterative process is performed with the coarse calibration matrix as an iterative initial value to obtain an iterative result; Repeat the steps of performing iterative processing based on the conversion relationship formula and taking the coarse calibration matrix as the iteration initial value to obtain the steps corresponding to the iterative result, until the calculation error corresponding to the iterative result is less than a preset threshold, and determine the iterative result as the target calibration result.

8. A calibration device for a manipulator end effector and a three-dimensional camera, characterized in that: The device comprises: The first unit is used to determine a rough calibration relationship between the manipulator end effector and the three-dimensional camera based on the installation direction of the three-dimensional camera and the installation orientation of the manipulator end effector; The second unit is used to determine data collection points based on the field of view of the three-dimensional camera, wherein the data collection points are used to capture different posture changes of the end effector of the manipulator; A third unit is used to calculate the spatial posture of the manipulator end effector running to the data collection point based on the rough calibration relationship; A fourth unit is used to determine a conversion relationship between a coordinate system corresponding to the three-dimensional camera and a coordinate system corresponding to the end effector of the manipulator based on the spatial posture; The fifth unit is used to calculate a target calibration result based on the data collection points, the rough calibration relationship and the conversion relationship, wherein the target calibration result is used to characterize the calibration relationship between the manipulator end effector and the three-dimensional camera.

9. A terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the method according to any one of claims 1 to 7.

10. A medium, characterized in that The medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method according to any one of claims 1 to 7.

Images