Tool teaching point position correction method and device and electronic equipment

Abstract

The invention relates to a tool teaching point position correction method and device and electronic equipment. The method comprises the following steps: acquiring a first image corresponding to a current installation tool by using an imaging system; determining a first pixel coordinate sequence corresponding to the first image according to the first image; acquiring a second pixel coordinate sequence corresponding to the historical installation tool, and acquiring an image conversion parameter of the imaging system; using the first pixel coordinate sequence, the second pixel coordinate sequence and an image conversion parameter of an imaging system to calculate a teaching point coordinate difference value of the historical installation tool and the current installation tool; and correcting the teaching point location of the historical installation tool by using the teaching point location coordinate difference value to obtain the teaching point location of the current installation tool. According to the scheme provided by the invention, the pose difference between the current installation tool and the historical installation tool can be measured based on vision, and the teaching point position is corrected by using the difference, so that the reusability of the teaching point position and the accuracy and stability of operation task execution of the tool are improved.

Description

Technical Field

[0001] This application relates to the field of mechanical automation technology, and particularly to a method, device, and electronic device for correcting the teaching point positions of tools. Background Art

[0002] There are various types of tools installed at the end of the robotic arm, such as electric grippers, pipettes, etc. During the execution of different operation tasks by the robotic arm, it is often necessary to replace the tool installed at the end of the robotic arm or the robotic arm itself. For example, when the tool fails, it is necessary to replace it with the same model to enable the operation task to continue. Another example is that when the robotic arm fails, it is necessary to replace the robotic arm itself.

[0003] However, even if the replaced tool is of the same model, due to factors such as tool installation error or tool production process error, after replacing the tool installed at the end of the robotic arm or the robotic arm itself, the position and posture of the tool installed at the end of the robotic arm or the robotic arm itself after the replacement are deviated from those before the replacement. After reusing the previous teaching point positions, due to the deviation, it is likely to affect the accuracy and stability of the robotic arm in executing operation tasks. Summary of the Invention

[0004] To solve or partially solve the problems existing in the related art, this application provides a method, device, and electronic device for correcting the teaching point positions of tools, which can measure the coordinate difference of the teaching point positions between the current installed tool and the historical installed tool by means of vision measurement, and use the coordinate difference of the teaching point positions to correct the teaching point positions of the historical installed tool, so as to reduce the difference in the teaching point positions between the current installed tool and the historical installed tool and improve the accuracy and stability of the robotic arm in executing operation tasks.

[0005] The first aspect of this application provides a method for correcting the teaching point positions of tools, and the method includes:

[0006] Collect a first image corresponding to the current installed tool by using an imaging system;

[0007] Determine a first pixel coordinate sequence corresponding to the first image according to the first image;

[0008] Obtain a second pixel coordinate sequence corresponding to the historical installed tool, and obtain the image conversion parameters of the imaging system;

[0009] Calculate the coordinate difference of the teaching point positions between the historical installed tool and the current installed tool by using the first pixel coordinate sequence, the second pixel coordinate sequence, and the image conversion parameters of the imaging system;

[0010] Correct the teaching point positions of the historical installed tool by using the coordinate difference of the teaching point positions to obtain the teaching point positions of the current installed tool.

[0011] In some embodiments, the current installation tool is a robotic arm or the current installation tool is installed on a robotic arm; before collecting the first image corresponding to the current installation tool by using the imaging system, the method further includes:

[0012] Performing a first calibration process on the imaging system to calculate the calibration parameters of the imaging system;

[0013] Using the calibration parameters to determine that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet a preset parallel requirement;

[0014] Performing a second calibration process on the imaging system to calculate the image conversion parameters of the imaging system.

[0015] In some embodiments, the using the calibration parameters to determine that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet a preset parallel requirement includes:

[0016] Using the calibration parameters to calculate the rotation parameters of the imaging system in the robotic arm base coordinate system, where the rotation parameters include: a first rotation angle of the camera of the imaging system rotating around the second axis direction of the robotic arm base coordinate system, and / or a second rotation angle of the camera of the imaging system rotating around the third axis direction of the robotic arm base coordinate system; wherein, the first axis direction, the second axis direction, and the third axis direction are pairwise orthogonal to each other;

[0017] Detecting whether the rotation parameters reach a preset precision threshold, and if so, determining that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet a preset parallel requirement.

[0018] In some embodiments, the optical axis of the camera is parallel to the first axis direction of the robotic arm base coordinate system; the first axis direction of the robotic arm base coordinate system is the Z-axis direction of the robotic arm base coordinate system, the second axis direction of the robotic arm base coordinate system is the X-axis direction of the robotic arm base coordinate system, and the third axis direction of the robotic arm base coordinate system is the Y-axis direction of the robotic arm base coordinate system.

[0019] In some embodiments, the method further includes:

[0020] If it is detected that the rotation parameters do not reach the preset precision threshold, performing an installation position correction process on the imaging system and re-performing the first calibration process on the imaging system until the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet the preset parallel requirement.

[0021] In some embodiments, performing a second calibration process on the imaging system and calculating the image conversion parameters of the imaging system includes:

[0022] Controlling the robotic arm to move at preset reference points within the same reference plane perpendicular to the first axis direction of the robotic arm base coordinate system. When the robotic arm moves to each preset reference point, taking pictures of the tool installed on the robotic arm through the imaging system to obtain a calibration image data set; the calibration image data set includes multiple calibration images;

[0023] Using the calibration image data set to calculate the image conversion parameters of the imaging system.

[0024] In some embodiments, using the calibration image data set to calculate the image conversion parameters of the imaging system includes:

[0025] Determining the feature points of each calibration image in the calibration image data set and obtaining the pixel coordinates of the feature points of each calibration image;

[0026] Obtaining the pixel coordinates of the feature points of the reference image; wherein, the reference image is obtained by taking pictures of the tool installed on the robotic arm through the imaging system when the robotic arm moves to a preset reference point within the reference plane;

[0027] Using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image to calculate the image conversion parameters of the imaging system.

[0028] In some embodiments, using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image to calculate the image conversion parameters of the imaging system includes:

[0029] Performing vector calculations on the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image respectively to obtain corresponding pixel coordinate vectors, and performing vector calculations on the robotic arm coordinates of each preset reference point and the robotic arm coordinates of the preset reference point respectively to obtain corresponding actual coordinate vectors;

[0030] According to each pixel coordinate vector and the corresponding actual coordinate vector, calculating the axis angle between the second axis direction of the camera coordinate system of the imaging system and the second axis direction of the robotic arm base coordinate system;

[0031] According to the axis angle, each pixel coordinate vector and the corresponding actual coordinate vector, calculating the image conversion parameters of the imaging system.

[0032] In some embodiments, calculating the image conversion parameters of the imaging system according to the included angle between coordinate axes, each of the pixel coordinate vectors, and the corresponding actual coordinate vector includes:

[0033] Substituting the included angle between coordinate axes, each of the pixel coordinate vectors, and the corresponding actual coordinate vector into the following formula to calculate the corresponding image conversion parameters respectively, and then averaging each of the image conversion parameters to finally obtain the image conversion parameters of the imaging system:

[0034]

[0035] where k is the image conversion parameter, beta is the included angle between coordinate axes, Δx i , Δy i is the vector value of the pixel coordinate vector, and Δx a , Δy a is the vector value of the actual coordinate vector corresponding to the pixel coordinate vector.

[0036] In some embodiments, determining the first pixel coordinate sequence corresponding to the first image according to the first image includes:

[0037] Performing feature matching or template matching on the first image to obtain a plurality of feature points included in the first image;

[0038] Constructing a first pixel coordinate sequence corresponding to the first image by using the pixel coordinates of the plurality of feature points.

[0039] In some embodiments, calculating the teaching point coordinate difference between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence, and the image conversion parameters of the imaging system includes:

[0040] Constructing a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence according to a preset rule; and constructing a second vector sequence corresponding to the second pixel coordinate sequence according to the pixel coordinates in the second pixel coordinate sequence according to the preset rule;

[0041] Calculating the installation rotation angle between the historical installation tool and the current installation tool according to the first vector sequence and the second vector sequence;

[0042] Determining a third pixel coordinate sequence corresponding to the current installation tool after reversely rotating the installation rotation angle according to the installation rotation angle and the first pixel coordinate sequence;

[0043] Calculate the coordinate difference between the teaching points of the historical installation tool and the current installation tool according to the pixel coordinate difference between the first pixel coordinate sequence and the third pixel coordinate sequence and the image conversion parameters;

[0044] The correcting the teaching points of the historical installation tool by using the coordinate difference between the teaching points, to obtain the teaching points of the current installation tool, includes:

[0045] Correct the teaching points of the historical installation tool by using the coordinate difference between the teaching points and the installation rotation angle to obtain the teaching points of the current installation tool.

[0046] In some embodiments, any pixel coordinate in the third pixel coordinate sequence is obtained according to the following two-dimensional coordinate rotation transformation formula:

[0047]

[0048] where x and y are the pixel coordinates of the first pixel coordinate sequence, alpha is the installation rotation angle, and x' and y' are the pixel coordinates of the third pixel coordinate sequence.

[0049] In some embodiments, the constructing a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence according to a preset rule includes:

[0050] Determine the positional relationship of the feature points corresponding to each pixel coordinate according to the pixel coordinates in the first pixel coordinate sequence;

[0051] Calculate the vectors of adjacent two feature points in turn according to the positional relationship of each feature point and the corresponding pixel coordinates, to construct a first vector sequence.

[0052] In some embodiments, the constructing a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence according to a preset rule includes:

[0053] Calculate the vectors of the feature points corresponding to any two pixel coordinates according to the pixel coordinates in the first pixel coordinate sequence, to construct a first vector sequence.

[0054] The second aspect of the present application provides a tool teaching point correction device, including:

[0055] A first acquisition module, configured to collect a first image corresponding to the current installation tool by using an imaging system;

[0056] A first calculation module, configured to determine a first pixel coordinate sequence corresponding to the first image according to the first image;

[0057] A second acquisition module, configured to acquire a second pixel coordinate sequence corresponding to a historical installation tool, and acquire image conversion parameters of the imaging system;

[0058] A second calculation module, configured to calculate a difference in taught point coordinates between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence, and the image conversion parameters of the imaging system;

[0059] A correction module, configured to correct the taught points of the historical installation tool by using the difference in taught point coordinates to obtain the taught points of the current installation tool.

[0060] A third aspect of the present application provides an electronic device, including:

[0061] A processor; and

[0062] A memory, storing executable code thereon, which when executed by the processor, causes the processor to execute the method as described above.

[0063] A fourth aspect of the present application provides a computer-readable storage medium, storing executable code thereon, which when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0064] The technical solution provided by the present application may include the following beneficial effects:

[0065] The technical solution of the present application can measure the difference in taught point coordinates between the current installation tool and the historical installation tool by using a vision measurement-based method, correct the taught points of the historical installation tool by using the difference in taught point coordinates, so as to reduce the difference in taught points between the current installation tool and the historical installation tool, and improve the accuracy and stability of the tool in performing operation tasks; and a large amount of manpower is required for the point teaching of the tool. Through the technical solution of the present application, the reusability of the taught points by manual teaching can be improved, the labor cost can be reduced, so that the replacement of the tool is more automated, and the operation efficiency of the tool is improved.

[0066] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] By describing the exemplary embodiments of the present application in more detail in conjunction with the drawings, the above and other objects, features, and advantages of the present application will become more obvious, wherein, in the exemplary embodiments of the present application, the same reference numerals generally represent the same components.

[0068] Figure 1It is a schematic flowchart of the tool teaching point correction method shown in the embodiments of the present application;

[0069] Figure 2 It is another schematic flowchart of the tool teaching point correction method shown in the embodiments of the present application;

[0070] Figure 3 It is a schematic diagram of the process of establishing a vector sequence in the tool teaching point correction method shown in the embodiments of the present application;

[0071] Figure 4 It is another schematic diagram of the process of establishing a vector sequence in the tool teaching point correction method shown in the embodiments of the present application;

[0072] Figure 5 It is a schematic structural diagram of the tool teaching point correction device shown in the embodiments of the present application;

[0073] Figure 6 It is another schematic structural diagram of the tool teaching point correction device shown in the embodiments of the present application;

[0074] Figure 7 It is a schematic structural diagram of the electronic device shown in the embodiments of the present application. Detailed Embodiments

[0075] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0076] The terms used in the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "the" and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0077] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this application, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0078] In the related art, due to factors such as tool installation errors or tool production process errors, after replacing the installation tool at the end of the robotic arm or replacing the robotic arm itself, the position and posture of the tool installed at the end of the robotic arm or the robotic arm itself after the above replacement process deviate from those before the replacement process. After continuing to reuse the previous taught points, due to the deviation, it is easy to affect the accuracy and stability of the robotic arm or the end tool when performing operation tasks.

[0079] In view of the above problems, the embodiments of this application provide a method for correcting taught points of a tool, which can measure the coordinate difference of the taught points of the current installed tool and the historical installed tool by means of vision measurement, and use the coordinate difference of the taught points to correct the taught points of the historical installed tool, so as to reduce the difference in the taught points between the current installed tool and the historical installed tool and improve the accuracy and stability of the tool when performing operation tasks.

[0080] The technical solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0081] Figure 1 is a schematic flowchart of the method for correcting taught points of a tool shown in the embodiments of this application. The method for correcting taught points of a tool in this application can be applied to correct the taught points of the installed tool after replacing the robotic arm itself or replacing the installation tool at the end of the robotic arm.

[0082] See Figure 1 , the method for correcting taught points of a tool in this application includes:

[0083] S101, using an imaging system to collect a first image corresponding to the current installed tool.

[0084] The imaging system is mainly used to take pictures of the current installed tool. A camera for taking pictures is set in the imaging system. That is to say, the imaging system can take pictures through the camera.

[0085] Collect a first image corresponding to the current installed tool through the camera in the imaging system.

[0086] The shooting direction of the first image can be from bottom to top or from top to bottom. It can be understood that the camera of the imaging system can take pictures of the current installation tool from below upwards or from above downwards. Of course, according to production requirements, the shooting direction of the camera in the imaging system can also be along the horizontal direction or other preset directions, and the shooting direction of the camera is not restricted here. For example, the camera in the imaging system takes pictures of the current installation tool along the horizontal direction from left to right. Another example is that the camera in the imaging system takes pictures of the current installation tool at an angle of 45° from top to bottom and inclined downwards.

[0087] It should be noted that the position of the camera of the imaging system can be fixed. For example, the camera of the imaging system is fixed on an installation plane by a fixed installation method.

[0088] Among them, the current installation tool can be the robotic arm itself; or, the current installation tool can be a tool installed on the robotic arm, such as an end effector of the robotic arm.

[0089] S102. Determine a first pixel coordinate sequence corresponding to the current installation tool according to the first image.

[0090] According to the obtained first image, image processing technology can be used to determine a first pixel coordinate sequence corresponding to the current installation tool.

[0091] Among them, the first pixel coordinate sequence can be a pixel coordinate sequence corresponding to the current installation tool part in the first image.

[0092] S103. Obtain a second pixel coordinate sequence corresponding to the historical installation tool, and obtain the image conversion parameters of the imaging system.

[0093] The historical installation tool refers to the installation tool corresponding to the current installation tool before the replacement process. Among them, the historical installation tool and the current installation tool can be the same type of tool with the same specifications, materials, functions and other characteristics. The image conversion parameters of the imaging system can refer to the relevant parameters for converting the pixels on the image taken by the camera of the imaging system into actual sizes. For example, the image conversion parameters can include pixel equivalent (that is, the conversion ratio between a single pixel on the image and the actual physical size).

[0094] The second pixel coordinate sequence can be obtained from the image acquired by photographing the historical installation tool through the imaging system before replacing the tool. In some embodiments, the second image of the historical installation tool acquired by photographing can be saved first, and then the second pixel coordinate sequence corresponding to the historical installation tool can be determined by acquiring the second image and based on the second image. In other embodiments, the second pixel coordinate sequence corresponding to the historical installation tool can be determined based on the second image first, and the second pixel coordinate sequence can be saved. When needed, the second pixel coordinate sequence can be directly acquired.

[0095] S104. Calculate the difference in the taught point coordinates between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence, and the image conversion parameters of the imaging system.

[0096] Based on the acquired first pixel coordinate sequence and second pixel coordinate sequence, the difference in pixel point coordinates between the historical installation tool and the current installation tool can be calculated. Then, combined with the image conversion parameters of the imaging system, the difference in the taught point coordinates between the historical installation tool and the current installation tool can be calculated and obtained.

[0097] It should be understood that the difference in the taught point coordinates can also refer to the actual coordinate difference between the historical installation tool and the current installation tool.

[0098] S105. Correct the taught point of the historical installation tool by using the difference in the taught point coordinates to obtain the taught point of the current installation tool.

[0099] After replacing and installing the current installation tool, by obtaining the difference in the taught point coordinates between the historical installation tool and the current installation tool, and performing corresponding addition and subtraction calculations on the taught point of the historical installation tool, the corrected taught point of the current installation tool can be obtained. That is, the corrected taught point is used as the taught point of the current installation tool.

[0100] Among them, it should be understood that obtaining the taught point requires a large amount of manpower. In this application, the taught point of the historical installation tool can be the taught point obtained in advance (before replacing the tool) through manual teaching or machine teaching. By using the tool taught point correction method of this application, the taught point applicable to the current installation tool can be obtained after correction based on the taught point of the historical installation tool, thereby improving the reusability and teaching efficiency of the taught point, reducing the labor cost, making the tool replacement process more automated, and improving the operation efficiency of the tool.

[0101] In this embodiment, the technical solution of the present application can measure the coordinate difference between the teaching points of the current installation tool and the historical installation tool in a vision measurement-based manner, and use the coordinate difference of the teaching points to correct the teaching points of the historical installation tool, so as to reduce the difference between the teaching points of the current installation tool and the historical installation tool, and improve the accuracy and stability of the tool to perform operation tasks.

[0102] Figure 2 It is another schematic flowchart of the tool teaching point correction method shown in the embodiment of the present application. The tool teaching point correction method of the present application can be applied to correct the teaching points of the installation tool after replacing the robotic arm itself or the installation tool at the end of the robotic arm.

[0103] In the tool teaching point correction method of the present application, an imaging system is applied. After replacing the robotic arm itself or the installation tool at the end of the robotic arm, in order to reduce the deviation between the camera coordinate system of the imaging system and the robotic arm base coordinate system, after replacing the robotic arm itself or the installation tool at the end of the robotic arm, the imaging system can also be calibrated first to improve the accuracy of the teaching point correction of the current installation tool by the method of the present application.

[0104] See Figure 2 , the tool teaching point correction method of the present application includes:

[0105] S201, perform a first calibration process on the imaging system, calculate the calibration parameters of the imaging system, and use the calibration parameters to determine that the first axis direction of the camera coordinate system of the imaging system reaches a preset parallel requirement with the first axis direction of the robotic arm base coordinate system.

[0106] The first calibration process is mainly used to calibrate the camera of the imaging system and calculate the calibration parameters of the imaging system. Among them, the calibration parameters of the imaging system can include the internal parameters and external parameters of the camera of the imaging system.

[0107] In some embodiments, the Zhang's calibration method or the camera calibration method based on active vision can be used to calculate the calibration parameters of the imaging system.

[0108] The Zhang's calibration method (Zhang Zhengyou calibration method) mainly uses a checkerboard calibration board to calibrate the camera. Taking the Zhang's calibration method to calculate the calibration parameters of the imaging system as an example, the process of obtaining the calibration parameters of the imaging system is described as follows:

[0109] A1, install the checkerboard calibration board at the bottom of the end of the robotic arm.

[0110] Among them, the installation method of the checkerboard calibration board can adopt methods such as pasting and screw fixation. The checkerboard calibration board is parallel to the plane at the bottom of the end of the robotic arm.

[0111] A2. Control the checkerboard calibration target to move to different poses and capture images through the camera of the imaging system to obtain multiple calibration target images.

[0112] Among them, control the movement of the robotic arm to achieve the control of the movement of the checkerboard calibration target. The number of calibration target images can be more than three. For example, the number of calibration target images is 20.

[0113] A3. Detect feature points on the multiple obtained calibration target images.

[0114] Among them, the feature points of the checkerboard calibration target can be all the corner points of the checkerboard calibration target.

[0115] A4. Solve all the internal and external parameters of the camera of the imaging system according to the results of detecting the feature points.

[0116] After obtaining the results of detecting the feature points, the internal and external parameters can be solved through corresponding functions. Among them, an open-source algorithm library can be used to solve the internal and external parameters. For example, OpenCV (Open Source Computer Vision Library) is used. After obtaining the results of detecting the feature points, call OpenCV to solve the internal and external parameters.

[0117] It can be understood that in the above Zhang's calibration method, a dot calibration target or other graphic calibration targets can also be used to replace the checkerboard calibration target to calibrate the camera, which is not limited here.

[0118] The camera calibration method based on active vision mainly calibrates the camera by using some motion information of the known camera. Specifically, control the camera to perform certain specific motions and capture multiple groups of images, and solve the internal and external parameters of the camera based on the image information and the known displacement changes. The following takes the calculation of the calibration parameters of the imaging system by the camera calibration method based on active vision as an example to elaborate the process of obtaining the calibration parameters of the imaging system:

[0119] B1. Control the camera or the object to be photographed to perform a preset displacement motion and capture multiple groups of images;

[0120] Among them, the preset displacement motion can be at least two groups of three orthogonal motions (three translational motions that are pairwise orthogonal), and the camera captures the images during the above displacement motion process.

[0121] B2. Solve the internal and external parameters of the camera according to the multiple groups of obtained images and the known displacement changes.

[0122] The displacement changes in the preset displacement motion are known, and the internal and external parameters of the camera are solved by combining the image information in the multiple groups of images.

[0123] After calibration, the internal parameter matrix and the external parameter matrix in the following formula can be obtained:

[0124]

[0125] Among them, the 3×4 matrix in the above formula (1) is the internal parameter matrix, and the 4×4 matrix is the external parameter matrix, where R is a 3×3 rotation matrix and T is a 3×1 vector (i.e., the translation vector).

[0126] Based on the obtained calibration parameters, it is determined that the direction of the first axis of the camera coordinate system of the imaging system and the direction of the first axis of the robotic arm base coordinate system meet the preset parallel requirement.

[0127] The preset parallel requirement may refer to whether the direction of the first axis of the camera coordinate system of the imaging system and the direction of the first axis of the robotic arm base coordinate system meet the preset parallel condition. For example, when the included angle between the direction of the first axis of the camera coordinate system of the imaging system and the direction of the first axis of the robotic arm base coordinate system is within a preset range, it can be regarded as meeting the preset parallel condition.

[0128] In some embodiments, the rotation parameters of the imaging system in the robotic arm base coordinate system can be calculated using the calibration parameters, and it is detected whether the rotation parameters reach a preset accuracy threshold. If so, it is determined that the direction of the first axis of the camera coordinate system of the imaging system and the direction of the first axis of the robotic arm base coordinate system meet the preset parallel requirement. It can be understood that the calculation process of the rotation parameters can be performed with the robotic arm base coordinate system as the world coordinate system.

[0129] Specifically, the rotation parameters may include a first rotation angle by which the camera of the imaging system rotates around the second axis direction of the robotic arm base coordinate system, and / or a second rotation angle by which the camera of the imaging system rotates around the third axis direction of the robotic arm base coordinate system.

[0130] In this application, the first axis direction, the second axis direction, and the third axis direction are pairwise orthogonal to each other. For example, the first axis direction, the second axis direction, and the third axis direction may correspond to the Z-axis direction, the X-axis direction, and the Y-axis direction in the same coordinate system.

[0131] Preferably, the optical axis of the camera can be parallel or tend to be parallel to the direction of the first axis of the base coordinate system of the robotic arm. When the shooting direction of the camera is from bottom to top or from top to bottom, the direction of the first axis of the base coordinate system of the robotic arm can be the Z-axis direction of the base coordinate system of the robotic arm. Correspondingly, the direction of the first axis of the camera coordinate system is the Z-axis direction of the camera coordinate system; the direction of the second axis of the base coordinate system of the robotic arm can be the X-axis direction of the base coordinate system of the robotic arm. Correspondingly, the direction of the second axis of the camera coordinate system is the X-axis direction of the camera coordinate system; the direction of the third axis of the base coordinate system of the robotic arm can be the Y-axis direction of the base coordinate system of the robotic arm. Correspondingly, the direction of the third axis of the camera coordinate system is the Y-axis direction of the camera coordinate system. It can be understood that according to the actual photographing requirements, the directions of each axis can be adaptively adjusted. For example, the direction of the first axis of the base coordinate system of the robotic arm can also be the X-axis direction or the Y-axis direction of the base coordinate system of the robotic arm. When the direction of the first axis of the base coordinate system of the robotic arm is the X-axis direction of the base coordinate system of the robotic arm, at this time, the optical axis of the camera is parallel to the X-axis of the base coordinate system of the robotic arm, and the camera can photograph the front-mounted tool from the horizontal direction.

[0132] It can be understood that it can be achieved by calculating whether the plane perpendicular to the first axis direction in the camera coordinate system of the imaging system and the plane perpendicular to the first axis direction in the base coordinate system of the robotic arm reach the preset parallel requirement, that is, by calculating the rotation angle between the second axis direction of the camera coordinate system of the imaging system and the second axis direction of the base coordinate system of the robotic arm, and / or calculating the rotation angle between the third axis direction of the camera coordinate system of the imaging system and the third axis direction of the base coordinate system of the robotic arm, so as to determine whether the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the base coordinate system of the robotic arm reach the preset parallel requirement.

[0133] For example, when the first axis direction is the Z-axis direction, the rotation angle of the camera of the imaging system around the X-axis of the base coordinate system of the robotic arm (i.e., the first rotation angle) can be calculated, and the rotation angle of the camera of the imaging system around the Y-axis of the base coordinate system of the robotic arm (i.e., the second rotation angle) can be calculated. By calculating the above two rotation angles and determining whether both reach the preset precision threshold, if so, it is determined that the first axis direction (Z-axis direction) of the camera coordinate system of the imaging system and the first axis direction (Z-axis direction) of the base coordinate system of the robotic arm reach the preset parallel requirement. Among them, the preset precision threshold can be 0.3 degrees. When the rotation angle is less than 0.3 degrees, it is considered that the rotation angle reaches the preset precision threshold.

[0134] The following takes the external parameter matrix in the calibration parameters obtained in Equation (1) as an example to describe the process of calculating the rotation angle of the camera of the imaging system around the X-axis of the base coordinate system of the robotic arm and the rotation angle of the camera of the imaging system around the Y-axis of the base coordinate system of the robotic arm:

[0135] Assume that the rotation matrix in the external parameter matrix is known, and denote:

[0136]

[0137] Assume that the rotation sequence of the camera of the imaging system around the base coordinate system of the robotic arm is to first rotate by θ around the X-axis of the base coordinate system of the robotic arm 1 and then rotate by θ around the Y-axis of the base coordinate system of the robotic arm 2 and finally rotate by θ around the z-axis of the robotic arm 3 , the following formula can be obtained:

[0138]

[0139] In formula (3), θ 1 is the rotation angle of the camera of the imaging system around the X-axis of the base coordinate system of the robotic arm, and θ 2 is the rotation angle of the camera of the imaging system around the Y-axis of the base coordinate system of the robotic arm, and θ 3 is the rotation angle of the camera of the imaging system around the Z-axis of the base coordinate system of the robotic arm.

[0140] It should be noted that when the first-axis direction (such as the Z-axis direction) of the camera coordinate system of the imaging system and the first-axis direction (such as the Z-axis direction) of the base coordinate system of the robotic arm do not meet the preset parallel requirement, for example, the rotation parameter does not reach the preset accuracy threshold, it can be considered that the deviation between the first-axis direction (such as the Z-axis direction) of the camera coordinate system of the imaging system and the first-axis direction (such as the Z-axis direction) of the base coordinate system of the robotic arm is large, and the installation position of the camera must be corrected.

[0141] In some embodiments, if it is detected that the rotation parameter does not reach the preset accuracy threshold, the installation position of the imaging system can be corrected, and the imaging system can be re-calibrated for the first time until the first-axis direction of the camera coordinate system and the first-axis direction of the base coordinate system of the robotic arm meet the preset parallel requirement.

[0142] It can be understood that the camera of the imaging system is installed on the installation plane. Among them, the optical axis of the camera can be parallel or tend to be parallel to the first-axis direction of the base coordinate system of the robotic arm. The optical axis of the camera refers to the center line of the camera lens during the photographing process. By making the optical axis of the camera parallel to the first-axis direction of the base coordinate system of the robotic arm, the imaging plane during the camera's photographing can be perpendicular to the first-axis direction of the base coordinate system of the robotic arm.

[0143] When it is detected that the rotation parameters do not reach the preset precision threshold, the installation position of the imaging system is corrected, for example, by manual correction, and the imaging system is recalibrated to ensure that the first axis direction of the camera coordinate system is parallel to the first axis direction of the robot arm base coordinate system as required, so as to achieve the first calibration of the imaging system, and further ensure that the error between the camera coordinate system and the robot arm base coordinate system of the imaging system is within the allowable range, thereby improving the accuracy of the subsequent tool teaching point correction process.

[0144] For example, when correcting the installation position of the imaging system by manual correction, the real-time image can be previewed by turning on the camera's photographing function. While adjusting the installation position of the camera, the image can be observed for auxiliary adjustment to make the first axis direction of the camera coordinate system (such as the optical axis direction of the camera) parallel to the first axis direction of the robot arm base coordinate system as required. It can be understood that the installation position of the imaging system can also be corrected by machine correction, which is not limited here.

[0145] S202. Perform the second calibration process on the imaging system to calculate the image conversion parameters of the imaging system.

[0146] The second calibration process is mainly used to calculate the image conversion parameters of the imaging system. It should be understood that the image conversion parameters refer to the relevant calculation parameters for converting pixels in the image captured by the camera in the imaging system into actual physical distances. The image conversion parameters may include, but are not limited to, pixel equivalent.

[0147] In some embodiments, the second calibration process may include the following steps:

[0148] S2021. Control the robot arm to move at preset reference points in the same reference plane perpendicular to the first axis direction of the robot arm base coordinate system. When the robot arm moves to each preset reference point, take pictures of the tool installed on the robot arm through the imaging system to obtain a calibration image dataset.

[0149] The preset reference points may be multiple grid points arranged at intervals in the same reference plane perpendicular to the first axis direction of the robot arm base coordinate system, and the positional relationship between different preset reference points is known. When the robot arm moves according to the above preset reference points, that is, the robot arm moves at a specified step length, where the step length corresponds to the distance between different preset reference points. When the robot arm moves to each preset reference point, that is, when the robot arm moves at a specified step length and pauses each time, take pictures of the tool installed on the robot arm through the imaging system to obtain a calibration image dataset. It can be understood that the calibration image dataset includes multiple calibration images.

[0150] For example, the direction of the first axis of the robotic arm base coordinate system is the Z-axis direction of the robotic arm base coordinate system. The robotic arm can be controlled to move several times along the X-axis direction of the robotic arm base coordinate system in steps of 1 mm, and the robotic arm can also be controlled to move several times along the Y-axis direction of the robotic arm base coordinate system in steps of 1 mm. After each movement of the robotic arm, it pauses and the tool installed on the robotic arm is photographed by the imaging system, so as to obtain a calibration image dataset including multiple calibration images.

[0151] S2022. Using the calibration image dataset, calculate the image conversion parameters of the imaging system.

[0152] Perform relevant calculations using the obtained calibration image dataset to obtain the image conversion parameters of the imaging system.

[0153] Specifically, the calculation process of obtaining the image conversion parameters of the imaging system may include the following steps:

[0154] S2022a. Determine the feature points of each calibration image in the calibration image dataset, and obtain the pixel coordinates of the feature points of each calibration image.

[0155] Select feature points from each calibration image in the obtained calibration image dataset. The feature points can be points on the tool installed on the robotic arm in each calibration image. It should be understood that the feature points of each calibration image are the same positions corresponding to the tool installed on the robotic arm. Among them, the feature points do not exceed the edge of the calibration image. The feature points in each calibration image can be one or more.

[0156] S2022b. Obtain the pixel coordinates of the feature points of the reference image; among them, the reference image is obtained by photographing the tool installed on the robotic arm by the imaging system when the robotic arm moves to a preset reference point in the reference plane.

[0157] The preset reference point can be a preset position in the direction directly facing the optical axis of the camera of the imaging system when the robotic arm moves. It should be noted that when the robotic arm moves to the preset reference point and the preset reference point in the reference plane, the bottom plane of the tool installed on the robotic arm photographed by the camera of the imaging system is clear.

[0158] S2022c. Using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image, calculate the image conversion parameters of the imaging system.

[0159] According to the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image, calculate the image conversion parameters of the imaging system. For example, the image conversion parameters of the imaging system can be calculated according to the difference between the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image.

[0160] In some embodiments, the process of calculating the image conversion parameters of the imaging system by using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image includes the following steps:

[0161] S2022c1: Perform vector calculations on the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image respectively to obtain the corresponding pixel coordinate vectors, and perform vector calculations on the robotic arm coordinates of each preset reference point and the robotic arm coordinates of the preset reference point respectively to obtain the corresponding actual coordinate vectors.

[0162] Assume that A and B are the pixel coordinate vector and the actual coordinate vector obtained according to any calibration image respectively. The pixel coordinate vector A can be obtained by performing vector difference calculation on the pixel coordinates of the feature points of the calibration image and the pixel coordinates of the feature points of the reference image. The actual coordinate vector B represents the actual vector between the preset reference point corresponding to the calibration image and the preset reference point corresponding to the reference image. Among them, the robotic arm coordinates of each preset reference point and the robotic arm coordinates of the preset reference point are all coordinates based on the robotic arm base coordinate system, and the corresponding actual coordinate vector is obtained by calculating the difference between the robotic arm coordinates of any preset reference point and the preset reference point.

[0163] S2022c2: Calculate the axis angle between the second axis direction of the camera coordinate system of the imaging system and the second axis direction of the robotic arm base coordinate system according to each pixel coordinate vector and the corresponding actual coordinate vector.

[0164] Preferably, if the first axis direction is the Z-axis direction, the above axis angle can be the angle between the X-axis direction of the camera coordinate system of the imaging system and the X-axis direction of the robotic arm base coordinate system.

[0165] According to the inner product formula of plane vectors:

[0166] A·B = |A||B|cos(beta) (4)

[0167] The cosine formula of the included angle beta can be obtained:

[0168]

[0169] In the two-dimensional case, let A = (x 1 , y 1 ), B = (x 2 , y 2 ), then:

[0170]

[0171] It should be understood that since both the preset reference point and the preset datum point are artificially set, and the vectors formed by the actual coordinates of the robotic arm moving between different preset reference points and between the preset reference point and the preset datum point are also known, the vector formed by the difference between the pixel coordinates of the feature points of the calibration image and the pixel coordinates of the feature points of the reference image needs to be multiplied by the pixel equivalent k to be converted into the actual physical distance. Then, A = (Δx i ·k, Δy i ·k), B = (Δx a , Δy a ).

[0172] Substituting the obtained vectors A and B into Equation (6), we can get:

[0173]

[0174] By using the inverse cosine function and combining with the above Equation (7), the axis angle beta can be calculated.

[0175] Among them, an axis angle beta can be calculated for each calibration image, and the average value of the axis angles beta calculated from the obtained multiple calibration images can be used as the axis angle beta between the second axis direction of the camera coordinate system of the imaging system and the second axis direction of the robotic arm base coordinate system.

[0176] S2022c3. According to the axis angle, each pixel coordinate vector, and the corresponding actual coordinate vector, calculate the image conversion parameters of the imaging system.

[0177] After determining the axis angle, according to the two-dimensional coordinate rotation transformation formula:

[0178]

[0179] Combining with Equation (6), and according to the pixel coordinate vector and the actual coordinate vector, it can be known that x' = Δx i ·k, y' = Δy i ·k, x = Δx a , y = Δy a . Substituting into Equation (8), we can get:

[0180]

[0181] Among them, k is the image conversion parameter, beta is the axis angle, Δx i , Δy i are the vector values of the pixel coordinate vector, and Δx a , Δy a are the vector values of the actual coordinate vector corresponding to the pixel coordinate vector.

[0182] In this way, for each calibration image, two pixel equivalents can be calculated using the above formula (9). Among them, the average value of the pixel equivalents calculated from multiple calibration images can be used as the pixel equivalent k of the imaging system.

[0183] Through the above step S201 and step S202, the first calibration process and the second calibration process can be respectively implemented for the imaging system. It can be understood that the main functions and steps of the first calibration process and the second calibration process are different, and the first calibration process and the second calibration process for the imaging system can be selected according to the actual application situation.

[0184] For example, when the current installation tool is a robotic arm, after replacing the robotic arm, the first calibration process and the second calibration process need to be performed on the imaging system again.

[0185] For example, when the current installation tool is installed on the robotic arm and the distances between the bottoms of the current installation tool and the historical installation tool and the camera of the imaging system are relatively small, after replacing the current installation tool, the first calibration process and the second calibration process may not be performed on the imaging system, that is, the results of the first calibration process and the second calibration process performed on the imaging system before the tool replacement are reused.

[0186] For another example, when the current installation tool is installed on the robotic arm and the distances between the bottoms of the current installation tool and the historical installation tool and the camera of the imaging system are relatively large, after replacing the current installation tool, only the second calibration process can be performed on the imaging system to recalculate the pixel equivalent k of the imaging system, while reusing the result of the first calibration process performed on the imaging system before the tool replacement.

[0187] S203, collect a first image corresponding to the current installation tool by the imaging system.

[0188] Collect a first image corresponding to the current installation tool by the camera in the imaging system. The first image may include a partial position of the current installation tool. For example, the first image may include the end face of the current installation tool facing the camera.

[0189] S204, perform feature matching or template matching on the first image to obtain multiple feature points included in the first image.

[0190] In some embodiments, the feature points in the first image can be obtained by recognizing the first image according to preset feature points. The feature points may be points on the current installation tool in the first image. For example, the feature points may be the geometric center points of a part of the current installation tool in the first image, or for another example, the feature points may be the edge end points of a part of the current installation tool in the first image.

[0191] In some embodiments, the matching process for obtaining multiple feature points can also adopt a template matching method.

[0192] For example, multiple pre-made image templates are used to match the first image, so as to match and obtain multiple matching regions in the obtained first image, and feature points are obtained according to the multiple matching regions, where the geometric centers of the multiple matching regions can be used as the feature points.

[0193] Of course, other methods can also be used to implement the above-mentioned feature point matching process. For example, a feature matching method can also be used to implement the feature point matching process, where the feature matching method is to find the feature correspondence between two images.

[0194] S205, using the pixel coordinates of multiple feature points, construct a first pixel coordinate sequence corresponding to the current installation tool.

[0195] According to the multiple obtained feature points, a first pixel coordinate sequence of the first image is composed of the pixel coordinates corresponding to the multiple feature points.

[0196] S206, obtain a second image corresponding to the historical installation tool, and determine a second pixel coordinate sequence corresponding to the historical installation tool according to the second image.

[0197] Among them, the principle process of obtaining the second pixel coordinate sequence according to the second image is the same as that of obtaining the first pixel coordinate sequence according to the first image, and will not be elaborated here.

[0198] Among them, the second pixel coordinate sequence can be a pixel coordinate sequence corresponding to the part of the historical installation tool in the second image. It should be understood that the pixel points in the first pixel coordinate sequence and the second pixel coordinate sequence correspond one by one, that is, the first pixel coordinate sequence and the second pixel coordinate sequence correspond to the same part of the current installation tool and the historical installation tool.

[0199] Among them, the second image can be obtained by taking a photo with the camera of the imaging system. In some embodiments, the second image can be stored in the data storage structure before replacing the current installation tool, so that it is convenient to call the second image.

[0200] In other embodiments, it can also be to obtain the second pixel coordinate sequence corresponding to the historical installation tool by obtaining the above-mentioned pre-determined second pixel coordinate sequence determined in advance according to the second image corresponding to the historical installation tool. Among them, the second pixel coordinate sequence determined in advance according to the second image can be stored in the data storage structure before replacing the current installation tool, so that the calculation amount can be reduced.

[0201] S207. Construct a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence, and construct a second vector sequence corresponding to the second pixel coordinate sequence according to the pixel coordinates in the second pixel coordinate sequence according to a preset rule.

[0202] It can be understood that the preset rules for constructing the first vector sequence corresponding to the first pixel coordinate sequence and the second vector sequence corresponding to the second pixel coordinate sequence are the same.

[0203] The position between the current installation tool in the first image and the historical installation tool in the second image can be decomposed into translation and rotation transformations. Therefore, after obtaining the first pixel coordinate sequence corresponding to the current installation tool and the second pixel coordinate sequence corresponding to the historical installation tool, the shape and size formed by connecting the corresponding pixel points in the first pixel coordinate sequence and the second pixel coordinate sequence according to the same rule remain unchanged.

[0204] Figure 3 It is a schematic diagram of the process of establishing a vector sequence in the tool teaching point correction method shown in the embodiments of the present application.

[0205] See together Figure 3 , specifically, the establishment of the first vector sequence may include the following steps:

[0206] S2071. Determine the positional relationship of the feature points corresponding to each pixel coordinate according to the pixel coordinates in the first pixel coordinate sequence.

[0207] It can be understood that the positional relationship of the feature points can be used to determine two adjacent feature points. Among them, the positional relationship of the feature points may include but is not limited to the distribution order between the feature points. The distribution order may be the distribution order in the same reference direction. For example, two adjacent feature points are determined according to the distribution order from left to right, or two adjacent feature points are determined according to the distribution order from top to bottom. The positional relationship of the feature points may also include the distance relationship between the feature points. For example, two adjacent feature points are determined according to the size of the interval distance.

[0208] S2072. Calculate the vectors of two adjacent feature points in turn according to the positional relationship of each feature point and the corresponding pixel coordinates to construct the first vector sequence.

[0209] According to the obtained positional relationship of each feature point and the corresponding pixel coordinates, connect the feature points according to the obtained positional relationship of each feature point, and calculate the vectors of two adjacent feature points in turn, so as to construct the first vector sequence.

[0210] Figure 4 It is another schematic diagram of the process of establishing a vector sequence in the tool teaching point correction method shown in the embodiments of the present application.

[0211] See also Figure 4 , in some embodiments, according to each pixel coordinate in the first pixel coordinate sequence, the vectors of the feature points corresponding to any two pixel coordinates can be calculated to construct a first vector sequence. That is to say, by connecting the feature points corresponding to any two pixel coordinates in a way that does not limit the positional relationship and calculating the vectors, more vector data can be obtained, further increasing the data volume of the first vector sequence, providing more reference data for subsequent calculations, and thus improving the accuracy of the calculation results.

[0212] In this application, the second vector sequence can be constructed in the same construction manner as the first vector sequence. The specific construction process can refer to steps S2071 and S2072, which will not be elaborated here.

[0213] S208, according to the first vector sequence and the second vector sequence, obtain the installation rotation angle between the historical installation tool and the current installation tool.

[0214] It can be understood that the vectors in the first vector sequence and the second vector sequence correspond one by one, and there is an installation rotation angle between the corresponding vectors in the first vector sequence and the second vector sequence.

[0215] By calculating the installation rotation angles between the corresponding vectors in the above-mentioned first vector sequence and the second vector sequence, multiple installation rotation angles can be obtained, and the average value can be taken as the installation rotation angle between the historical installation tool and the current installation tool.

[0216] S209, according to the installation rotation angle and the first pixel coordinate sequence, determine the third pixel coordinate sequence corresponding to the current installation tool after reverse rotation by the installation rotation angle.

[0217] After obtaining the installation rotation angle, perform a reverse rotation transformation on the first pixel coordinate sequence according to the installation rotation angle, which is equivalent to reversely rotating the image of the current installation tool according to the installation rotation angle, that is, the third pixel coordinate sequence corresponding to the current installation tool after reverse rotation by the installation rotation angle can be obtained. In this way, there is only a translation transformation between the rotated image of the current installation tool and the image of the historical installation tool.

[0218] In some embodiments, any pixel coordinate in the third pixel coordinate sequence can be obtained according to the following two-dimensional coordinate rotation transformation formula:

[0219]

[0220] where x and y are the pixel coordinates of the first pixel coordinate sequence, alpha is the installation rotation angle, and x' and y' are the pixel coordinates of the third pixel coordinate sequence.

[0221] S210. Calculate the difference in teaching point coordinates between the historical installation tool and the current installation tool based on the pixel coordinate difference between the second pixel coordinate sequence and the third pixel coordinate sequence, and the image conversion parameters.

[0222] The second pixel coordinate sequence corresponds to the position of the historical installation tool, and the third pixel coordinate sequence corresponds to the position of the current installation tool after translation relative to the historical installation tool. By calculating the pixel coordinate difference between the second pixel coordinate sequence and the third pixel coordinate sequence and taking the average value, and then combining with the pixel equivalent k of the above imaging system, the difference in teaching point coordinates between the current installation tool and the historical installation tool can be obtained.

[0223] Specifically, in the process of obtaining the difference in teaching point coordinates between the current installation tool and the historical installation tool, it can be calculated by the following formula:

[0224]

[0225] In formula (11), Δx j and Δy j are the average values of the pixel coordinate differences between the second pixel coordinate sequence and the third pixel coordinate sequence, and k is the pixel equivalent of the imaging system. Δx b and Δy b are the differences in teaching point coordinates between the current installation tool and the historical installation tool.

[0226] S211. Use the difference in teaching point coordinates and the installation rotation angle to correct the teaching point of the historical installation tool to obtain the teaching point of the current installation tool.

[0227] Among them, the installation rotation angle corresponds to the angle of rotation of the current installation tool relative to the historical installation tool. Using the difference in teaching point coordinates and the installation rotation angle, correction is performed on the basis of the teaching point of the historical installation tool to obtain the teaching point of the current installation tool.

[0228] Among them, after obtaining the teaching point of the current installation tool, corresponding rotation and translation operations can be controlled on the robotic arm to realize the correction process of the teaching point.

[0229] Assume that the teaching point of the historical installation tool before replacement is (x 0 , y 0 , z 0 , rx 0 , ry 0 , rz 0 ). Among them, the installation rotation angle alpha obtained through the previous steps, and the difference in teaching point coordinates between the historical installation tool and the current installation tool is Δx b 、Δyb , then the taught point of the current installation tool after correction is (x 0 -Δx b , y 0 -Δy b , z 0 , rx 0 , ry 0 , rz 0 -alpha).

[0230] In this embodiment, the technical solution of the present application, after replacing the robotic arm itself or the installation tool at the end of the robotic arm, performs the first calibration process and the second calibration process on the imaging system first, reduces the deviation between the camera coordinate system of the imaging system and the robotic arm base coordinate system, and improves the accuracy of correcting the taught point of the current installation tool by the method of the present application.

[0231] Corresponding to the foregoing application function implementation method embodiment, the present application also provides a tool taught point correction device, an electronic device, and corresponding embodiments.

[0232] Figure 5 is a schematic structural diagram of the tool taught point correction device shown in the embodiment of the present application.

[0233] See Figure 5 , the tool taught point correction device of the present application can be applied to correct the taught point of the installation tool after replacing the robotic arm itself or the installation tool at the end of the robotic arm. The tool taught point correction device 300 of the present application includes: a first acquisition module 310, a first calculation module 320, a second acquisition module 330, a second calculation module 340, and a correction module 350.

[0234] The first acquisition module 310 is used to collect a first image corresponding to the current installation tool by using the imaging system.

[0235] The first calculation module 320 is used to determine a first pixel coordinate sequence corresponding to the first image according to the first image;

[0236] The second acquisition module 330 is used to acquire a second pixel coordinate sequence corresponding to the historical installation tool, and acquire the image conversion parameters of the imaging system;

[0237] The second calculation module 340 is used to calculate the taught point coordinate difference between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence, and the image conversion parameters of the imaging system;

[0238] The correction module 350 is used to correct the taught point of the historical installation tool by using the taught point coordinate difference to obtain the taught point of the current installation tool.

[0239] Figure 6 It is another structural schematic diagram of the tool teaching point correction shown in the embodiments of the present application.

[0240] Refer to Figure 6 , on the basis of the device shown in Figure 5 , the tool teaching point correction device 300 of the present application further includes: a first calibration module 360 and a second calibration module 370.

[0241] The first calibration module 360 is used to perform a first calibration process on the imaging system, calculate the calibration parameters of the imaging system; and use the calibration parameters to determine that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet the preset parallel requirement;

[0242] The second calibration module 370 is used to perform a second calibration process on the imaging system, and calculate the image conversion parameters of the imaging system.

[0243] In some embodiments, the first calibration module 360 can specifically be used to calculate the rotation parameters of the imaging system in the robotic arm base coordinate system by using the calibration parameters. The rotation parameters include: a first rotation angle by which the camera of the imaging system rotates around the second axis direction of the robotic arm base coordinate system, and / or a second rotation angle by which the camera of the imaging system rotates around the third axis direction of the robotic arm base coordinate system; wherein, the first axis direction, the second axis direction, and the third axis direction are pairwise orthogonal; detect whether the rotation parameters reach a preset precision threshold, and if so, determine that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet the preset parallel requirement.

[0244] In some embodiments, the optical axis of the camera is parallel to the first axis direction of the robotic arm base coordinate system; the first axis direction of the robotic arm base coordinate system is the Z-axis direction of the robotic arm base coordinate system, the second axis direction of the robotic arm base coordinate system is the X-axis direction of the robotic arm base coordinate system, and the third axis direction of the robotic arm base coordinate system is the Y-axis direction of the robotic arm base coordinate system.

[0245] In some embodiments, if it is detected that the rotation parameters do not reach the preset precision threshold, the first calibration module 360 can also perform an installation position correction process on the imaging system, and then re-perform the first calibration process on the imaging system until the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robotic arm base coordinate system meet the preset parallel requirement.

[0246] In some embodiments, the second calibration module 370 may specifically be configured to control the robotic arm to move at a preset reference point within the same reference plane perpendicular to the first axis direction of the robotic arm base coordinate system. When the robotic arm moves to each preset reference point, the imaging system takes pictures of the tool installed on the robotic arm to obtain a calibration image data set; the calibration image data set includes multiple calibration images; and the image conversion parameters of the imaging system are calculated using the calibration image data set.

[0247] In some embodiments, the second calibration module 370 calculates the image conversion parameters of the imaging system using the calibration image data set, including: determining the feature points of each calibration image in the calibration image data set, and obtaining the pixel coordinates of the feature points of each calibration image; obtaining the pixel coordinates of the feature points of the reference image; wherein the reference image is obtained by taking pictures of the tool installed on the robotic arm by the imaging system when the robotic arm moves to a preset reference point within the reference plane; and calculating the image conversion parameters of the imaging system using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image.

[0248] In some embodiments, the second calibration module 370 calculates the image conversion parameters of the imaging system using the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image, including: performing vector calculations on the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image respectively to obtain corresponding pixel coordinate vectors, and performing vector calculations on the robotic arm coordinates of each preset reference point and the robotic arm coordinates of the preset reference point respectively to obtain corresponding actual coordinate vectors; calculating the coordinate axis angle between the second axis direction of the camera coordinate system of the imaging system and the second axis direction of the robotic arm base coordinate system according to each pixel coordinate vector and the corresponding actual coordinate vector; and calculating the image conversion parameters of the imaging system according to the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector.

[0249] In some embodiments, the second calibration module 370 calculates the image conversion parameters of the imaging system according to the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector, including: substituting the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector into the following formula respectively to calculate the corresponding image conversion parameters, and then averaging each of the image conversion parameters to finally obtain the image conversion parameters of the imaging system:

[0250]

[0251] where k is the image conversion parameter, beta is the coordinate axis angle, Δx i , Δy i is the vector value of the pixel coordinate vector, Δx a , Δya is the vector value of the actual coordinate vector corresponding to the pixel coordinate vector.

[0252] In some embodiments, the first calculation module 320 may specifically be configured to perform feature matching or template matching on the first image to obtain a plurality of feature points included in the first image; and construct a first pixel coordinate sequence corresponding to the current installation tool by using the pixel coordinates of the plurality of feature points.

[0253] In some embodiments, the second calculation module 340 may specifically be configured to construct a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence according to a preset rule; and construct a second vector sequence corresponding to the second pixel coordinate sequence according to the pixel coordinates in the second pixel coordinate sequence according to a preset rule; calculate the installation rotation angle between the historical installation tool and the current installation tool according to the first vector sequence and the second vector sequence; determine a third pixel coordinate sequence corresponding to the current installation tool after reverse rotation of the installation rotation angle according to the installation rotation angle and the first pixel coordinate sequence; calculate the teaching point coordinate difference between the historical installation tool and the current installation tool according to the pixel coordinate difference between the second pixel coordinate sequence and the third pixel coordinate sequence and the image conversion parameter;

[0254] Correspondingly, the correction module 350 may specifically be configured to correct the teaching points of the historical installation tool by using the teaching point coordinate difference and the installation rotation angle to obtain the teaching points of the current installation tool.

[0255] In some embodiments, any pixel coordinate in the third pixel coordinate sequence is obtained according to the following two-dimensional coordinate rotation transformation formula:

[0256]

[0257] where x and y are the pixel coordinates of the first pixel coordinate sequence, alpha is the installation rotation angle, and x' and y' are the pixel coordinates of the third pixel coordinate sequence.

[0258] In some embodiments, the second calculation module 340 constructs a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence according to a preset rule, including: determining the positional relationship of the feature points corresponding to the respective pixel coordinates according to the pixel coordinates in the first pixel coordinate sequence; and sequentially calculating the vectors of adjacent two feature points according to the positional relationship of the respective feature points and the corresponding pixel coordinates to construct the first vector sequence.

[0259] In some embodiments, the second computing module 340 constructs a first vector sequence corresponding to the first pixel coordinate sequence according to the pixel coordinates in the first pixel coordinate sequence, including: calculating vectors of feature points corresponding to any two pixel coordinates according to each pixel coordinate in the first pixel coordinate sequence to construct the first vector sequence.

[0260] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0261] Figure 7 It is a schematic structural diagram of an electronic device shown in an embodiment of the present application.

[0262] Refer to Figure 7 , the electronic device 1000 includes a memory 1010 and a processor 1020.

[0263] The processor 1020 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0264] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device employs a mass storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all of the instructions and data required by the processor during operation. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be employed. In some embodiments, the memory 1010 may include a removable storage device that is readable and / or writable, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, a super density disc, a flash memory card (such as SD card, min SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and instantaneous electronic signals transmitted wirelessly or by wire.

[0265] Executable code is stored on the memory 1010, and when the executable code is processed by the processor 1020, it may cause the processor 1020 to execute some or all of the methods described above.

[0266] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps of the above method according to the present application.

[0267] Alternatively, the present application may also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium), on which executable code (or a computer program or computer instruction code) is stored. When the executable code (or the computer program or computer instruction code) is executed by a processor of an electronic device (or a server, etc.), it causes the processor to execute some or all of the steps of the above method according to the present application.

[0268] The embodiments of the present application have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to technologies in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A tool teaching point correction method, characterized in that: The method comprises: Using an imaging system to capture a first image corresponding to the current installation tool; Determining, according to the first image, a first pixel coordinate sequence corresponding to the current installation tool; Acquiring a second pixel coordinate sequence corresponding to the historical installation tool, and acquiring image conversion parameters of the imaging system; Calculating the coordinate difference of the teaching point between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence and the image conversion parameter of the imaging system; The teaching point of the historical installation tool is corrected using the teaching point coordinate difference to obtain the teaching point of the current installation tool.

2. The method according to claim 1, characterized in that The current installation tool is a robotic arm or the current installation tool is installed on a robotic arm; before using the imaging system to capture the first image corresponding to the current installation tool, the method further includes: Performing a first calibration process on the imaging system, calculating calibration parameters of the imaging system, and using the calibration parameters to determine whether the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robot base coordinate system meet a preset parallel requirement; A second calibration process is performed on the imaging system to calculate and obtain image conversion parameters of the imaging system.

3. The method according to claim 2, characterized in that The method of using the calibration parameters to determine that the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robot base coordinate system meet a preset parallel requirement includes: The calibration parameters are used to calculate the rotation parameters of the imaging system in the manipulator base coordinate system, wherein the rotation parameters include: a first rotation angle of the camera of the imaging system rotating around the second axis direction of the manipulator base coordinate system, and / or a second rotation angle of the camera of the imaging system rotating around the third axis direction of the manipulator base coordinate system; wherein the first axis direction, the second axis direction, and the third axis direction are mutually orthogonal in pairs; Detect whether the rotation parameter reaches a preset accuracy threshold, and if so, determine whether the first axis direction of the camera coordinate system of the imaging system and the first axis direction of the robot base coordinate system meet a preset parallel requirement.

4. The method according to claim 3, characterized in that The optical axis of the camera is parallel to the first axis direction of the robotic arm base coordinate system; the first axis direction of the robotic arm base coordinate system is the Z axis direction of the robotic arm base coordinate system, the second axis direction of the robotic arm base coordinate system is the X axis direction of the robotic arm base coordinate system, and the third axis direction of the robotic arm base coordinate system is the Y axis direction of the robotic arm base coordinate system.

5. The method according to claim 3, characterized in that: The method further comprises: If it is detected that the rotation parameter does not reach the preset accuracy threshold, the imaging system is subjected to installation position correction processing, and the imaging system is re-calibrated until the first axis direction of the camera coordinate system and the first axis direction of the robot arm base coordinate system meet the preset parallel requirement.

6. The method according to claim 2, characterized in that The performing a second calibration process on the imaging system to calculate and obtain image conversion parameters of the imaging system includes: Control the robot arm to move according to preset reference points in the same reference plane perpendicular to the first axis direction of the robot arm base coordinate system, and take a picture of the tool installed on the robot arm through the imaging system every time the robot arm moves to a preset reference point to obtain a calibration image data set; the calibration image data set includes multiple calibration images; The image conversion parameters of the imaging system are calculated using the calibration image data set.

7. The method according to claim 6, characterized in that The step of calculating the image conversion parameters of the imaging system by using the calibration image data set includes: Determining feature points of each calibration image in the calibration image data set, and obtaining pixel coordinates of the feature points of each calibration image; Acquire pixel coordinates of feature points of a reference image; wherein the reference image is obtained by photographing a tool mounted on the robotic arm through the imaging system when the robotic arm moves to a preset reference point in the reference plane; The image conversion parameters of the imaging system are calculated using the pixel coordinates of the feature points of each of the calibration images and the pixel coordinates of the feature points of the reference image.

8. The method according to claim 7, characterized in that The step of calculating the image conversion parameters of the imaging system by using the pixel coordinates of the feature points of each of the calibration images and the pixel coordinates of the feature points of the reference image comprises: Performing vector calculation on the pixel coordinates of the feature points of each calibration image and the pixel coordinates of the feature points of the reference image to obtain corresponding pixel coordinate vectors, and performing vector calculation on the mechanical arm coordinates of each preset reference point and the mechanical arm coordinates of the preset reference point to obtain corresponding actual coordinate vectors; Calculate, according to each of the pixel coordinate vectors and the corresponding actual coordinate vector, a coordinate axis angle between a second axis direction of the camera coordinate system of the imaging system and a second axis direction of the robot arm base coordinate system; The image conversion parameters of the imaging system are calculated according to the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector.

9. The method according to claim 8, characterized in that The calculating and obtaining the image conversion parameters of the imaging system according to the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector comprises: Substitute the coordinate axis angle, each pixel coordinate vector and the corresponding actual coordinate vector into the following formula to calculate the corresponding image conversion parameters, then average the image conversion parameters to finally obtain the image conversion parameters of the imaging system: Wherein, k is the image conversion parameter, beta is the coordinate axis angle, Δx i , Δy i is the vector value of the pixel coordinate vector, Δx a , Δy a is the vector value of the actual coordinate vector corresponding to the pixel coordinate vector.

10. The method according to claim 1, characterized in that The step of determining a first pixel coordinate sequence corresponding to the current installation tool according to the first image includes: Performing feature matching or template matching on the first image to obtain a plurality of feature points contained in the first image; The pixel coordinates of the plurality of feature points are used to construct a first pixel coordinate sequence corresponding to the current installation tool.

11. The method according to any one of claims 1 to 10, characterized in that: The method of calculating the difference between the teaching point coordinates of the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence and the image conversion parameter of the imaging system comprises: Constructing a first vector sequence corresponding to the first pixel coordinate sequence according to a preset rule based on the pixel coordinates in the first pixel coordinate sequence, and constructing a second vector sequence corresponding to the second pixel coordinate sequence according to the preset rule based on the pixel coordinates in the second pixel coordinate sequence; Calculating an installation rotation angle between the historical installation tool and the current installation tool according to the first vector sequence and the second vector sequence; Determine, according to the installation rotation angle and the first pixel coordinate sequence, a third pixel coordinate sequence corresponding to the current installation tool after being reversely rotated by the installation rotation angle; Calculating the difference in coordinates of the teaching points between the historical installation tool and the current installation tool according to the pixel coordinate difference between the second pixel coordinate sequence and the third pixel coordinate sequence and the image conversion parameter; The method of correcting the teaching point of the historical installation tool by using the teaching point coordinate difference to obtain the teaching point of the current installation tool includes: The teaching point of the historical installation tool is corrected using the teaching point coordinate difference and the installation rotation angle to obtain the teaching point of the current installation tool.

12. The method according to claim 11, characterized in that Any pixel coordinate in the third pixel coordinate sequence is obtained according to the following two-dimensional coordinate rotation transformation formula: Among them, x and y are the pixel coordinates of the first pixel coordinate sequence, alpha is the installation rotation angle, and x′ and y′ are the pixel coordinates of the third pixel coordinate sequence.

13. The method according to claim 11, characterized in that The step of constructing a first vector sequence corresponding to the first pixel coordinate sequence according to a preset rule based on the pixel coordinates in the first pixel coordinate sequence includes: Determining, according to each pixel coordinate in the first pixel coordinate sequence, a positional relationship of a feature point corresponding to each pixel coordinate; According to the positional relationship of each feature point and the corresponding pixel coordinates, vectors of two adjacent feature points are calculated in sequence to construct a first vector sequence.

14. The method according to claim 11, characterized in that The step of constructing a first vector sequence corresponding to the first pixel coordinate sequence according to a preset rule based on the pixel coordinates in the first pixel coordinate sequence includes: According to each pixel coordinate in the first pixel coordinate sequence, vectors of feature points corresponding to any two pixel coordinates are calculated to construct a first vector sequence.

15. A tool teaching point correction device, characterized in that: The device comprises: A first acquisition module, used for acquiring a first image corresponding to the current installation tool by using an imaging system; A first calculation module, configured to determine a first pixel coordinate sequence corresponding to the first image according to the first image; A second acquisition module, used to acquire a second pixel coordinate sequence corresponding to the historical installation tool, and to acquire an image conversion parameter of the imaging system; A second calculation module, configured to calculate a difference in coordinates of a teaching point between the historical installation tool and the current installation tool by using the first pixel coordinate sequence, the second pixel coordinate sequence and an image conversion parameter of the imaging system; A correction module is used to correct the teaching point of the historical installation tool by using the teaching point coordinate difference to obtain the teaching point of the current installation tool.

16. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having executable code stored thereon, characterized in that: When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method according to any one of claims 1 to 14.

Images