A method and system for automatic visual alignment and lamination of a robot

Abstract

A method and system for automatic visual alignment and lamination of a manipulator, wherein the method comprises: calibrating a first camera and a second camera, respectively obtaining the conversion relationship between the image coordinate systems of the first camera and the second camera and the coordinate system of the manipulator terminal; extracting the position and direction of the feature points on the lamination object from the moving image of the manipulator holding the lamination object captured by the first camera, and converting it to the manipulator terminal coordinate system according to the conversion relationship; extracting the position and direction of the feature points on the lamination target from the lamination target image captured by the second camera, and converting it to the manipulator terminal coordinate system according to the conversion relationship; calculating the lamination posture of the manipulator according to the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, and aligning the lamination object and the lamination target according to the lamination posture. The present invention improves the efficiency and accuracy of alignment and lamination.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection technology, and in particular to a method and system for automatic visual alignment and lamination of a robot. Background Art

[0002] Visual alignment technology is a technology that uses cameras to collect images of the object and target to be bonded, and algorithms to locate the corresponding object and target positions, thereby guiding the robot to automatically align and assemble. On the production line, high-precision components are assembled at high speed, which cannot be achieved with the accuracy and efficiency of the human eye, so automatic alignment and bonding technology based on visual guidance is often used.

[0003] The existing vision-based robot alignment and bonding method uses hand-eye calibration, template creation, and manual teaching, which is inefficient and limits the application conditions. Because the alignment and bonding method based on teaching, the teaching accuracy completely determines the final alignment and bonding accuracy. If the teaching is inaccurate, the final system accuracy cannot meet the assembly requirements. Moreover, in many application scenarios, there are mechanical obstructions or other restrictions during bonding, and humans cannot observe in real time and manually adjust the teaching. This teaching-based method is difficult to apply and the accuracy cannot be guaranteed. Summary of the invention

[0004] In order to solve the defects of the prior art, the purpose of the present invention is to provide a robot automatic visual alignment and bonding method and system, which can align and bond the bonding object and the bonding target without teaching modeling, thereby improving the alignment and bonding accuracy.

[0005] In order to achieve the above object, the present invention provides a method for automatic visual alignment and lamination of a robot, comprising the following steps:

[0006] Calibrate the first camera and the second camera to obtain the conversion relationship between the image coordinate system of the first camera and the second camera and the coordinate system of the manipulator end;

[0007] Extracting the position and direction of feature points on the object from the moving image of the manipulator gripping the object captured by the first camera, and converting them into the manipulator end coordinate system according to the conversion relationship;

[0008] Extracting the position and direction of feature points on the fitted target from the fitted target image captured by the second camera, and converting them into the manipulator end coordinate system according to the conversion relationship;

[0009] The bonding posture of the robot is calculated according to the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, and the bonding object and the bonding target are aligned and bonded according to the bonding posture.

[0010] Furthermore, the step of calibrating the first camera and the second camera to obtain the conversion relationship between the image coordinate system of the first camera and the second camera and the manipulator end coordinate system further includes:

[0011] In the field of view of the first camera and the second camera, the end of the manipulator holds the marker and moves in a horizontal plane according to a predetermined movement point. The first camera and the second camera collect multiple corresponding images, extract the feature points on the marker from the images, select the calibration reference position, establish an intermediate coordinate system, and obtain the affine transformation matrix from the image coordinate system of the first camera and the second camera to the corresponding intermediate coordinate system. , ;

[0012] At the calibration reference positions of the first camera and the second camera, the manipulator end clamps the marker and rotates on the horizontal plane according to the predetermined movement point position. The first camera and the second camera collect multiple corresponding images, and extract the feature points on the marker from the images to obtain the offset of the origin of the manipulator end coordinate system in the intermediate coordinate system, and obtain the offset matrix from the intermediate coordinate system to the manipulator end coordinate system. , ;

[0013] Affine transformation matrix from the image coordinate system to the intermediate coordinate system , , and the offset matrix from the intermediate coordinate system to the manipulator end coordinate system , , get the transformation matrix from the image coordinate system of the first camera and the second camera to the coordinate system of the manipulator end , .

[0014] Furthermore, in the field of view of the first camera and the second camera, the end of the manipulator holds the marker and moves in a horizontal plane according to a predetermined movement point, and the first camera and the second camera collect a plurality of corresponding images, extract the feature points on the marker from the images, select the calibration reference position, establish an intermediate coordinate system, and obtain the affine transformation matrix from the image coordinate system of the first camera and the second camera to the corresponding intermediate coordinate system. , The steps further include:

[0015] Extracting feature points on the marker from the captured image to obtain coordinates in the image coordinate system;

[0016] Selecting a calibration reference position; the calibration reference position is the manipulator coordinate when the feature point of the marker is located at the center of the camera field of view;

[0017] On the plane where the motion point is located, an intermediate coordinate system is established with the calibrated reference position as the origin and the directions parallel to the horizontal and vertical axes of the manipulator base coordinate system as the horizontal and vertical axes;

[0018] Subtract the coordinates of the calibrated reference position from the coordinates of all the manipulator translation points to obtain the coordinates of the movement points in the intermediate coordinate system;

[0019] The affine transformation matrix from the image coordinate system to the intermediate coordinate system is calculated using the least squares method through one-to-one correspondence between the point coordinates in the image coordinate system and the point coordinates in the intermediate coordinate system.

[0020] Furthermore, at the calibration reference positions of the first camera and the second camera, the manipulator end clamps the marker and rotates on the horizontal plane according to the predetermined movement point position, and the first camera and the second camera collect a plurality of corresponding images, and extract the feature points on the marker from the images, and obtain the offset of the origin of the manipulator end coordinate system in the intermediate coordinate system, that is, the offset matrix from the intermediate coordinate system to the manipulator end coordinate system is obtained. , The steps further include:

[0021] Extracting feature points on the marker from the captured image to obtain coordinates in the image coordinate system;

[0022] Apply the obtained affine transformation matrix from the image coordinate system to the intermediate coordinate system , , transform the coordinates in the image coordinate system to the intermediate coordinate system;

[0023] In the intermediate coordinate system, the least squares method is used to fit the coordinates of the center of the circle, that is, the coordinates of the origin of the manipulator end coordinate system in the intermediate coordinate system, and the offset matrix from the intermediate coordinate system to the manipulator end coordinate system are obtained. , .

[0024] Furthermore, by establishing the intermediate coordinate system, the affine transformation matrix from the camera image coordinate system to the intermediate coordinate system is completed in two steps: , Calibration, and the offset matrix from the intermediate coordinate system to the robot end coordinate system , Calibrate, and then obtain the transformation matrix from the image coordinate system of the first camera and the second camera to the coordinate system of the manipulator end , .

[0025] Furthermore, the step of extracting the position and direction of feature points on the fitting object and the fitting target further includes: using a feature positioning method based on line search, circle search, or template matching to extract the position and direction of feature points from the corresponding image.

[0026] Furthermore, the step of obtaining the fitting posture of the manipulator according to the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, further includes:

[0027] The following formula is used to calculate the robot's fitting posture :

[0028] ,

[0029] in, are the coordinates and angles of the end of the robot when it is fitted. are the coordinates and angles of the feature points of the fitted object in the robot end coordinate system, are the coordinates and angles of the feature points of the fitting target in the robot end coordinate system, are the horizontal and vertical coordinates and angles of the first camera calibration reference position, The horizontal and vertical coordinates and angles of the reference position are calibrated for the second camera respectively.

[0030] On the other hand, the present invention also provides a robot automatic visual alignment and bonding system, which adopts the robot automatic visual alignment and bonding method as described above, comprising:

[0031] A robot arm having a mobile gripping unit;

[0032] A first camera unit, used for collecting images of the bonding object clamped by the end of the manipulator within the moving range;

[0033] A second camera unit, used to collect images of the fitting target;

[0034] An image extraction unit, configured to receive images captured by the first camera unit and the second camera unit, extract feature point information from the images, and output the feature point information;

[0035] A data processing unit, used to receive the feature point information sent by the image extraction unit, and respectively establish a conversion relationship between the image coordinate system of the first camera unit and the second camera unit and the manipulator end coordinate system; according to the conversion relationship between the image coordinate system and the manipulator end coordinate system, the feature point information is converted to the manipulator end coordinate system, and the manipulator fitting posture is calculated and output;

[0036] The control unit is used to control the robot to align and bond the bonding object with the bonding target according to the bonding posture of the robot.

[0037] The automatic visual alignment and laminating method of the robot provided by the present invention has the following beneficial effects compared with the prior art:

[0038] By calibrating the first camera and the second camera, the conversion relationship between the image coordinate system of the first camera and the second camera and the coordinate system of the manipulator end is obtained respectively; from the moving image of the manipulator holding the bonding object captured by the first camera, the position and direction of the feature points on the bonding object are extracted, and converted to the manipulator end coordinate system according to the conversion relationship; from the bonding target image captured by the second camera, the position and direction of the feature points on the bonding target are extracted, and converted to the manipulator end coordinate system according to the conversion relationship; according to the calibration data of the first camera and the second camera, and the corresponding feature point position and direction, the bonding posture of the manipulator is calculated, and the bonding object and the bonding target are aligned and bonded according to the bonding posture. Without manual teaching and modeling, the alignment and bonding of the object and the target can be completed accurately, which improves the efficiency and accuracy of the alignment and bonding of the manipulator.

[0039] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0041] Figure 1 This is a flow chart of a method for automatic visual alignment and bonding of a robot according to an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of camera calibration according to an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of a straight line search method according to an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a circle search method according to an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a template matching search method according to an embodiment of the present invention;

[0046] Figure 6 Schematic diagram of the structure of a robot automatic visual alignment and bonding system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0048] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0049] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0050] It should be noted that the concepts of “first”, “second”, etc. may be mentioned in the present invention only to distinguish different devices, components or parts, and are not used to limit the order or interdependence of the functions performed by these devices, components or parts.

[0051] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more". "Plurality" should be understood as two or more.

[0052] In an embodiment of the present invention, a method for automatic visual alignment and bonding of a robot is provided, comprising the following steps: calibrating a first camera and a second camera to obtain the conversion relationship between the image coordinate systems of the first camera and the second camera and the coordinate system of a robot end; extracting the position and direction of feature points on the bonding object from the moving image of the robot gripping the bonding object captured by the first camera, and converting them to the coordinate system of the robot end according to the conversion relationship; extracting the position and direction of feature points on the bonding target from the bonding target image captured by the second camera, and converting them to the coordinate system of the robot end according to the conversion relationship; calculating the bonding posture of the robot based on the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, and aligning and bonding the bonding object and the bonding target according to the bonding posture.

[0053] Example 1

[0054] Figure 1 The flowchart of the automatic visual alignment and bonding method of the robot according to the embodiment of the present invention is as follows. Figure 1 The specific embodiments of the present invention are described in further detail.

[0055] In step S1 , the hand-eye calibration is performed on the first camera 1 (the positioning camera for the end of the manipulator to fit the object 3 ) to obtain the conversion relationship between the image coordinate system of the first camera 1 and the coordinate system of the end of the manipulator.

[0056] Figure 2 FIG. 1 is a schematic diagram of camera calibration according to an embodiment of the present invention. Figure 2 As shown, the end of the manipulator clamps the bonding object 3 (i.e., the marker) and moves horizontally in the field of view of the first camera 1 according to the predetermined movement point, and a plurality of corresponding images are collected by the first camera 1. Specifically, according to different embodiments, the camera field of view is different from the movement range of the manipulator, and the movement point can be different.

[0057] In the embodiment of the present invention, the translational motion point of the manipulator is recorded as , , , the angle remains unchanged, that is Since the center of the camera field of view has the smallest distortion and the highest accuracy, the calibration reference position should be selected at the manipulator coordinates when the feature point of the marker is located at the center of the camera field of view to improve the alignment accuracy. The coordinates are marked as , the angle is recorded as The moving points should be evenly distributed around the calibration reference position and completely cover the possible range of variation of the bonding object 3 within the field of view of the first camera 1 to ensure the detection accuracy of the subsequent actual movement of the bonding object 3. At the same time, in order to reduce the system error and ensure accuracy, the angle of the manipulator during translation should be consistent with the selected calibration reference angle, that is, .

[0058] Preferably, 9 moving points are selected and distributed in a 3×3 grid pattern, and the first point coincides with the calibration reference position, such as Figure 2 shown.

[0059] In each collected image, extract the feature point position on the fitting object 3 Specifically, according to different embodiments, feature positioning methods such as line search, circle search, or template matching may be used to extract feature point positions, and other positioning algorithms having the same function as the present embodiment may also be selected according to actual conditions.

[0060] refer to Figure 3When there are fixed straight line features in the marker image, the straight line search method can be used to extract feature points. The main steps are: S1. Define the approximate straight line detection direction and area; S2. Generate positioning detection moments at equal intervals in the area according to the set parameters; S3. Take points in each positioning detection moment to generate a profile contour line; S4. Derivate the contour line to find the extreme value point of the derivative, that is, the edge point; S5. Perform straight line fitting on the edge points to obtain the edge straight line equation; S6. Repeat the above steps to obtain two non-parallel straight lines; S7. Find the horizontal coordinate u and vertical coordinate v of the intersection of the two straight lines as the feature point .

[0061] refer to Figure 4 When there are fixed circle features in the marker image, the circle search method can be used to extract feature points. The main steps are: S1. Define the approximate circle detection area; S2. Generate positioning detection moments in the area at equal intervals along the circumference according to the set parameters; S3. Take points in each positioning detection moment to generate a profile contour line; S4. Derivate the contour line to find the extreme value point of the derivative, that is, the edge point; S5. Perform circle fitting on the edge point to obtain the center of the circle as the feature point .

[0062] refer to Figure 5 When there are fixed irregular texture features in the marker image, the 2D template matching method can be used to extract feature points. The main steps are: S1. Use the area with fixed and obvious texture features to create a grayscale matching template; S2. Perform template matching in the image and obtain the matching position as the feature point .

[0063] Continue to refer Figure 2 The image coordinate system of the camera and the coordinate system of the end of the manipulator are in an affine transformation relationship. The coordinate system of the end of the manipulator is generally the center of its end flange, which cannot be directly observed. Therefore, it is necessary to solve the positioning through calibration. In the plane where the motion point is located, take the calibration reference position as the origin, and take the horizontal and vertical axes parallel to the horizontal and vertical axes of the manipulator base coordinate system as the horizontal and vertical axes to establish an intermediate coordinate system. ,like Figure 2 As shown, the end coordinates of the manipulator in this coordinate system are ,in, is the coordinate of the translational motion point at the end of the original manipulator, To calibrate the reference position, the affine transformation matrix from the image coordinate system to the intermediate coordinate system is The least square method can be used to find , Transform all image points to the intermediate coordinate system After the actual coordinate value The sum of the squares of the distances represents the transformation error. The optimal transformation matrix is ​​obtained by minimizing this error. ,in is the image coordinate of the feature point, is the number of feature points, The value range is , is the affine transformation matrix between two plane coordinate systems, .

[0064] There is only an x, y offset relationship between the robot end coordinate system and the intermediate coordinate system. In order to calibrate the offset, it is sufficient to calibrate the rotation center of the robot end. , rotates around the z-axis according to a predetermined angle, and collects corresponding images through the first camera 1. Specifically, according to different embodiments, the camera field of view is different from the movement range of the manipulator, and the rotation angle may be different, which is recorded as , , The larger the rotation angle range, the higher the calibration accuracy.

[0065] Preferably, the rotation angle is 5, and the reference position angle is calibrated Distribute evenly on both sides.

[0066] In each collected rotation image, similarly, the feature point positions on the fitted object 3 are extracted. . For image coordinates Apply the affine transformation matrix obtained above ,get , The intermediate coordinate system The coordinates on . The least squares method is used to fit a circle to the coordinate array, that is, , Substitute the sum of squares of the errors of all points into the algebraic equation of the circle to represent the fitting error. The optimal circle equation is obtained by minimizing this error, where is the number of points, and the value range of i is , are the first-order terms of the target circle algebraic equation and coefficient, is the constant term of the target circular algebraic equation, that is, the circular algebraic equation is obtained , and thus obtain the coordinates of the circle center , , that is, the rotation center of the end of the robot flange (that is, the origin of the robot end coordinate system) is in the middle coordinate system The coordinates are obtained by the offset matrix from the intermediate coordinate system to the robot end coordinate system. .

[0067] Affine transformation matrix from image coordinate system to intermediate coordinate system , and the offset matrix from the intermediate coordinate system to the robot end coordinate system , we can get the direct transformation matrix from the image coordinate system to the robot end coordinate system , complete the hand-eye calibration of the first camera 1.

[0068] In step S2, the hand-eye calibration of the second camera 2 (the positioning camera for the attached target 4) is performed to obtain the conversion relationship between the image coordinate system of the second camera 2 and the coordinate system of the end of the manipulator.

[0069] In the embodiment of the present invention, referring to the calibration method of the first camera 1 in step S1 above, the hand-eye calibration of the camera 2 is completed, and the calibration reference position of the second camera 2 is obtained. ,angle , and the transformation matrix from the image coordinate system of the second camera 2 to the manipulator end coordinate system .

[0070] In step S3, the calibrated first camera 1 is used to capture images, obtain the positions and directions of feature points on the bonding object, and convert them into the robot end coordinate system.

[0071] In the embodiment of the present invention, according to different embodiments, feature positioning methods such as line search, circle search, or template matching can be used (refer to the above step S1 for the specific process) to extract the feature point position, and other positioning algorithms with the same function as this embodiment can also be selected according to actual conditions. The conversion relationship between the image coordinate system of the first camera 1 calibrated in step S1 and the coordinate system of the end of the manipulator , and convert the extracted feature point position and direction into the robot end coordinate system.

[0072] For example, we can use the straight line search method to extract feature points. , where one straight line direction is used as the characteristic direction ; or use the circle search method to extract feature points , the direction of the line connecting the two centers or the straight line is used as the feature direction ; or use 2D template matching method to extract feature points , matching angle as feature direction . Use the hand-eye calibration information of the first camera 1 to fit the feature points of the object 3 The image coordinates are converted to the robot end coordinate system to obtain the coordinate point , feature direction Convert to the robot end coordinate system to get the angle direction ,in Representation vector angular direction.

[0073] In step S4, the calibrated second camera 2 is used to capture images, obtain the positions and directions of feature points on the fitting target 4, and convert them into the robot end coordinate system.

[0074] In the embodiment of the present invention, according to different embodiments, feature positioning methods such as line search, circle search, template matching, etc. can be used (refer to the above step S1 for the specific process) to extract the feature point position, and other positioning algorithms with the same function as this embodiment can also be selected according to actual conditions. The conversion relationship between the image coordinate system of the second camera 2 calibrated in step S2 and the coordinate system of the end of the manipulator , the extracted feature point position and direction are converted to the robot end coordinate system. Similarly, the feature points that fit the target 4 can be obtained. Coordinates in the robot end coordinate system , feature direction Convert to the robot end coordinate system to get the direction .

[0075] In step S5, the robot arm bonding posture is obtained according to the hand-eye calibration data of the first camera 1 and the second camera 2, and the corresponding feature point information, and the bonding object 3 and the bonding target 4 are aligned and bonded according to the bonding posture.

[0076] In the embodiment of the present invention, it is assumed that the (current) end posture of the manipulator during lamination is , where x and y are the horizontal and vertical coordinates of the end of the robot during bonding, is the angle of the end of the robot during bonding;

[0077] At this time, the coordinates of the feature points of the bonding object 3 in the robot base coordinate system are

[0078] ,

[0079] That is, the first camera 1 calibrates the feature point rotation in the manipulator end coordinate system After the angle, add the current robot end coordinates.

[0080] The feature direction of the bonding object 3 in the robot base coordinate system is

[0081] ;

[0082] The original angle Rotation The angle value after angle.

[0083] The coordinates of the feature points of the fitting target 4 in the manipulator base coordinate system are

[0084] ,

[0085] That is, the coordinates of the feature points in the end coordinate system of the robot calibrated by the second camera 2 are added to the coordinates of the reference position of the robot calibrated by the second camera 2.

[0086] The feature direction of the fitting target 4 in the manipulator base coordinate system is

[0087] ,

[0088] Since the coordinate system of the end of the calibrated manipulator is parallel to the coordinate axis of the manipulator base coordinate system, the angle in the manipulator base coordinate system is consistent with the angle in the coordinate system of the end of the calibrated manipulator.

[0089] When the features correspond one to one, the features of the fitting object 3 and the fitting target 4 coincide with each other, that is,

[0090]

[0091]

[0092]

[0093] The robot arm fitting posture can be obtained .

[0094] The automatic vision alignment and bonding method for a robot provided by the present invention does not require manual teaching and modeling, and solves the problem that in the application of automatic vision-based alignment and bonding for a robot, due to occlusion or other factors, manual observation and adjustment are impossible during the bonding action, resulting in difficulty in manual teaching, thereby improving the detection efficiency and alignment and bonding accuracy.

[0095] Example 2

[0096] In an embodiment of the present invention, a robot automatic visual alignment and bonding system is also provided, which adopts the robot automatic visual alignment and bonding method steps of the above embodiment to achieve high-precision robot alignment and bonding without the need for teaching modeling.

[0097] Figure 6 Schematic diagram of the structure of the automatic visual alignment and bonding system of the robot according to an embodiment of the present invention. Figure 6 As shown, the robot automatic visual alignment and bonding system 600 of the embodiment of the present invention includes:

[0098] A robot 601 having a mobile gripping unit;

[0099] The first camera unit 602 is used to capture images of the bonding object held by the end of the manipulator 601 within the moving range;

[0100] The second camera unit 603 is used to collect images of the fitting target;

[0101] The image extraction unit 604 is used to receive the images captured by the first camera unit 602 and the second camera unit 603, extract feature point information (including position and direction) from the images and output them;

[0102] The data processing unit 605 is used to receive the feature point information sent by the image extraction unit 604, establish the conversion relationship between the image coordinate system and the robot end coordinate system; according to the conversion relationship between the image coordinate system and the robot end coordinate system, convert the feature point information to the robot end coordinate system, calculate the robot fitting posture and output it;

[0103] The control unit 606 is used to control the robot 601 to complete the alignment and bonding of the bonding object and the bonding target according to the bonding posture of the robot.

[0104] In the embodiment of the present invention, the image extraction unit 604 extracts feature point information from the image using a feature location method based on line search, circle search, or template matching. The specific method steps based on line search, circle search, or template matching refer to the above embodiment.

[0105] Those skilled in the art can understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions recorded in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A robot automatic visual alignment and lamination method, characterized in that: The following steps are involved: Calibrate the first camera and the second camera to obtain the conversion relationship between the image coordinate system of the first camera and the second camera and the coordinate system of the manipulator end; Extracting the position and direction of feature points on the object from the moving image of the manipulator gripping the object captured by the first camera, and converting them into the manipulator end coordinate system according to the conversion relationship; Extracting the position and direction of feature points on the fitted target from the fitted target image captured by the second camera, and converting them into the manipulator end coordinate system according to the conversion relationship; According to the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, the bonding posture of the manipulator is calculated, and the bonding object and the bonding target are aligned and bonded according to the bonding posture; The step of calibrating the first camera and the second camera to obtain the conversion relationship between the image coordinate system of the first camera and the second camera and the coordinate system of the manipulator end further includes: In the field of view of the first camera and the second camera, the end of the manipulator holds the marker and moves in a horizontal plane according to a predetermined movement point. The first camera and the second camera collect multiple corresponding images, extract the feature points on the marker from the images, select the calibration reference position, establish an intermediate coordinate system, and obtain the affine transformation matrix from the image coordinate system of the first camera and the second camera to the corresponding intermediate coordinate system. , ; At the calibration reference positions of the first camera and the second camera, the manipulator end clamps the marker and rotates on the horizontal plane according to the predetermined movement point position. The first camera and the second camera collect multiple corresponding images, and extract the feature points on the marker from the images to obtain the offset of the origin of the manipulator end coordinate system in the intermediate coordinate system, and obtain the offset matrix from the intermediate coordinate system to the manipulator end coordinate system. , ; Affine transformation matrix from the image coordinate system to the intermediate coordinate system , , and the offset matrix from the intermediate coordinate system to the manipulator end coordinate system , , get the transformation matrix from the image coordinate system of the first camera and the second camera to the coordinate system of the manipulator end , ; The calibration reference position is the coordinates of the manipulator when the feature point of the marker is located at the center of the camera's field of view; The moving points are evenly distributed around the calibration reference position and completely cover the possible variation range of the lamination object within the field of view of the first camera; The angle of the manipulator during translational movement is consistent with the angle of the selected calibration reference position; The step of establishing the intermediate coordinate system is to establish the intermediate coordinate system in the plane where the moving point is located, with the calibration reference position as the origin, and with the horizontal axis and vertical axis parallel to the horizontal axis of the manipulator base coordinate system as the horizontal axis and vertical axis.

2. The automatic visual alignment and laminating method of a robot according to claim 1, characterized in that: In the field of view of the first camera and the second camera, the end of the manipulator holds the marker and moves in a horizontal plane according to a predetermined movement point, and the first camera and the second camera collect a plurality of corresponding images, extract the feature points on the marker from the images, select the calibration reference position, establish an intermediate coordinate system, and obtain the affine transformation matrix from the image coordinate system of the first camera and the second camera to the corresponding intermediate coordinate system , The steps further include: Extracting feature points on the marker from the captured image to obtain coordinates in the image coordinate system; Selecting a calibration reference position; the calibration reference position is the manipulator coordinate when the feature point of the marker is located at the center of the camera field of view; On the plane where the motion point is located, an intermediate coordinate system is established with the calibration reference position as the origin and directions parallel to the horizontal and vertical axes of the manipulator base coordinate system as the horizontal and vertical axes; Subtract the coordinates of the calibrated reference position from the coordinates of all the manipulator translation points to obtain the coordinates of the movement points in the intermediate coordinate system; The affine transformation matrix from the image coordinate system to the intermediate coordinate system is calculated using the least squares method through one-to-one correspondence between the point coordinates in the image coordinate system and the point coordinates in the intermediate coordinate system.

3. The automatic visual alignment and laminating method of a robot according to claim 1, characterized in that: At the calibration reference position of the first camera and the second camera, the manipulator end clamps the marker and rotates on the horizontal plane according to the predetermined movement point position, collects multiple corresponding images through the first camera and the second camera, and extracts the feature points on the marker from the image, and obtains the offset of the origin of the manipulator end coordinate system in the intermediate coordinate system, that is, the offset matrix from the intermediate coordinate system to the manipulator end coordinate system , The steps further include: Extracting feature points on the marker from the captured image to obtain coordinates in the image coordinate system; Apply the obtained affine transformation matrix from the image coordinate system to the intermediate coordinate system , , transform the coordinates in the image coordinate system to the intermediate coordinate system; In the intermediate coordinate system, the least squares method is used to fit the coordinates of the center of the circle, the coordinates of the origin of the manipulator end coordinate system in the intermediate coordinate system, and the offset matrix from the intermediate coordinate system to the manipulator end coordinate system. , .

4. The automatic visual alignment and lamination method of a robot according to claim 1, characterized in that: By establishing the intermediate coordinate system, the affine transformation matrix from the camera image coordinate system to the intermediate coordinate system is completed in two steps: , Calibration, and the offset matrix from the intermediate coordinate system to the robot end coordinate system , Calibrate, and then obtain the transformation matrix from the image coordinate system of the first camera and the second camera to the coordinate system of the manipulator end , .

5. The automatic visual alignment and lamination method of a robot according to claim 1, characterized in that: The step of extracting the position and direction of feature points on the fitting object and the fitting target further includes: using a feature positioning method based on line search, circle search, or template matching to extract the position and direction of feature points from the corresponding image.

6. The automatic visual alignment and bonding method of a robot according to claim 1, characterized in that: The step of obtaining the fitting posture of the manipulator according to the calibration data of the first camera and the second camera, and the corresponding feature point positions and directions, further comprises: The following formula is used to calculate the robot's fitting posture : ， in, are the coordinates and angles of the end of the robot when it is fitted. are the coordinates and angles of the feature points of the fitted object in the robot end coordinate system, are the coordinates and angles of the feature points of the fitting target in the robot end coordinate system, are the horizontal and vertical coordinates and angles of the first camera calibration reference position, are the horizontal and vertical coordinates of the reference position for calibrating the second camera respectively.

7. A robot automatic visual alignment and laminating system, using the robot automatic visual alignment and laminating method according to any one of claims 1 to 6, characterized in that: include: A manipulator having a mobile gripping unit; A first camera unit, used for collecting images of the bonding object clamped by the end of the manipulator within the moving range; A second camera unit, used to collect images of the fitting target; An image extraction unit, configured to receive images captured by the first camera unit and the second camera unit, extract feature point information from the images, and output the feature point information; A data processing unit, used to receive the feature point information sent by the image extraction unit, and respectively establish a conversion relationship between the image coordinate system of the first camera unit and the second camera unit and the manipulator end coordinate system; according to the conversion relationship between the image coordinate system and the manipulator end coordinate system, the feature point information is converted to the manipulator end coordinate system, and the manipulator fitting posture is calculated and output; The control unit is used to control the robot to align and bond the bonding object with the bonding target according to the bonding posture of the robot.

Images