Visual robot fusion method

Abstract

The application discloses a visual robot fusion method, aiming at overcoming the problem of inaccurate positioning in the prior art, and comprises a workpiece coordinate system and tool coordinate system establishment process, an N-point calibration process, a template manufacturing process, a feature image processing process, a posture self-adapting process and a height automatic adjusting process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, in particular to a vision robot fusion method. BACKGROUND

[0002] The robot positioning accuracy is an important index for measuring its working performance, at present, the vision robots generated by domestic and foreign manufacturers are mostly low in positioning accuracy due to manufacturing, installation and other factors, and cannot meet the needs of high-precision processing, therefore, analyzing various factors causing the robot positioning error and maximizing the robot absolute positioning accuracy have become the core content of the research on machine vision technology.

[0003] At present, the commonly used robot calibration method at home and abroad usually completes with the help of advanced measuring equipment in the external world, but this leads to the problems of high cost, complex measuring process and the need for professional operation, and in the process of involving coordinate system conversion, the coordinate system conversion error is easily introduced, which also leads to the problem of large measuring error.

[0004] In view of the above status, the application provides a vision robot fusion method to solve the problem of positioning calibration. SUMMARY

[0005] In order to overcome the deficiencies and existing problems of the prior art, the application provides a vision robot system and a fusion method thereof.

[0006] In order to achieve the above purpose, the application adopts the following technical scheme:

[0007] On the one hand, the application provides a vision robot system for executing a workpiece coordinate system and a tool coordinate system establishment process, executing a template making process, a template making process, a feature point position approximation process, a feature image processing process, a posture self-adaptive process and a height automatic adjustment process, the system comprising:

[0008] A robot comprising an executing mechanism;

[0009] A two-dimensional camera arranged at the end of the executing mechanism;

[0010] A workbench for positioning a workpiece thereon;

[0011] A feature pattern comprising feature points and feature line segments.

[0012] As a preferred, the workpiece coordinate system and tool coordinate system establishment process comprises a vision robot system establishing a workpiece coordinate system and a tool coordinate system process, wherein the tool coordinate system comprises a TCP position.

[0013] As preferred, the N-point calibration procedure comprises that the visual robot system establishes a conversion matrix between an image coordinate system and a robot physical coordinate system, wherein N is a positive integer.

[0014] As preferred, the template making procedure comprises that the visual robot system adjusts a shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to take a picture and acquires an image, processes the image to obtain a pixel contour of the feature pattern, takes the pixel contour of the feature pattern as a positioning template, and calculates the positioning template to obtain physical geometric parameters of the positioning template, wherein the feature pattern comprises feature points and feature line segments.

[0015] As preferred, the feature point position approximation procedure comprises that the visual robot system sets the feature pattern on the workpiece, triggers the camera to take a picture and acquires an image, executes a feature image processing procedure on the image to obtain physical geometric parameters of a feature contour, and controls the camera to translate on the X-axis and the Y-axis of the robot physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0016] As preferred, the feature image processing procedure comprises that the visual robot system processes the image to obtain a contour of the feature pattern, judges whether the contour of the feature pattern matches the positioning template, if yes, takes the contour of the feature pattern as a feature contour and calculates the physical geometric parameters of the feature contour according to the feature contour, and if not, adjusts the shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to take a picture and acquires an image, wherein the feature contour is a pixel contour of the feature pattern.

[0017] As preferred, the pose self-adaption procedure comprises that the visual robot system respectively controls the two-dimensional camera to rotate around the X-axis, the Y-axis and the Z-axis of the robot physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0018] As preferred, the height automatic adjustment procedure comprises that the visual robot system controls the two-dimensional camera to translate on the Z-axis of the robot physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0019] In another aspect, the present application also provides a visual robot fusion method, which is executed by the above-mentioned visual robot system and comprises the following steps:

[0020] The workpiece coordinate system and tool coordinate system establishment procedure: a workpiece coordinate system and tool coordinate system establishment procedure, wherein the tool coordinate system comprises a TCP position;

[0021] The N-point calibration procedure: a conversion matrix between an image coordinate system and a robot physical coordinate system is established, wherein N is a positive integer;

[0022] The template making process comprises the following steps: adjusting the shooting pose of the two-dimensional camera on the feature pattern, triggering the two-dimensional camera to take a picture and obtaining an image, processing the image to obtain a pixel contour of the feature pattern, taking the pixel contour of the feature pattern as a positioning template, and calculating the positioning template to obtain physical geometric parameters of the positioning template, wherein the feature pattern comprises feature points and feature line segments.

[0023] The feature point position approximation process comprises the following steps: setting the feature pattern on the workpiece, triggering the camera to take a picture and obtaining an image, executing a feature image processing process on the image to obtain physical geometric parameters of a feature contour, and controlling the camera to translate on the X axis and the Y axis of the machine physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0024] The feature image processing process comprises the following steps: processing the image to obtain a contour of the feature pattern, judging whether the contour of the feature pattern matches the positioning template, taking the contour of the feature pattern as a feature contour and calculating the physical geometric parameters of the feature contour if the contour of the feature pattern matches the positioning template, and readjusting the shooting pose of the two-dimensional camera on the feature pattern, triggering the two-dimensional camera to take a picture and obtaining an image if the contour of the feature pattern does not match the positioning template, wherein the feature contour is a pixel contour of the feature pattern.

[0025] The pose self-adaption process comprises the following steps: respectively controlling the two-dimensional camera to rotate around the X axis, the Y axis and the Z axis of the machine physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0026] The height automatic adjustment process comprises the following steps: controlling the two-dimensional camera to translate on the Z axis of the machine physical coordinate system so that the physical geometric parameters of the feature contour approximate the physical geometric parameters of the positioning template.

[0027] Preferably, the workpiece coordinate system and tool coordinate system establishment process comprises the following steps:

[0028] Step 111: establishing the tool coordinate system and determining the TCP position, and setting the TCP position on the visual center point, wherein the visual center refers to the center of the lower end face of the two-dimensional camera lens.

[0029] Step 112: determining the working surface of the robot through the workpiece coordinate system, and establishing the XY plane of the workpiece coordinate system on the machining surface, wherein the machining surface is a tangent plane on the feature pattern, and the common point of the machining surface and the feature pattern is a feature point.

[0030] Preferably, the N-point calibration process comprises the following process:

[0031] Step 121: setting the TCP position on the visual center.

[0032] Step 122: controlling the two-dimensional camera to be perpendicular to the machining surface, and controlling the object distance to reach a set value.

[0033] Step 123: Set N calibration points on the machining surface. The physical coordinates of the N calibration points are (X1, Y1, Z1), (X2, Y2, Z2)...(X... N ,Y N Z N The system controls the TCP to move to N calibration points and acquire calibration images. Based on the calibration images, the pixel coordinates of the TCP at the N calibration points are determined. The pixel coordinates of the TCP at the N calibration points are (x1, y1), (x2, y2), ..., (x...). N ,y N );

[0034] Step 124: Substitute the physical coordinates of the N calibration points and the pixel coordinates of the TCP at the N calibration points into the following formula and calculate the transformation matrix using the least squares method:

[0035]

[0036] In the formula, the transformation matrix is a, d, b, and e are rotational components, and c and f are translational components.

[0037] Preferably, the template creation process specifically includes the following steps:

[0038] Step 131: Adjust the shooting pose of the 2D camera for the feature pattern, trigger the 2D camera to take a picture and acquire the image, wherein the feature pattern includes feature points and feature line segments; Step 132: Acquire the pixel contour of the feature pattern and use the pixel contour of the feature pattern as the positioning template; Step 133: Calculate the pixel geometric parameters of the positioning template according to the positioning template, and convert the pixel geometric parameters of the positioning template into the physical geometric parameters of the positioning template through a transformation matrix, wherein the geometric parameters of the positioning template include the position parameters of the feature points and the size parameters of the feature line segments.

[0039] Preferably, the feature point location approximation process specifically includes the following steps:

[0040] Step 211: Set the feature pattern on the workpiece and position the workpiece on the worktable;

[0041] Step 212: Trigger the 2D camera to take a picture and acquire the image;

[0042] Step 213: Perform a feature image processing procedure on the image to obtain the physical geometric parameters of the feature contour;

[0043] Step 214: Calculate the offset based on the physical geometric parameters of the positioning template and the physical geometric parameters of the feature contour. The offset includes ΔT. X and ΔT Y ΔT XLet ΔT be the offset of the feature contour relative to the positioning template on the X-axis of the robot's physical coordinate system. Y This represents the offset of the feature contour relative to the positioning template on the Y-axis of the robot's physical coordinate system.

[0044] Step 215: Determine ΔT X Is it less than the threshold ΔX? i If ΔT X Not less than the threshold ΔX i Then determine ΔT X Does it exceed the threshold? If ΔT X Exceeding the threshold Then control the 2D camera to translate along the X-axis of the robot's physical coordinate system to ΔT. X Not exceeding the threshold If ΔT X Not exceeding the threshold Then control the 2D camera to iteratively translate along the X-axis of the robot's physical coordinate system. The distance until ΔT X Less than If ΔT X Less than Then, the 2D camera is controlled to translate ΔT along the X-axis of the robot's physical coordinate system. X The distance, where the threshold Greater than 4 times the threshold ΔX i ;

[0045] Step 216: If ΔT X Less than the threshold ΔX i Then determine ΔT Y Is it less than the threshold ΔY? i If ΔT Y Not less than the threshold ΔY i Then determine ΔT Y Does it exceed the threshold? If ΔT Y Exceeding the threshold Then control the 2D camera to translate along the Y-axis of the robot's physical coordinate system to ΔT. Y Not exceeding the threshold If ΔT Y Not exceeding the threshold Then control the 2D camera to iteratively translate along the Y-axis of the robot's physical coordinate system. The distance until ΔT Y Less than If ΔT Y Less than Then, the 2D camera is controlled to translate ΔT along the Y-axis of the robot's physical coordinate system. Y The distance, where the threshold Greater than 4 times the threshold ΔYi .

[0046] As preferred, the feature image processing procedure specifically comprises the following steps:

[0047] S11: The visual robot system extracts the edge points of the pattern in the image according to the Canny algorithm;

[0048] S12: The visual robot system generates the pattern contour according to the extracted edge points of the pattern, wherein the pattern contour is composed of a plurality of edge points;

[0049] S13: The visual robot system calculates the number of edge points and the aspect ratio of the pattern contour according to the pattern contour, judges whether the number of edge points of the pattern contour is within the safe number range and whether the aspect ratio of the pattern contour is within the safe aspect ratio range, if yes, the corresponding pattern contour is retained, if not, the corresponding pattern contour is rejected;

[0050] S14: The visual robot system judges whether the difference between one edge point and the adjacent edge point of the retained pattern contour is within the gradient change threshold, if yes, the edge point is determined as the contour intersection point, judges whether the contour intersection point of the pattern contour is within the safe intersection point number range, if yes, the contour of the feature pattern is reconstructed according to the contour intersection point of the pattern contour;

[0051] S15: The visual robot system calculates the feature pattern pixel geometry parameters according to the contour of the feature pattern, converts the feature pattern pixel geometry parameters into feature pattern physical geometry parameters according to the conversion matrix, judges whether the feature pattern pixel geometry parameters and the positioning template physical geometry parameters constitute a similar relationship, if yes, it is determined that the feature pattern and the positioning template match successfully and the contour of the feature pattern is taken as the feature contour, if not, it is determined that the feature pattern and the positioning template do not match successfully and the shooting pose of the two-dimensional camera to the feature pattern is adjusted and the image is acquired.

[0052] As preferred, the pose self-adaptive procedure specifically comprises the following steps:

[0053] Step 221: Trigger the two-dimensional camera to take a picture and acquire an image;

[0054] Step 222: Perform the feature image processing procedure on the image to obtain the feature contour physical geometry parameters;

[0055] Step 223: Calculate the offset according to the feature contour physical geometry parameters and the positioning template physical geometry parameters, the offset includes θ Z1 , θ X1 and θ Y1 , θ Z1 is the offset angle of the feature contour relative to the positioning template on the Z-axis of the robot physical coordinate system, θ X1θ is the offset angle of the feature contour relative positioning template on the X-axis of the robot's physical coordinate system. Y1 The offset angle of the feature contour relative to the positioning template on the Y-axis of the robot's physical coordinate system;

[0056] Step 224: Determine θ Z1 Is it less than the threshold θ? Z Otherwise, control the 2D camera to rotate around the robot's physical coordinate system Z-axis until it is less than the threshold θ. Z ;

[0057] Step 225: If θ Z1 Less than threshold θ Z Then determine θ X1 Is it less than the threshold θ? X If θ X1 Not less than the threshold θ X Then determine θ X1 Is it greater than the threshold? If θ X1 Greater than the threshold Then control the 2D camera to rotate around the X-axis of the robot's physical coordinate system to θ. X1 Not greater than the threshold If θ X1 Not greater than the threshold Then control the 2D camera to iteratively rotate around the X-axis of the robot's physical coordinate system. Angle up to θ X1 Less than If θ Z1 Less than Then control the 2D camera to rotate θ around the X-axis of the robot's physical coordinate system. X1 The angle, where the threshold θ X Greater than 4 times the threshold

[0058] Step 226: If θ X1 Less than threshold θ X Then execute the feature point approximation process, and then determine θ. Y1 Is it less than the threshold θ? Y If θ Y1 Not less than the threshold θ Y Then determine θ Y1 Is it greater than the threshold? If θ Y1 Greater than the threshold Then control the 2D camera to rotate around the robot's physical coordinate system Y-axis to θ. Y1 Not greater than the threshold If θ Y1 Not greater than the threshold Then control the 2D camera to iteratively rotate around the Y-axis of the robot's physical coordinate system. Angle up to θY1 less than if θ Y1 less than then control the two-dimensional camera to rotate θ Y1 around the Y-axis of the robot physical coordinate system, wherein the threshold value θ Y is greater than 4 times the threshold value

[0059] Step 227: if θ Y1 is less than the threshold value θ Y , execute the feature point position approximation process, and record the current posture information of the robot.

[0060] As a preferred, the height automatic adjustment process comprises the following steps:

[0061] Step 241: trigger the two-dimensional camera to take a picture and obtain an image;

[0062] Step 242: execute a feature image processing process on the image to obtain feature contour physical geometric parameters;

[0063] Step 243: calculate an offset according to the feature contour physical geometric parameters and the positioning template physical geometric parameters, the offset comprising ΔH, ΔH being a height offset of the feature contour relative to the positioning template;

[0064] Step 244: determine whether ΔH is less than a threshold value ΔH1, if ΔH is less than the threshold value ΔH1, determine whether ΔH is greater than a threshold value ΔH2, if ΔH is greater than the threshold value ΔH2, control the two-dimensional camera to translate on the Z-axis of the robot physical coordinate system until ΔH is not greater than the threshold value ΔH2, if ΔH is not greater than the threshold value ΔH2, control the two-dimensional camera to iteratively translate on the Z-axis of the robot physical coordinate system a distance until ΔH is less than if ΔH is less than , control the two-dimensional camera to translate on the Z-axis of the robot physical coordinate system by a distance of ΔH, wherein the threshold value ΔH2 is greater than 4 times the threshold value ΔH1.

[0065] The present application has the following prominent and beneficial technical effects compared with the prior art:

[0066] (1) In the present application, the image captured by the two-dimensional camera is processed to realize the position adjustment of the two-dimensional camera, the position positioning of the workpiece, and the posture adjustment of the robot, thereby realizing the effect that the executing mechanism positions and processes the workpiece at the feature position, and the present application is suitable for workpiece positioning and processing with complex surfaces, and therefore has the advantages of high positioning efficiency and high positioning accuracy.

[0067] (2) In the present application, in the positioning process, the method of successive approximation and coarse-fine adjustment is used to realize the step-by-step approximation positioning of the pixel contour of the template, which on the one hand improves the positioning efficiency of the two-dimensional camera, and on the other hand reduces the positioning error of the two-dimensional camera.

[0068] (3) In the present application, the pose of the robot and the actuator is strictly positioned, avoiding the abnormal interference of the pose to the subsequent robot processing of the workpiece, and ensuring the processing precision and quality of the workpiece. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a structural schematic diagram of the visual robot system of the present application;

[0070] Figure 2 is a structural schematic diagram of the feature pattern of the present application;

[0071] Figure 3 is a total flow schematic diagram of the visual robot fusion method of the present application;

[0072] Figure 4 is a distribution schematic diagram of the nine calibration points on the workpiece in the visual robot fusion method of the present application;

[0073] Figure 5 is a schematic diagram of the template making process in the visual robot fusion method of the present application;

[0074] Figure 6 is a schematic diagram of the feature point position approximation process in the visual robot fusion method of the present application;

[0075] Figure 7 is a schematic diagram of the feature point position in the coarse-fine adjustment process in the X direction of the present application;

[0076] Figure 8 is a schematic diagram of the pose self-adaptive process in the visual robot fusion method of the present application;

[0077] Figure 9 is a schematic diagram of the height automatic adjustment process in the visual robot fusion method of the present application;

[0078] In the figure: 1-robot, 2-two-dimensional camera, 3-workbench, 4-workpiece, 5-feature pattern, 6-tool, 11-actuator. DETAILED DESCRIPTION

[0079] In order to facilitate the understanding of those skilled in the art, the present application will be further described below in conjunction with the drawings and specific embodiments.

[0080] As shown in Figure 1 , a visual robot system includes a robot, a two-dimensional camera, a workbench, a workpiece, a feature pattern, a photoelectric switch and a touch screen.

[0081] The robot comprises an execution mechanism for converting the control signal of the robot into a corresponding posture action of the mechanical arm, which is the main entity for the robot to complete the work task. In the embodiment, the execution mechanism belongs to the prior art and is a mechanical arm composed of a series of connecting rods, mechanical joints, and pose sensors. The bottom of the execution mechanism is provided with a base, and the end of the execution mechanism is provided with a tool, which is a main appliance for processing the workpiece during work. In the embodiment, the tool is a cutting tool.

[0082] A two-dimensional camera is used to shoot images, and the robot generates a control signal for controlling the execution mechanism by processing the images. The two-dimensional camera is also installed on the end of the execution mechanism, and the end of the execution mechanism has an end flange. The tool and the two-dimensional camera are arranged on the end flange, and the axial direction of the tool and the shooting direction of the two-dimensional camera are parallel to each other to facilitate subsequent positioning and calibration.

[0083] The workbench is used to place and clamp the workpiece and plays a role in positioning the workpiece. The workbench is arranged near the robot. In the embodiment, the workbench comprises a plurality of movable positioning blocks. In actual use, when the workpiece is correctly placed on the workbench, the positioning blocks move to abut against the workpiece, and the positioning blocks position the workpiece on the workbench.

[0084] A photoelectric switch (not shown in the figure) is used to detect the position of the workpiece on the workbench to help the workpiece to be correctly positioned on the workbench. The photoelectric switch is electrically connected to the robot, and the photoelectric switch is arranged on the workbench. The detection range of the photoelectric switch is arranged on the workbench, so that the photoelectric switch detects the position of the workpiece on the workbench.

[0085] A touch screen (not shown in the figure) is used to realize human-computer interaction. In actual use, the touch screen can display the current pose state of the execution mechanism in real time.

[0086] The workpiece is a component that needs to be processed by the robot system. The workpiece has a processing surface on which the tool can start processing. In the embodiment, the workpiece is an automobile, and the processing surface is a plane tangent to the feature pattern.

[0087] The feature pattern is used to be attached to the workpiece. In actual use, the feature pattern is attached to a specific position of the workpiece, which can be determined in advance and artificially. The camera shoots the feature pattern, and the robot identifies and positions the feature pattern, thereby realizing the identification and positioning of the workpiece. In the embodiment, the specific position of the workpiece can be the processing surface of the workpiece.

[0088] In the embodiment, the feature pattern is a soft film with a specific pattern on the surface. For example, the feature pattern is a soft film with a specific pattern on the surface. Figure 2As shown, the specific pattern includes a rectangle and an equilateral triangle intersecting together, one end point of the equilateral triangle is arranged on the center of the rectangle, the feature pattern includes feature points and feature line segments, the end points of the feature line segments are the feature points, the feature points are denoted by g, h, i, j and k, the feature line segments are denoted by A1, A2, B1 and B2, the feature point g is located at the intersection of the equilateral triangle and the rectangle, and is also located at the midpoint of the side of the equilateral triangle, the feature points h and k are located at the end points of the rectangle, the feature points i and j are located at the corners of the equilateral triangle, the feature line segment A1 is the line segment between the feature points i and g, the feature line segment A2 is the line segment between the feature points g and j, the feature line segment B1 is the line segment between the feature points h and g, and the feature line segment B2 is the line segment between the feature points g and k. In actual use, the feature pattern can be fully attached to the machining surface of the workpiece, thereby having accurate positioning effect on the curved surface.

[0089] The visual robot system is used for a workpiece coordinate system and a tool coordinate system establishment process, an execution template making process, a template making process, a feature point position approximation process, a feature image processing process, a posture self-adaption process and a height automatic adjustment process.

[0090] The workpiece coordinate system and the tool coordinate system establishment process includes a visual robot system establishing a workpiece coordinate system and a tool coordinate system process, wherein the tool coordinate system includes a TCP position.

[0091] The N-point calibration process includes a visual robot system establishing a conversion matrix between an image coordinate system and a robot physical coordinate system, wherein N is a positive integer.

[0092] The template making process includes a visual robot system adjusting a shooting pose of a two-dimensional camera on a feature pattern, triggering the two-dimensional camera to take a picture and obtaining an image, processing the image to obtain a pixel contour of the feature pattern, taking the pixel contour of the feature pattern as a positioning template, and calculating the positioning template to obtain physical geometric parameters of the positioning template, wherein the feature pattern includes feature points and feature line segments.

[0093] In the template making process, the two-dimensional camera can obtain images at multiple shooting poses of the feature pattern and make multiple positioning templates. In the subsequent feature image processing process, the contour of the feature pattern can be matched with the positioning templates at multiple shooting poses, and the two-dimensional camera can also accurately match the corresponding positioning templates at different poses, thereby improving the matching range.

[0094] The feature point position approximation process includes a visual robot system setting a feature pattern on a workpiece, triggering a camera to take a picture and obtaining an image, executing a feature image processing process on the image to obtain feature contour physical geometric parameters, and controlling the camera to translate on the robot physical coordinate system X-axis and Y-axis to make the feature contour physical geometric parameters approximate the positioning template physical geometric parameters.

[0095] The feature image processing procedure comprises that the visual robot system processes the image to obtain a contour of the feature pattern, judges whether the contour of the feature pattern and the positioning template match, if yes, takes the contour of the feature pattern as a feature contour, calculates the feature contour physical geometric parameters according to the feature contour, and if not, readjusts the shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to shoot and acquires the image, wherein the feature contour is a pixel contour of the feature pattern.

[0096] The posture self-adapting procedure comprises that the visual robot system respectively controls the two-dimensional camera to rotate around the X-axis, Y-axis and Z-axis of the physical coordinate system of the robot so that the feature contour physical geometric parameters approximate the positioning template physical geometric parameters.

[0097] The height automatic adjusting procedure comprises that the visual robot system controls the two-dimensional camera to translate on the Z-axis of the physical coordinate system of the robot so that the feature contour physical geometric parameters approximate the positioning template physical geometric parameters.

[0098] On the other hand, the application further provides a visual robot fusion method, which is executed by the above visual robot system. The steps of the visual robot fusion method comprise a preprocessing procedure and a real-time processing procedure. The specific steps of the preprocessing comprise a workpiece coordinate system and tool coordinate system establishing procedure, an N-point calibration procedure and a template making procedure. The real-time processing procedure comprises a feature point position approximation procedure, a feature image processing procedure, a posture self-adapting procedure and a height automatic adjusting procedure.

[0099] As shown in Figure 3 In the actual positioning process, the visual robot system starts, judges whether the preprocessing procedure needs to be executed, if not, enters the judgment whether the real-time processing procedure needs to be executed, if yes, displays the interface of the preprocessing procedure on the touch screen, judges whether the workpiece coordinate system and tool coordinate system establishing needs to be executed, if yes, executes the workpiece coordinate system and tool coordinate system establishing procedure, if not, judges whether the N-point calibration procedure needs to be executed, if yes, executes the N-point calibration procedure, if not, judges whether the template making procedure needs to be executed, if yes, executes the template making procedure and returns to the judgment whether the preprocessing procedure needs to be executed, if not, returns to the judgment whether the preprocessing procedure needs to be executed. The visual robot system judges whether the real-time processing procedure needs to be executed, if yes, executes the feature point position approximation procedure and the posture self-adapting procedure in turn, after the posture self-adapting procedure is executed, judges whether the posture adjustment angle (i.e. the offset θ Z1 , θ X1 and θ Y1 ) exceeds the threshold value (i.e. the threshold value θ Z , the threshold value θ X and the threshold value θ Y), if yes, re-perform the feature point position approximation procedure and the pose self-adaption procedure, if no, perform the height automatic adjustment procedure, then judge whether the height total adjustment value is less than the threshold AH3, if yes, the visual robot system ends, if no, perform the feature point approximation procedure and then end. If the real-time processing procedure is not needed to be performed, judge whether the system needs to be exited, if yes, the visual robot system ends, if no, return to the judgment of whether the pre-processing procedure needs to be performed.

[0100] Workpiece coordinate system and tool coordinate system establishment procedure: the visual robot system establishes the workpiece coordinate system and the tool coordinate system procedure, wherein the tool coordinate system includes the TCP position. In the embodiment, as shown in the figure, the X axis and the Y axis in the figure represent the workpiece coordinate system X axis and Y axis respectively. By establishing the workpiece coordinate system, the workpiece coordinate system X axis and Y axis can be set on the machining surface of the workpiece, so as to facilitate accurate positioning of the feature pattern on the machining surface. Figure 2

[0101] N-point calibration procedure: the visual robot system establishes the conversion matrix between the image coordinate system and the robot physical coordinate system, wherein N is a positive integer. In the embodiment, the value of N is 9. Through the N-point calibration procedure, the conversion between the image coordinate system and the robot physical coordinate system is realized.

[0102] Template making procedure: the visual robot system adjusts the shooting pose of the two-dimensional camera on the feature pattern, triggers the two-dimensional camera to take a picture and acquires an image, processes the image to obtain the pixel contour of the feature pattern, takes the pixel contour of the feature pattern as a positioning template, and calculates the positioning template physical geometric parameters, wherein the feature pattern includes feature points and feature line segments. In the embodiment, the positioning template is used to adjust the pose of the two-dimensional camera, the actuator and the robot relative to the workpiece after the workpiece is subsequently fed onto the workbench.

[0103] In the template making procedure, the two-dimensional camera can acquire images at multiple shooting poses of the feature pattern and make multiple positioning templates. In the subsequent feature image processing procedure, the contour of the feature pattern can be matched with the positioning templates at multiple shooting poses, and the two-dimensional camera can also accurately match the corresponding positioning template at different poses, thereby improving the matching range.

[0104] Feature point position approximation procedure: the visual robot system sets the feature pattern on the workpiece, triggers the camera to take a picture and acquires an image, performs the feature image processing procedure on the image to obtain the feature contour physical geometric parameters, and controls the two-dimensional camera to translate on the robot physical coordinate system X axis and Y axis so that the feature contour physical geometric parameters approximate the positioning template physical geometric parameters.

[0105] ​The feature image processing flow is that the visual robot system processes the image to obtain the contour of the feature pattern, judges whether the contour of the feature pattern and the positioning template are matched, if yes, takes the contour of the feature pattern as the feature contour and calculates the feature contour physical geometric parameters according to the feature contour, if not, adjusts the shooting pose of the two-dimensional camera to the feature pattern, triggers the two-dimensional camera to shoot and acquires the image, wherein the feature contour is the pixel contour of the feature pattern.

[0106] The pose self-adaption flow is that the visual robot system respectively controls the two-dimensional camera to rotate around the X axis, the Y axis and the Z axis of the robot physical coordinate system to make the feature contour physical geometric parameters approximate to the positioning template physical geometric parameters.

[0107] The height automatic adjustment flow is that the visual robot system controls the two-dimensional camera to translate on the Z axis of the robot physical coordinate system to make the feature contour physical geometric parameters approximate to the positioning template physical geometric parameters.

[0108] The workpiece coordinate system and tool coordinate system establishment flow, specifically includes the following steps:

[0109] Step 111: the visual robot system establishes the tool coordinate system and determines the TCP position, sets the TCP position on the visual center point, wherein the visual center refers to the center of the lower end surface of the two-dimensional camera lens;

[0110] Step 112: the lower end surface of the two-dimensional camera lens is the end surface exposed outside the two-dimensional camera. The visual robot system determines the machining surface position of the workpiece through the workpiece coordinate system, and the XY plane of the workpiece coordinate system is established on the machining surface, wherein the machining surface is the tangent plane on the feature pattern, and the common point of the machining surface and the feature pattern is the feature point.

[0111] As preferred, the N-point calibration flow, the value of N is 9, specifically includes the following flow:

[0112] Step 121: the visual robot system sets the TCP position on the visual center, wherein the visual center refers to the center of the lower end surface of the two-dimensional camera lens;

[0113] Step 122: the visual robot system controls the two-dimensional camera to be perpendicular to the machining surface, and controls the object distance to reach the set value;

[0114] In the above steps, the position of the two-dimensional camera can be adjusted by the robot. If the two-dimensional camera is perpendicular to the machining surface, the lower end surface of the two-dimensional camera lens and the machining surface are parallel to each other. The set value can be pre-set in the visual robot system, if the object distance reaches the set value, the focal length of the two-dimensional camera and the machining surface is appropriate, and the two-dimensional camera is in focus.

[0115] Step 123: The vision robot system sets N calibration points on the processing surface. The physical coordinates of the N calibration points are (X1, Y1, Z1), (X2, Y2, Z2)...(X... N ,Y N Z N The system controls the TCP to move to N calibration points and acquire calibration images. Based on the calibration images, the pixel coordinates of the TCP at the N calibration points are determined. The pixel coordinates of the TCP at the N calibration points are (x1, y1), (x2, y2), ..., (x...). N ,y N );

[0116] In the above steps, such as Figure 4 The diagram shows the distribution of nine calibration points on the machined surface. From left to right and from top to bottom, the points are the 1st, 2nd, ..., Nth calibration points, respectively. In this embodiment, the calibration points can be set on feature point a of the feature pattern. (X1, Y1, Z1), (X2, Y2, Z2)...(X N ,Y N Z N ) can be represented as (x1,y1), (x2,y2)...(x N ,y N ) can be represented as and

[0117] Step 124: The vision robot system substitutes the physical coordinates of N calibration points and the pixel coordinates of TCP at the N calibration points into the following formula and calculates the transformation matrix using the least squares method:

[0118]

[0119] In the formula, the transformation matrix is a, d, b, and e are rotational components, and c and f are translational components, respectively.

[0120] In the above steps, the physical coordinates of the calibration point, the pixel coordinates of the TCP at the calibration point, and the transformation matrix satisfy the following formula:

[0121]

[0122] Expanding the above formula yields the following formula:

[0123] ax + by + c = X(1),

[0124] dx+ey+f=Y(2),

[0125] In the present embodiment, N is 9, the physical coordinates of the 9 calibration points and the pixel coordinates of the TCP on the 9 calibration points are substituted into the above formula (1) to obtain the conversion formula between the X axis of the physical coordinate system and the X axis of the pixel coordinate system as follows:

[0126] ax1+by1+c=X1

[0127] ax2+by2+c=X2

[0128] ax3+by3+c=X3

[0129] ax4+by4+c=X4

[0130] ax5+by5+c=X5

[0131] ax6+by6+c=X6

[0132] ax7+by7+c=X7

[0133] ax8+by8+c=X8,

[0134] ax9+by9+c=X9

[0135] Similarly, the physical coordinates of the 9 calibration points and the pixel coordinates of the TCP on the 9 calibration points are substituted into the above formula (2) to obtain the conversion formula between the Y axis of the physical coordinate system and the Y axis of the pixel coordinate system.

[0136] According to the least square method, the variance on both sides of the conversion formula between the X axis of the physical coordinate system and the X axis of the pixel coordinate system is minimized to obtain the following formula:

[0137]

[0138] In the formula, S(a, b, c) is the variance on the X axis;

[0139] The minimum value of S(a, b, c) is solved by calculating the partial derivative of S(a, b, c) and making the first derivative zero, and then a, b and c are obtained. The values of a, b and c are obtained by calculating the three linear equations.

[0140] Similarly, according to the least square method, the variance on both sides of the conversion formula between the Y axis of the physical coordinate system and the Y axis of the pixel coordinate system is minimized to obtain the following formula:

[0141]

[0142] In the formula, S(d, e, f) is the variance on the Y axis;

[0143] Similarly, the values ​​of d, e, and f can be obtained by calculating S(d,e,f), which will not be repeated here. The transformation matrix can then be obtained based on the values ​​of a, b, c, d, e, and f.

[0144] The template creation process specifically includes the following steps:

[0145] Step 131: The visual robot system adjusts the shooting pose of the 2D camera for the feature pattern, triggers the 2D camera to take a picture and acquire the image, wherein the feature pattern includes feature points and feature line segments;

[0146] In the above steps, the feature pattern is placed against a background without other objects for shooting, and the acquired image contains only the feature pattern as the scene, which facilitates the subsequent acquisition of the pixel outline of the feature pattern. In this embodiment, the 2D camera can take pictures and acquire images relative to multiple poses of the feature pattern, and the images acquired by the 2D camera in multiple poses can be processed to obtain multiple positioning templates.

[0147] Step 132: The visual robot system acquires the pixel contour of the feature pattern and uses the pixel contour of the feature pattern as a localization template.

[0148] In the above steps, the visual robot system can obtain the pixel contours of feature patterns in the image based on the Canny algorithm.

[0149] Step 133: The visual robot system calculates the pixel geometric parameters of the positioning template based on the positioning template, and converts the pixel geometric parameters of the positioning template into physical geometric parameters of the positioning template through a transformation matrix. The geometric parameters of the positioning template include the position parameters of the feature points and the size parameters of the feature line segments.

[0150] like Figure 5 The diagram illustrates the template creation process. The visual robot system begins the template creation process. Its actuator adjusts the 2D camera's pose for capturing the feature pattern, triggering the camera to take a picture and acquire the image. Then, it acquires the pixel contour of the feature pattern and determines if the contour acquisition was successful. If not, the visual robot system's touchscreen displays: "Template matching failed, please adjust the position." If successful, it sets the template as a positioning template and acquires the feature point positions and feature line segment pixel dimensions, converting them to physical dimensions. The feature points and feature line segments of the positioning template correspond to the feature points and feature line segments of the feature pattern. Finally, it checks if each dimension value (i.e., the feature line segments in the positioning template) is within the threshold range of the actual dimensions (i.e., the feature line segments of the feature pattern). If so, it records the feature point positions and feature line segment physical dimensions of the positioning template as the physical geometric parameters of the positioning template. Otherwise, it displays an error message indicating that the physical geometric parameters of the positioning template are incorrect and requests adjustment of the position (i.e., adjusting the 2D camera's pose for capturing the feature pattern).

[0151] The feature point position approximation process specifically comprises the following steps.

[0152] Step 211: The visual robot system sets a feature pattern on a workpiece and positions the workpiece on a workbench.

[0153] In the above step, the visual robot system can monitor the position of the workpiece on the workbench through a photoelectric switch. When the workpiece reaches a predetermined position on the workbench, the photoelectric switch sends a trigger signal, and the visual robot system determines that the workpiece is positioned on the workbench.

[0154] Step 212: The visual robot system triggers a two-dimensional camera to take a picture and acquire an image.

[0155] Step 213: The visual robot system performs a feature image processing process on the image to obtain feature contour physical geometric parameters.

[0156] In the above step, after matching, the deflection angle of the pixel contour relative to the positioning template can also be recorded, and the two-dimensional camera is adjusted to rotate according to the deflection angle, so that the pixel contour can be directly opposite to the positioning template, to facilitate subsequent correction of the offset.

[0157] Step 214: The visual robot system calculates an offset according to the positioning template physical geometric parameters and the feature contour physical geometric parameters, wherein the offset comprises ΔT X and ΔT Y , ΔT X is the offset of the feature contour relative to the positioning template on the X-axis of the robot physical coordinate system, and ΔT Y is the offset of the feature contour relative to the positioning template on the Y-axis of the robot physical coordinate system.

[0158] Step 215: The visual robot system determines whether ΔT X is less than a threshold value ΔX i . If ΔT X is not less than the threshold value ΔX i , it is determined whether ΔT X exceeds a threshold value If ΔT X exceeds the threshold value , the two-dimensional camera is controlled to translate on the X-axis of the robot physical coordinate system to ΔT X If ΔT X does not exceed the threshold value , the two-dimensional camera is controlled to iteratively translate on the X-axis of the robot physical coordinate system by a distance of until ΔT X is less than If ΔT X is less than , the two-dimensional camera is controlled to translate on the Y-axis of the robot physical coordinate system to ΔT X .Then, the 2D camera is controlled to translate ΔT along the X-axis of the robot's physical coordinate system. X The distance, where the threshold Greater than 4 times the threshold ΔX i ;

[0159] Step 216: If ΔT X Less than the threshold ΔX i The visual robot system then determines ΔT. Y Is it less than the threshold ΔY? i If ΔT Y Not less than the threshold ΔY i Then determine ΔT Y Does it exceed the threshold? If ΔT Y Exceeding the threshold Then control the 2D camera to translate along the Y-axis of the robot's physical coordinate system to ΔT. Y Not exceeding the threshold If ΔT Y Not exceeding the threshold Then control the 2D camera to iteratively translate along the Y-axis of the robot's physical coordinate system. The distance until ΔT Y Less than If ΔT Y Less than Then, the 2D camera is controlled to translate ΔY along the Y-axis of the robot's physical coordinate system. i The distance, where the threshold Greater than 4 times the threshold ΔY i .

[0160] like Figure 6 The diagram illustrates the feature point location approximation process. The vision robot system begins the feature point location approximation process, positioning the workpiece in place (step 211). The vision robot system then triggers the 2D camera to capture an image (step 212). The vision robot system then performs feature image processing and obtains the offset (steps 213 and 214). Finally, the vision robot system determines the image X-direction deviation (ΔT). X Is it less than ΔX? i Otherwise, determine whether the image X-direction deviation exceeds [the specified value]. If yes, control the robot to make coarse adjustments in the X direction and record the adjustment value; otherwise, control the robot to make fine adjustments in the X direction and record the adjustment value, i.e., step 215. Coarse adjustment in the X direction refers to controlling the 2D camera to translate to ΔT along the X-axis of the robot's physical coordinate system. X Not exceeding the threshold Fine-tuning the robot in the X direction refers to controlling the 2D camera to iteratively translate along the X-axis of the robot's physical coordinate system. The distance until ΔTX Less than If the X-direction deviation of the image is less than ΔX i The visual robot system then determines the deviation in the Y direction of the image (i.e., ΔY). i Is it less than ΔY? i Otherwise, determine whether the image's Y-direction deviation exceeds [the specified value]. If yes, then control the robot to make coarse adjustments in the Y direction and record the adjustment value; otherwise, control the robot to make fine adjustments in the Y direction and record the adjustment value, i.e., step 216. Coarse adjustments in the Y direction refer to controlling the 2D camera to translate to ΔT along the Y-axis of the robot's physical coordinate system. Y Not exceeding the threshold Fine-tuning the robot in the Y direction refers to controlling the 2D camera to iteratively translate along the Y-axis of the robot's physical coordinate system. The distance until ΔT Y Less than If the image deviates in the Y direction (i.e., ΔY) i Less than ΔY i Alternatively, after completing step 216, the visual robot system terminates the feature point position approximation process. An adjustment process combining coarse and fine adjustments in the X and Y directions is employed, balancing adjustment efficiency and positioning accuracy.

[0161] like Figure 7 The diagram shows the location of feature points during the coarse and fine adjustment processes in the X direction. In the diagram, X... i For ΔT x The real-time value, T x This represents the target deviation value adjusted in the X direction, ΔX. i The threshold for position adjustment in the X direction. This is also the threshold for position adjustment in the X direction. This is still the critical point between the coarse and fine adjustments. The detailed process for coarse and fine adjustments is as follows:

[0162] Visual robot system determines X i Is it greater than If so, perform coarse adjustments, controlling the robot to adjust in the X direction. Ensure that the distance between the robot and the target position after adjustment is within... Within the range;

[0163] Visual robot system re-determines X i Is it less than If so, then fine-tune the adjustments; the fine-tune area is... The area is divided into four equal parts, and the robot's current position is determined in real time. If the position falls within one of the last three areas, the robot is controlled to move in the X direction. The distance allows it to enter the previous area until it enters... Within the range; if within If the current offset X is within the range, then control the robot to run the current offset X i . In the feature point position approximation process, The threshold range must be greater than 4 times ΔX i .

[0164] Similarly, according to the coarse and fine adjustment process in the X direction, the coarse and fine adjustment process in the Y direction can be known.

[0165] The feature image processing process,

[0166] Specifically includes the following steps:

[0167] S11: The visual robot system extracts the edge points of the pattern in the image according to the Canny algorithm;

[0168] In the actual processing process, the image captured by the two-dimensional camera not only contains the feature pattern but also includes the patterns of other scenes. Using the Canny algorithm not only extracts the edge points of the feature pattern, but also extracts the edge points of the patterns of other scenes.

[0169] S12: The visual robot system generates a pattern contour according to the extracted edge points of the pattern, wherein the pattern contour is composed of a plurality of continuous edge points;

[0170] In the above steps, the visual robot not only generates the contour of the feature pattern, but also generates the contour of the pattern of other scenes. In this embodiment, the number of pattern contours is at least two, one of which is the contour of the feature pattern and the remaining is the contour of the pattern of other scenes.

[0171] S13: The visual robot system calculates the number of edge points and the aspect ratio of the pattern contour according to the pattern contour, judges whether the number of edge points of the pattern contour is within the safe number range and whether the aspect ratio of the pattern contour is within the safe aspect ratio range, if yes, then retains the corresponding pattern contour, if not, then rejects the corresponding pattern contour.

[0172] In the above step, in order to distinguish the feature pattern contour in the pattern contour, in the embodiment, a safe number range and a safe aspect ratio range are set in the vision robot system in advance. The safe number range represents a value range of the number of edge points of the contour of the feature pattern. If the number of edge points of the pattern contour is out of the safe number range, the corresponding pattern contour is not the contour of the feature pattern, and thus the pattern contour is rejected. The safe aspect ratio range represents a value range of the aspect ratio of the feature pattern. If the aspect ratio of the pattern contour is out of the safe aspect ratio range, the corresponding pattern contour is not the contour of the feature pattern, and thus the pattern contour is rejected. If the number of edge points of the pattern contour is in the safe number range and the aspect ratio of the pattern contour is in the safe aspect ratio range, the corresponding pattern contour is likely to be the contour of the feature pattern, and thus the pattern contour is retained.

[0173] In the above step, since the direction of the contour line segment near the contour intersection point changes, the existence of the contour intersection point is determined by judging the vector variation trend of the adjacent edge points. In the embodiment, a gradient variation threshold is set in advance. The gradient variation threshold represents a value range of the vector difference between the adjacent edge points on two adjacent contour line segments. If the difference between one edge point of the retained pattern contour and the adjacent edge point is in the gradient variation threshold, it is proved that the edge point is the contour intersection point.

[0174] In the above step, since the direction of the contour line segment near the contour intersection point changes, the existence of the contour intersection point is determined by judging the vector variation trend of the adjacent edge points. In the embodiment, a gradient variation threshold is set in advance. The gradient variation threshold represents a value range of the vector difference between the adjacent edge points on two adjacent contour line segments. If the difference between one edge point of the retained pattern contour and the adjacent edge point is in the gradient variation threshold, it is proved that the edge point is the contour intersection point.

[0175] In the above step, since the direction of the contour line segment near the contour intersection point changes, the existence of the contour intersection point is determined by judging the vector variation trend of the adjacent edge points. In the embodiment, a gradient variation threshold is set in advance. The gradient variation threshold represents a value range of the vector difference between the adjacent edge points on two adjacent contour line segments. If the difference between one edge point of the retained pattern contour and the adjacent edge point is in the gradient variation threshold, it is proved that the edge point is the contour intersection point.

[0176] S15: The visual robot system calculates the feature pattern pixel geometry parameters according to the contour of the feature pattern, converts the feature pattern pixel geometry parameters into feature pattern physical geometry parameters according to the conversion matrix, judges whether the feature pattern pixel geometry parameters and the positioning template physical geometry parameters constitute a similar relationship, if yes, determines that the feature pattern and the positioning template match successfully and takes the contour of the feature pattern as the feature contour, if not, determines that the feature pattern and the positioning template do not match successfully and adjusts the shooting pose of the two-dimensional camera to the feature pattern and acquires the image again.

[0177] In the above steps, the feature pattern pixel geometry parameters and the positioning template physical geometry parameters constitute a similar relationship refers to that when the similarity of the feature pattern and the positioning template is less than a similarity threshold, the shooting pose of the camera to the feature pattern is adjusted and the photographing is taken again. In the embodiment, the similarity of the feature pattern and the positioning template is determined according to the ratio of different feature line segments of the feature pattern and different feature line segments of the positioning template, for example, the similarity of the feature pattern and the positioning template is equal to the ratio of A1 of the feature pattern and A1 of the positioning template minus the ratio of B1 of the feature pattern and B1 of the positioning template.

[0178] In the feature image processing flow, the visual robot system matches the contour of the feature pattern in the image with the plurality of positioning templates, so as to quickly and accurately match the two, and then quickly and accurately calibrate the actuator.

[0179] As shown in FIG. 1, it is a schematic diagram of the pose adaptive flow, which is used to realize the pose adjustment of the actuator and the two-dimensional camera in the physical coordinate system X axis, Y axis and Z axis, and ensure that the feature point position is unchanged. The pose adaptive flow specifically includes the following steps: Figure 8

[0180] Step 221: The visual robot system triggers the two-dimensional camera to take a photograph and acquires an image;

[0181] Step 222: The visual robot system executes the feature image processing flow on the image to obtain the feature contour physical geometry parameters;

[0182] In the above steps, the feature contour physical geometry parameters include the physical values of A1, A2, B1 and B2.

[0183] Step 223: The visual robot system calculates the offset according to the feature contour physical geometry parameters and the positioning template physical geometry parameters, and the offset includes θ Z1 , θ X1 and θ Y1 , θ Z1 is the offset angle of the feature contour relative to the positioning template in the Z axis of the machine physical coordinate system, θ X1 ​θ is the offset angle of the feature profile relative to the template on the X-axis of the machine physical coordinate system Y1 θ is the offset angle of the feature profile relative to the template on the Y-axis of the machine physical coordinate system

[0184] In the above step, the offset θ is adjusted by a step-by-step method. X1 and θ Y1 are calculated by the proportional relationship formula of A1, A2 and B1, B2. The proportional relationship formula is as follows:

[0185]

[0186]

[0187] In the formula, f(θ Y1 ) is the offset θ Y1 , are the ratios of A1 and A2 obtained in the nth time and the ith time, respectively, a n ...a i ...a0 are the weight coefficients in the nth time, the ith time and the 0th time, respectively, and f(θ X1 ) is the offset θ X1 , are the ratios of A1 and A2 obtained in the nth time and the ith time, respectively, b n ...b i ...b0 are the weight coefficients in the nth time, the ith time and the 0th time, respectively.

[0188] In step 224, the vision robot system determines whether θ Z1 is less than the threshold θ Z , and if not, controls the two-dimensional camera to rotate around the Z-axis of the machine physical coordinate system until it is less than the threshold θ Z .

[0189] In the above step, the vision robot system controls the two-dimensional camera to rotate around the Z-axis of the machine physical coordinate system until θ Z is less than the threshold θ Z1 , which is a step-by-step adjustment method.

[0190] In step 225, if θ Z1 is less than the threshold θ Z , the vision robot system determines whether θ X1 is less than the threshold θ X , and if θ X1 is not less than the threshold θ X , it determines whether θ X1 is greater than the threshold If θ X1 is greater than the threshold then control the two-dimensional camera to rotate around the X axis of the machine physical coordinate system to θ X1 not greater than a threshold value if θ X1 is not greater than a threshold value then control the two-dimensional camera to rotate around the X axis of the machine physical coordinate system iteratively by an angle of θ until θ X1 is less than if θ Z1 is less than then control the two-dimensional camera to rotate around the X axis of the machine physical coordinate system by an angle of θ X1 , wherein the threshold value θ X is greater than 4 times the threshold value

[0191] In the above steps, the visual robot system also uses the coarse adjustment and fine adjustment manner to adjust the movement of the two-dimensional camera on the X axis of the machine physical coordinate system.

[0192] Step 226: if θ X1 is less than a threshold value θ X , the visual robot system executes a feature point approximation process, and then determines whether θ Y1 is less than a threshold value θ Y , if θ Y1 is not less than a threshold value θ Y , determines whether θ Y1 is greater than a threshold value if θ Y1 is greater than a threshold value then control the two-dimensional camera to rotate around the Y axis of the machine physical coordinate system to θ Y1 not greater than a threshold value if θ Y1 is not greater than a threshold value then control the two-dimensional camera to rotate around the Y axis of the machine physical coordinate system iteratively by an angle of θ until θ Y1 is less than if θ Y1 is less than then control the two-dimensional camera to rotate around the Y axis of the machine physical coordinate system by an angle of θ Y1 , wherein the threshold value θ Y is greater than 4 times the threshold value

[0193] In the above steps, the visual robot system also uses the coarse adjustment and fine adjustment manner to adjust the movement of the two-dimensional camera on the Y axis of the machine physical coordinate system.

[0194] Step 227: if θ Y1 is less than a threshold value θ YIf so, the visual robot system executes the feature point position approximation procedure, and records the current pose information of the robot;

[0195] In the above step, the above θ Z1 , θ X1 , θ Y1 , θT X and ΔT Y After the adjustment is completed, the visual robot system can record the pose of the robot as the final pose for positioning and processing the workpiece.

[0196] As shown in FIG. 2, it is a schematic diagram of the height automatic adjustment procedure. The height automatic adjustment procedure includes the following steps: Figure 9

[0197] Step 241: The visual robot system triggers the two-dimensional camera to take a picture and obtain an image;

[0198] Step 242: The visual robot system executes the feature image processing procedure on the image to obtain the feature contour physical geometric parameters;

[0199] In the above step, the feature contour physical geometric parameters further include the height of the feature contour, and the height of the feature contour is the vertical distance between the camera and the feature pattern.

[0200] In the above step, the height in the feature contour physical geometric parameters satisfies the following formula:

[0201] f(A1) = c n (A1) n +... + c i (A1) i +... + c0,

[0202] f(A2) = d n (A2) n +... + d i (A2) i +... + d0

[0203] f(B1) = e n (B1) n +... + e i (B1) i +... + e0

[0204] f(B2) = f n (B2) n +... + f i (B2) i +... + f0

[0205] In the formula, f(A1) is the height of the feature contour calculated according to A1, c n ... c​i ...c0is the weight coefficient of the n-th time… the i-th time… the 0-th time, (A1) n ...(A1) i ...is the physical value of A1 obtained in the n-th time… the i-th time, f(A2) is the feature profile height calculated according to A2, d n ...d i ...d0is the weight coefficient of the n-th time… the i-th time… the 0-th time, (A2) n ...(A2) i ...is the physical value of A2 obtained in the n-th time… the i-th time, f(B1) is the feature profile height calculated according to B1, e n ...e i ...e0is the weight coefficient of the n-th time… the i-th time… the 0-th time, (B1) n ...(B1) i ...is the physical value of B1 obtained in the n-th time… the i-th time, f(B2) is the feature profile height calculated according to B2, f n ...f i ...f0is the weight coefficient of the n-th time… the i-th time… the 0-th time, (B2) n ...(B2) i ...is the physical value of B2 obtained in the n-th time… the i-th time. The height of the feature profile can be obtained according to the above formula.

[0206] Step 243: The visual robot system calculates the offset according to the feature profile physical geometry parameter and the positioning template physical geometry parameter, the offset includes ΔH, ΔH is the height offset of the feature profile relative to the positioning template;

[0207] In the above steps, the positioning template physical geometry parameter includes the height of the positioning template, the height of the positioning template is the vertical distance between the feature pattern and the camera calculated in advance, and the height of the positioning template can also be calculated by the formula in step 242.

[0208] Step 244: The visual robot system judges whether ΔH is less than threshold ΔH1, if ΔH is less than threshold ΔH1, judges whether ΔH is greater than threshold ΔH2, if ΔH is greater than threshold ΔH2, controls the two-dimensional camera to translate on the Z-axis of the robot physical coordinate system until ΔH is not greater than threshold ΔH2, if ΔH is not greater than threshold ΔH2, controls the two-dimensional camera to iteratively translate on the Z-axis of the robot physical coordinate system if ΔH is less than threshold ΔH1, controls the two-dimensional camera to iteratively translate on the Z-axis of the robot physical coordinate system ​then controlling the two-dimensional camera to translate a distance of ΔH on the Z-axis of the machine physical coordinate system, wherein the threshold value ΔH2 is greater than 4 times the threshold value ΔH1;

[0209] Step 245: The visual robot system determines whether the total height adjustment value is less than ΔH3. If yes, it is determined that the robot height adjustment is too much to cause the feature point position to be offset, and the feature point position approximation process is executed again to adjust the feature point position. The total height adjustment value is equal to the difference between the final two-dimensional camera height and the initial two-dimensional camera height.

[0210] The above embodiments are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore, any equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A visual robot fusion method, characterized in that, The method comprises the following steps: The workpiece coordinate system and tool coordinate system establishment process: the workpiece coordinate system and tool coordinate system establishment process, wherein the tool coordinate system comprises a TCP position; The N-point calibration process: a conversion matrix between the image coordinate system and the machine physical coordinate system is established, wherein N is a positive integer; The template making process: adjust the shooting pose of the two-dimensional camera to the feature pattern, trigger the two-dimensional camera to take a picture and obtain an image, process the image to obtain the pixel contour of the feature pattern, take the pixel contour of the feature pattern as a positioning template, and calculate the positioning template physical geometric parameters, wherein the feature pattern comprises feature points and feature lines; The feature point position approximation process: set the feature pattern on the workpiece, trigger the camera to take a picture and obtain an image, execute the feature image processing process on the image to obtain the feature contour physical geometric parameters, and control the camera to translate on the machine physical coordinate system X-axis and Y-axis to make the feature contour physical geometric parameters approximate to the positioning template physical geometric parameters; The feature image processing process: process the image to obtain the feature contour, judge whether the feature contour and the positioning template match, if yes, calculate the feature contour physical geometric parameters according to the feature contour, if not, adjust the shooting pose of the two-dimensional camera to the feature pattern, trigger the two-dimensional camera to take a picture and obtain an image, wherein the feature contour is the pixel contour of the feature pattern; The pose self-adaption process: control the two-dimensional camera to rotate around the machine physical coordinate system X-axis, Y-axis and Z-axis respectively to make the feature contour physical geometric parameters approximate to the positioning template physical geometric parameters; The height automatic adjustment process: control the two-dimensional camera to translate on the machine physical coordinate system Z-axis to make the feature contour physical geometric parameters approximate to the positioning template physical geometric parameters; The height automatic adjustment process comprises the following steps: Step 241: trigger the two-dimensional camera to take a picture and obtain an image; Step 242: execute the feature image processing process on the image to obtain the feature contour physical geometric parameters; The feature contour physical geometric parameters further comprise the height of the feature contour, and the height of the feature contour is the vertical distance between the camera and the feature pattern; In the above steps, the height in the feature contour physical geometric parameters satisfies the following formula: f(A1) = c n (A1) n +... + c i (A1) i +... + c0, f(A2) = d n (A2) n +... + d i (A2) i +... + d0, f(B1) = e n (B1) n +...+e i (B1) i +...+e0, f(B2) = f n (B2) n +... + f i (B2) i +... + f0, wherein f(A1) is a feature profile height calculated from A1, c n ...c i ...c0is a weight coefficient of the nth time, the ith time, and the 0th time, respectively, (A1) n ...(A1) i is a physical value of A1 obtained in the nth time, a physical value of A1 obtained in the ith time, respectively, f(A2) is a feature profile height calculated from A2, d n ...d i ...d0is a weight coefficient of the nth time, the ith time, and the 0th time, respectively, (A2) n ...(A2) i is a physical value of A2 obtained in the nth time, a physical value of A2 obtained in the ith time, respectively, f(B1) is a feature profile height calculated from B1, e n ...e i ...e0is a weight coefficient of the nth time, the ith time, and the 0th time, respectively, (B1) n ...(B1) i is a physical value of B1 obtained in the nth time, a physical value of B1 obtained in the ith time, respectively, f(B2) is a feature profile height calculated from B2, f n ...f i ...f0is a weight coefficient of the nth time, the ith time, and the 0th time, respectively, (B2) n ...(B2) i is a physical value of B2 obtained in the nth time, a physical value of B2 obtained in the ith time, respectively; the height of the feature profile can be obtained according to the above formula; Step 243: calculate the offset according to the feature contour physical geometric parameters and the positioning template physical geometric parameters, and the offset comprises ΔH, wherein ΔH is the height offset of the feature contour relative to the positioning template; The positioning template physical geometric parameters comprise the height of the positioning template, and the height of the positioning template is the vertical distance between the feature pattern and the camera, which is calculated in advance, and the height of the positioning template is calculated by the formula in step 242; Step 244: judging whether ΔH is less than a threshold value ΔH1, if ΔH is less than the threshold value ΔH1, judging whether ΔH is greater than a threshold value ΔH2, if ΔH is greater than the threshold value ΔH2, controlling the two-dimensional camera to translate on the Z axis of the machine physical coordinate system until ΔH is not greater than the threshold value ΔH2, if ΔH is not greater than the threshold value ΔH2, controlling the two-dimensional camera to iteratively translate on the Z axis of the machine physical coordinate system by a distance of ΔH until ΔH is less than the threshold value ΔH1 if ΔH is less than the threshold value ΔH1, controlling the two-dimensional camera to translate on the Z axis of the machine physical coordinate system by a distance of ΔH if ΔH is less than the threshold value ΔH1, controlling the two-dimensional camera to translate on the Z axis of the machine physical coordinate system by a distance of ΔH wherein the threshold value ΔH2 is greater than 4 times the threshold value ΔH1. Step 245: judge whether the total height adjustment value is less than ΔH3, if yes, it is determined that the feature point position is shifted due to excessive height adjustment of the robot, and the feature point position approximation process is executed again to adjust the feature point position, and the total height adjustment value is equal to the difference between the final two-dimensional camera height and the initial two-dimensional camera height.

2. The visual robot fusion method of claim 1, wherein, The workpiece coordinate system and tool coordinate system establishment process comprises the following steps: Step 111: establish a tool coordinate system and determine a TCP position, and set the TCP position on a visual center point, wherein the visual center refers to a center of an end face under a two-dimensional camera lens; Step 112: determine a working face of the robot through a workpiece coordinate system, and establish an XY plane of the workpiece coordinate system on a machining surface, wherein the machining surface is a section on the feature pattern, and a common point of the machining surface and the feature pattern is a feature point.

3. The visual robot fusion method of claim 1, wherein, The N-point calibration process specifically includes the following process: Step 121: set the TCP position on the visual center; Step 122: control the two-dimensional camera to be perpendicular to the machining surface, and control the object distance to reach a set value; Step 123: set N calibration points on the processing surface, the physical coordinates of the N calibration points are (X1, Y1, Z1), (X2, Y2, Z2)…(XN, YN, ZN) respectively, control the TCP to move to the N calibration points and acquire calibration images, determine the pixel coordinates of the TCP on the N calibration points according to the calibration images, the pixel coordinates of the TCP on the N calibration points are (x1, y1), (x2, y2)…(xN, yN) respectively. N N N N N ​​​​​ Step 124: according to the physical coordinates of the N calibration points and the pixel coordinates of the TCP on the N calibration points, substitute into the following formula and calculate the conversion matrix according to the least square method: In the formula, the conversion matrix is a, d, b, and e are respectively rotation components, and c and f are respectively translation components.

4. The visual robot fusion method of claim 1, wherein, The template making process specifically includes the following steps: Step 131: adjust the shooting pose of the two-dimensional camera on the feature pattern, trigger the two-dimensional camera to take a picture and obtain an image, wherein the feature pattern includes a feature point and a feature line segment; Step 132: obtain a pixel contour of the feature pattern, and take the pixel contour of the feature pattern as a positioning template; Step 133: calculate positioning template pixel geometric parameters from the positioning template, and convert the positioning template pixel geometric parameters into positioning template physical geometric parameters through the conversion matrix, wherein the positioning template geometric parameters include a position parameter of the feature point and a size parameter of the feature line segment.

5. The visual robot fusion method of claim 1, wherein, The feature point position approximation process specifically includes the following steps: Step 211: set the feature pattern on the workpiece, and position the workpiece on the workbench; Step 212: trigger the two-dimensional camera to take a picture and obtain an image; Step 213: execute a feature image processing process on the image to obtain feature contour physical geometric parameters; Step 214: calculating the offset according to the physical geometry parameters of the positioning template and the physical geometry parameters of the feature profile, wherein the offset includes ΔT X and ΔT Y , ΔT X is the offset of the feature profile relative to the positioning template on the X-axis of the machine physical coordinate system, and ΔT Y is the offset of the feature profile relative to the positioning template on the Y-axis of the machine physical coordinate system. Step 215: Determine ΔT X Is it less than the threshold ΔX? i If ΔT X Not less than the threshold ΔX i Then determine ΔT X Does it exceed the threshold? If ΔT X Exceeding the threshold Then control the 2D camera to translate along the X-axis of the robot's physical coordinate system to ΔT. X Not exceeding the threshold If ΔT X Not exceeding the threshold Then control the 2D camera to iteratively translate along the X-axis of the robot's physical coordinate system. The distance until ΔT X Less than If ΔT X Less than Then, the 2D camera is controlled to translate ΔT along the X-axis of the robot's physical coordinate system. X The distance, where the threshold Greater than 4 times the threshold ΔX i ; Step 216: If ΔT X Less than the threshold ΔX i Then determine ΔT Y Is it less than the threshold ΔY? i If ΔT Y Not less than the threshold ΔY i Then determine ΔT Y Does it exceed the threshold? If ΔT Y Exceeding the threshold Then control the 2D camera to translate along the Y-axis of the robot's physical coordinate system to ΔT. Y Not exceeding the threshold If ΔT Y Not exceeding the threshold Then control the 2D camera to iteratively translate along the Y-axis of the robot's physical coordinate system. The distance until ΔT Y Less than If ΔT Y Less than Then, the 2D camera is controlled to translate ΔT along the Y-axis of the robot's physical coordinate system. Y The distance, where the threshold Greater than 4 times the threshold ΔY i .

6. The visual robot fusion method of claim 1, wherein, The feature image processing process specifically includes the following steps: S11: the vision robot system extracts edge points of a pattern in the image according to a Canny algorithm; S12: the vision robot system generates a pattern contour according to the extracted edge points of the pattern, wherein the pattern contour is composed of a plurality of edge points; S13: the vision robot system calculates the number of edge points and the aspect ratio of the pattern contour according to the pattern contour, judges whether the number of edge points of the pattern contour is within a safe number range and whether the aspect ratio of the pattern contour is within a safe aspect ratio range, if yes, retains the corresponding pattern contour, and if not, eliminates the corresponding pattern contour; S14: the vision robot system judges whether a difference between one edge point and an adjacent edge point of the retained pattern contour is within a gradient change threshold, if yes, determines that the edge point is a contour intersection point, judges whether the contour intersection point of the pattern contour is within a safe intersection point number range, if yes, reconstructs a contour of the feature pattern according to the contour intersection point of the pattern contour. S15: The visual robot system calculates the feature pattern pixel geometry parameters according to the contour of the feature pattern, converts the feature pattern pixel geometry parameters into feature pattern physical geometry parameters according to the conversion matrix, judges whether the feature pattern pixel geometry parameters and the positioning template physical geometry parameters constitute a similar relationship, if yes, determines that the feature pattern and the positioning template match successfully and takes the contour of the feature pattern as the feature contour, if not, determines that the feature pattern and the positioning template do not match successfully and re-adjusts the shooting pose of the two-dimensional camera to the feature pattern and acquires the image.

7. The visual robot fusion method of claim 1, wherein, The pose adaptive process specifically includes the following steps: Step 221: Trigger the two-dimensional camera to take a picture and acquire an image; Step 222: Perform feature image processing on the image to obtain feature contour physical geometry parameters; Step 223: Calculate the offset according to the feature profile physical geometry parameters and the positioning template physical geometry parameters, the offset includes θ Z1 , θ X1 and θ Y1 , θ Z1 is the offset angle of the feature profile relative to the positioning template on the Z axis of the machine physical coordinate system, θ X1 is the offset angle of the feature profile relative to the positioning template on the X axis of the machine physical coordinate system, and θ Y1 is the offset angle of the feature profile relative to the positioning template on the Y axis of the machine physical coordinate system. Step 224: judging whether θ Z1 is less than a threshold value θ Z , if not, controlling the two-dimensional camera to rotate around the Z axis of the machine physical coordinate system until it is less than the threshold value θ Z ; Step 225: if θ Z1 is less than a threshold θ Z , determine whether θ X1 is less than a threshold θ X , if θ X1 is not less than a threshold θ X , determine whether θ X1 is greater than a threshold , if θ X1 is greater than a threshold , control the two-dimensional camera to rotate around the X axis of the physical coordinate system of the machine character by an angle of θ X1 is not greater than a threshold , if θ X1 is not greater than a threshold , control the two-dimensional camera to rotate around the X axis of the physical coordinate system of the machine character iteratively by an angle of θ until θ X1 is less than , if θ Z1 is less than , control the two-dimensional camera to rotate around the X axis of the physical coordinate system of the machine character by an angle of θ X1 , wherein the threshold θ X is greater than 4 times the threshold θ Step 226: If θ X1 Less than threshold θ X Then execute the feature point approximation process, and then determine θ. Y1 Is it less than the threshold θ? Y If θ Y1 Not less than the threshold θ Y Then determine θ Y1 Is it greater than the threshold? If θ Y1 Greater than the threshold Then control the 2D camera to rotate around the robot's physical coordinate system Y-axis to θ. Y1 Not greater than the threshold If θ Y1 Not greater than the threshold Then control the 2D camera to iteratively rotate around the Y-axis of the robot's physical coordinate system. Angle up to θ Y1 Less than If θ Y1 Less than Then control the 2D camera to rotate θ around the robot's physical coordinate system Y-axis. Y1 The angle, where the threshold θ Y Greater than 4 times the threshold Step 227: If θ Y1 is smaller than threshold θ Y , execute the feature point position approximation process, and record the current pose information of the robot.

Images