Vision-based workpiece accurate placement method, equipment and device

Abstract

The invention provides an initial visual guidance position correction method and system for workpiece grabbing. The initial visual guidance position correction method comprises the steps that firstly, internal parameters of a camera are calibrated, and a space geometric model with three-dimensional coordinates is established; a plurality of workpiece images are collected, and workpieces in the images are marked through the rectangular target detection frame; training a target detection model based on a deep learning method based on the marked data, and obtaining a target detection frame of the workpiece in the scene and a coordinate position of the center of the workpiece; the target position is converted into a three-dimensional coordinate position according to internal and external parameters of the mechanical arm and the camera, and finally the three-dimensional coordinate position is converted into a final guide coordinate; by means of the target detection method based on deep learning, efficient and high-precision positioning of the workpiece can be achieved, high robustness is achieved, other pose estimation devices can be connected after positioning through accurate positioning of the device, and the pose recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision positioning technology, and in particular to a method, equipment and device for precise workpiece placement based on vision. Background Art

[0002] In the existing robot grasping system, in the workpiece position recognition-grasping-correction-placement task, placing the workpiece on a fixed fixture is the last step of the entire process. Whether the workpiece placement link is successful determines whether the entire recognition-grasping-correction-placement task can be completed, which is a crucial link.

[0003] However, in the existing workpiece grasping and placement solutions, there are errors in the calibration of the camera, hand-eye matrix and gripper. As a result, even if the predicted posture and posture correction algorithms are very accurate, due to the errors (approximately 2 to 3 mm) in the calibrated hand-eye matrix and gripper matrix, it is difficult to achieve precise placement for fixtures with high placement accuracy requirements (error requirement within 1 mm).

[0004] There are some problems with the existing technical solutions for solving the problem of accurate workpiece access:

[0005] 1. Use a camera to photograph the workpiece, and use feature matching to correct the workpiece's position to the state of the template workpiece. This solution requires high-precision calibration. Even if the feature matching algorithm is highly accurate, the existing calibration solution makes it difficult for the hand-eye matrix and the gripper matrix to meet strict placement accuracy requirements (within 1 mm).

[0006] 2. Use force sensors to output force feedback during placement to guide the placement of the workpiece. This solution requires high-precision force sensors, which are more expensive than cameras.

[0007] Therefore, in order to address the problems existing in the above solutions, it is necessary to design an efficient and accurate visual positioning guidance mechanism to achieve precise placement of workpieces. Summary of the invention

[0008] The present invention provides a method, equipment and device for precise workpiece placement based on vision, which captures an image of a workpiece when it is placed on a fixture, uses a deep learning algorithm to determine the degree of fit between the workpiece and the locating pins on the fixture, and outputs the workpiece posture that needs to be updated by the robot arm TCP, thereby avoiding the influence of precision errors during tooling calibration on the entire process and solving the problem of precise workpiece placement at a low cost.

[0009] In order to achieve the above object, the present invention provides a method for accurate workpiece placement based on vision, comprising:

[0010] Place a camera with multiple viewing angles on the fixture attachment to take photos of the fixture positioning pins;

[0011] Grasp the workpiece with the collaborative arm, teach the process of accurately placing the workpiece on the fixture multiple times, and obtain multi-view video and TCP readings;

[0012] The relative pose of the workpiece is calculated based on the front and rear TCP readings, and a training data set of relative pose recorded during multiple recordings and precise placement under multiple perspectives is established. A deep learning model is trained based on the training data set. The deep learning model inputs the workpiece position map at the fixture locating pin and outputs the TCP position and relative pose that should be adjusted.

[0013] Furthermore, during the shooting process of workpiece placement, each camera shoots K frames of images and records K TCP readings in d teaching trajectories to obtain N*K*d groups of (image, TCP) data sets.

[0014] Furthermore, during the training process of the deep model, the relative posture deltaT of the front and rear tcp readings is calculated, and the (image, tcp) data set is converted into an (image, deltaT) data set to obtain N*K*d groups of (image, deltaT) training data sets to train the deep learning model.

[0015] Furthermore, according to the deep learning model, the image of the workpiece arranged at the fixture locating pin is input, and the relative posture deltaT that should be adjusted is output.

[0016] Furthermore, the test process of the deep learning model is also included, including:

[0017] Take an image of the workpiece at the fixture locating pin and input it into the deep learning model to obtain the deltaT pose that needs to be adjusted;

[0018] Adjust the tcp coordinates of the workpiece based on the output deltaT posture to change the workpiece posture so that its positioning hole and the fixture positioning pin are more closely matched;

[0019] Take an image of the adjusted workpiece and determine whether the adjusted workpiece position deltaT is less than the accuracy tolerance, wherein: if deltaT is less than the accuracy tolerance, the placement is completed; if deltaT is greater than the accuracy tolerance, readjust the positioning.

[0020] On the other hand, the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0021] On the other hand, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the steps of the above method.

[0022] The present invention has the following beneficial effects and technical advances.

[0023] 1. The present invention can avoid the calibration errors of the hand-eye matrix and the gripper matrix, and directly obtain the TCP posture that needs to be adjusted from the image. The posture that needs to be adjusted is obtained by using the obvious features between the fixture positioning pins and the placement holes on the workpiece and the deep learning algorithm. Compared with the previous vision-based solutions, this solution is not affected by the calibration error and achieves precise placement.

[0024] 2. The data collection method for training the deep learning model of the present invention is simple and efficient. It uses multiple cameras to simulate the placement data of workpieces in different postures, and can obtain a large amount of data with a wide distribution through a small amount of teaching.

[0025] 3. The present invention is based on a pure visual device, does not require a force sensor, and can achieve accurate placement of workpieces at a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 The present invention is a flowchart of a method for precise workpiece placement based on vision according to an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of camera installation during testing of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0031] Figure 1 The flowchart of the method of the present invention is a method of precise workpiece placement based on vision. Figure 1 As shown, a vision-based workpiece precision placement method of the present invention includes a calibration and training process and a testing process.

[0032] Among them, the training process is used to build a deep learning model that outputs the updated posture of the robot arm. The deep learning model judges the degree of fit between the workpiece and the locating pins on the fixture based on the image taken when the workpiece is placed on the fixture, and outputs the workpiece posture that the robot arm TCP needs to update.

[0033] The testing process adjusts the TCP coordinates of the robot arm based on the workpiece posture output by the deep learning model, and continuously changes the workpiece posture through iteration to make it more accurately placed.

[0034] Specifically, in this embodiment, the training process includes the following steps:

[0035] Step 1.1, set up multiple cameras (N) near the fixture, shoot at the fixture positioning pins to obtain the distance between the workpiece placement hole and the fixture positioning pins.

[0036] like Figure 2 As shown in the figure, in order to increase the amount of training data collected and the breadth of data distribution, N cameras are arranged around the fixture to shoot at the fixture locating pins. The multi-view images taken by multiple cameras are used to simulate the situation when the workpiece is taught in different postures, thereby increasing data distribution.

[0037] Step 1.2, the teaching process is used to provide a training data set for training the deep learning model. The training data set includes data recording the robot arm TCP reading trajectory and multi-view shooting video.

[0038] Specifically, this embodiment uses a collaborative arm to teach the grasping of workpieces. After the collaborative arm grasps the workpiece, it teaches the process of accurately placing the workpiece on the fixture d times. In this process, the robot arm TCP reading trajectory and the video taken by multiple viewing angle cameras are recorded. Among them, each video captures K frames of images, corresponding to the K TCP coordinate readings in the current robot arm teaching trajectory.

[0039] The training data set includes N*K*d groups of images and TCP readings, where N is the number of cameras with different viewing angles, d is the number of teaching times, and the Kth frame image and the Kth corresponding TCP reading are obtained during each teaching process.

[0040] Step 1.3, transform N*K*d groups of images and TCP data sets to obtain the relative position deltaT of the workpiece based on the front and rear TCP readings, and obtain N*K*d (image, deltaT) training data sets.

[0041] The converted (image, deltaT) training data set is used as the training data set to train the deep learning model. The model directly outputs deltaT, which is convenient for using the threshold to determine when to stop TCP adjustment.

[0042] Step 1.4, use the collected data to train the deep learning model, input the workpiece-fixture image taken at the fixture positioning pin, and output the position deltaT that should be adjusted and the TCP coordinates of the robot arm.

[0043] Specifically, the testing process of the deep learning model includes the following steps:

[0044] Step 2.1 Take an image of the workpiece at the fixture locating pin and input it into the deep learning model to obtain the deltaT pose that needs to be adjusted.

[0045] Step 2.2, adjust the tcp coordinates of the workpiece based on the output deltaT posture to change the workpiece posture so that its positioning hole and the fixture positioning pin are more closely matched.

[0046] Step 2.3 captures the image of the adjusted workpiece and determines whether the adjusted workpiece position deltaT is less than the accuracy tolerance, wherein: if deltaT is less than the accuracy tolerance, placement is completed; if deltaT is greater than the accuracy tolerance, readjust the positioning.

[0047] On the other hand, the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0048] On the other hand, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the steps of the above method.

[0049] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for accurate workpiece placement based on vision, characterized in that: include: Place a camera with multiple viewing angles on the fixture attachment to take photos of the fixture locating pins and the prevention holes on the workpiece; Grasp the workpiece with the collaborative arm, teach the process of accurately placing the workpiece on the fixture multiple times, and obtain multi-view video and TCP readings; The relative pose of the workpiece is calculated based on the front and rear TCP readings, and a training data set of relative pose recorded during multiple recordings and precise placement under multiple perspectives is established. A deep learning model is trained based on the training data set. The deep learning model inputs the workpiece position map at the fixture locating pin and outputs the TCP position and relative pose that should be adjusted.

2. A method for accurate workpiece placement based on vision as claimed in claim 1, characterized in that: During the shooting process of workpiece placement, each camera takes K frames of images and records K TCP readings in d teaching trajectories to obtain N*K*d groups of (image, TCP) data sets.

3. A method for accurate workpiece placement based on vision as described in claim 2, characterized in that: During the training process of the deep model, the relative position deltaT of the front and rear tcp readings is calculated, and the (image, tcp) data set is converted into an (image, deltaT) data set to obtain N*K*d groups of (image, deltaT) training data sets to train the deep learning model.

4. A method for accurate workpiece placement based on vision as claimed in claim 3, characterized in that: According to the deep learning model, the image of the workpiece arranged at the fixture positioning pin is input, and the relative posture deltaT that should be adjusted is output.

5. A method for accurate workpiece placement based on vision as claimed in claim 4, characterized in that: It is characterized in that Also included is a testing process for the deep learning model, including: Take an image of the workpiece at the fixture locating pin and input it into the deep learning model to obtain the deltaT pose that needs to be adjusted; Adjust the tcp coordinates of the workpiece based on the output deltaT posture to change the workpiece posture so that its positioning hole and the fixture positioning pin are more closely matched; Take an image of the adjusted workpiece and determine whether the adjusted workpiece position deltaT is less than the accuracy tolerance, wherein: if deltaT is less than the accuracy tolerance, the placement is completed; if deltaT is greater than the accuracy tolerance, readjust the positioning.

6. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images