Initial visual guidance position correction method and system for workpiece grabbing

Abstract

The invention provides an initial visual guidance position correction method and system for workpiece grabbing. The initial visual guidance position correction method comprises the steps that firstly, internal parameters of a camera are calibrated, and a space geometric model with three-dimensional coordinates is established; a plurality of workpiece images are collected, and workpieces in the images are marked through the rectangular target detection frame; training a target detection model based on a deep learning method based on the marked data, and obtaining a target detection frame of the workpiece in the scene and a coordinate position of the center of the workpiece; the target position is converted into a three-dimensional coordinate position according to internal and external parameters of the mechanical arm and the camera, and finally the three-dimensional coordinate position is converted into a final guide coordinate; by means of the target detection method based on deep learning, efficient and high-precision positioning of the workpiece can be achieved, high robustness is achieved, other pose estimation devices can be connected after positioning through accurate positioning of the device, and the pose recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision positioning, and in particular to a method and system for correcting an initial vision-guided position of a workpiece grasping. Background Art

[0002] Among the existing robot grasping systems, the mainstream grasping solutions are mostly based on 2D solutions based on template matching and pure 3D solutions. The 3D solution is expensive. In actual application, the 2D solution that can achieve functions at a lower cost in the production environment occupies a certain market. The 2D solution's disordered material grabbing is mostly based on the template matching method to locate the workpiece. Most 2D template matching has the following problems:

[0003] 1. Dependence on the object's posture: Template matching usually assumes that the target object has a specific posture or orientation in the image. If the object rotates during the grasping process, template matching may fail. Therefore, general template matching methods are not used for complex workpieces, because complex workpieces will have large changes in pitch and roll angles, and changes in perspective relationships will cause template matching to be small;

[0004] 2. Sensitivity to occlusion: If the target object is partially or completely occluded by other objects, template matching may not be able to find the correct matching position or may not be able to match at all, and positioning will fail.

[0005] 3. Lack of generalization ability: Template matching relies on predefined templates, so it usually only works for objects that are very similar to the template. For objects with slight changes in shape, size or color, new templates may need to be created, which limits its generalization ability.

[0006] 4. Computational complexity: In large images or complex scenes, template matching may require a lot of computing resources to search and match target objects. This may lead to a decrease in real-time performance and is not suitable for application scenarios that require fast response.

[0007] 5. Sensitivity to background noise: Background noise in the image, such as texture, color changes, or reflections, may interfere with the template matching algorithm, causing localization and failure.

[0008] 6. Insufficient robustness of initial pose: Template matching methods generally require the mechanical gripper to maintain a common plane with the object. It is difficult to guarantee such harsh conditions in complex application scenarios in actual production, thus resulting in reduced grasping accuracy.

[0009] Therefore, in order to address the problems existing in the above solutions, it is necessary to design an efficient and accurate visual positioning guidance mechanism to achieve precise positioning of grasped objects in complex scenes and improve the grasping accuracy of the entire grasping system. Summary of the invention

[0010] The present invention provides an initial vision-guided position correction method and system for workpiece grasping, which aims to achieve accurate positioning of grasped objects in complex scenes.

[0011] In order to achieve the above object, the present invention provides an initial visual guidance position correction method for workpiece grasping, which is characterized by comprising:

[0012] Calibrate the internal parameters of the camera and establish a spatial geometric model with three-dimensional coordinates, wherein the spatial geometric model has a positional relationship of corresponding points in the image;

[0013] Collect multiple workpiece images and mark the workpieces in the images using rectangular target detection frames;

[0014] Based on the labeled data, a target detection model based on deep learning method is trained. According to the target detection model, the target detection frame of the workpiece in the scene is obtained. The coordinate position of the center of the workpiece is calculated through the positions of the upper left corner and the lower right corner of the target detection frame.

[0015] Calculate the relative position of the camera and the robotic arm as external parameters and set the height of the camera movement, calculate the external parameters of the camera, and then convert the camera target position into a three-dimensional coordinate position, and finally convert the three-dimensional coordinates into the final guidance coordinates;

[0016] The robot arm is moved according to the guide coordinates to complete the grasping of the workpiece.

[0017] Furthermore, the internal parameters of the camera include the focal length and principal point position of the camera.

[0018] Furthermore, according to the external parameters of the robot The relative position of the calibrated robot arm and camera Calculate the camera's extrinsic parameters The calculation process is as follows:

[0019]

[0020] Further, according to the predetermined moving height Z of the camera c , the predetermined two-dimensional plane target position P in the camera image uv Convert to three-dimensional coordinates in the camera coordinate system The calculation process is as follows:

[0021]

[0022] Furthermore, the target position in the camera's three-dimensional coordinate system is converted into the final guidance coordinates The calculation process is as follows:

[0023]

[0024] Further, the end of the robot arm is moved to the final guide coordinate P obj The center is shifted 70 degrees along the X-axis. The angle can be modified according to the data collected during subsequent training, and the range of the data collected during training can be kept consistent. Since the present invention selects a 70-degree cone range during collection, it is moved to a 70-degree angle position.

[0025] Furthermore, the target detection model adopts a yolo or retinanet model.

[0026] On the other hand, the present invention also provides an initial visual guidance position correction system for workpiece grasping, including a camera, the camera is connected through the top of a jig robot arm, a clamp is connected to the end of the robot arm, the robot arm is connected to a control device, and the control device executes an initial visual guidance position correction method for workpiece grasping as described above.

[0027] The present invention provides an initial visual guidance position correction method and system for workpiece grasping, which firstly calibrates the internal parameters of a camera and establishes a spatial geometric model with three-dimensional coordinates; collects multiple workpiece images, and annotates the workpieces in the images through a rectangular target detection frame; trains a target detection model based on a deep learning method based on the annotated data, and obtains the target detection frame of the workpiece in the scene and the coordinate position of the center of the workpiece; converts the target position into a three-dimensional coordinate position according to the internal and external parameters of a robotic arm and a camera, and finally converts it into a final guidance coordinate according to the three-dimensional coordinates; moves the robotic arm according to the guidance coordinates to complete the grasping of the workpiece. The present invention uses a target detection method based on deep learning, and can achieve efficient and high-precision positioning of the workpiece, and has strong robustness. Through the precise positioning of the device, other posture estimation devices can be connected after positioning to improve the accuracy of posture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 The present invention is a flowchart of a method for correcting an initial vision-guided position of a workpiece grasping according to an embodiment of the present invention.

[0030] Figure 2 It is a schematic structural diagram of a workpiece grasping robot arm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0033] The purpose of the present invention is to provide a method and system for correcting the initial visual guidance position of workpiece grasping, which aims to achieve accurate positioning of the grasped object in complex scenes.

[0034] Figure 1 This is a method flow chart of a method for correcting the initial visual guidance position of a workpiece grasping method of the present invention. Figure 1 At any time, an initial vision-guided position correction method for workpiece grasping of the present invention includes a calibration and training process and an inference process.

[0035] The calibration and training process includes the following steps:

[0036] Step 1: Camera Calibration

[0037] Camera calibration is used to calculate the camera's intrinsic and extrinsic parameters. The camera's intrinsic parameters include the camera's focal length and principal point position, and the extrinsic parameters include the camera's position and orientation in the world coordinate system. By modeling the camera, the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image can be determined. Since the precise position needs to be calculated, the calibration process needs to be as accurate as possible. In the present invention, it is particularly necessary to calibrate the camera's intrinsic parameters and the relationship between the end of the robotic arm and the camera.

[0038] Step 2: Data Collection

[0039] In a standard scenario, multiple images containing workpieces are collected, and the image data needs to have certain differences, such as the relative relationship between the camera and the workpiece needs to change within a certain range.

[0040] Step 3: Data Labeling

[0041] The data collected in step 2 is standardized, and the artifact appearing in the image is marked as a rectangular target recognition frame, and its category is indicated.

[0042] Step 4: Train the target detection model

[0043] According to the image data annotated in step 3, train the target detection model based on deep learning method, and the model adopts models such as yolo / retinanet.

[0044] Step 5: Calculate the target center point in the image

[0045] Based on the object detection model trained by PlayCar, the object detection box in the newly acquired image is inferred. The position of the center point of the object can be calculated through the pixel positions of the upper left corner and the lower right corner of the object detection box, and the center point position is marked as C. obj .

[0046] Step 6: Calculation of 3D guide position

[0047] When obtaining the center point position P uv After that, the robot arm can be used according to the external parameters The relationship between the end-arm and the camera has been calibrated Camera Intrinsic Parameters I cam , we can calculate the camera external At this time, you can set the height Z you want the camera to move to. c , convert the camera position back to the three-dimensional coordinates in the camera coordinate system Then convert the three-dimensional coordinates in the camera coordinate system into the final guide position coordinates

[0048] Step 7: Robotic arm moves

[0049] Then move the end of the robot arm to With the workpiece as the center, it can be offset 70 degrees along the x-axis, so that the workpiece can be positioned more clearly.

[0050] Figure 2 The schematic diagram of the structure of a workpiece grasping robot arm according to an embodiment of the present invention includes a camera, which is connected to the top of the fixture robot arm, and a clamp is connected to the end of the robot arm. The robot arm is connected to the control device to implement the above workpiece grasping method

[0051] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for correcting the initial visual guidance position of a workpiece grasping, characterized in that: include: Calibrate the internal parameters of the camera and establish a spatial geometric model with three-dimensional coordinates, wherein the spatial geometric model has a positional relationship of corresponding points in the image; Collect multiple workpiece images and mark the workpieces in the images using rectangular target detection frames; Based on the labeled data, a target detection model based on deep learning method is trained. According to the target detection model, the target detection frame of the workpiece in the scene is obtained. The coordinate position of the center of the workpiece is calculated through the positions of the upper left corner and the lower right corner of the target detection frame. Calculate the relative position of the camera and the robotic arm as external parameters and set the height of the camera movement, calculate the external parameters of the camera, and then convert the camera target position into a three-dimensional coordinate position, and finally convert the three-dimensional coordinates into the final guidance coordinates, where the camera external parameters are used to represent the position of the camera itself in the coordinate system; The robot arm is moved according to the guide coordinates to complete the grasping of the workpiece.

2. The method for correcting the initial visual guidance position of a workpiece grasping according to claim 1, characterized in that: The internal parameters of the camera include the focal length and the principal point position of the camera.

3. A method for correcting the initial visual guidance position of a workpiece grasping as described in claim 1, characterized in that: According to the external parameters of the robot The relative position of the calibrated robot arm and camera Calculate the camera's extrinsic parameters The calculation process is as follows:

4. A method for correcting the initial visual guidance position of a workpiece grasping as claimed in claim 3, characterized in that: According to the camera's predetermined moving height Z c , the predetermined two-dimensional plane target position P in the camera image uv Convert to three-dimensional coordinates in the camera coordinate system The calculation process is as follows:

5. A method for correcting the initial visual guidance position of a workpiece grasping according to claim 4, characterized in that: Convert the target position in the camera's 3D coordinate system to the final guidance coordinate The calculation process is as follows:

6. A method for correcting the initial visual guidance position of a workpiece grasping as claimed in claim 5, characterized in that: Move the end of the robot arm to the final guide coordinate The center is offset by 70 degrees along the X axis. The 70-degree position is because in the subsequent posture model training, the 70-degree cone range of the workpiece center is selected for data collection. Therefore, choosing 70 degrees can maintain the relative consistency between training and reasoning. This angle can be modified according to the data collection during subsequent training, and it can be kept consistent with the range of data collected during training.

7. The method for correcting the initial visual guidance position of a workpiece grasping according to claim 1, characterized in that: The target detection model adopts models such as yolo or retinanet.

8. An initial visual guidance position correction system for workpiece grasping, characterized in that: It includes a camera, which is connected through the top of a fixture robot arm, a clamp is connected to the end of the robot arm, and the robot arm is connected to a control device. The control device executes an initial visual guidance position correction method for workpiece grasping as described in claims 1 to 7.

Images