A mechanical arm grabbing crawler plate method based on monocular vision

Abstract

The application discloses a mechanical arm grabbing crawler plate method based on monocular vision, and the method comprises the following steps: creating a deformable template with calibration according to the relationship between the crawler plate plane and the camera coordinate system and the scaling ratio and other conditions; collecting the current working picture of the piled material, calculating the matching search area in the piled material picture based on the scaling coefficient and the preset ROI area, matching the deformable template with calibration, obtaining all the graspable crawler plate poses, and finally converting the obtained crawler plate pose according to the relationship between the grabbing space coordinate system and the camera coordinate system and the relationship between the crawler plate plane and the camera coordinate system, obtaining the pose of the crawler plate in the grabbing coordinate system. The application unifies all the coordinate systems to the grabbing space coordinate system, thereby simplifying the entire vision device and avoiding the direct conversion of the coordinate systems between the camera and the mechanical arm.

Description

Technical Field

[0001] This invention relates to a method for a robotic arm to grasp track plates based on monocular vision, belonging to the field of machine vision system technology. Background Technology

[0002] Currently, most vision-based grasping methods are quite cumbersome. For example, binocular vision-based grasping requires a very complex calibration process, necessitates rigid interconnection between cameras, and is costly. Secondly, grasping methods where the camera is mounted on a robotic arm suffer from wear and tear on the camera due to prolonged robotic arm operation, and the vibration of the robotic arm affects the already calibrated internal parameters of the camera. Furthermore, calibration with the robotic arm's coordinate system is also required, making the process quite complex. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for a robotic arm to grasp track plates based on monocular vision. By unifying all coordinate systems into the grasping space coordinate system, the entire vision device is simplified, avoiding direct conversion between the coordinate systems of the camera and the robotic arm. The operation process is simple and easy to operate.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] This invention provides a method for a robotic arm to grasp a track plate based on monocular vision, comprising:

[0006] S1. The camera is calibrated to obtain the camera's intrinsic parameters; the camera is installed directly above the trolley, the trolley is placed on a full stack of track plates, and the camera's field of view can include the full stack of track plates.

[0007] S2. Determine the relationship between the track plate plane and the camera coordinate system, and construct a calibrated deformable template based on the pose of the track plate under the camera.

[0008] S3. Collect a stack image during the current operation, calculate the matching search area in the stack image based on the scaling factor and the preset ROI area, match it with the calibrated deformable template, obtain the poses of all grabbable track plates, calculate the average height of the current track plate layer, and update the scaling factor.

[0009] S4. Determine whether the number of all acquired grabbable track plate poses matches the set number of poses.

[0010] If the conditions are met, output the poses of all the grabbable track plates under the camera.

[0011] If the number of poses is incorrect, the track plate corresponding to the abnormal pose is determined. Based on the scaling factor and the separately set ROI region, the matching search region of the track plate corresponding to the abnormal pose is calculated and matched with the calibrated deformable template to obtain the abnormal track plate pose and summarize it into the track plate pose.

[0012] S5. Based on the relationship between the grasping space coordinate system and the camera coordinate system, and the relationship between the track plate plane and the camera coordinate system, perform coordinate system transformation on the obtained track plate pose to obtain the track plate pose in the grasping coordinate system.

[0013] Furthermore, determining the relationship between the track plate plane and the camera coordinate system, and constructing a calibrated deformable template based on the track plate's pose under the camera, includes:

[0014] Place a single track plate on the trolley, place a dot calibration plate on the surface of the track plate, and make the origin of the dot calibration plate coincide with the center of the track plate;

[0015] The image of the dot calibration plate is acquired, and the pose of the origin of the dot calibration plate in the camera coordinate system is determined using the EPnP algorithm. 42 ,T 42 ];

[0016] Based on the thickness of the dot calibration plate, the pose of the track plate under the camera is obtained [R]. ′ 42 T4 ′ 2];

[0017] Remove the dot calibration plate and acquire an image of a single track plate with intact surface features;

[0018] Position of the track plate under the camera [R] ′ 42 T4 ′ 2] Use the reference pose to establish a calibrated deformable template.

[0019] Furthermore, after acquiring the images of the stacked materials, the process also includes:

[0020] Determine whether it is the first loading. If it is the first loading, use the initial scaling factor multiplied by the preset ROI area as the matching search area for the stacking image and match it with the calibrated deformable template.

[0021] If it is not the first time the material is loaded, the scaling factor of the last update is multiplied by the preset ROI area as the matching search area for the stacking image, and it is matched with the calibrated deformable template.

[0022] Furthermore, after obtaining the poses of all grabbable track plates, the process also includes:

[0023] Abnormal poses are removed, and then the average height of the current track plate layer is calculated.

[0024] Furthermore, the update scaling factor includes:

[0025] Based on the current average height of the track plate layer and the template height of the calibrated deformable template, the scaling factor is updated as follows:

[0026] R1 = H0 / H;

[0027] Where R1 is the updated scaling factor, H0 is the template height, and H is the average height of the current track plate layer.

[0028] Furthermore, the ROI area for each individual track plate is determined as follows:

[0029] A single track plate is placed on the bottom frame of the vehicle to capture images of the track plate;

[0030] The central feature region of the track plate image is selected as a template using Halcon's template matching function;

[0031] The bounding rectangle of the template is determined as the ROI area for a single track plate.

[0032] Furthermore, 15 images of the dot calibration plate were acquired to calibrate the camera and obtain the camera intrinsic parameter K.

[0033] Furthermore, the relationship between the grasping space coordinate system and the camera coordinate system is determined as follows:

[0034] Determine the top left corner of the trolley as the origin of the grasping space coordinate system, and place the calibration plate at the origin of the grasping space coordinate system;

[0035] The calibration board image is acquired, and the EPnP algorithm is used to solve for the 3D coordinates of the projection points in the camera coordinate system, transforming the problem into a 2D to 3D pose estimation problem. The pose of the origin of the grasping space coordinate system in the camera is determined through pose calculation. 43 ,T 43 ].

[0036] Furthermore, based on the relationship between the grasping space coordinate system and the camera coordinate system, and the relationship between the track plate plane and the camera coordinate system, the obtained track plate pose is transformed to obtain the track plate pose in the grasping coordinate system, as follows:

[0037]

[0038] Among them, [R 31 ,T 31 [R] represents the pose of the track plate in the grasping coordinate system.41 ,T 41 [ ] represents the position of the track plate under the camera.

[0039] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0040] This invention proposes a method for a robotic arm to grasp track plates based on monocular vision. By unifying all coordinate systems into the grasping space coordinate system, the entire vision device is simplified, avoiding direct coordinate system conversion between the camera and the robotic arm. In actual operation, the scaling factor of the previous layer is used to calculate the matching search area of ​​the next layer based on whether it is the first time the material is loaded, which greatly speeds up the search and improves the system efficiency.

[0041] This invention uses a dot calibration plate to obtain the position and pose of the track plate under the camera, and then uses this as a reference position and pose to establish a calibration deformable template. If there are different types of track plates in the future, it is only necessary to recalibrate the track plate, establish a new template, and make simple adjustments before use.

[0042] This invention uses only one camera, saving costs, maximizing benefits, and making it easy to add new production lines later. Attached Figure Description

[0043] Figure 1 This is a flowchart of a method for a robotic arm to grasp a track plate based on monocular vision, provided by the present invention.

[0044] Figure 2 The diagram shows the calibration of the track plates. In the diagram, Xw, Yw, and Zw are the grasping space coordinate system, Xc, Yc, and Zc are the camera coordinate system, and Xobj, Yobj, and Zobj are the track plate coordinate systems.

[0045] Figure 3 A schematic diagram of a robotic arm device for grasping track plates based on monocular vision provided by the present invention;

[0046] Figure 4 This is an example of the pose of the calibration plate origin in the camera coordinate system in this invention;

[0047] Figure 5 This is an example of the pose of the center of the track plate template in the camera coordinate system in this invention;

[0048] Figure 6 This is an example of the pose of the track plate in the grasping space coordinate system in this invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0050] This invention provides a robotic arm device for grasping track plates based on monocular vision, see [link / reference]. Figure 3 It includes a gripping space plane 3, a robotic arm, and an industrial camera 4. The target workpiece 6 is placed on the gripping space plane, the industrial camera is placed directly above the gripping space plane, and the robotic arm is placed to one side of the gripping space plane.

[0051] The robotic arm includes a base 5, a flange 7 at the end of the robotic arm, and an electromagnet chuck 8 fixed on the flange 7. The electromagnet chuck 8 is used to grip the track plate.

[0052] Based on the above-described device, the present invention provides a method for a robotic arm to grasp a track plate based on monocular vision, such as... Figure 1 As shown, the specific implementation includes the following steps:

[0053] Step 1: Calibrate the camera to obtain its intrinsic parameters.

[0054] Step 2: Install an industrial camera directly above the trolley and place the full stack of track plates on the trolley so that the camera's field of view includes the full stack of track plates.

[0055] It should be noted that the camera's Z-coordinate system should be kept perpendicular to the plane of the trolley to ensure that the trolley's position is correct in the camera's field of view. As the number of stacks increases, the top of the stack will be magnified in the trolley's field of view. In order to not exceed the camera's field of view, the camera height should be adjusted to accommodate the full stack's field of view.

[0056] Step 3: Determine the relationship between the track plate plane and the camera coordinate system, and construct a calibrated deformable template based on the pose of the track plate under the camera.

[0057] Step 4: Acquire a stack image during the current operation. Calculate the matching search region in the stack image based on the scaling factor and the preset region of interest (ROI). Match the region with the calibrated deformable template to obtain the poses of all grabbable track plates. Calculate the average height of the current layer of track plates based on the poses of all grabbable track plates, and update the scaling factor.

[0058] It should be noted that step four also includes:

[0059] When collecting images of the stacked materials during the work process, it is determined whether it is the first time the materials are loaded. If it is the first time the materials are loaded, the preset ROI region in the stacked material image is template matched using the initialized scaling factor. If it is not the first time the materials are loaded, the preset ROI region in the stacked material image is template matched using the previously updated scaling factor.

[0060] It should be noted that step four also includes:

[0061] After obtaining the poses of all grabbable track plates, abnormal poses are removed, and then the average height of the current track plate layer is calculated.

[0062] It should be noted that in step four of this procedure, updating the scaling factor means:

[0063] Based on the current average height of the track plates in this layer and the template height of the calibrated deformable template, the scaling factor is updated as follows:

[0064] R1 = H0 / H;

[0065] Where R1 is the updated scaling factor, H0 is the template height, and H is the average height of the track plate in this layer.

[0066] It should be noted that step four also includes:

[0067] If it is the first time loading materials, the matching search area of ​​the material stacking image is based on the scaling factor R0 multiplied by the preset ROI area in the material stacking image, and then matched with the template.

[0068] It should be noted that the ROI area is the approximate area where the track plates are located.

[0069] If it is not the first time the material is loaded, the matching search area of ​​the material loading image is based on the previously updated scaling factor R1 multiplied by the preset ROI area in the material loading image, and matched with the template, thereby reducing the search space and speeding up the search efficiency.

[0070] Step 5: Determine whether the number of track plate poses matched in Step 4 matches the number of poses set for the current layer.

[0071] If the number of poses is correct, output the poses of all track plates matched in step four under the camera, denoted as [R]. 41 ,T 41 ];

[0072] If the number of poses is incorrect, the track plate corresponding to the abnormal pose is determined based on the abnormal pose and its corresponding number. Then, based on the scaling factor, a separately set ROI region is used to calculate the matching lookup region for the track plate corresponding to the abnormal pose. Template matching is performed to obtain the abnormal track plate pose, and the results are summarized in the output pose [R]. 41 ,T 41 ]middle.

[0073] It should be noted that the specific implementation process for obtaining the pose of abnormal track plates is as follows:

[0074] S1. Acquire the track plate template image, as follows:

[0075] Place a single track plate on the bottom frame of the trolley (e.g.) Figure 5Images are collected and subsequently used to create calibrated templates.

[0076] S2. Determine the template outline and draw the bounding rectangle of the outline based on the track plate template image, as follows:

[0077] The template matching function of Halcon software is used to select the central area with obvious features of the track plate as a template, and the size of the bounding rectangle of the template is determined.

[0078] S3. Determine the template area for a single track plate;

[0079] This area is used to subsequently identify regions with abnormal poses and perform matching again in this area.

[0080] S4, Affine transformation of the track plate region:

[0081] Perform pose coordinate system transformation based on calibration parameters.

[0082] S5. Obtain the location of the abnormal track plate area, as follows:

[0083] Since the size and number of track plates are fixed, the pose of abnormal areas can be determined based on the already obtained pose.

[0084] Step Six: Based on the relationship between the grasping space coordinate system and the camera coordinate system, and the relationship between the track plate plane and the camera coordinate system, perform coordinate system transformation on the obtained output pose to obtain the pose of the track plate in the grasping coordinate system [R]. 31 ,T 31 ].

[0085] For the relationships between the coordinate systems, see Figure 2 .

[0086] It should be noted that, in order to prevent errors during offline operation from causing deviations in the robotic arm's grasping position, compensation parameters need to be added to the X, Y, and Z directions of the final pose sent to the PLC for debugging to obtain the optimal pose.

[0087] In one preferred embodiment of the present invention, 15 images of a dot calibration plate are acquired to calibrate the camera and obtain the camera intrinsic parameter K.

[0088] In one preferred embodiment of the present invention, the relationship between the grasping space coordinate system and the camera coordinate system is determined as follows:

[0089] Determine the top left corner of the trolley as the origin of the grasping space coordinate system, and place the calibration plate at the origin of the grasping space coordinate system;

[0090] Acquire images of the calibration board to determine the pose of the origin of the capture space coordinate system under the camera [R]. 43 ,T43 ].

[0091] In this embodiment, EPnP is used to solve for the 3D coordinates of the projection point in the camera coordinate system. Finally, the problem is transformed into a 2D to 3D pose estimation problem. The pose of the origin of the grasping space coordinate system in the camera is obtained through pose calculation. 43 ,T 43 ].

[0092] As a preferred embodiment, in one instance of the present invention, the relationship between the track plate plane and the camera coordinate system is determined, and a calibrated deformable template is constructed based on the pose of the track plate under the camera. The specific method is as follows:

[0093] See Figure 2 To establish the relationship between the coordinate systems, place a single track plate 1 on the trolley and place a dot calibration plate 2 on the surface of the track plate, aligning the origin of the dot calibration plate (i.e., the center point of the dot calibration plate) with the center of the track plate as much as possible. Figure 4 As shown;

[0094] Acquire images of the dot calibration plate and calculate the pose of the center origin of the dot calibration plate in the camera coordinate system. 42 ,T 42 Then, based on the thickness of the dot calibration plate itself, the position and pose of the track plate under the camera are obtained [R]. ′ 42 T4 ′ 2];

[0095] Remove the dot calibration plate and acquire an image of a single track plate with intact surface features, such as... Figure 5 As shown, the position of the track plate under the camera [R] ′ 42 T4 ′ 2] Use the reference pose to establish a calibrated deformable template.

[0096] It should be noted that:

[0097] The calibrated deformable template is a calibrated template that can be recognized even if the matched object has slight deformation.

[0098] In this embodiment, EPnP is used to solve for the 3D coordinates of the projection point in the camera coordinate system. Finally, the problem is transformed into a 2D to 3D pose estimation problem. The pose of the calibration board center origin in the camera is obtained through pose calculation. 42 ,T 42 ].

[0099] To achieve more accurate pose, the 3mm thickness of the calibration plate was taken into account, resulting in the pose of the track plate under the camera [R]. ′42 T4 ′ 2).

[0100] The specific conversion process is as follows:

[0101] Based on the calibration plate's own thickness of 3mm, the translation matrix changes after the calibration plate is moved to a new position as follows:

[0102]

[0103] Therefore, the relationship between the new position calibration board and the camera pose transformation is as follows:

[0104]

[0105] Using the new position of the calibration plate as a reference pose, a deformable template can be created to obtain the pose transformation relationship of the track plate under the camera:

[0106]

[0107] M is the transformation matrix from the new pose of the calibration plate to the center of gravity pose of the track plate template.

[0108] Therefore, the pose transformation relationship between the track plate and the grasping space coordinate system is as follows:

[0109]

[0110] Finally, the pose of the track plate in the grasping coordinate system is obtained, such as Figure 6 As shown.

[0111] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A monocular vision-based mechanical arm grabbing track plate method, characterized in that, The application relates to a method for determining the position of a track shoe in a camera coordinate system. The method comprises the following steps: S1, calibrating a camera to obtain camera intrinsic parameters; The camera is installed directly above a trolley, and a full stack of track shoes is placed on the trolley, and the camera field of view can contain the full stack of track shoes; S2, determining the relationship between the track shoe plane and the camera coordinate system, and constructing a deformable template with calibration based on the pose of the track shoe under the camera; S3, collecting a stack picture at the current working time, calculating a matching search area in the stack picture based on a scaling factor and a preset ROI area, matching the deformable template with calibration, obtaining all the graspable track shoe poses, calculating the average height of the current track shoe layer, and updating the scaling factor; The matching comprises: determining whether it is the first time to load, if it is the first time to load, multiplying the preset ROI area by the initialized scaling factor as the matching search area of the stack picture, and matching the deformable template with calibration; if it is not the first time to load, multiplying the preset ROI area by the last updated scaling factor as the matching search area of the stack picture, and matching the deformable template with calibration; The updating of the scaling factor comprises: updating the scaling factor based on the average height of the current track shoe layer and the template height of the deformable template with calibration as follows: R1=H0 / H; wherein R1 is the updated scaling factor, H0 is the template height, and H is the average height of the current track shoe layer; S4, determining whether the number of all the obtained graspable track shoe poses meets the set pose number, if yes, outputting all the graspable track shoe poses under the camera; if the pose number is incorrect, determining the track shoe corresponding to the abnormal pose, calculating the matching search area of the track shoe corresponding to the abnormal pose based on the scaling factor and the ROI area set for the single track shoe, matching the deformable template with calibration, obtaining the abnormal track shoe pose, and integrating the abnormal track shoe pose into the track shoe pose; 2. The monocular vision-based mechanical arm gripper track plate method according to claim 1, wherein, S5, performing coordinate system conversion on the obtained track shoe pose according to the relationship between the grasping space coordinate system and the camera coordinate system and the relationship between the track shoe plane and the camera coordinate system, to obtain the track shoe pose in the grasping coordinate system. The determination of the relationship between the track shoe plane and the camera coordinate system and the construction of the deformable template with calibration based on the pose of the track shoe under the camera comprise: Collecting the circle dot calibration board image, and determining the pose of the circle dot calibration board origin in the camera coordinate system by using an EPnP algorithm ; According to the thickness of the circle dot calibration board itself, the pose of the track shoe under the camera is obtained ; placing a single track shoe on the trolley, placing a circular dot calibration plate on the surface of the track shoe, and making the origin of the circular dot calibration plate coincide with the center of the track shoe; Pose of track plate under camera A deformable template with a calibration is established as a reference pose.

3. The monocular vision-based mechanical arm gripper track plate method according to claim 2, wherein, taking away the circular dot calibration plate and collecting the single track shoe image with perfect surface features; after obtaining all the graspable track shoe poses, the method further comprises:

4. The monocular vision-based mechanical arm gripper track plate method according to claim 1, wherein, removing the abnormal pose, and then calculating the average height of the current track shoe layer. The ROI area set for the single track shoe is determined as follows: placing the single track shoe on the bottom frame of the trolley to collect the track shoe image; using the template matching function of halcon to frame out the central feature area of the track shoe image as a template; 5. The monocular vision-based mechanical arm gripper track plate method according to claim 1, wherein, determining the circumscribed rectangle of the template as the ROI area set for the single track shoe. Fifteen circular dot calibration plate images are collected, and the camera is calibrated to obtain camera intrinsic parameters K.

6. The monocular vision-based mechanical arm gripper track plate method according to claim 1, wherein, The relationship between the grabbing space coordinate system and the camera coordinate system is determined as follows: The upper left corner of the trolley is determined as the origin of the grabbing space coordinate system, and the calibration board is placed at the origin of the grabbing space coordinate system; The calibrated plate image is collected, an EPnP algorithm is used to solve 3D coordinates of the projection point in a camera coordinate system, a problem is converted into a 2D to 3D pose estimation problem, and a pose is calculated to determine a pose of the space coordinate system origin under the camera .

7. The monocular vision-based mechanical arm gripper track plate method according to claim 6, wherein, According to the relationship between the grabbing space coordinate system and the camera coordinate system, and the relationship between the track plate plane and the camera coordinate system, the obtained track plate pose is converted in the coordinate system to obtain the pose of the track plate in the grabbing coordinate system, and the specific process is as follows: ； wherein, is the pose of the track shoe in the grasp coordinate frame, is the pose of the track shoe under the camera resulting from creating a deformable template with the new position of the calibration plate as the reference pose.

Images