A monocular vision guidance method for dual-robot flexible shaft-hole assembly

Abstract

The present invention discloses a monocular vision guidance method for dual-robot flexible shaft-hole assembly, comprising: adopting a master-slave dual-robot collaborative assembly strategy, capturing images of the shaft system workpiece to be assembled from a monocular camera at the end of the robot, transmitting the captured image to a host computer, inputting it into a Mask-RCNN target recognition network model constructed based on a convolutional neural network, and outputting a mask image of the shaft part to be assembled. The mask image serves as the ROI region of interest for a further image processing algorithm. The image processing algorithm extracts key features of the shaft part in the ROI and fits feature parameters. The pose is estimated using a spatial pose estimation algorithm based on single-circle features, and then the slave robot's target movement position is generated to control the slave robot to align the axis of the shaft-hole part. After the slave robot is in place, the master robot grips the shaft part and moves it along the axis by a set distance until the shaft is fully inserted into the hole, completing the shaft-hole assembly. The present invention improves the flexibility and intelligence of automated shaft-hole assembly.

Description

Technical Field

[0001] The present invention relates to the technical field of robot collaborative assembly, and in particular to a monocular vision guidance method for dual-robot flexible shaft hole assembly. Background Art

[0002] Advanced technologies such as robotics, artificial intelligence, information technology, and intelligent manufacturing are developing rapidly, and the manufacturing industry is moving towards intelligent production models. The development of modern industry and advances in robotics have continuously enhanced the capabilities of robots, and the fields and scope of robot applications are constantly expanding.

[0003] Docking assembly is a key process in the manufacturing of aerospace and marine equipment. In assembly tasks, shaft-hole assembly is a typical basic assembly unit in robotic assembly operations. In many cases, due to working condition limitations, the capabilities of a single robot are insufficient. Most of them are targeted at specific assembly objects, and the assembly trajectories are mostly fixed and cannot be adaptively adjusted for different assembly postures. The flexibility is low and cannot meet production needs.

[0004] Inspired by human cooperative behavior, we can consider using multiple existing robots to form a team, with members coordinating to complete tasks. This type of coordination and collaborative operation between two or even multiple robots is a hot topic in robotics research. Summary of the Invention

[0005] The purpose of the present invention is to provide a monocular vision guidance method for dual-robot flexible shaft hole assembly to address the deficiencies in the above-mentioned prior art, which can realize autonomous visual guidance of the assembly robot, thereby improving the flexibility and automation level of the shaft hole assembly task.

[0006] 1. A monocular vision guidance method for dual-robot flexible shaft-hole assembly, characterized by comprising the following steps:

[0007] S1: A dual-robot collaborative assembly strategy based on a master-slave system is adopted. The master robot is responsible for grasping the shaft-based workpiece, and the slave robot is responsible for grasping the hole-based workpiece. An industrial camera is installed at the end of the slave robot. The master robot grasps the shaft-based workpiece and moves it to the assembly position, while the slave robot grasps the hole-based workpiece and moves it to the acquisition position.

[0008] S2: The industrial camera at the end of the robot collects image data of the shaft workpiece and transmits the image data to the host computer;

[0009] S3: The host computer uses the Mask-RCNN model built based on the convolutional neural network to detect and identify the shafting workpiece in the image data and outputs a binary mask image with the area around the shafting workpiece to be assembled as 1 and the rest of the area as 0.

[0010] S4: using an OpenCV-based image processing algorithm to multiply the mask image and the image to be processed to obtain an image of the region of interest, then performing edge extraction and fitting algorithms on the image of the region of interest to extract the ellipse edge formed by the front and rear end face circles of the shafting workpiece on the imaging plane, and fitting the ellipse feature parameters;

[0011] S5: Based on the ellipse feature parameters, a method based on spatial perspective geometry is used to calculate the spatial posture of the shaft system workpiece, thereby calculating the posture of the slave robot to be assembled, and sending the posture instruction to the slave robot controller to control the slave robot to grab the hole system workpiece and align it with the axis of the shaft system workpiece and maintain a pre-assembly distance with the end face of the shaft system workpiece. The distance can be set to a fixed value in advance.

[0012] S6: After the slave robot is in place, the master robot moves a specified distance along the axis of the workpiece. This distance is the sum of the pre-assembly distance and the shaft hole assembly depth. After the master robot moves into place, it releases the clamping jaws to complete the shaft hole assembly task.

[0013] Furthermore, step S3 includes the following steps:

[0014] S301: Use industrial cameras to capture images of different types of shaft workpieces under different lighting and environments.

[0015] S302: labeling the collected shaft workpiece image using labeling tool software to create a data set;

[0016] S303: Perform random adjustments and transformations on the labeled data set, such as image flipping, image scaling, image translation, image rotation, image noise addition, and image brightness / contrast / saturation, to expand the data samples.

[0017] S304: Construct a Mask-RCNN network model based on the backbone network of ResNet50 and FPN network, use the transfer learning method to train the prepared dataset, and generate the target recognition and instance segmentation network model.

[0018] S305: Using the trained network model to identify and segment the shafting workpiece, and outputting a mask image of the shafting workpiece.

[0019] Furthermore, step S4 includes the following steps:

[0020] S401: Read the mask image output by the MaskRCNN network model, multiply the mask image with the image to be processed to obtain the region of interest image. The image value in the region of interest remains unchanged, while the image values ​​outside the region are all 0. Use the weighted average method to grayscale the region of interest image to obtain grayscale;

[0021] S402: Using a Gaussian filtering algorithm to filter the grayscale image to remove noise, using a 5×5 filter to calculate a weighted average of pixels around the pixel point to obtain a final filtering effect.

[0022] S403: using a Canny operator to extract the edge of the shafting workpiece to obtain an edge image of the shafting workpiece;

[0023] S403: Using an ellipse fitting algorithm to detect the ellipse edge formed by the front and rear end circles of the shafting workpiece on the image plane, and fitting the ellipse parameters including the major axis, minor axis, rotation angle, and center coordinates of the ellipse.

[0024] Furthermore, step S5 includes the following steps:

[0025] S501: using ellipse characteristic parameters, respectively establish general ellipse equations of the projections of the two end circles of the shafting workpiece onto the imaging plane;

[0026] S502: Convert the pixel coordinates in the image coordinate system into coordinates in a coordinate system with the center point of the imaging plane as the origin;

[0027] S503: using the spatial projection principle to convert the 2D pixel coordinates into spatial 3D coordinates in a camera coordinate system with the camera optical center as the origin;

[0028] S504: Based on the spatial 3D coordinates, establish the spatial circular projection elliptical cone equation, convert it into a quadratic form and normalize it to obtain its eigenvalues; use the eigenvalues ​​to calculate the coordinates of the center of the front end face of the shaft workpiece and the normal direction vector of the spatial circular plane, but due to the ambiguous nature of the spatial circular pose solution, the obtained solutions are respectively O front,1 , O front,2 , O rar,1 , O rar,2 and N fornt,1 , N fornt,2 , N rear,1 , N rear,2 .

[0029] S504: Eliminate the ambiguity of the posture solution to obtain the true posture of the axis workpiece, and calculate the distance h between the center of the front face circle and the center of the rear face circle according to the spatial coordinates of the center of each group of solutions 11 , h 12 , h 21 , h 22 , calculate the candidate point set after the two center solutions of the front face circle are moved along the direction of the two normal vector solutions by the distance h, and then calculate the distance between the candidate point set and the two center solutions of the rear face circle. The minimum result is the combination of the true solution to obtain the true pose solution of the workpiece, which includes the coordinates O of the center of the front face circle and the normal vector N of the circle end face.

[0030] S504: Calculate the pose of the axis workpiece in the camera coordinate system based on the desired pose solution, and calculate the target position of the slave robot:

[0031]

[0032] The target pose is sent to the slave robot controller for visual guidance. After reaching the target pose, the master robot moves a distance h along the axis of the workpiece to complete the shaft-hole assembly task.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] When used, the present invention adopts a master-slave dual-robot collaborative solution to simplify the control strategy, in which the slave robot has the functions of part target detection and visual guidance, which greatly improves the flexibility and intelligence of the assembly system; a network model based on Mask-RCNN is used to perform pixel-level target detection on the workpiece, and the mask image of the shaft workpiece can be output as the ROI area for image processing, which greatly reduces the impact of changes in the working environment and the complexity of the workpiece recognition background on image processing, and greatly improves the flexibility and robustness of the workpiece pose estimation method; in the pose estimation method, the part pose is solved by the common features of the shaft part, which greatly improves the versatility of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention, the following briefly introduces the drawings required for use in the specific embodiments.

[0036] Figure 1 Schematic diagram of the dual-robot flexible shaft-hole assembly platform.

[0037] Figure 2 Flowchart of the master-slave dual-robot collaborative shaft-hole assembly method.

[0038] Figure 3 This is a flow chart of the workpiece pose estimation method based on the common features of the shaft-based workpiece.

[0039] Figure 4 This is the imaging principle diagram of the axis-based workpiece in the camera imaging plane. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0041] Dual robot flexible shaft hole assembly platform Figure 2 As shown, the master robot 2 is responsible for grabbing the shaft workpiece 5, and the slave robot 1 is responsible for grabbing the hole workpiece 4. An industrial camera 3 is installed at the end of the slave robot. Figure 1 As shown, a monocular vision guidance method for dual-robot flexible shaft-hole assembly includes the following steps:

[0042] S1: A master-slave dual-robot collaborative assembly strategy is adopted, wherein the master robot grabs the shaft workpiece and moves it to the assembly position, and the slave robot grabs the hole workpiece and moves it to the collection position;

[0043] S2: The industrial camera at the end of the robot collects image data of the shaft workpiece and transmits the image data to the host computer;

[0044] S3: The host computer uses the Mask-RCNN model built based on the convolutional neural network to detect and identify the shafting workpiece in the image data and outputs a binary mask image with the area around the shafting workpiece to be assembled as 1 and the rest of the area as 0.

[0045] S4: using an OpenCV-based image processing algorithm to multiply the mask image and the image to be processed to obtain an image of the region of interest, then performing edge extraction and fitting algorithms on the image of the region of interest to extract the ellipse edge formed by the front and rear end face circles of the shafting workpiece on the imaging plane, and fitting the ellipse feature parameters;

[0046] S5: Based on the ellipse feature parameters, a method based on spatial perspective geometry is used to calculate the spatial posture of the shaft system workpiece, thereby calculating the posture of the slave robot to be assembled, and sending the posture instruction to the slave robot controller to control the slave robot to grab the hole system workpiece and align it with the axis of the shaft system workpiece and maintain a pre-assembly distance with the end face of the shaft system workpiece. The distance can be set to a fixed value in advance.

[0047] S6: After the slave robot is in place, the master robot moves a specified distance along the axis of the workpiece. This distance is the sum of the pre-assembly distance and the shaft hole assembly depth. After the master robot moves into place, it releases the clamping jaws to complete the shaft hole assembly task.

[0048] In order to achieve the goal of step S3, step S3 includes the following steps:

[0049] S301: Use industrial cameras to capture images of different types of shaft workpieces under different lighting and environments.

[0050] S302: labeling the collected shaft workpiece image using labeling tool software to create a data set;

[0051] S303: Perform random adjustments and transformations on the labeled data set, such as image flipping, image scaling, image translation, image rotation, image noise addition, and image brightness / contrast / saturation, to expand the data samples.

[0052] S304: Construct a Mask-RCNN network model based on the backbone network of ResNet50 and FPN network, use the transfer learning method to train the prepared dataset, and generate the target recognition and instance segmentation network model.

[0053] S305: Using the trained network model to identify and segment the shafting workpiece, and outputting a mask image of the shafting workpiece.

[0054] In order to extract the key features of the shafting workpiece, step S4 includes the following steps:

[0055] S401: Read the mask image output by the MaskRCNN network model, multiply the mask image with the image to be processed to obtain the region of interest image. The image value in the region of interest remains unchanged, while the image values ​​outside the region are all 0. Use the weighted average method to grayscale the region of interest image to obtain grayscale;

[0056] S402: Using a Gaussian filtering algorithm to filter the grayscale image to remove noise, using a 5×5 filter to calculate a weighted average of pixels around the pixel point to obtain a final filtering effect.

[0057] S403: using a Canny operator to extract the edge of the shafting workpiece to obtain an edge image of the shafting workpiece;

[0058] S403: Using an ellipse fitting algorithm to detect the ellipse edge formed by the front and rear end circles of the shafting workpiece on the image plane, and fitting the ellipse parameters including the major axis, minor axis, rotation angle, and center coordinates of the ellipse.

[0059] like Figure 3 FIG. 1 is a flow chart of a method for eliminating the ambiguity of a spatial circular pose solution and obtaining a true pose solution in step S5, wherein step S5 includes the following steps:

[0060] S501: Using the ellipse characteristic parameters, establish the ellipse equations of the two end circles of the shaft system workpiece projected onto the imaging plane as shown in the following equations:

[0061] au 2 +bv 2 +cuv+du+ev+f=0

[0062] Among them, u and v are the pixel coordinates in the image coordinate system with the upper left corner as the origin, and a, b, c, d, and e are the coefficients of the equation calculated based on the ellipse feature parameters.

[0063] S502: Convert the coordinates in the image coordinate system into coordinates in a coordinate system with the center point of the imaging plane as the origin using the following formula:

[0064]

[0065]

[0066] Among them, X p 、Y p are the coordinates in the coordinate system with the center of the imaging plane as the origin, dx and dy are the pixel sizes of the camera, and u0 and v0 are the pixel coordinates of the center of the image in the image coordinate system.

[0067] S503: Convert the 2D pixel coordinates into 3D coordinates in the camera coordinate system with the camera optical center as the origin using the following formula:

[0068]

[0069] Among them, x, y, z are the spatial coordinates of the target point in the camera coordinate system, and f is the focal length of the camera.

[0070] S504: Combining the above formulas, we can obtain the equation of the spatial circular projection elliptical cone:

[0071] Ax 2 +Bxy+Cy 2 +Dxz+Eyz+Fz 2 =0

[0072] Among them, A, B, C, D, and E are the coefficients of the spatial elliptical cone equation.

[0073] Convert it to quadratic form:

[0074] [xyz]Q[xyz] T =0

[0075]

[0076] Convert the quadratic form into the standard form: λ1x' 2 +λ2y' 2 +λ3z' 2 =0

[0077] Use the following formula to calculate the center coordinates o of the front end face of the shaft workpiece front,1 , O front,2 , O rear,1 , O rear,2 and the normal direction vector N of the space circular plane front,1 , N front,2 , N rear,1 , N rear,2 2 groups of solutions:

[0078]

[0079]

[0080] S504: Eliminate the ambiguity of the posture solution to obtain the true posture of the axis workpiece, and calculate the distance h between the center of the front face circle and the center of the rear face circle according to the spatial coordinates of the center of each group of solutions 11 , h 12 , h 21 , h 22 ,use

[0081] O=O+N·h

[0082] Calculate the candidate point set after moving the center distance h of the two circle centers of the front face circle along the direction of the two normal vector solutions, and then calculate the distance between the candidate point set and the two circle centers of the rear face. The smallest result is the combination of the true solution to obtain the true posture solution of the workpiece, which includes the center coordinates O of the front face and the normal vector N of the circle end face.

[0083] S504: Calculate the pose of the axis workpiece in the camera coordinate system based on the desired pose solution, and calculate the target position of the slave robot:

[0084]

[0085] The target pose is sent to the slave robot controller for visual guidance. After reaching the target pose, the master robot moves a distance h along the axis of the workpiece to complete the shaft-hole assembly task.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A monocular vision guidance method for dual-robot flexible shaft-hole assembly, characterized in that: The steps include: S1: A dual-robot collaborative assembly strategy using a master-slave system is adopted, where the master robot is responsible for grasping the shaft-based workpiece, and the slave robot is responsible for grasping the hole-based workpiece. An industrial camera is installed at the end of each slave robot. The master robot grasps the shaft-based workpiece and moves it to the assembly position, while the slave robot grasps the hole-based workpiece and moves it to the acquisition position. S2: The industrial camera collects image data of the shaft workpiece and transmits the collected image data to the host computer; S3: The host computer uses the Mask-RCNN model built based on the convolutional neural network to detect and identify the shafting workpiece in the image data and outputs a binary mask image with the area around the shafting workpiece to be assembled as 1 and the rest of the area as 0; S4: using an OpenCV-based image processing algorithm to multiply the binary mask image and the image to be processed to obtain an image of the region of interest; performing edge extraction and a fitting algorithm on the image of the region of interest to extract the ellipse edge formed by the front and rear end face circles of the shafting workpiece on the imaging plane, and fitting the ellipse feature parameters; S5: Based on the ellipse feature parameters, a method based on spatial perspective geometry is used to calculate the spatial pose of the shafting workpiece, and the to-be-assembled pose of the slave robot is calculated. The command of the to-be-assembled pose is sent to the slave robot controller, and the slave robot is controlled to grasp the hole workpiece and align the axis of the shafting workpiece with the axis of the shafting workpiece and maintain a pre-assembly distance with the end face of the shafting workpiece, where the pre-assembly distance is a fixed value set in advance. S6: After the slave robot is in place, the master robot moves a specified distance along the axis of the workpiece. The specified distance is the sum of the pre-assembly distance and the shaft hole assembly depth. After the master robot moves into place, it releases the clamping jaws to complete the shaft hole assembly task.

2. The monocular vision guidance method for dual-robot flexible shaft-hole assembly according to claim 1, characterized in that: The step S3 comprises the following steps: S301: Using an industrial camera to capture images of different types of shafting workpieces under different lighting conditions and environments; S302: labeling the collected shaft workpiece image using labeling tool software to create a data set; S303: Perform image flipping, image scaling, image translation, image rotation, image noise addition, random adjustment of image brightness / contrast / saturation, and transformation on the labeled dataset to expand the data samples; S304: Build a Mask-RCNN network model based on the backbone network of ResNet50 and FPN network, use the transfer learning method to train the prepared dataset, and generate the object recognition and instance segmentation network model; S305: Using the trained network model to identify and segment the shafting workpiece, and outputting a mask image of the shafting workpiece.

3. The monocular vision guidance method for dual-robot flexible shaft-hole assembly according to claim 1, characterized in that: The step S4 comprises the following steps: S401: Read the mask image output by the MaskRCNN network model, multiply the mask image by the image to be processed to obtain an image of the region of interest, where the image values ​​within the region of interest remain unchanged, while the image values ​​outside the region are all 0; grayscale the image of the region of interest using a weighted average method to obtain a grayscale image; S402: Filtering the grayscale image using a Gaussian filtering algorithm to remove noise, and calculating a weighted average of pixels surrounding the pixel using a 5×5 filter to obtain a final filtering effect; S403: using a Canny operator to extract the edge of the shafting workpiece to obtain an edge image of the shafting workpiece; The ellipse fitting algorithm is used to detect the ellipse edge formed by the front and rear end circles of the shafting workpiece in the image plane, and the ellipse parameters including the major axis, minor axis, rotation angle and ellipse center coordinates are fitted.

4. The monocular vision guidance method for dual-robot flexible shaft-hole assembly according to claim 1, characterized in that: The step S5 comprises the following steps: S501: using ellipse characteristic parameters, respectively establish ellipse equations of the projections of the two end circles of the shafting workpiece onto the imaging plane; S502: Convert the pixel coordinates in the image coordinate system into coordinates in a coordinate system with the center point of the imaging plane as the origin; S503: using the spatial projection principle to convert the 2D pixel coordinates into spatial 3D coordinates in a camera coordinate system with the camera optical center as the origin; S504: establishing a spatial circular projection elliptical cone equation based on the spatial 3D coordinates, converting the equation into a quadratic form and normalizing it to obtain its eigenvalues; using the eigenvalues, respectively calculating the coordinates of the center of the front end face of the shafting workpiece and the normal direction vector of the spatial circular plane; Eliminate the ambiguity of the posture solution to obtain the true posture of the axis workpiece. Calculate the distance between the center of the front face circle and the center of the rear face circle according to the spatial coordinates of the center of each group of solutions. Calculate the distance between the centers of the two front face circles along the direction of the two normal vector solutions. h The candidate point set is calculated by calculating the distance between the candidate point set and the two circle center solutions of the rear end face. The smallest result is the combination of the true solution to obtain the true pose solution of the workpiece, which includes the coordinates of the center of the front end face. O and the normal vector of the circular end face N ; Calculate the pose of the axis workpiece in the camera coordinate system according to the pose solution, and calculate the target position of the robot pose s : n x 、n y 、n z They are the coordinates of the axis workpiece in the camera coordinate system; the target posture is sent to the slave robot controller for visual guidance, and the master robot moves along the axis of the axis workpiece after it is in place h distance to complete the shaft-hole assembly task.

Images