An assembly control system and method for a collaborative robot based on dual vision detection

Abstract

The present invention discloses an assembly control system and method for a collaborative robot based on dual-vision detection. The system includes multiple workspaces, a three-dimensional camera, a two-dimensional camera, and a host computer. The workspace includes a workpiece grasping space and an assembly control space. The workpiece to be grasped is located in the workpiece grasping space, and an assembly axis is provided in the assembly control space. The shooting range of the three-dimensional camera at least covers the workpiece grasping space and the assembly control space. The two-dimensional camera is installed at the end of the collaborative robot and moves with the robot to the top of the workpiece to be grasped and the top of the assembly axis respectively. The host computer is connected to the three-dimensional camera, the two-dimensional camera, and the collaborative robot. The present invention adopts three-dimensional vision combined with two-dimensional vision to realize workpiece assembly operations in complex environments. The three-dimensional vision can obtain the complete six-dimensional posture of the workpiece and realize the grasping of the workpiece in different states. The secondary positioning based on two-dimensional vision can ensure the reliability of assembly accuracy and compensate for errors in the robot movement process.

Description

Technical Field

[0001] The present invention relates to the field of industrial automated assembly, and in particular to an assembly control system and method of a collaborative robot based on dual-vision detection. Background Art

[0002] As a key technology for future intelligent manufacturing, robots are widely used in various fields. Parts assembly, in particular, is a prime example of intelligent robotic systems. With the aging of my country's population and rising labor costs, an increasing number of industrial robots are being used in industrial production. These robots primarily perform simple, repetitive tasks, assembling specific products. Compared to traditional manual assembly, they offer greater efficiency and precision. However, traditional industrial robots have a low level of intelligent assembly capabilities. During operation, they simply replicate pre-programmed paths. Changes to the product being assembled or the assembly location require replanning the robot's trajectory, significantly limiting their level of intelligence.

[0003] Robotic assembly involves replacing manual labor with robots to perform component assembly, achieving automation and standardization. Visual inspection equipment can be combined with high-precision visual recognition algorithms to locate various components and control robotic assembly operations through hand-eye calibration. Vision-based assembly technology is widely used in robotic assembly. It uses vision to perceive external information and then feeds it back to the robotic system, enabling the assembly of various parts, demonstrating its high level of intelligence. With the maturity of vision technology, various vision technologies have been gradually applied to various industrial fields. These technologies can be categorized into two main categories: two-dimensional (2D) vision and three-dimensional (3D) vision. 2D vision, due to its lack of depth information, can only be applied to flat surfaces. However, its advantage lies in its simplicity and the low cost of the industrial 2D cameras it uses, making it suitable for a variety of flat scenes, such as assembly lines. 3D vision, on the other hand, can acquire depth information of workpieces through techniques such as structured light, making it applicable to unstructured scenes. However, the 3D cameras used are significantly more expensive and technically complex.

[0004] However, there are some shortcomings in the current methods of visual recognition and positioning. For example, patent CN202010029843.X mainly introduces a workpiece assembly method, device and storage medium based on visual positioning. The method includes the following steps: obtaining a first position of a first workpiece in a first three-dimensional camera coordinate system, and a second position of a second workpiece in a second three-dimensional camera coordinate system; transferring the first workpiece by a first robot according to the first position, and transferring the second workpiece by a second robot according to the second position; obtaining a third position of the transferred first workpiece in the first two-dimensional camera coordinate system, and a fourth position of the transferred second workpiece in the second two-dimensional camera coordinate system; grabbing the transferred first workpiece by the first robot according to the third position, grabbing the transferred second workpiece by the second robot according to the fourth position, and docking and assembling the transferred first workpiece and the transferred second workpiece. For example, patent CN104057290A discloses a robot assembly system based on vision and force feedback, which includes: an industrial robot, a workpiece contour detection unit, an assembly force detection unit, a clamping unit and a system control host, wherein: the industrial robot drives the end movement according to the control instructions of the system control host; the assembly force detection unit is used to obtain the contact force between the shaft workpiece and the hole during the assembly process; the clamping unit is used to clamp the shaft workpiece; the workpiece contour detection unit is used to obtain measurement data of the workpiece contour; the system control host is used to receive position and force data, and position the assembly workpiece according to the received data, and generate control instructions to send to the industrial robot.

[0005] The first patent requires four different cameras to fully position the workpiece, which is extremely costly and requires four cameras with high precision to complete the assembly task. This method also requires the workpiece to be transferred through four locations, which will cause position errors during the process and cannot achieve high-precision assembly. The second patent can only achieve the positioning of parts in fixed positions, and cannot accurately estimate the posture of parts that change their posture state. Moreover, it is limited to a plane and cannot achieve assembly operations in three-dimensional space.

[0006] Therefore, it is of great significance to combine the advantages of 2D vision and 3D vision to provide a collaborative robot assembly control system and method based on dual-vision detection to meet the needs of industrial assembly. Summary of the Invention

[0007] The main purpose of the present invention is to provide an assembly control system and method for a collaborative robot based on dual-vision detection, thereby overcoming the shortcomings of the existing technology and achieving the purpose of high-precision assembly.

[0008] To achieve the aforementioned object of the invention, the technical solution adopted by the present invention includes: an assembly control system of a collaborative robot based on dual visual detection, comprising:

[0009] A plurality of working spaces, wherein the working spaces include a workpiece grasping space and an assembly control space, the workpiece to be grasped is located in the workpiece grasping space, and an assembly axis is provided in the assembly control space;

[0010] A three-dimensional camera, wherein the shooting range of the three-dimensional camera at least covers the workpiece grasping space and the assembly control space;

[0011] a two-dimensional camera, mounted on the end of the collaborative robot, and moving with the robot to above the workpiece to be grasped and above the assembly axis;

[0012] a host computer, connected to the three-dimensional camera, the two-dimensional camera, and the collaborative robot;

[0013] The three-dimensional camera is used to capture a point cloud image and transmit the obtained three-dimensional point cloud data to the host computer for processing. The host computer is used to divide the three-dimensional point cloud data into three-dimensional pose information of the workpiece and three-dimensional pose information of the assembly axis, and send the three-dimensional pose information of the workpiece to the collaborative robot; the collaborative robot is used to guide the end of the collaborative robot to the position of the workpiece to be grasped according to the three-dimensional pose information of the workpiece; the two-dimensional camera is used to capture a two-dimensional image of the workpiece as the end of the collaborative robot reaches the position of the workpiece to be grasped and transmit the two-dimensional image of the workpiece to the host computer for processing, and the host computer is used to send the processed final pose information of the workpiece to the collaborative robot for grasping; the collaborative robot is used to guide the end of the collaborative robot to the top of the assembly axis according to the three-dimensional pose information of the assembly axis, the two-dimensional camera is used to capture a two-dimensional image of the assembly axis as the end of the collaborative robot reaches the position of the workpiece to be grasped and transmit the two-dimensional image of the assembly axis to the host computer for processing, and the host computer is used to send the processed final pose information of the assembly axis to the collaborative robot, and the collaborative robot is used to clamp the workpiece according to the final pose information of the assembly axis for assembly operation.

[0014] In a preferred embodiment, the workpieces to be grasped are placed in disorder in the workpiece grasping space, and the assembly axis is fixed vertically or obliquely on the work surface.

[0015] In a preferred embodiment, an industrial light source is installed around the two-dimensional camera.

[0016] On the other hand, the present invention also discloses another technical solution: an assembly control method of a collaborative robot based on dual-vision detection, the method including a workpiece grasping process and a workpiece assembly process, the workpiece grasping process including:

[0017] S10, using a 3D camera to capture a point cloud image, and transmitting the obtained 3D point cloud data to a host computer for processing;

[0018] S11, the host computer divides the three-dimensional point cloud data into three-dimensional pose information of the workpiece and three-dimensional pose information of the assembly axis, and sends the three-dimensional pose information of the workpiece to the collaborative robot;

[0019] S12, the collaborative robot guides the end of the collaborative robot to the position of the workpiece to be grasped according to the three-dimensional pose information of the workpiece;

[0020] S13, after the collaborative robot's end reaches the position of the workpiece to be grasped, the two-dimensional camera captures a two-dimensional image of the workpiece and transmits the two-dimensional image of the workpiece to the host computer for processing;

[0021] S14, the host computer sends the final pose information of the workpiece obtained through processing to the collaborative robot for grasping;

[0022] The workpiece assembly process includes:

[0023] S20, the collaborative robot guides the end of the collaborative robot to the top of the assembly axis according to the three-dimensional posture information of the assembly axis;

[0024] S21, after the collaborative robot terminal reaches the position of the workpiece to be grasped, the two-dimensional camera captures a two-dimensional image of the assembly axis and transmits the two-dimensional image of the assembly axis to the host computer for processing;

[0025] S22, the host computer is used to send the processed final posture information of the assembly axis to the collaborative robot;

[0026] S23, the collaborative robot clamps the workpiece according to the final posture information of the assembly axis to perform assembly operation.

[0027] In a preferred embodiment, in S11, the host computer uses a straight-through filter to separate the workpiece and the assembly axis according to the different coordinates.

[0028] In a preferred embodiment, in S11, the host computer further processes the three-dimensional pose information of the workpiece as follows:

[0029] S111a, preprocessing the workpiece three-dimensional point cloud data, wherein the preprocessing includes separating the background point cloud, removing noise points, and reducing the point cloud density;

[0030] S112a, segmenting the pre-processed three-dimensional point cloud data of the workpiece to obtain a single workpiece point cloud cluster that is easiest to grasp;

[0031] S113a, matching the obtained single workpiece point cloud cluster with the CAD template. If matched, combining the matched workpiece 3D pose information with the workpiece hand-eye calibration matrix to obtain the workpiece 3D pose information for the collaborative robot to grasp.

[0032] In a preferred embodiment, in S112a, after segmentation, a grasping score is used to judge and select the single workpiece point cloud cluster that is easiest to grasp. The grasping score G is expressed as:

[0033]

[0034] Among them, Fz is the z-axis score, which means F area is the area fraction, which means S j is the projection area of ​​one of the clusters of the segmented point cloud on the xoy plane, S max It is the maximum area of ​​the workpiece on the xoy plane.

[0035] In a preferred embodiment, in S11, the host computer further processes the three-dimensional pose information of the assembly axis as follows:

[0036] S111b, preprocessing the assembly shaft three-dimensional point cloud data, wherein the preprocessing includes separating the background point cloud, removing noise points, and reducing the point cloud density;

[0037] S112b, after preprocessing, identifying the assembly axis, and estimating the axial position and posture of the assembly axis, combining the axial position and posture with the assembly axis hand-eye calibration matrix to obtain the three-dimensional position and posture information of the assembly axis for assembly by the collaborative robot.

[0038] In a preferred embodiment, in S112b, a cylindrical fitting method based on PCA algorithm and RANSAC algorithm is used to estimate the axial posture.

[0039] In a preferred embodiment, in S13, the process of the host computer processing the two-dimensional image of the workpiece includes:

[0040] S131, first performing filtering processing on the two-dimensional image of the workpiece;

[0041] S132, extracting the ROI region with complete workpiece information;

[0042] S133, performing template matching on the ROI area to obtain workpiece pose information, and scoring it. If the matching score is greater than a preset matching score threshold, the workpiece pose information is the final pose information of the workpiece in the collaborative robot coordinate system.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention adopts three-dimensional vision technology combined with two-dimensional vision technology to realize workpiece assembly operations in complex environments. The three-dimensional vision technology can obtain the complete six-dimensional position of the workpiece and realize the grasping of the workpiece in different states. The secondary positioning based on two-dimensional vision can ensure the reliability of assembly accuracy and compensate for the errors in the robot movement process.

[0045] 2. The present invention adopts a method of setting a grasping score, which can match the workpiece with the best current posture and avoid interference from other disorderly stacked workpieces.

[0046] 3. The present invention sets up a grasping space and an assembly space. After shooting a point cloud, the workpiece and the assembly axis can be processed separately, and the positions of the workpiece and the assembly axis can be obtained at the same time without interfering with each other, thereby improving efficiency.

[0047] 4. The present invention can get rid of plane restrictions and estimate the position and posture of the assembly axis in an inclined state. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 1 is a simplified structural diagram of an assembly control system of a collaborative robot based on dual-vision detection according to an embodiment of the present invention;

[0050] Figure 2 is a simplified flow chart of an assembly control method of a collaborative robot based on dual-vision detection according to an embodiment of the present invention;

[0051] Figure 3 This is a specific flow chart of a collaborative robot assembly control method based on dual-vision detection using three-dimensional vision technology according to an embodiment of the present invention;

[0052] Figure 4 This is a specific flow chart of a collaborative robot assembly control method based on dual-vision detection using two-dimensional vision technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be more fully understood through the following detailed description, which should be read in conjunction with the accompanying drawings. Detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the present invention, which can be embodied in various forms. Therefore, the specific functional details disclosed herein should not be construed as limiting, but rather as a basis for the claims and as a representative basis for teaching those skilled in the art to employ the present invention in various ways in virtually any appropriately detailed embodiment.

[0054] The present invention discloses a collaborative robot assembly control system and method based on dual-vision detection, enabling workpiece assembly in three-dimensional space, freeing it from the constraints of a single plane and achieving precision that meets various assembly requirements. By using 3D vision technology to capture the full six-degree-of-freedom position of the workpiece and assembly axes, combined with 2D vision technology for secondary positioning, the system compensates for errors in the robot's motion and achieves reliable assembly.

[0055] Combine Figure 1 As shown, an embodiment of the present invention discloses an assembly control system for a collaborative robot based on dual-vision inspection, capable of grasping workpieces in a disordered state and visually adjusting their positions to achieve precise assembly. The system includes multiple workspaces, a three-dimensional camera, a two-dimensional camera, and a host computer. The workspace includes at least a workpiece grasping space and an assembly control space. The workpiece to be grasped is located in the workpiece grasping space, and an assembly axis is provided in the assembly control space. In this embodiment, the workpiece to be grasped and the assembly axis are specifically arranged on a workbench. Both the workpiece grasping space and the assembly control space are within global vision, and information about the workpiece and the assembly axis position can be accurately captured by the three-dimensional camera. In the workpiece grasping space, multiple workpieces are arranged in a disordered state. That is, the workpieces can be placed in any position and some stacking is allowed, but all workpieces must be placed within the global field of view of the three-dimensional camera 1. In the assembly control space, the assembly axis is fixed to the workbench, specifically, it can be fixed vertically or tilted on the workbench surface, and the assembly axis must also be placed within the global field of view of the three-dimensional camera 1.

[0056] The 3D camera is installed directly above the workbench. As a global vision, its shooting range includes the entire workspace, that is, the workpiece grasping space and the assembly control space. The 3D camera is connected to the host computer and is used to transmit the captured 3D point cloud data to the host computer for processing to obtain the 3D position information of the target object (i.e. the workpiece to be grasped). The host computer is then connected to the collaborative robot, and the position information is sent to the robot, guiding the end of the robot to reach above the workpiece to be grasped. During implementation, the global 3D camera needs to be installed at a higher place in order to obtain information about the workpiece and the assembly position. The lens should be a short focal length, wide-angle lens to maximize the detection range.

[0057] A high-precision 2D camera is mounted at the end of the collaborative robot. Serving as a local vision sensor, it moves as the robot's end position changes. When the host computer outputs the 3D position information of the assembly axis, guiding the robot's end to the target object (i.e., the assembly axis), the 2D camera also moves over the axis and captures a 2D image of the axis. This precise position information is then sent to the host computer for processing, which then transmits it to the collaborative robot for movement. In implementation, in contrast to global vision, local vision primarily meets the needs of local high-precision inspection. A 2D camera with an appropriate frame rate and high resolution is sufficient to improve workpiece inspection accuracy. Preferably, an industrial light source is installed around the 2D camera to provide light compensation. Because 3D cameras use structured light to illuminate the target object to obtain specific information, they are less dependent on environmental factors such as lighting. However, 2D cameras require a stable lighting environment to prevent errors. Therefore, a stable industrial light source is required around the 2D camera to obtain clear images. Preferably, whether it is 3D vision or 2D vision, a suitable matching algorithm should be selected according to changes in the environment and the target object, so as to improve accuracy while improving processing efficiency.

[0058] During implementation, the collaborative robot 8 can be a six-degree-of-freedom robot, specifically the TM14 collaborative robot from Daming. The 3D camera 1 is a structured light camera, specifically the RVC X 3D camera. Both the collaborative robot and the 3D camera are connected to the host computer 9 via Ethernet.

[0059] The interaction between 3D camera 1 and 2D camera 2 and collaborative robot 8 requires hand-eye calibration. 3D camera 1 and collaborative robot 8 perform hand-eye calibration with the eye outside the hand, obtaining the transformation matrix from the 3D camera coordinate system to the robot's base coordinate system. 2D camera 2 and collaborative robot 8 perform hand-eye calibration with the eye inside the hand, obtaining the transformation matrix from the 2D camera coordinate system to the robot's end-user coordinate system. The host computer, through coordinate system conversion, transmits the resulting pose information to robot 8, enabling grasping and assembly.

[0060] It should be noted that before the two cameras start working, a corresponding template needs to be made. In 3D vision, the required template is made through CAD. The template is also three-dimensional, showing the specific shape of the target object, and the grasping point and grasping posture can be defined according to one's own needs. In this embodiment, the grasping point is defined at the center of the object (i.e., the workpiece to be grasped), and the grasping posture of the gripper is the posture defined according to the needs; in 2D vision, the template of the two-dimensional image is obtained by photographing the target object with a camera and performing denoising processing. The image must be clear. In this embodiment, the grasping point is positioned at the center to create the template.

[0061] Combine Figures 2 to 4 As shown, based on the above assembly control system, the present invention discloses an assembly control method of a collaborative robot based on dual visual detection, including a workpiece grasping process and a workpiece assembly process, wherein the workpiece grasping process includes:

[0062] S10, using a 3D camera to shoot a point cloud image, and transmitting the obtained 3D point cloud data to a host computer for processing.

[0063] Specifically, the assembly axes are manually installed in the assembly control space, and the workpieces are randomly placed in the workpiece gripping space. A 3D camera captures a point cloud image, and the resulting 3D point cloud data is transmitted to a host computer for processing.

[0064] S11, the host computer divides the three-dimensional point cloud data into three-dimensional pose information of the workpiece and three-dimensional pose information of the assembly axis, and sends the three-dimensional pose information of the workpiece to the collaborative robot.

[0065] Specifically, because the 3D camera 1 captures the entire work surface, a single point cloud image contains both the workpiece point cloud and the assembly axis point cloud. Therefore, before processing the point cloud data, these two components must be separated and processed separately to obtain the pose of the target workpiece and the pose of the assembly axis. This separation is achieved using a straight-through filter. Since the workpiece and assembly axis reside on the same surface and differ only in their coordinates, their x-coordinates can be used to separate them.

[0066] The three-dimensional point cloud data in the point cloud image is segmented by the algorithm in the host computer into three-dimensional pose information of the workpiece and three-dimensional pose information of the assembly axis, which are processed separately to obtain the initial pose information of the assembly axis and the workpiece.

[0067] In the workpiece point cloud, the workpieces are arranged in a disordered manner, with many of them stacked together. Direct template matching would result in significant errors. First, the workpiece 3D point cloud data is preprocessed. This includes separating the background point cloud, removing noise, and reducing point cloud density. This preprocessing significantly reduces the amount of workpiece point cloud data, and removing some useless points does not affect the accuracy of pose estimation.

[0068] If template matching is performed on all workpieces within a workpiece point cloud, matching efficiency will be significantly reduced and mismatches may occur. Therefore, before template matching, the pre-processed workpiece 3D point cloud data needs to be segmented to create individual point cloud clusters for matching. This not only improves efficiency but also increases matching accuracy. After segmentation, the individual workpiece point cloud clusters that are easiest to capture need to be selected.

[0069] In this embodiment, a crawl score is used To judge, among them, F z is the z-axis score, which means F area is the area fraction, which means S j is the projection area of ​​one of the clusters of the segmented point cloud on the xoy plane, S max It is the maximum area of ​​the actual object on the xoy plane.

[0070] A larger G indicates a more easily graspable workpiece. The grasp score algorithm indicates that objects higher up and flatter are easiest to grasp. After obtaining a single point cloud cluster, it is matched to the CAD template and a matching threshold, H, is set. If the matching score exceeds the threshold, the matching result meets the requirements. The matched workpiece 3D pose information is then combined with the workpiece hand-eye calibration matrix to obtain the workpiece 3D pose information for the collaborative robot 8 to grasp.

[0071] In the assembly axis point cloud section, the assembly axis can be fixed vertically or tilted on the work surface. Like the workpiece point cloud section, the point cloud needs to be preprocessed first to reduce the amount of data in the point cloud and remove isolated points. After the preprocessing is completed, the assembly axis needs to be identified and the axial position of the assembly axis needs to be estimated. Specifically, a cylindrical fitting method based on the PCA (Principal Component Analysis, PCA) algorithm and the RANSAC (Random Sample Consensus, RANSAC) algorithm is used to estimate the axial position. A point on the axis and its normal direction are obtained. Combined with the hand-eye calibration matrix, the three-dimensional position information of the assembly axis for collaborative robot assembly is obtained, and the collaborative robot 8 is guided to reach above the assembly axis.

[0072] Among them, the PCA algorithm finds the three main axes of the assembly axis point cloud through global dimensionality reduction, calculates the covariance matrix of the midpoint of the point cloud, and obtains three eigenvalues ​​and corresponding eigenvectors. The eigenvector corresponding to the minimum eigenvalue is the axis direction vector of the cylinder, and this direction vector is used as the initial direction vector of the RANSAC algorithm.

[0073] The RANSAC algorithm obtains the axis direction vector through iterative optimization of interior points. Since it is a greedy algorithm, fitting the cylinder axis using the RANSAC algorithm alone will result in reduced efficiency, and the accuracy of the fitted axis will not increase with the increase in the number of iterations. The initial direction vector provided by the PCA algorithm can reduce the iterations of the RANSAC algorithm, and the fitted axis will only be more accurate than the initial axis. The principle of fitting the axis is as follows:

[0074] 1) All points on the cylindrical surface are perpendicular to the cylindrical axis. You can choose any two points and their normals and find a unique line perpendicular to the normals of the two points.

[0075] 2) Cylindrical surface equation: Among them, (x, y, z) is a point on the cylindrical surface, (x0, y0, z0) is a point on the cylindrical axis, is the direction vector of the axis, and r0 is the radius of the cylinder base. This equation represents the distance between all points on the cylindrical surface and the axis equal to the radius of the cylinder base. Therefore, this equation can be used to determine how many points are within r0 of the axis generated in step 1) and to set a distance error, err, with points within (r0 ± err) considered inliers.

[0076] 3) Using the cylindrical surface equation, find the points within the assembly axis point cloud that are within (r0 ± err) of the initial axis. Let num be the number of points. The number of inliers generated by the two-point fitted axis in step 2) is N. If N > num, the fitted axis is superior to the initial axis; if N < num, the fitted axis is inferior and should be discarded.

[0077] 4) Through continuous iterative optimization, the best axis is found, which contains the most internal points. Its direction vector is obtained and can be converted into the axial position of the assembly axis.

[0078] S12, the collaborative robot guides the end of the collaborative robot to the position of the workpiece to be grasped according to the three-dimensional posture information of the workpiece.

[0079] S13, after the collaborative robot's end reaches the position of the workpiece to be grasped, the two-dimensional camera takes a two-dimensional image of the workpiece and transmits the two-dimensional image of the workpiece to the host computer for processing.

[0080] Specifically, a 2D camera 2 is mounted on the end of collaborative robot 8. When the robot's end reaches the workpiece, it begins operating and turns on the industrial ring light source 7 at the end. 2D camera 2 receives the information and captures a 2D image, which must contain complete information about workpiece 3. 2D camera 2 sends the image to a host computer for processing. Host computer 9 performs secondary positioning based on the pose information obtained from the image. Host computer 9 processes the image by first filtering it and then extracting the ROI region containing complete workpiece information. Template matching is performed on this region to obtain the center point coordinates (x, y) and the rotation angle θ of Rz. This is then scored, with a matching score threshold of T. If the matching score is less than T, the matching result does not meet assembly requirements, requiring a new image capture, workpiece information extraction, and template matching again until the matching score exceeds the matching score threshold T. The coordinates (x, y) combined with the hand-eye calibration matrix provide the final pose information of the workpiece in the robot 8 coordinate system. Using this pose information, robot 8 can eliminate global 3D vision errors and accurately grasp the target workpiece 3.

[0081] S14, the host computer sends the final posture information of the workpiece obtained through processing to the collaborative robot for grasping.

[0082] The workpiece assembly process includes:

[0083] S20, the collaborative robot guides the end of the collaborative robot to the top of the assembly axis according to the three-dimensional posture information of the assembly axis.

[0084] Specifically, after the collaborative robot 8 grasps the workpiece 3 , it guides the end to the top of the assembly axis 4 using the assembly axis position obtained by the three-dimensional camera 1 .

[0085] S21, after the collaborative robot terminal reaches the position of the workpiece to be grasped, the two-dimensional camera takes a two-dimensional image of the assembly axis and transmits the two-dimensional image of the assembly axis to the host computer for processing.

[0086] Specifically, after the collaborative robot's end reaches the position of the workpiece to be grasped, the two-dimensional camera 2 starts working again and takes a two-dimensional image containing the assembly axis.

[0087] S22, the host computer is used to send the processed final posture information of the assembly axis to the collaborative robot.

[0088] Specifically, the host computer 9 processes the two-dimensional image and obtains the center point coordinates (x1, y1) of the assembly axis and the rotation angle θ1 relative to the template through template matching.

[0089] S23, the collaborative robot clamps the workpiece according to the final posture information of the assembly axis to perform assembly operation.

[0090] Specifically, through these position and posture information, the robot 8 clamps the workpiece to perform assembly operations, that is, assembles the workpiece 3 onto the assembly axis 4 .

[0091] The present invention has the following technical effects: 1. The present invention adopts three-dimensional vision technology combined with two-dimensional vision technology to realize workpiece assembly operations in complex environments. The three-dimensional vision technology can obtain the complete six-dimensional posture of the workpiece and realize the grasping of the workpiece in different states. The secondary positioning based on two-dimensional vision can ensure the reliability of assembly accuracy and make up for the errors in the robot movement process. 2. The present invention adopts a method of setting the grasping score, which can match the workpiece with the best current posture and avoid interference from other disorderly stacked workpieces. 3. The present invention sets a grasping space and an assembly space. After taking a point cloud, the workpiece and the assembly axis can be processed separately, and the posture of the workpiece and the assembly axis can be obtained at the same time without interfering with each other, thereby improving efficiency. 4. The present invention can get rid of the plane limitation and estimate the posture of the assembly axis in an inclined state.

[0092] Therefore, compared to existing assembly technologies, this invention applies vision systems to industrial robotic assembly technology, improving the level of intelligence and production efficiency of industrial robotic assembly. By using a dual-vision inspection system, combining the advantages of 3D vision and 2D vision technologies, it can use visual feedback to perform assembly globally, achieving high precision and wide-scale results.

[0093] The present invention solves the problem of difficulty in positioning when traditional robot positioning methods are restricted at a low cost. It obtains two-axis acceleration information respectively through two acceleration sensors, establishes homogeneous coordinates of the two acceleration sensors in the vehicle coordinate system and the global coordinate system, and solves the current position of the mobile robot according to the homogeneous coordinate conversion relationship, thereby realizing the indoor local positioning of the mobile robot. As an auxiliary positioning of the traditional positioning method, it improves the accuracy and reliability of the robot's local positioning.

[0094] The various aspects, embodiments, features and examples of the present invention should be considered as illustrative in all respects and are not intended to limit the present invention, the scope of which is defined solely by the claims. Other embodiments, modifications and uses will be apparent to those skilled in the art without departing from the spirit and scope of the invention as claimed.

Claims

1. An assembly control system for a collaborative robot based on dual visual detection, characterized in that: The system comprises: A plurality of working spaces, wherein the working spaces include a workpiece grasping space and an assembly control space, the workpiece to be grasped is located in the workpiece grasping space, and an assembly axis is provided in the assembly control space; A three-dimensional camera, wherein the shooting range of the three-dimensional camera at least covers the workpiece grasping space and the assembly control space; a two-dimensional camera, mounted on the end of the collaborative robot, and moving with the robot to above the workpiece to be grasped and above the assembly axis; a host computer, connected to the three-dimensional camera, the two-dimensional camera, and the collaborative robot; The 3D camera is used to capture a point cloud image, and transmit the obtained 3D point cloud data to a host computer for processing. The host computer is used to divide the 3D point cloud data into 3D pose information of the workpiece and 3D pose information of the assembly axis, and transmit the 3D pose information of the workpiece to the collaborative robot; the collaborative robot is used to guide the end of the collaborative robot to the position of the workpiece to be grasped according to the 3D pose information of the workpiece; the 2D camera is used to capture a 2D image of the workpiece as the end of the collaborative robot reaches the position of the workpiece to be grasped, and transmit the 2D image of the workpiece to the host computer for processing, and the host computer is used to send the processed final pose information of the workpiece to the collaborative robot for grasping; the collaborative robot is used to guide the end of the collaborative robot to the top of the assembly axis according to the 3D pose information of the assembly axis, and the 2D camera is used to capture a 2D image of the assembly axis as the end of the collaborative robot reaches the position of the workpiece to be grasped, and transmit the 2D image of the assembly axis to the host computer for processing, and the host computer is used to send the processed final pose information of the assembly axis to the collaborative robot, and the collaborative robot is used to clamp the workpiece according to the final pose information of the assembly axis for assembly operation; The host computer is used to send the final pose information of the workpiece obtained through processing to the collaborative robot for grasping, including: the host computer processes the image, first filters the image, then extracts the ROI area with complete workpiece information, performs template matching on it, obtains the center point coordinates (x, y) and the rotation angle θ of Rz, and scores them, setting the matching score threshold to T. If its matching score is less than T, it means that the matching result does not meet the assembly requirements, and it is necessary to re-take the image, extract the workpiece information and perform template matching again until the matching score is greater than the matching score threshold T; the coordinates (x, y) combined with the hand-eye calibration matrix are the final pose information of the workpiece in the robot coordinate system. Through this pose information, the robot can eliminate the error of global 3D vision and accurately grasp the target workpiece; The host computer is used to send the processed final posture information of the assembly axis to the collaborative robot, including: the host computer processes the two-dimensional image, and obtains the center point coordinates (x1, y1) of the assembly axis and the rotation angle θ1 relative to the template through template matching.

2. The assembly control system of a collaborative robot based on dual visual detection according to claim 1, characterized in that: The workpieces to be grasped are placed in disorder in the workpiece grasping space, and the assembly axis is fixed vertically or obliquely on the work table.

3. The assembly control system of a collaborative robot based on dual visual detection according to claim 1, characterized in that: An industrial light source is installed around the two-dimensional camera.

4. An assembly control method of an assembly control system of a collaborative robot based on dual vision detection according to any one of claims 1 to 3, characterized in that: The method includes a workpiece grasping process and a workpiece assembling process, wherein the workpiece grasping process includes: S10, using a 3D camera to capture a point cloud image, and transmitting the obtained 3D point cloud data to a host computer for processing; S11, the host computer divides the three-dimensional point cloud data into three-dimensional pose information of the workpiece and three-dimensional pose information of the assembly axis, and sends the three-dimensional pose information of the workpiece to the collaborative robot; S12, the collaborative robot guides the end of the collaborative robot to the position of the workpiece to be grasped according to the three-dimensional pose information of the workpiece; S13, after the collaborative robot's end reaches the position of the workpiece to be grasped, the two-dimensional camera captures a two-dimensional image of the workpiece and transmits the two-dimensional image of the workpiece to the host computer for processing; S14, the host computer sends the final pose information of the workpiece obtained through processing to the collaborative robot for grasping, including: the host computer processes the image, first filters the image, then extracts the ROI area with complete workpiece information, performs template matching on it, obtains the center point coordinates (x, y) and the rotation angle θ of Rz, and scores them, setting the matching score threshold to T. If its matching score is less than T, it means that the matching result does not meet the assembly requirements, and it is necessary to re-take the image, extract the workpiece information and perform template matching again until the matching score is greater than the matching score threshold T; the coordinates (x, y) combined with the hand-eye calibration matrix are the final pose information of the workpiece in the robot coordinate system. Through this pose information, the robot can eliminate the error of global 3D vision and accurately grasp the target workpiece; The workpiece assembly process includes: S20, the collaborative robot guides the end of the collaborative robot to the top of the assembly axis according to the three-dimensional posture information of the assembly axis; S21, after the collaborative robot terminal reaches the position of the workpiece to be grasped, the two-dimensional camera captures a two-dimensional image of the assembly axis and transmits the two-dimensional image of the assembly axis to the host computer for processing; S22, the host computer is used to send the processed final posture information of the assembly axis to the collaborative robot, including: the host computer processes the two-dimensional image, and obtains the center point coordinates (x1, y1) of the assembly axis and the rotation angle θ1 relative to the template through template matching; S23, the collaborative robot clamps the workpiece according to the final posture information of the assembly axis to perform assembly operation.

5. The assembly control method of the collaborative robot assembly control system based on dual visual detection according to claim 4, characterized in that: In S11, the host computer uses a straight-through filter to separate the workpiece and the assembly axis according to the different coordinates.

6. The assembly control method of the collaborative robot assembly control system based on dual visual detection according to claim 4, characterized in that: In S11, the host computer further processes the three-dimensional pose information of the workpiece as follows: S111a, preprocessing the workpiece three-dimensional point cloud data, wherein the preprocessing includes separating the background point cloud, removing noise points, and reducing the point cloud density; S112a, segmenting the pre-processed three-dimensional point cloud data of the workpiece to obtain a single workpiece point cloud cluster that is easiest to grasp; S113a, matching the obtained single workpiece point cloud cluster with the CAD template. If matched, combining the matched workpiece 3D pose information with the workpiece hand-eye calibration matrix to obtain the workpiece 3D pose information for the collaborative robot to grasp.

7. The assembly control method of the collaborative robot assembly control system based on dual visual detection according to claim 6, characterized in that: In S112a, after segmentation, a grasping score is used to judge and select the single workpiece point cloud cluster that is easiest to grasp. The grasping score G is expressed as: Among them, Fz is the z-axis score, which means F area is the area fraction, which means S j is the projection area of ​​one of the clusters of the segmented point cloud on the xoy plane, S max It is the maximum area of ​​the workpiece on the xoy plane.

8. The assembly control method of the collaborative robot assembly control system based on dual visual detection according to claim 4, characterized in that: In S11, the host computer further processes the three-dimensional pose information of the assembly axis as follows: S111b, preprocessing the assembly shaft three-dimensional point cloud data, wherein the preprocessing includes separating the background point cloud, removing noise points, and reducing the point cloud density; S112b, after preprocessing, identifying the assembly axis, and estimating the axial position and posture of the assembly axis, combining the axial position and posture with the assembly axis hand-eye calibration matrix to obtain the three-dimensional position and posture information of the assembly axis for assembly by the collaborative robot.

9. The assembly control method of the collaborative robot assembly control system based on dual visual detection according to claim 8, characterized in that: In S112b, a cylindrical fitting method based on the PCA algorithm and the RANSAC algorithm is used to estimate the axial posture.

Images