Workpiece posture alignment method and device, electronic equipment and storage medium

Abstract

The invention discloses a workpiece posture alignment method and device, electronic equipment and a storage medium, and relates to the technical field of intelligent robos.The workpiece posture alignment method comprises the steps that after a mechanical arm of a composite robot grabs a target workpiece, the actual posture of the target workpiece is adjusted to a target assembly posture; and multiple pieces of point cloud data of the target workpiece under the target assembly posture are collected, template point cloud data are determined based on the multiple pieces of point cloud data, then the posture deviation value between the current posture of the new workpiece grabbed by the mechanical arm and the target assembly posture of the target workpiece is determined, and the tail end of the mechanical arm of the composite robot is driven to move according to the posture deviation value. According to the invention, the target point cloud data of the new workpiece under the current attitude and the template point cloud data are subjected to target attitude alignment, so that the problems of high computing resource consumption, particularly difficulty in adaptation when an assembly system processes the new-type workpiece, reduction of universality and reliability of the assembly system and the like in the prior art are solved; therefore, the flexibility and applicability of the assembly system are improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent robots, and particularly to a workpiece attitude alignment method, device, electronic device, and storage medium. Background Art

[0002] In the automation of industrial assembly lines, compound robots need to estimate the pose of the grasped object with extremely high precision because they cannot directly determine their own base coordinate system, so as to provide accurate basic data for subsequent operations such as insertion and precise placement.

[0003] In the related art, the grasping and installation of memory modules are completed by a robotic arm combined with internal and external dual vision sensors. That is, by using the internal and external dual vision sensors combined with the robotic arm, the memory module is initially located by the internal vision, its attitude is accurately detected by the external vision, and the three-dimensional spatial attitude of the memory module is calculated using the PnP (Perspective-n-Point) algorithm and the pre-stored high-precision CAD (Computer-Aided Design) model of the memory module. Then, the end pose of the robotic arm is dynamically adjusted in combination with closed-loop control, and finally, the high-precision grasping and installation of the memory module are realized by fusing slot positioning and force feedback.

[0004] However, in the related art, when determining the attitude of the memory module, it is easily affected by light interference. Therefore, it relies on a large number of image processing algorithms, resulting in high consumption of computing resources and a decrease in algorithm accuracy, reducing the flexibility of the assembly system. Especially the dependence on high-precision CAD models makes it difficult for the assembly system to adapt to new model workpieces, and the processing error between the CAD model and the actual workpiece introduces new uncertainties, reducing the versatility and reliability of the assembly system, which urgently needs to be solved. Summary of the Invention

[0005] This application provides a workpiece attitude alignment method, device, electronic device, and storage medium to at least solve the technical problems in the related art such as high consumption of computing resources, decreased algorithm accuracy, reduced flexibility of the assembly system, especially the dependence on high-precision CAD models, which makes it difficult for the assembly system to adapt to new model workpieces, and reduced versatility and reliability of the assembly system.

[0006] The present application provides a workpiece attitude alignment method, including: after the robotic arm of the composite robot grabs the target workpiece, adjusting the actual attitude of the target workpiece to the target assembly attitude; collecting a plurality of point cloud data of the target workpiece in the target assembly attitude, and determining template point cloud data based on the plurality of point cloud data; determining a pose deviation value between the current attitude of the new workpiece grabbed by the robotic arm of the composite robot and the target assembly attitude of the target workpiece based on the template point cloud data, so as to drive the movement of the end of the robotic arm of the composite robot according to the pose deviation value, so that the target point cloud data of the new workpiece in the current attitude is aligned with the template point cloud data in the target attitude.

[0007] Optionally, in an embodiment of the present application, the determining template point cloud data based on the plurality of point cloud data includes: performing downsampling processing on the plurality of point cloud data to obtain downsampled point cloud data; using the downsampled point cloud data as the template point cloud data.

[0008] Optionally, in an embodiment of the present application, the determining a pose deviation value between the current attitude of the new workpiece grabbed by the robotic arm of the composite robot and the target assembly attitude of the target workpiece includes: using the robotic arm to grab the new workpiece and moving the new workpiece to the target rough positioning area to determine the initial pose of the new workpiece after moving; based on the initial pose, performing a preliminary adjustment on the end of the robotic arm to obtain the current pose of the new workpiece; determining the pose deviation value between the current attitude of the new workpiece and the target assembly attitude of the target workpiece based on the target point cloud data of the new workpiece in the current pose and the template point cloud data.

[0009] Optionally, in an embodiment of the present application, the using the robotic arm to grab the new workpiece includes: identifying the image information and point cloud information of the new workpiece; determining the point cloud centroid position of the new workpiece based on the image information and the point cloud information; driving the robotic arm to grab the new workpiece based on the rough positioning coordinates of the point cloud centroid position of the new workpiece in the target base coordinate system.

[0010] Optionally, in an embodiment of the present application, the moving the new workpiece to the target rough positioning area includes: determining the original pose of the new workpiece, and performing rough alignment between the original pose of the new workpiece and the target pose of the template point cloud to determine the pose change data when the new workpiece is roughly aligned; adjusting the attitude of the end of the robotic arm according to the pose change data to move the new workpiece to the target rough positioning area.

[0011] Optionally, in an embodiment of the present application, the determining the pose change data when the new workpiece is roughly aligned includes: segmenting the collected image of the new workpiece to obtain the target point cloud data of the new workpiece; determining the target center point of the target point cloud data of the new workpiece, and based on the target center point, determining the pose change data when the new workpiece is roughly aligned.

[0012] Optionally, in an embodiment of the present application, the driving the end of the robotic arm of the composite robot according to the pose deviation value includes: performing iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative closest point algorithm to obtain the pose deviation value that meets the preset error condition; using the pose deviation value to determine the target adjustment value of the end of the robotic arm, and driving the end of the robotic arm to move according to the target adjustment value.

[0013] Optionally, in an embodiment of the present application, the driving the end of the robotic arm to move according to the target adjustment value includes: planning multiple motion paths of the end of the robotic arm based on the target adjustment value; using the multiple motion paths to determine the target motion path of the end of the robotic arm, and controlling the end of the robotic arm to move according to the target motion path.

[0014] The present application also provides a workpiece pose alignment device, including: an adjustment module, configured to adjust the actual pose of the target workpiece to the target assembly pose after the robotic arm of the composite robot grabs the target workpiece; a determination module, configured to collect a plurality of point cloud data of the target workpiece in the target assembly pose, and determine template point cloud data based on the plurality of point cloud data; a registration module, configured to determine the pose deviation value between the current pose of the new workpiece grabbed by the robotic arm of the composite robot and the target assembly pose of the target workpiece based on the template point cloud data, and drive the end of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current pose is aligned with the template point cloud data in the target pose.

[0015] Optionally, in an embodiment of the present application, the determination module includes: a first processing unit, configured to perform downsampling processing on the plurality of point cloud data to obtain downsampled point cloud data; a first determination unit, configured to use the downsampled point cloud data as the template point cloud data.

[0016] Optionally, in an embodiment of the present application, the registration module includes: a moving unit, configured to use the robotic arm to grasp a new workpiece and move the new workpiece to a target rough positioning area to determine the initial pose of the new workpiece after movement; an acquisition unit, configured to preliminarily adjust the end of the robotic arm based on the initial pose to obtain the current pose of the new workpiece; a second determination unit, configured to determine the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece based on the target point cloud data and the template point cloud data of the new workpiece in the current pose.

[0017] Optionally, in an embodiment of the present application, the moving unit includes: an identification subunit, configured to identify the image information and point cloud information of the new workpiece; a determination subunit, configured to determine the centroid position of the point cloud of the new workpiece based on the image information and the point cloud information; a driving subunit, configured to drive the robotic arm to grasp the new workpiece based on the rough positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0018] Optionally, in an embodiment of the present application, the moving unit includes: a processing subunit, configured to determine the original pose of the new workpiece and perform rough alignment between the original pose of the new workpiece and the target pose of the template point cloud to determine the pose change data when the new workpiece is subjected to rough alignment; a moving subunit, configured to adjust the end pose of the robotic arm according to the pose change data to move the new workpiece to the target rough positioning area.

[0019] Optionally, in an embodiment of the present application, the processing subunit is further configured to perform segmentation processing on the collected image of the new workpiece to obtain the target point cloud data of the new workpiece; determine the target center point of the target point cloud data of the new workpiece, and determine the pose change data when the new workpiece is subjected to rough alignment based on the target center point.

[0020] Optionally, in an embodiment of the present application, the registration module includes: a second processing unit, configured to perform iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative closest point algorithm to obtain the pose deviation value that meets the preset error condition; a third determination unit, configured to determine the target adjustment value of the end of the robotic arm by using the pose deviation value, and drive the movement of the end of the robotic arm according to the target adjustment value.

[0021] Optionally, in an embodiment of the present application, the third determination unit includes: a planning subunit, configured to plan multiple motion paths of the end of the robotic arm based on the target adjustment value; and a control subunit, configured to determine a target motion path of the end of the robotic arm using the multiple motion paths, so as to control the motion of the end of the robotic arm according to the target motion path.

[0022] The present application further provides an electronic device, including: a memory, configured to store a computer program; and a processor, configured to implement the steps of any one of the above workpiece attitude alignment methods when executing the computer program.

[0023] The present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above workpiece attitude alignment methods are implemented.

[0024] The present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above workpiece attitude alignment methods are implemented.

[0025] Through the present application, after the robotic arm of the composite robot grasps the target workpiece, the actual attitude of the target workpiece is adjusted to the target assembly attitude, then multiple point cloud data of the target workpiece in the target assembly attitude are collected, and template point cloud data is determined based on the multiple point cloud data, and further the pose deviation value between the current attitude of the newly grasped workpiece by the robotic arm and the target assembly attitude of the target workpiece is determined, so as to drive the motion of the end of the robotic arm of the composite robot according to the pose deviation value, so that the target point cloud data of the new workpiece in the current attitude is aligned with the template point cloud data in the target attitude. Therefore, the technical problems in the related art such as large consumption of computing resources and decline of algorithm accuracy, reduction of the flexibility of the assembly system, especially the dependence on high-precision CAD models, making it difficult for the assembly system to adapt to new model workpieces, and reduction of the versatility and reliability of the assembly system can be solved, and the technical effect of improving the flexibility and applicability of the assembly system is achieved. Description of the Drawings

[0026] In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 A schematic diagram of an assembly system provided by an embodiment of the present application;

[0028] Figure 2 A flowchart of a workpiece attitude alignment method provided by an embodiment of the present application;

[0029] Figure 3 Schematic diagram of the misaligned template point cloud and the new workpiece point cloud in a specific embodiment of the present application;

[0030] Figure 4 Schematic diagram of the roughly aligned template point cloud and the new workpiece point cloud in a specific embodiment of the present application;

[0031] Figure 5 Schematic diagram of the precisely aligned template point cloud and the new workpiece point cloud in a specific embodiment of the present application;

[0032] Figure 6 Structural diagram of a workpiece attitude alignment device provided by an embodiment of the present application.

[0033] Reference numerals: 10 - assembly system, 101 - composite robot, 102 - visual perception unit, 103 - control unit, 1011 - multi - degree - of - freedom robotic arm, and 1012 - hand - eye camera; 20 - workpiece attitude alignment device, 100 - adjustment module, 200 - determination module, and 300 - registration module. Detailed Description of the Embodiment

[0034] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0035] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variation thereof are intended to cover non - exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0036] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] Next, a workpiece attitude alignment method, device, electronic device, and storage medium proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0038] Before introducing the workpiece attitude alignment method in the embodiments of the present application, the assembly system involved in the workpiece attitude alignment method of the present application will be briefly introduced.

[0039] As Figure 1 shown, this application establishes an assembly system 10, which includes a composite robot 101, a visual perception unit 102, and a control unit 103. Each component works together through physical connections and communication protocols.

[0040] Among them, the composite robot 101 includes a mobile chassis and a multi-degree-of-freedom robotic arm 1011 mounted on the chassis. The end of the robotic arm is equipped with a gripper and an eye-in-hand camera 1012, where the eye-in-hand camera 1012 is a depth camera;

[0041] The visual perception unit 102, that is, a depth camera is set at a fixed position on the workbench to collect workpiece point cloud data;

[0042] The control unit 103, that is, an industrial computer is wired to the depth camera and is responsible for point cloud processing and pose calculation; the robot controller is built into the composite robot 101 body and is used to control the movement of the robotic arm and the mobile chassis; the control unit 103, that is, the industrial computer and the robot controller perform data interaction through a wireless network to transmit pose adjustment instructions in real time.

[0043] As Figure 2 shown, Figure 2 is a flowchart of a workpiece pose alignment method according to an embodiment of this application. The workpiece pose alignment method includes the following steps:

[0044] In step S201, after the robotic arm of the composite robot grasps the target workpiece, the actual pose of the target workpiece is adjusted to the target assembly pose.

[0045] In the embodiment of this application, the target workpiece is the workpiece providing the template point cloud; the target assembly pose is the ideal assembly pose, which can be specifically set by those skilled in the relevant art.

[0046] It can be understood that in the embodiment of this application, first, the operator manually controls the composite robot in the above assembly system to grasp the workpiece to be assembled. After the robotic arm of the composite robot grasps the workpiece to be assembled, the pose of the end of the robotic arm is adjusted to make the pose of the workpiece accurately reach the ideal position required by the assembly process.

[0047] In the embodiment of this application, after manually controlling to grasp the workpiece, the pose of the end of the robotic arm is adjusted to accurately align with the ideal position, combining the flexibility of manual operation to ensure efficient and accurate assembly effects.

[0048] In step S202, multiple point cloud data of the target workpiece in the target assembly pose are collected, and the template point cloud data is determined based on the multiple point cloud data.

[0049] It can be understood that in the embodiment of the present application, the externally mounted depth camera fixed to the workbench in the above-mentioned assembly system can be used to photograph the positioned workpiece, so as to collect multiple point cloud data of the workpiece in the ideal assembly posture, such as, complete surface point cloud data, and determine the multiple point cloud data as template point cloud data. Let the workpiece pose of the current template point cloud be:

[0050] T grasp = g || g ∈SE(3)

[0051] wherein, R g represents the rotation matrix of the nominal grasping pose relative to the world coordinate system, t g represents the translation vector of the nominal grasping pose relative to the world coordinate system, and SE(3) represents the Special Euclidean Group.

[0052] In the embodiment of the present application, the ideal assembly posture point cloud data of the workpiece can be collected by using the external depth camera and used as the template point cloud data, ensuring high-precision template establishment and improving the accuracy and consistency of the subsequent assembly process.

[0053] Among them, in one embodiment of the present application, determining the template point cloud data based on multiple point cloud data includes: performing downsampling processing on the multiple point cloud data to obtain the downsampled point cloud data; using the downsampled point cloud data as the template point cloud data.

[0054] In the actual execution process, the embodiment of the present application can perform data preprocessing on the multiple point cloud data, that is, the complete surface point cloud data. For example, methods such as statistical filtering and radius filtering are used to remove outliers and noise, the voxel grid filter is used to reduce the point cloud density, the mean or center point is used to replace the points within each voxel, and the normal direction of each point in the point cloud is calculated. Among them, the voxel grid filter can be used in the embodiment of the present application to reduce the point cloud density.

[0055] Among them, after the original point cloud is downsampled by the voxel grid, the point cloud density is homogenized, that is:

[0056] P down ={p i |p i =VoxelCenter(v j ), v j ∈Grid(P raw , ρ)}

[0057] wherein, ρ is the side length of the voxel, p i is a point in the downsampled point cloud, and P downis the point cloud set after downsampling, and VoxelCenter(v j ) is the center position of voxel v j . v j is a voxel in the voxel grid, P raw is the original point cloud data, and Grid(P raw , ρ) is the process of voxel grid quantization for the original point cloud P raw .

[0058] It should be noted that the point clouds mentioned in the embodiments of the present application are all point clouds after voxel grid downsampling, which effectively reduces the data volume, improves the subsequent processing efficiency, simplifies the template point cloud data, reduces the computational resource requirements, and ensures efficient and accurate pose alignment for new workpieces.

[0059] In step S203, based on the template point cloud data, the pose deviation value between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece is determined, so as to drive the movement of the end of the robotic arm of the composite robot according to the pose deviation value, so that the target point cloud data of the new workpiece in the current pose is aligned with the template point cloud data in the target pose.

[0060] In the embodiments of the present application, the target point cloud data is local point cloud data; the target pose alignment is accurate pose alignment.

[0061] It can be understood that after the calibration of the template point cloud in the above steps is completed in the embodiments of the present application, the composite robot enters the autonomous assembly stage. Among them, the robotic arm of the composite robot collects new workpiece data in real time based on the hand-eye camera. For example, the new workpiece is identified through a target detection algorithm, so as to drive the robotic arm to move and grasp the new workpiece. For example, as Figure 3 shown, they are the unaligned workpiece point cloud (i.e., the new workpiece point cloud) and the template point cloud.

[0062] Then, in the embodiments of the present application, the new workpiece can be moved to the rough positioning area to make the new workpiece roughly within the visible area of the fixed depth camera, and the fixed depth camera is used to process the new workpiece to obtain workpiece pose information with large errors, and the end of the robotic arm is initially adjusted to roughly align the new workpiece and the template pose.

[0063] Secondly, determine the local point cloud data of the new workpiece in the current pose through a fixed-depth camera, that is, the new workpiece point cloud in this application, and use the ICP (Iterative Closest Point) algorithm to calculate the pose deviation value between the local point cloud data and the template point cloud data, so as to drive the movement of the end of the robotic arm of the composite robot according to the pose deviation value, so that the pose of the new workpiece is adjusted to the expected pose, so as to achieve accurate pose alignment between the local point cloud data of the new workpiece in the current pose and the template point cloud data.

[0064] In the embodiment of the present application, the embodiment of the present application can also determine the complete point cloud of the new workpiece in the current pose through a fixed-depth camera, so as to more comprehensively understand the actual state of the new workpiece, which helps to further verify the accuracy of pose alignment, or provide more detailed data support when higher-precision operations are required.

[0065] The embodiment of the present application can accurately calculate the pose deviation and adjust the robotic arm movement, ensuring that the workpiece can accurately reach the predetermined assembly position and pose. Compared with the related technology, the present application completes fixture-free and model-independent precision assembly, improves the assembly quality, realizes efficient and accurate workpiece assembly, and enhances the robustness and adaptability of the assembly system, which is of great significance for improving the level of industrial automation.

[0066] Among them, in an embodiment of the present application, determining the pose deviation value between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece includes: using the robotic arm to grasp the new workpiece and move the new workpiece to the target rough positioning area to determine the initial pose of the new workpiece after moving; based on the initial pose, make a preliminary adjustment to the end of the robotic arm to obtain the current pose of the new workpiece; based on the target point cloud data and the template point cloud data of the new workpiece in the current pose, determine the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece.

[0067] As a possible implementation, the embodiment of the present application can use the robotic arm to grasp the new workpiece, move the new workpiece from the initial position to a preset rough positioning area to determine the initial pose of the new workpiece after moving. Then, based on the initial pose, make a preliminary adjustment to the end of the robotic arm to obtain the current pose of the new workpiece, use a fixed-depth camera to collect the local point cloud data in the current pose, and compare and analyze the local point cloud data of the new workpiece in the current pose and the template point cloud data to determine the pose deviation value between the current pose of the new workpiece and the ideal assembly pose of the target workpiece. Therefore, the embodiment of the present application can significantly reduce the search space and time required to find the correct position of the new workpiece by first performing rough positioning and then fine adjustment, can significantly improve the workpiece positioning accuracy and assembly efficiency, and at the same time reduce the operation complexity.

[0068] Optionally, in an embodiment of the present application, using a robotic arm to grasp a new workpiece includes: identifying the image information and point cloud information of the new workpiece; determining the centroid position of the point cloud of the new workpiece based on the image information and point cloud information; driving the robotic arm to grasp the new workpiece based on the rough positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0069] In some embodiments, the robotic arm of the composite robot according to the embodiments of the present application, based on the new workpiece data collected in real time by the hand-eye camera, through target detection algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), point-line features, template matching, Yolo, Faster RCNN (Region-based Convolutional Neural Network), SSD (Single Shot MultiBox Detector), etc., identifies the image information and point cloud information of the new workpiece to determine the centroid position of the point cloud of the new workpiece, so that the rough positioning coordinates of the centroid position of the point cloud of the new workpiece in the base coordinate system can be combined to drive the robotic arm to move to complete the grasping of the new workpiece, thereby realizing the fast and accurate positioning and grasping of the new workpiece and improving the operation efficiency and automation level.

[0070] Optionally, in an embodiment of the present application, moving the new workpiece to the target rough positioning area includes: determining the original pose of the new workpiece, and roughly aligning the original pose of the new workpiece with the target pose of the template point cloud to determine the pose change data when the new workpiece is roughly aligned; adjusting the pose of the end of the robotic arm according to the pose change data to move the new workpiece to the target rough positioning area.

[0071] For example, after the robotic arm finishes grasping, the new workpiece is no longer visible in the field of view of the hand-eye camera. Therefore, the new workpiece can be positioned by a fixed-depth camera. Let the original pose of the new workpiece after actual grasping, that is, the pose of the new workpiece T obj Due to various factors such as the motion error of the robotic arm, the positioning error during grasping, and the relative sliding after grasping, the pose of the new workpiece in the grasped state is not determined, and there may be a large error from the expected pose, that is:

[0072] T obj = T grasp ·ΔT A ·ΔT G ·ΔT F ·…·T C = T grasp ·ΔT E·T C

[0073] Among them, T obj represents the pose of the new workpiece, T grasp represents the workpiece pose of the template point cloud, and ΔT A , ΔT G , ΔT F and... respectively represent the attitude error caused by the motion error of the robotic arm, the attitude error caused by the positioning error during grasping, the attitude error caused by the relative sliding after grasping, and other uncertain attitude errors; T C represents the attitude transformation required for rough alignment of the pose of the new workpiece after grasping with the template point cloud. For simplicity of representation, the attitude errors caused by all errors are written as ΔT E .

[0074] Next, the embodiment of the present application can roughly align the original pose of the new workpiece with the target pose of the template point cloud, that is, the workpiece pose T grasp of the template point cloud, to determine the pose change data when the new workpiece is roughly aligned, and adjust the attitude of the end of the robotic arm according to the pose change data to move the new workpiece to the rough positioning area, thereby efficiently realizing the preliminary positioning and movement of the new workpiece, simplifying the adjustment process before precise assembly, and improving the overall operation efficiency and automation level.

[0075] Optionally, in an embodiment of the present application, determining the pose change data when the new workpiece is roughly aligned includes: segmenting the collected image of the new workpiece to obtain the target point cloud data of the new workpiece; determining the target center point of the target point cloud data of the new workpiece, and based on the target center point, determining the pose change data when the new workpiece is roughly aligned.

[0076] During the actual execution process, the embodiment of the present application first adjusts the position of the end of the robotic arm to make the new workpiece roughly in the field of view of the fixed-depth camera, and segments the collected image of the new workpiece using the RGB-D image aligned by the fixed-depth camera. For example, use Yolo for object detection and the GrabCut algorithm for RGB object segmentation, and obtain the point cloud data p = {p i} of the new workpiece through RGB-D image alignment, i = 1, 2, 3... n. Assume that the plane equation of the new workpiece is n·p + d = 0, where n is the unit normal vector and d is the distance from the origin to the plane. Optimizing the following equation can obtain the plane equation, that is:

[0077] min∑(n·p i +d) 2 s.t.||n|| = 1

[0078] Calculate the center point c of the point cloud in the new workpiece area as:

[0079]

[0080] Among them, c is the center point.

[0081] T can be obtained according to Rodrigues’s Formula C =[R C |t C , where

[0082] R C =I + sinθ·[k] × +(1 - cosθ)·[k] × 2

[0083] t C =t g -R C c

[0084] Among them, [k] × is the cross - product matrix of k, R C is the rotation matrix, t C is the translation vector, I is the identity matrix, θ is the rotation angle, k is the unit vector of the rotation axis, t g is the translation vector in the target coordinate system.

[0085] Among them,

[0086]

[0087] Among them, R g is the target rotation matrix, n T is the transpose of n, ||n|| and ||R g n|| are the magnitudes of n and R g n respectively.

[0088] The ΔT for the attitude adjustment required at the end of the robotic arm can be obtained through the real - time calibrated system coordinate relationship relative is:

[0089]

[0090] Among them, ΔT base is the pose change matrix relative to the base coordinate system, T C is the attitude transformation required for the rough alignment of the pose after grasping the new workpiece and the template point cloud, ΔT relative is the pose change matrix relative to the tool coordinate system, is the transformation matrix from the camera coordinate system to the base coordinate system, It is the transformation matrix from the base coordinate system to the end coordinate system obtained by using the kinematics of the robotic arm and the DH (Denavit-Hartenberg) model.

[0091] As Figure 4 shown, they are the workpiece point cloud after rough alignment (i.e., the new workpiece point cloud) and the template point cloud. Due to various errors ΔT E , therefore, it is very difficult for the new workpiece point cloud and the template point cloud to completely coincide during rough alignment. Generally, there will be a part that coincides.

[0092] Optionally, in an embodiment of the present application, driving the movement of the end of the robotic arm of the composite robot according to the pose deviation value includes: based on the target iterative closest point algorithm, performing iterative registration processing on the target point cloud data and the template point cloud data to obtain a pose deviation value that meets the preset error condition; using the pose deviation value to determine the target adjustment value of the end of the robotic arm, so as to drive the movement of the end of the robotic arm according to the target adjustment value.

[0093] As a possible implementation manner, the embodiment of the present application can start fine alignment after completing rough alignment, that is, eliminating various errors ΔT E . At time k, use the ICP algorithm to register the point cloud to obtain the registration error Adjust the movement of the end of the robotic arm by using the following , that is:

[0094]

[0095] Among them, represents the pose change matrix relative to the tool coordinate system at time k, represents the registration error.

[0096] Then, repeat the ICP algorithm to register the point cloud again to obtain a new registration error Adjust the robotic arm again, and so on in a loop until the registration error is within the allowable error range or reaches the maximum number of iterations I t .

[0097] During the actual operation process, due to factors such as an unsatisfactory shooting angle, the calibration error has a greater impact on the assembly system. If the attitude adjustment is directly performed according to the method described above, it may not converge to the desired attitude, that is:

[0098]

[0099] Among them, ΔT ca represents the calibration error matrix, represents the true coordinate transformation matrix.

[0100] When actually controlling the movement of the robotic arm, the composite robot should be in a stationary state. Therefore, ΔT ca is an unknown time-invariant constant matrix.

[0101] Write the above in the following form:

[0102]

[0103] Increase the error gain matrix K, then the error after iterative adjustment and the current error are related as:

[0104]

[0105] where represents the adjoint matrix of the calibration error ΔT ca ; represents the error after iterative adjustment, represents the current error, and K represents the error gain matrix.

[0106] Assume that the calibration error is small enough, then:

[0107]

[0108] where I is the identity matrix.

[0109] Define the error gain matrix K as a diagonal matrix, that is:

[0110] K = diag(k1, k2, k3, k4, k5, k6)

[0111] By configuring the parameters to make the spectral radius of the error gain matrix, the error convergence can be guaranteed. As Figure 5 shown, for the template point cloud and the workpiece point cloud (i.e., the new workpiece point cloud) in fine alignment, through iterative loops, control the movement of the robotic arm to make the error converge and complete the registration.

[0112] Among them, in an embodiment of the present application, driving the movement of the end of the robotic arm according to the target adjustment value includes: planning multiple movement paths of the end of the robotic arm based on the target adjustment value; determining the target movement path of the end of the robotic arm using the multiple movement paths to control the movement of the end of the robotic arm according to the target movement path.

[0113] In the embodiment of the present application, the target movement path is the best movement path for the end of the robotic arm to move.

[0114] In some embodiments, the embodiments of the present application can generate multiple potential motion paths for workpiece pose alignment according to the target adjustment value, and fully consider factors such as the working space limit of the robotic arm, joint limits, obstacle positions, and self-collision avoidance when generating the paths to ensure that each path is feasible; in addition, a series of evaluation criteria can be set for each path, such as path length, execution time, energy consumption, smoothness, and obstacle avoidance performance, etc., so as to conduct a comprehensive score and select the path with the highest score as the best motion path, thus realizing an efficient, accurate, and safe operation process.

[0115] Therefore, compared with the related art, the present application does not require customized processing tooling fixtures, reducing costs and improving the flexibility of the production line; and does not require a high-precision CAD model. Only through iterative registration and closed-loop control, the registration error converges. In addition, for the point cloud registration of the workpiece, only part of the point cloud needs to be extracted, and sub-pixel positioning of feature points is not required, and there is a great tolerance for errors such as insufficient accuracy of target foreground segmentation (point cloud missing), thus realizing model-free precision assembly in a dynamic scenario.

[0116] According to the workpiece pose alignment method proposed by the embodiments of the present application, after the robotic arm of the composite robot grabs the target workpiece, the actual pose of the target workpiece is adjusted to the target assembly pose, then multiple point cloud data of the target workpiece in the target assembly pose are collected, and the template point cloud data is determined based on the multiple point cloud data, and further the pose deviation value between the current pose of the new workpiece grabbed by the robotic arm and the target assembly pose of the target workpiece is determined, so as to drive the movement of the end of the robotic arm of the composite robot according to the pose deviation value, so that the target point cloud data of the new workpiece in the current pose is aligned with the template point cloud data in the target pose. Therefore, it can solve the technical problems in the related art such as large consumption of computing resources and decreased algorithm accuracy, reducing the flexibility of the assembly system, especially the dependence on high-precision CAD models, making it difficult for the assembly system to adapt to new model workpieces, and reducing the versatility and reliability of the assembly system, and achieving the technical effect of improving the flexibility and applicability of the assembly system.

[0117] Next, a workpiece pose alignment device according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0118] Figure 6 It is a block diagram of a workpiece pose alignment device according to an embodiment of the present application.

[0119] As Figure 6 shown, the workpiece pose alignment device 20 includes: an adjustment module 100, a determination module 200, and a registration module 300.

[0120] Specifically, the adjustment module 100 is configured to adjust the actual pose of the target workpiece to the target assembly pose after the robotic arm of the composite robot grasps the target workpiece.

[0121] The determination module 200 is configured to collect a plurality of point cloud data of the target workpiece in the target assembly pose and determine template point cloud data based on the plurality of point cloud data.

[0122] The registration module 300 is configured to determine the pose deviation value between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece based on the template point cloud data, so as to drive the movement of the end of the robotic arm of the composite robot according to the pose deviation value, so that the target point cloud data of the new workpiece in the current pose is aligned with the template point cloud data in the target pose.

[0123] Optionally, in an embodiment of the present application, the determination module 200 includes: a first processing unit and a first determination unit.

[0124] Wherein, the first processing unit is configured to perform downsampling processing on the plurality of point cloud data to obtain the downsampled point cloud data.

[0125] The first determination unit is configured to use the downsampled point cloud data as the template point cloud data.

[0126] Optionally, in an embodiment of the present application, the registration module 300 includes: a moving unit, an obtaining unit, and a second determination unit.

[0127] Wherein, the moving unit is configured to grasp the new workpiece by using the robotic arm and move the new workpiece to the target rough positioning area to determine the initial pose of the new workpiece after moving.

[0128] The obtaining unit is configured to perform a preliminary adjustment on the end of the robotic arm based on the initial pose to obtain the current pose of the new workpiece.

[0129] The second determination unit is configured to determine the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece based on the target point cloud data and the template point cloud data of the new workpiece in the current pose.

[0130] Optionally, in an embodiment of the present application, the moving unit includes: an identification subunit, a determination subunit, and a driving subunit.

[0131] Wherein, the identification subunit is configured to identify the image information and point cloud information of the new workpiece.

[0132] The determination subunit is configured to determine the centroid position of the point cloud of the new workpiece based on the image information and the point cloud information.

[0133] A driving subunit, configured to drive a robotic arm to grasp a new workpiece based on the rough positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0134] Optionally, in an embodiment of the present application, the moving unit includes: a processing subunit and a moving subunit.

[0135] Wherein, the processing subunit is configured to determine the original pose of the new workpiece, and roughly align the original pose of the new workpiece with the target pose of the template point cloud to determine the pose change data when the new workpiece is roughly aligned.

[0136] The moving subunit is configured to adjust the pose of the end of the robotic arm according to the pose change data to move the new workpiece to the target rough positioning area.

[0137] Optionally, in an embodiment of the present application, the processing subunit is further configured to perform segmentation processing on the collected image of the new workpiece to obtain the target point cloud data of the new workpiece; determine the target center point of the target point cloud data of the new workpiece, and based on the target center point, determine the pose change data when the new workpiece is roughly aligned.

[0138] Optionally, in an embodiment of the present application, the registration module 300 includes: a second processing unit and a third determination unit.

[0139] Wherein, the second processing unit is configured to perform iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative closest point algorithm to obtain a pose deviation value that meets the preset error condition.

[0140] The third determination unit is configured to use the pose deviation value to determine the target adjustment value of the end of the robotic arm, and drive the movement of the end of the robotic arm according to the target adjustment value.

[0141] Optionally, in an embodiment of the present application, the third determination unit includes: a planning subunit and a control subunit.

[0142] Wherein, the planning subunit is configured to plan multiple movement paths of the end of the robotic arm based on the target adjustment value.

[0143] The control subunit is configured to determine the target movement path of the end of the robotic arm using the multiple movement paths, and control the movement of the end of the robotic arm according to the target movement path.

[0144] It should be noted that for the description of the features in the corresponding embodiments of the workpiece pose alignment device, reference can be made to the relevant descriptions in the corresponding embodiments of the workpiece pose alignment method, which will not be elaborated here one by one.

[0145] The workpiece attitude alignment device proposed according to the embodiments of the present application adjusts the actual attitude of the target workpiece to the target assembly attitude after the robotic arm of the composite robot grasps the target workpiece. Then, it collects a plurality of point cloud data of the target workpiece in the target assembly attitude, determines the template point cloud data based on the plurality of point cloud data, and further determines the pose deviation value between the current attitude of the new workpiece grasped by the robotic arm and the target assembly attitude of the target workpiece. Based on the pose deviation value, it drives the movement of the end of the robotic arm of the composite robot, so that the target point cloud data of the new workpiece in the current attitude is aligned with the template point cloud data in the target attitude. Therefore, it can solve the technical problems in the related art, such as large consumption of computing resources, decreased algorithm accuracy, reduced flexibility of the assembly system, especially the dependence on high-precision CAD models, making it difficult for the assembly system to adapt to new model workpieces, and reducing the versatility and reliability of the assembly system, achieving the technical effect of improving the flexibility and applicability of the assembly system.

[0146] Embodiments of the present application also provide an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the workpiece attitude alignment method.

[0147] Embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above embodiments of the workpiece attitude alignment method when running.

[0148] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs and other various media that can store computer programs.

[0149] Embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the workpiece attitude alignment method.

[0150] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above embodiments of the workpiece attitude alignment method.

[0151] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0152] The above has introduced in detail a workpiece attitude alignment method, device, electronic device, and storage medium provided by this application. Specific examples have been used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only applicable to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A workpiece posture alignment method, characterized in that: Applied to a composite robot, wherein the method comprises the following steps: After the mechanical arm of the composite robot grasps the target workpiece, adjusting the actual posture of the target workpiece to the target assembly posture; Collecting a plurality of point cloud data of the target workpiece in the target assembly posture, and determining template point cloud data based on the plurality of point cloud data; Based on the template point cloud data, the posture deviation value between the current posture of the new workpiece grasped by the manipulator arm of the composite robot and the target assembly posture of the target workpiece is determined, so as to drive the end of the manipulator arm of the composite robot to move according to the posture deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data in the target posture.

2. The method according to claim 1, characterized in that: The step of determining the template point cloud data based on the plurality of point cloud data comprises: Downsampling the plurality of point cloud data to obtain downsampled point cloud data; The downsampled point cloud data is used as the template point cloud data.

3. The method according to claim 1, characterized in that: The step of determining a posture deviation value between a current posture of a new workpiece grasped by the mechanical arm of the composite robot and a target assembly posture of the target workpiece includes: Using the robot arm to grab a new workpiece, and moving the new workpiece to a target rough positioning area to determine an initial position of the new workpiece after the movement; Based on the initial posture, preliminarily adjusting the end of the robot arm to obtain the current posture of the new workpiece; The posture deviation value between the current posture of the new workpiece and the target assembly posture of the target workpiece is determined based on the target point cloud data of the new workpiece at the current posture and the template point cloud data.

4. The method according to claim 3, characterized in that The method of using the mechanical arm to grab a new workpiece includes: Identify image information and point cloud information of the new workpiece; Determine the point cloud centroid position of the new workpiece based on the image information and the point cloud information; Based on the coarse positioning coordinates of the point cloud centroid position of the new workpiece in the target base coordinate system, the robotic arm is driven to grasp the new workpiece.

5. The method according to claim 3, characterized in that: The step of moving the new workpiece to a target rough positioning area comprises: Determine the original pose of the new workpiece, and roughly align the original pose of the new workpiece with the target pose of the template point cloud to determine pose change data of the new workpiece when the rough alignment is performed; The posture of the end of the robot arm is adjusted according to the posture change data to move the new workpiece to the target rough positioning area.

6. The method according to claim 5, characterized in that The determining of the posture change data of the new workpiece when performing rough alignment includes: Segmenting the acquired image of the new workpiece to obtain the target point cloud data of the new workpiece; A target center point of the target point cloud data of the new workpiece is determined, so as to determine, based on the target center point, position and posture change data of the new workpiece when performing rough alignment.

7. The method according to claim 1, characterized in that The step of driving the end of the manipulator arm of the composite robot to move according to the posture deviation value comprises: Based on the target iterative closest point algorithm, the target point cloud data and the template point cloud data are iteratively registered to obtain the posture deviation value that meets the preset error condition; The target adjustment value of the end of the robotic arm is determined by using the posture deviation value, so as to drive the end of the robotic arm to move according to the target adjustment value.

8. A workpiece posture alignment device, characterized in that: Applied to a composite robot, wherein the device comprises: An adjustment module, used for adjusting the actual posture of the target workpiece to a target assembly posture after the mechanical arm of the composite robot grasps the target workpiece; A determination module, configured to collect a plurality of point cloud data of the target workpiece in the target assembly posture, and determine template point cloud data based on the plurality of point cloud data; A registration module is used to determine the posture deviation value between the current posture of the new workpiece grasped by the manipulator arm of the composite robot and the target assembly posture of the target workpiece based on the template point cloud data, so as to drive the end of the manipulator arm of the composite robot to move according to the posture deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data in the target posture.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the workpiece posture alignment method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the workpiece posture alignment method as described in any one of claims 1 to 7.

Images