Workpiece posture alignment method and device, electronic equipment and storage medium

Abstract

The application discloses a workpiece posture alignment method and device, electronic equipment and storage medium, and relates to the technical field of intelligent robots, which comprises the following steps: after a target workpiece is grabbed by a mechanical arm of a composite robot, adjusting the actual posture of the target workpiece to a target assembly posture; collecting multiple point cloud data of the target workpiece under the target assembly posture, and determining template point cloud data based on the multiple point cloud data, and then determining a pose deviation value between the current posture of a new workpiece grabbed by the mechanical arm and the target assembly posture of the target workpiece, so as to drive the end of the mechanical arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece under the current posture is aligned with the template point cloud data in the target posture. The problems, such as large consumption of computing resources in the related art, especially the difficulty of adaptation of the assembly system when processing new models of workpieces, and the reduction of the versatility and reliability of the assembly system, are solved, thereby improving the flexibility and applicability of the assembly system.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to workpiece posture alignment methods, devices, electronic devices, and storage media. Background Technology

[0002] In industrial assembly line automation, composite robots cannot directly determine their own base coordinate system. They need to estimate the pose of the grasped object with extremely high precision in order to provide accurate basic data for subsequent operations such as insertion and precise placement.

[0003] In related technologies, memory modules are grasped and installed using a robotic arm combined with internal and external dual vision sensors. Specifically, the internal vision sensor is used in conjunction with the robotic arm to initially locate the memory module, while the external vision sensor accurately detects its posture. The PnP (Perspective-n-Point) algorithm and a pre-stored high-precision CAD (Computer-Aided Design) model of the memory module are used to calculate the three-dimensional spatial posture of the memory module. Then, closed-loop control is used to dynamically adjust the posture of the robotic arm's end effector. Finally, slot positioning and force feedback are integrated to achieve high-precision grasping and installation of the memory module.

[0004] However, in related technologies, the determination of the orientation of memory modules is easily affected by lighting interference, thus relying on a large number of image processing algorithms, resulting in high consumption of computing resources and decreased algorithm accuracy, which reduces the flexibility of the assembly system. In particular, the reliance on high-precision CAD models makes it difficult for the assembly system to adapt when handling new types of workpieces. Furthermore, the processing error between the CAD model and the actual workpiece introduces new uncertainties, reducing the versatility and reliability of the assembly system, which urgently needs to be addressed. Summary of the Invention

[0005] This application provides a workpiece posture alignment method, apparatus, electronic device, and storage medium to at least solve the technical problems in related technologies, such as high consumption of computing resources and decreased algorithm accuracy, which reduce the flexibility of assembly systems, especially the dependence on high-precision CAD models, making it difficult for assembly systems to adapt when handling new types of workpieces, and reducing the versatility and reliability of assembly systems.

[0006] This application provides a workpiece posture alignment method, comprising: after the robotic arm of the composite robot grasps a target workpiece, adjusting the actual posture of the target workpiece to a target assembly posture; acquiring multiple point cloud data of the target workpiece in the target assembly posture, and determining template point cloud data based on the multiple point cloud data; determining a pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece based on the template point cloud data, so as to drive the end effector of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data.

[0007] Optionally, in one embodiment of this application, determining template point cloud data based on the plurality of point cloud data includes: performing downsampling processing on the plurality of point cloud data to obtain downsampled point cloud data; and using the downsampled point cloud data as the template point cloud data.

[0008] Optionally, in one embodiment of this application, determining the pose deviation value between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece includes: using the robotic arm to grasp the new workpiece and moving the new workpiece to a target coarse positioning area to determine the initial pose of the new workpiece after movement; based on the initial pose, making preliminary adjustments to the end effector of the robotic arm to obtain the current pose of the new workpiece; and based on the target point cloud data of the new workpiece in the current pose and the template point cloud data, determining the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece.

[0009] Optionally, in one embodiment of this application, the step of using the robotic arm to grasp a new workpiece includes: identifying image information and point cloud information of the new workpiece; determining the centroid position of the point cloud of the new workpiece based on the image information and the point cloud information; and driving the robotic arm to grasp the new workpiece based on the coarse positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0010] Optionally, in one embodiment of this application, moving the new workpiece to the target coarse positioning area includes: determining the original pose of the new workpiece, and coarsely aligning the original pose of the new workpiece with the target pose of the template point cloud to determine the pose change data of the new workpiece during coarse alignment; adjusting the posture of the robotic arm end effector according to the pose change data to move the new workpiece to the target coarse positioning area.

[0011] Optionally, in one embodiment of this application, determining the pose change data of the new workpiece during coarse alignment includes: segmenting the acquired image of the new workpiece to obtain the target point cloud data of the new workpiece; determining the target center point of the target point cloud data of the new workpiece, so as to determine the pose change data of the new workpiece during coarse alignment based on the target center point.

[0012] Optionally, in one embodiment of this application, driving the end effector of the composite robot according to the pose deviation value includes: performing iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative nearest point algorithm to obtain the pose deviation value that satisfies the preset error condition; and using the pose deviation value to determine the target adjustment value of the end effector, so as to drive the end effector of the robot according to the target adjustment value.

[0013] Optionally, in one embodiment of this application, driving the robotic arm end effector to move according to the target adjustment value includes: planning multiple motion paths for the robotic arm end effector based on the target adjustment value; determining a target motion path for the robotic arm end effector using the multiple motion paths, so as to control the movement of the robotic arm end effector according to the target motion path.

[0014] This application also provides a workpiece posture alignment device, comprising: an adjustment module, configured to adjust the actual posture of the target workpiece to a target assembly posture after the robotic arm of the composite robot grasps the target workpiece; a determination module, configured to collect multiple point cloud data of the target workpiece in the target assembly posture, and determine template point cloud data based on the multiple point cloud data; and a registration module, configured to determine a pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece based on the template point cloud data, so as to drive the end effector of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data.

[0015] Optionally, in one embodiment of this application, the determining module includes: a first processing unit, configured to perform downsampling processing on the plurality of point cloud data to obtain downsampled point cloud data; and a first determining unit, configured to use the downsampled point cloud data as the template point cloud data.

[0016] Optionally, in one embodiment of this application, the registration module includes: a moving unit, used to use the robotic arm to grasp a new workpiece and move the new workpiece to a target coarse positioning area to determine the initial pose of the new workpiece after movement; an acquisition unit, used to make preliminary adjustments to the end effector of the robotic arm based on the initial pose to obtain the current pose of the new workpiece; and a second determining unit, used to determine the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece based on the target point cloud data of the new workpiece in the current pose and the template point cloud data.

[0017] Optionally, in one embodiment of this application, the moving unit includes: an identification subunit for identifying image information and point cloud information of the new workpiece; a determination subunit for determining the centroid position of the point cloud of the new workpiece based on the image information and the point cloud information; and a driving subunit for driving the robotic arm to grasp the new workpiece based on the coarse positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0018] Optionally, in one embodiment of this application, the moving unit includes: a processing subunit, configured to determine the original pose of the new workpiece and coarsely align the original pose of the new workpiece with the target pose of the template point cloud to determine pose change data of the new workpiece during coarse alignment; and a moving subunit, configured to adjust the end effector posture of the robotic arm according to the pose change data to move the new workpiece to the target coarse positioning area.

[0019] Optionally, in one embodiment of this application, the processing subunit is further configured to segment the acquired image of the new workpiece to obtain the target point cloud data of the new workpiece; determine the target center point of the target point cloud data of the new workpiece, so as to determine the pose change data of the new workpiece during coarse alignment based on the target center point.

[0020] Optionally, in one embodiment of this application, the registration module includes: a second processing unit, configured to perform iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative nearest point algorithm to obtain the pose deviation value that satisfies the preset error condition; and a third determining unit, configured to determine the target adjustment value of the robotic arm end effector using the pose deviation value, so as to drive the robotic arm end effector to move according to the target adjustment value.

[0021] Optionally, in one embodiment of this application, the third determining unit includes: a planning subunit, configured to plan multiple motion paths of the robotic arm end effector based on the target adjustment value; and a control subunit, configured to determine a target motion path of the robotic arm end effector using the multiple motion paths, so as to control the movement of the robotic arm end effector according to the target motion path.

[0022] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described workpiece orientation alignment methods.

[0023] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described workpiece orientation alignment methods.

[0024] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described workpiece orientation alignment methods.

[0025] This application addresses the following technical issues: after the robotic arm of a composite robot grasps the target workpiece, the actual posture of the target workpiece is adjusted to the target assembly posture. Then, multiple point cloud data of the target workpiece in the target assembly posture are collected, and template point cloud data is determined based on these data. This allows for the determination of the pose deviation between the current posture of the new workpiece grasped by the robotic arm and the target assembly posture of the target workpiece. The end effector of the composite robot's robotic arm is then driven according to this pose deviation, aligning the target point cloud data of the new workpiece in its current posture with the template point cloud data. Therefore, this approach solves the problems of high computational resource consumption and decreased algorithm accuracy in related technologies, which reduce the flexibility of the assembly system, especially its reliance on high-precision CAD models. This makes it difficult for the assembly system to adapt to new workpiece models, reducing its versatility and reliability. Ultimately, this approach improves the flexibility and applicability of the assembly system. Attached Figure Description

[0026] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A schematic diagram of an assembly system provided in an embodiment of this application;

[0028] Figure 2 A flowchart of a workpiece posture alignment method provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of an misaligned template point cloud and a new workpiece point cloud according to a specific embodiment of this application;

[0030] Figure 4 This is a schematic diagram of a coarsely aligned template point cloud and a new workpiece point cloud according to a specific embodiment of this application;

[0031] Figure 5 This is a schematic diagram of a finely aligned template point cloud and a new workpiece point cloud according to a specific embodiment of this application;

[0032] Figure 6 This is a structural diagram of a workpiece posture alignment device provided in an embodiment of this application.

[0033] Reference numerals: 10-assembly system, 101-composite robot, 102-visual perception unit, 103-control unit, 1011-multi-degree-of-freedom robotic arm and 1012-hand-eye camera; 20-workpiece posture alignment device, 100-adjustment module, 200-determination module and 300-registration module. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0035] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0036] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The following description, with reference to the accompanying drawings, outlines a workpiece orientation alignment method, apparatus, electronic device, and storage medium according to embodiments of this application.

[0038] Before introducing the workpiece posture alignment method of the embodiments of this application, let me briefly introduce the assembly system involved in the workpiece posture alignment method of this application.

[0039] like Figure 1 As shown, this application establishes an assembly system 10, which includes a composite robot 101, a vision perception unit 102, and a control unit 103. The components work together through physical connections and communication protocols.

[0040] The composite robot 101 includes a mobile chassis and a multi-degree-of-freedom robotic arm 1011 mounted on the chassis. The end of the robotic arm is equipped with a gripper and a hand-eye camera 1012, wherein the hand-eye camera 1012 is a depth camera.

[0041] The visual perception unit 102 is a depth camera set at a fixed position on the worktable to collect point cloud data of the workpiece.

[0042] The control unit 103, i.e., the industrial computer, is wired to the depth camera and is responsible for point cloud processing and pose calculation; the robot controller is built into the body of the composite robot 101 and is used to control the movement of the robotic arm and the mobile chassis; the control unit 103, i.e., the industrial computer, and the robot controller interact with each other through a wireless network and transmit pose adjustment commands in real time.

[0043] like Figure 2 As shown, Figure 2 This is a flowchart of a workpiece orientation alignment method according to an embodiment of this application. The workpiece orientation alignment method includes the following steps:

[0044] In step S201, after the robotic arm of the composite robot grasps the target workpiece, the actual posture of the target workpiece is adjusted to the target assembly posture.

[0045] In this embodiment, the target workpiece is the workpiece that provides the template point cloud; the target assembly posture is the ideal assembly posture, which can be set by relevant technical personnel.

[0046] It is understood that in the embodiments of this application, the operator first manually controls the composite robot in the above-mentioned assembly system to grasp the workpiece to be assembled. After the robotic arm of the composite robot grasps the workpiece to be assembled, the position of the end of the robotic arm is adjusted so that the posture of the workpiece accurately reaches the ideal position required by the assembly process.

[0047] This embodiment of the application combines manual control to grasp the workpiece and then adjusts the position of the robotic arm's end effector to precisely align it with the ideal position, thus ensuring efficient and accurate assembly results.

[0048] In step S202, multiple point cloud data of the target workpiece in the target assembly posture are collected, and template point cloud data are determined based on the multiple point cloud data.

[0049] It is understood that, in the embodiments of this application, an external depth camera fixedly installed on the worktable in the above-described assembly system can be used to photograph the positioned workpiece, thereby collecting multiple point cloud data of the workpiece in an ideal assembly posture, such as complete surface point cloud data, and determining multiple point cloud data as template point cloud data. Let the workpiece pose of the current template point cloud be:

[0050] T grasp =[R g ||t g ]∈SE(3)

[0051] Among them, R g The nominal capture pose rotation matrix relative to the world coordinate system is represented by t. g SE(3) represents the translation vector of the nominal capture pose relative to the world coordinate system, and SE(3) represents the special Euclidean group.

[0052] This application embodiment can use an external depth camera to collect point cloud data of the ideal assembly posture of the workpiece and use it as template point cloud data, which ensures high-precision template establishment and improves the accuracy and consistency of the subsequent assembly process.

[0053] In one embodiment of this application, determining template point cloud data based on multiple point cloud data includes: performing downsampling processing on multiple point cloud data to obtain downsampled point cloud data; and using the downsampled point cloud data as template point cloud data.

[0054] In actual implementation, the embodiments of this application can perform data preprocessing on multiple point cloud data, i.e., complete surface point cloud data. For example, outliers and noise can be removed by using statistical filtering, radius filtering, etc., point cloud density can be reduced by using voxel grid filtering, replacing points in each voxel with the mean or center point, and calculating the normal direction of each point in the point cloud. In the embodiments of this application, voxel grid filtering can be used to reduce point cloud density.

[0055] In this process, the original point cloud is downsampled using a voxel grid to achieve uniform point cloud density, i.e.:

[0056] P down ={p i |p i =VoxelCenter(v j ),v j ∈Grid(P raw ,ρ)}

[0057] Where ρ is the side length of the voxel, p i P is a point in the downsampled point cloud. downFor the downsampled point cloud set, VoxelCenter(v j ) is voxel v j The center position, v j P is a voxel in a voxel grid. raw For raw point cloud data, Grid(P) raw ,ρ) represents the original point cloud P raw The process of voxel rasterization.

[0058] It should be noted that the point clouds mentioned in the embodiments of this application are all point clouds after voxel grid downsampling, which effectively reduces the amount of data, improves the efficiency of subsequent processing, simplifies the template point cloud data, reduces the computing resource requirements, and ensures efficient and accurate attitude alignment of new workpieces.

[0059] In step S203, based on the template point cloud data, the pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece is determined, so as to drive the end effector of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data.

[0060] In this embodiment, the target point cloud data is local point cloud data; the target pose alignment is precise pose alignment.

[0061] It is understood that, in the embodiments of this application, after the calibration of the template point cloud in the above steps is completed, the composite robot enters the autonomous assembly stage. The robotic arm of the composite robot collects new workpiece data in real time based on a hand-eye camera. For example, it identifies the new workpiece using a target detection algorithm, thereby driving the robotic arm to move and grasp the new workpiece. Figure 3 As shown, this is the misaligned workpiece point cloud (i.e., the new workpiece point cloud) and the template point cloud.

[0062] Next, in this embodiment of the application, the new workpiece can be moved to the coarse positioning area so that the new workpiece is roughly within the visible area of ​​the fixed depth camera. The new workpiece is then processed by the fixed depth camera to obtain workpiece posture information with large errors, and the end effector of the robotic arm is initially adjusted to roughly align the new workpiece and the template pose.

[0063] Secondly, a fixed depth camera is used to determine the local point cloud data of the new workpiece in the current posture, i.e., the new workpiece point cloud in this application. The pose deviation value between the local point cloud data and the template point cloud data is calculated using the ICP (Iterative Closest Point) algorithm. The pose deviation value is used to drive the end effector of the composite robot to move, so that the pose of the new workpiece is adjusted to the expected posture, thereby achieving accurate posture alignment between the local point cloud data of the new workpiece in the current posture and the template point cloud data.

[0064] In this embodiment of the application, a fixed depth camera can also be used to determine the complete point cloud of the new workpiece in the current pose, so as to have a more comprehensive understanding of the actual state of the new workpiece, which helps to further verify the accuracy of the pose alignment, or to provide more detailed data support when higher precision operations are required.

[0065] The embodiments of this application can accurately calculate the pose deviation and adjust the robot arm's movements to ensure that the workpiece can accurately reach the predetermined assembly position and posture. Compared with related technologies, this application completes precision assembly without fixtures or model dependence, improves assembly quality, realizes efficient and accurate workpiece assembly, and enhances the robustness and adaptability of the assembly system, which is of great significance for improving the level of industrial automation.

[0066] In one embodiment of this application, determining the pose deviation between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece includes: using the robotic arm to grasp the new workpiece and moving the new workpiece to the target coarse positioning area to determine the initial pose of the new workpiece after movement; based on the initial pose, making preliminary adjustments to the end effector of the robotic arm to obtain the current pose of the new workpiece; and based on the target point cloud data and template point cloud data of the new workpiece in the current pose, determining the pose deviation between the current pose of the new workpiece and the target assembly pose of the target workpiece.

[0067] As one possible implementation, this embodiment utilizes a robotic arm to grasp a new workpiece, moving it from its initial position to a pre-defined coarse positioning area to determine its initial pose. Then, based on this initial pose, the robotic arm's end effector is preliminarily adjusted to obtain the workpiece's current pose. A fixed depth camera is used to acquire local point cloud data at this pose, and the local point cloud data of the new workpiece at its current pose is compared and analyzed with template point cloud data to determine the pose deviation between the new workpiece's current pose and the ideal assembly pose of the target workpiece. Therefore, this embodiment significantly reduces the search space and time required to find the correct position of the new workpiece by performing coarse positioning followed by fine adjustment, thereby significantly improving workpiece positioning accuracy and assembly efficiency while reducing operational complexity.

[0068] Optionally, in one embodiment of this application, using a robotic arm to grasp a new workpiece includes: identifying image information and point cloud information of the new workpiece; determining the centroid position of the point cloud of the new workpiece based on the image information and point cloud information; and driving the robotic arm to grasp the new workpiece based on the coarse positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0069] In some embodiments, the robotic arm of the composite robot in this application uses real-time data of new workpieces collected by a hand-eye camera to identify image and point cloud information of the new workpieces through target detection algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), point-line features, template matching, YOLO, Faster RCNN (Region-based Convolutional Neural Network), and SSD (Single Shot MultiBox Detector). This determines the centroid position of the point cloud of the new workpiece. By combining the coarse positioning coordinates of the centroid position of the point cloud of the new workpiece in the base coordinate system, the robotic arm can be driven to move and complete the grasping of the new workpiece. This achieves fast and accurate positioning and grasping of the new workpiece, improving operational efficiency and automation.

[0070] Optionally, in one embodiment of this application, moving a new workpiece to a target coarse positioning area includes: determining the original pose of the new workpiece and coarsely aligning the original pose of the new workpiece with the target pose of the template point cloud to determine the pose change data of the new workpiece during coarse alignment; and adjusting the end effector posture of the robotic arm according to the pose change data to move the new workpiece to the target coarse positioning area.

[0071] For example, after the robotic arm has finished grasping the new workpiece, it is no longer visible in the field of view of the hand-eye camera. Therefore, the new workpiece can be located using a fixed depth camera. Let the original pose of the new workpiece after actual grasping be T. obj Due to various factors such as the robotic arm's motion error, positioning error during gripping, and relative sliding after gripping, the posture of the new workpiece in the gripping state is uncertain and may deviate significantly from the expected posture.

[0072] T obj =T grasp ·ΔT A ·ΔT G ·ΔT F ·…·T C =T grasp ·ΔT E·T C

[0073] Among them, T obj Indicates the new workpiece pose, T grasp The workpiece pose, ΔT, represents the template point cloud. A ΔT G ΔT F And… respectively represent the attitude error caused by the robotic arm's motion error, the attitude error caused by the positioning error during grasping, the attitude error caused by the relative sliding after grasping, and other uncertain attitude errors; T C This represents the attitude transformation required for coarse alignment between the new workpiece and the template point cloud after the workpiece is grasped. For simplicity, the attitude error caused by all errors is written as ΔT. E .

[0074] Next, in this embodiment of the application, the original pose of the new workpiece can be compared with the target pose of the template point cloud, i.e., the workpiece pose T of the template point cloud. grasp Coarse alignment is performed to determine the pose change data of the new workpiece during coarse alignment. Based on the pose change data, the posture of the robotic arm end effector is adjusted to move the new workpiece to the coarse positioning area. This efficiently achieves the initial positioning and movement of the new workpiece, simplifies the adjustment process before precise assembly, and improves the overall operating efficiency and automation level.

[0075] Optionally, in one embodiment of this application, determining the pose change data of the new workpiece during coarse alignment includes: segmenting the acquired image of the new workpiece to obtain target point cloud data of the new workpiece; determining the target center point of the target point cloud data of the new workpiece, so as to determine the pose change data of the new workpiece during coarse alignment based on the target center point.

[0076] In actual implementation, this embodiment first adjusts the position of the robotic arm's end effector so that the new workpiece is roughly within the field of view of the fixed depth camera. Then, it uses the RGB-D image aligned with the fixed depth camera to segment the acquired image of the new workpiece. For example, it uses YOLO for target detection and GrabCut for RGB target segmentation. The point cloud data p = {p} of the new workpiece is obtained through RGB-D image alignment. i Let i = 1, 2, 3…n, and the plane equation of the new workpiece be n·p + d = 0, where n is the unit normal vector and d is the distance from the origin to the plane. The plane equation can be obtained by optimizing the following equation:

[0077] min∑(n·p i +d) 2 st||n||=1

[0078] The center point c of the point cloud of the new workpiece region is calculated as follows:

[0079]

[0080] Where c is the center point.

[0081] T can be obtained using Rodrigues's Formula. C =[R C |t C ],in,

[0082] R C =I + sinθ·[k] × +(1-cosθ)·[k] × 2

[0083] t C =t g -R C c

[0084] Where, [k] × Let R be the cross product matrix of k. C Let t be a rotation matrix. C Let I be the translation vector, θ be the identity matrix, θ be the rotation angle, k be the unit vector of the rotation axis, and t be the translation vector. g is the translation vector in the target coordinate system.

[0085] in,

[0086]

[0087] Among them, R g Let n be the target rotation matrix. T Let ||n|| and ||R be the transpose of n. g n|| represents n and R respectively. g The modulus of n.

[0088] The required attitude adjustment ΔT for the robotic arm's end effector can be obtained through the real-time calibrated system coordinate relationship. relative for:

[0089]

[0090] Where, ΔT base T is the pose transformation matrix relative to the base coordinate system. C The attitude transformation ΔT is required for coarse alignment of the new workpiece with the template point cloud after it has been grasped. relative Let be the pose transformation matrix relative to the tool coordinate system. This is the transformation matrix from the camera coordinate system to the base coordinate system. This is the transformation matrix from the base coordinate system to the end-effector coordinate system obtained using the kinematics of the robotic arm and the DH (Denavit-Hartenberg) model.

[0091] like Figure 4 The image shows the coarsely aligned workpiece point cloud (i.e., the new workpiece point cloud) and the template point cloud. Due to various errors ΔT, the results are inconsistent. E Therefore, it is difficult for the point cloud of the new workpiece and the point cloud of the template to completely overlap during coarse alignment; in general, there will be some overlap.

[0092] Optionally, in one embodiment of this application, driving the end effector of the composite robot based on the pose deviation value includes: performing iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative nearest point algorithm to obtain a pose deviation value that meets the preset error conditions; using the pose deviation value to determine the target adjustment value of the end effector, so as to drive the end effector of the robot based on the target adjustment value.

[0093] As one possible approach, embodiments of this application can begin fine alignment after coarse alignment is completed, i.e., eliminating various errors ΔT. E At time k, the point cloud is registered using the ICP algorithm, and the registration error is obtained. Using the following Adjusting the end effector motion of the robotic arm, i.e.:

[0094]

[0095] in, Let represent the pose change matrix relative to the tool coordinate system at time k. This indicates the registration error.

[0096] Then, the ICP algorithm is repeated to register the point cloud again, resulting in a new registration error. Adjust the robotic arm again, and repeat this process iteratively until the registration error is correct. Within the tolerance range or reaching the maximum number of iterations I t .

[0097] In actual operation, due to factors such as an unsatisfactory shooting angle, calibration error has a significant impact on the assembly system. If the attitude adjustment is performed directly according to the method described above, it may not converge to the desired attitude.

[0098]

[0099] Where, ΔT ca Represents the calibration error matrix. This represents the actual coordinate transformation matrix.

[0100] When actually controlling the robotic arm's movement, the composite robot should be in a stationary state; therefore, ΔT ca It is an unknown time-invariant constant matrix.

[0101] The above Write it in the following form:

[0102]

[0103] Increasing the error gain matrix K will result in an iteratively adjusted error. With current error The relationship is:

[0104]

[0105] in, Indicates the calibration error ΔT ca The adjoint matrix; This represents the error after iterative adjustment. Let K represent the current error, and K represent the error gain matrix.

[0106] Assuming the calibration error is sufficiently small, then:

[0107]

[0108] Where I is the identity matrix.

[0109] Define the error gain matrix K as a diagonal matrix, that is:

[0110] K = diag(k1,k2,k3,k4,k5,k6)

[0111] The spectral radius of the error gain matrix can be adjusted by configuring parameters. This ensures error convergence. For example... Figure 5 As shown, the template point cloud and workpiece point cloud (i.e., the new workpiece point cloud) are precisely aligned. Through iterative loops, the movement of the robotic arm is controlled to bring the error to converge and complete the registration.

[0112] In one embodiment of this application, driving the end effector of the robotic arm to move according to a target adjustment value includes: planning multiple motion paths for the end effector of the robotic arm based on the target adjustment value; and using the multiple motion paths to determine a target motion path for the end effector of the robotic arm, so as to control the motion of the end effector of the robotic arm according to the target motion path.

[0113] In this embodiment of the application, the target motion path is the optimal motion path of the robotic arm end effector.

[0114] In some embodiments, this application can generate multiple potential motion paths for workpiece orientation alignment using different path planning algorithms based on the target adjustment value. When generating paths, the workspace limitations of the robotic arm, joint limits, obstacle positions, and self-collision avoidance factors are fully considered to ensure that each path is feasible. In addition, a series of evaluation criteria can be set for each path, such as path length, execution time, energy consumption, smoothness, and obstacle avoidance performance, so as to conduct a comprehensive score and select the path with the highest score as the optimal motion path, thereby realizing an efficient, accurate, and safe operation process.

[0115] Therefore, compared with related technologies, this application does not require customized processing tooling fixtures, reducing costs and improving production line flexibility; and it does not require high-precision CAD models, but only achieves convergence of registration errors through iterative registration and closed-loop control. In addition, for point cloud registration of workpieces, only a portion of the point cloud needs to be extracted, and sub-pixel-level positioning of feature points is not required. It has a high tolerance for errors such as insufficient segmentation accuracy of the target foreground (missing point cloud), thereby realizing model-free precision assembly in dynamic scenes.

[0116] According to the workpiece posture alignment method proposed in this application, after the robotic arm of the composite robot grasps the target workpiece, the actual posture of the target workpiece is adjusted to the target assembly posture. Then, multiple point cloud data of the target workpiece under the target assembly posture are collected, and template point cloud data is determined based on the multiple point cloud data. Then, the pose deviation value between the current posture of the new workpiece grasped by the robotic arm and the target assembly posture of the target workpiece is determined. The end effector of the robotic arm of the composite robot is driven according to the pose deviation value, so that the target point cloud data of the new workpiece under the current posture and the template point cloud data are aligned. Therefore, it can solve the technical problems of high computational resource consumption and decreased algorithm accuracy in related technologies, which reduce the flexibility of the assembly system, especially the dependence on high-precision CAD models, which makes it difficult for the assembly system to adapt when handling new types of workpieces, and reduces the versatility and reliability of the assembly system. It achieves the technical effect of improving the flexibility and applicability of the assembly system.

[0117] Next, the workpiece posture alignment device according to the embodiments of this application is described with reference to the accompanying drawings.

[0118] Figure 6 This is a block diagram of a workpiece posture alignment device according to an embodiment of this application.

[0119] like Figure 6 As shown, the workpiece posture alignment device 20 includes: an adjustment module 100, a determination module 200, and a registration module 300.

[0120] Specifically, the adjustment module 100 is used to adjust the actual posture of the target workpiece to the target assembly posture after the robotic arm of the composite robot grasps the target workpiece.

[0121] The determination module 200 is used to collect multiple point cloud data of the target workpiece in the target assembly posture, and determine template point cloud data based on the multiple point cloud data.

[0122] The registration module 300 is used to determine the pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece based on the template point cloud data, so as to drive the end effector of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data.

[0123] Optionally, in one embodiment of this application, the determining module 200 includes a first processing unit and a first determining unit.

[0124] The first processing unit is used to downsample multiple point cloud data to obtain downsampled point cloud data.

[0125] The first determining unit is used to use the downsampled point cloud data as template point cloud data.

[0126] Optionally, in one embodiment of this application, the registration module 300 includes: a moving unit, an acquiring unit, and a second determining unit.

[0127] The moving unit is used to use a robotic arm to grasp a new workpiece and move it to the target coarse positioning area to determine the initial pose of the new workpiece after it has been moved.

[0128] The acquisition unit is used to make preliminary adjustments to the end effector of the robotic arm based on the initial pose in order to obtain the current pose of the new workpiece.

[0129] The second determining unit is used to determine the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece based on the target point cloud data and template point cloud data of the new workpiece in the current pose.

[0130] Optionally, in one embodiment of this application, the moving unit includes: an identification subunit, a determination subunit, and a driving subunit.

[0131] The recognition subunit is used to recognize the image information and point cloud information of the new workpiece.

[0132] The determination sub-unit is used to determine the centroid position of the point cloud of the new workpiece based on image information and point cloud information.

[0133] The drive subunit is used to drive the robotic arm to grasp the new workpiece based on the coarse positioning coordinates of the centroid position of the point cloud of the new workpiece in the target base coordinate system.

[0134] Optionally, in one embodiment of this application, the moving unit includes a processing subunit and a moving subunit.

[0135] The processing subunit is used to determine the original pose of the new workpiece and coarsely align the original pose of the new workpiece with the target pose of the template point cloud to determine the pose change data of the new workpiece during coarse alignment.

[0136] The moving subunit is used to adjust the posture of the robotic arm end effector based on the pose change data in order to move the new workpiece to the target coarse positioning area.

[0137] Optionally, in one embodiment of this application, the processing subunit is further configured to segment the acquired image of the new workpiece to obtain target point cloud data of the new workpiece; determine the target center point of the target point cloud data of the new workpiece, so as to determine the pose change data of the new workpiece during coarse alignment based on the target center point.

[0138] Optionally, in one embodiment of this application, the registration module 300 includes a second processing unit and a third determining unit.

[0139] The second processing unit is used to perform iterative registration processing on the target point cloud data and the template point cloud data based on the target iterative nearest point algorithm, so as to obtain the pose deviation value that meets the preset error conditions.

[0140] The third determining unit is used to determine the target adjustment value of the robotic arm end effector using the pose deviation value, so as to drive the robotic arm end effector to move according to the target adjustment value.

[0141] Optionally, in one embodiment of this application, the third determining unit includes a planning subunit and a control subunit.

[0142] The planning subunit is used to plan multiple motion paths at the end of the robotic arm based on the target adjustment value.

[0143] The control subunit is used to determine the target motion path of the robotic arm end effector using multiple motion paths, so as to control the movement of the robotic arm end effector according to the target motion path.

[0144] It should be noted that the description of the features in the embodiment corresponding to the workpiece posture alignment device can be found in the relevant description of the embodiment corresponding to the workpiece posture alignment method, and will not be repeated here.

[0145] According to the workpiece posture alignment device proposed in this application embodiment, after the robotic arm of the composite robot grasps the target workpiece, the actual posture of the target workpiece is adjusted to the target assembly posture. Then, multiple point cloud data of the target workpiece under the target assembly posture are collected, and template point cloud data is determined based on the multiple point cloud data. Then, the pose deviation value between the current posture of the new workpiece grasped by the robotic arm and the target assembly posture of the target workpiece is determined. The end effector of the robotic arm of the composite robot is driven to move according to the pose deviation value, so that the target point cloud data of the new workpiece under the current posture and the template point cloud data are aligned. Therefore, it can solve the technical problems of high computational resource consumption and decreased algorithm accuracy in related technologies, which reduce the flexibility of the assembly system, especially the dependence on high-precision CAD models, which makes it difficult for the assembly system to adapt when handling new types of workpieces, and reduces the versatility and reliability of the assembly system. It achieves the technical effect of improving the flexibility and applicability of the assembly system.

[0146] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described workpiece orientation alignment method embodiments.

[0147] Embodiments of this application also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described workpiece orientation alignment method embodiments during runtime.

[0148] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0149] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described workpiece orientation alignment method embodiments.

[0150] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described workpiece orientation alignment method embodiments.

[0151] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] The above provides a detailed description of a workpiece orientation alignment method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A workpiece orientation alignment method, characterized in that, Applied to composite robots, the method includes the following steps: After the robotic arm of the composite robot grasps the target workpiece, the actual posture of the target workpiece is adjusted to the target assembly posture. Multiple point cloud data of the target workpiece under the target assembly posture are collected, and template point cloud data is determined based on the multiple point cloud data, wherein the template point cloud data is the point cloud data of the target workpiece under the target assembly posture. Based on the template point cloud data, the pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece is determined. The robotic arm end effector is then driven to move according to the pose deviation value, aligning the target point cloud data of the new workpiece in the current posture with the template point cloud data. Driving the robotic arm end effector to move according to the pose deviation value includes: iteratively registering the target point cloud data and the template point cloud data based on a target iterative nearest-point algorithm to obtain the pose deviation value that meets a preset error condition; determining a target adjustment value for the robotic arm end effector using the pose deviation value, and driving the robotic arm end effector to move according to the target adjustment value; and driving the robotic arm end effector to move according to the target adjustment value includes: planning multiple motion paths for the robotic arm end effector based on the target adjustment value; comprehensively scoring the multiple motion paths based on a target evaluation criterion; determining the target motion path for the robotic arm end effector based on the comprehensive scoring result; and controlling the robotic arm end effector to move according to the target motion path.

2. The method according to claim 1, characterized in that, The step of determining template point cloud data based on the plurality of point cloud data includes: The multiple point cloud data are downsampled to obtain downsampled point cloud data; The downsampled point cloud data is used as the template point cloud data.

3. The method according to claim 1, characterized in that, The determination of the pose deviation value between the current pose of the new workpiece grasped by the robotic arm of the composite robot and the target assembly pose of the target workpiece includes: The robotic arm is used to grasp a new workpiece and move it to the target coarse positioning area to determine the initial pose of the new workpiece after the movement. Based on the initial pose, the end effector of the robotic arm is initially adjusted to obtain the current pose of the new workpiece; Based on the target point cloud data of the new workpiece in the current pose and the template point cloud data, the pose deviation value between the current pose of the new workpiece and the target assembly pose of the target workpiece is determined.

4. The method according to claim 3, characterized in that, The process of using the robotic arm to grasp a new workpiece includes: Identify the image information and point cloud information of the new workpiece; Based on the image information and the point cloud information, the centroid position of the point cloud of the new workpiece is determined; Based on the coarse positioning coordinates of the centroid of the point cloud of the new workpiece in the target base coordinate system, the robotic arm is driven to grasp the new workpiece.

5. The method according to claim 3, characterized in that, Moving the new workpiece to the target coarse positioning area includes: The original pose of the new workpiece is determined, and the original pose of the new workpiece is coarsely aligned with the target pose of the template point cloud to determine the pose change data of the new workpiece during coarse alignment. The robotic arm end effector is adjusted based on the pose change data to move the new workpiece to the target coarse positioning area.

6. The method according to claim 5, characterized in that, The determination of the pose change data during coarse alignment of the new workpiece includes: The acquired image of the new workpiece is segmented to obtain the target point cloud data of the new workpiece; The target center point of the target point cloud data of the new workpiece is determined, and the pose change data of the new workpiece during coarse alignment is determined based on the target center point.

7. A workpiece posture alignment device, characterized in that, Applied to composite robots, wherein the device includes: An adjustment module is used to adjust the actual posture of the target workpiece to the target assembly posture after the robotic arm of the composite robot grasps the target workpiece. The determination module is used to collect multiple point cloud data of the target workpiece under the target assembly posture, and determine template point cloud data based on the multiple point cloud data, wherein the template point cloud data is the point cloud data of the target workpiece under the target assembly posture. A registration module is used to determine, based on the template point cloud data, the pose deviation value between the current posture of the new workpiece grasped by the robotic arm of the composite robot and the target assembly posture of the target workpiece, so as to drive the end effector of the robotic arm of the composite robot to move according to the pose deviation value, so that the target point cloud data of the new workpiece in the current posture is aligned with the template point cloud data. The step of driving the end effector of the robotic arm according to the pose deviation value includes: performing iterative registration processing on the target point cloud data and the template point cloud data based on a target iterative nearest-point algorithm to obtain the pose deviation value that meets a preset error condition; determining a target adjustment value for the end effector of the robotic arm using the pose deviation value, so as to drive the end effector of the robotic arm to move according to the target adjustment value; and the step of driving the end effector of the robotic arm according to the target adjustment value includes: planning multiple motion paths for the end effector of the robotic arm based on the target adjustment value; comprehensively scoring the multiple motion paths based on a target evaluation criterion, and determining the target motion path for the end effector of the robotic arm based on the comprehensive scoring result, so as to control the end effector of the robotic arm to move according to the target motion path.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the workpiece orientation alignment method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the workpiece orientation alignment method as described in any one of claims 1-6.

Images