Object grabbing method and system based on point cloud deep learning

Abstract

The invention provides an object grabbing method and system based on point cloud deep learning, and the method comprises the steps: obtaining a target object point cloud of a target object, carrying out the point cloud matching of the target object point cloud through a point cloud registration model based on a deep learning network, obtaining an initial matching position, optimizing the matching position through a nearest iteration algorithm, and obtaining an optimal matching position; obtaining an accurate transformation matrix of the target object relative to the template under a camera coordinate system; based on a pre-obtained hand-eye transformation matrix, calculating a transformation relation of the point cloud attitude of the target object relative to a tool coordinate system through teaching; the hand-eye transformation matrix is used for indicating a transformation relation between a sensor coordinate system and a mechanical arm tail end coordinate system; and according to the conversion relation and the accurate transformation matrix, the grabbing position coordinates of the target object under the mechanical arm base coordinate system are solved. According to the invention, point cloud registration is carried out through the deep learning network, and the position of the target object can be determined more accurately.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and industrial automation, and in particular to an object grasping method and system based on point cloud deep learning. Background Art

[0002] With the continuous development of industrial automation and machine vision technology, object grasping and positioning are playing an increasingly important role in fields such as industrial production and logistics and transportation. Currently, the more common technical solutions are grasping systems based on 2D vision and grasping systems based on traditional 3D point clouds. The 2D grasping method is suitable for grasping planar objects. Due to the lack of depth information, the posture in the RX and RY directions cannot be obtained, resulting in low grasping efficiency and accuracy. Traditional 3D point cloud methods rely heavily on the integrity of the target object point cloud during processing. When changes in ambient lighting, etc., lead to image loss, the grasping error is large.

[0003] Based on this, the present application provides an object grasping method and system based on point cloud deep learning. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings of large grasping errors under changes in environmental lighting and other conditions, the present application provides an object grasping method and system based on point cloud deep learning.

[0005] The purpose of this application is achieved by the following technical solutions: In a first aspect, the present application provides an object grasping method based on point cloud deep learning, comprising: Obtaining a target point cloud of the target object, performing point cloud matching on the target point cloud using a point cloud registration model based on a deep learning network to obtain an initial matching position, and then optimizing the matching position using a nearest-repeat algorithm to obtain an accurate transformation matrix of the target object relative to the template in the camera coordinate system; Based on a pre-acquired hand-eye transformation matrix, the transformation relationship between the target object's point cloud pose and the tool coordinate system is calculated through teaching; the hand-eye transformation matrix is ​​used to indicate the transformation relationship between the sensor coordinate system and the robot arm end coordinate system; The grasping position coordinates of the target object in the robot arm base coordinate system are calculated based on the conversion relationship and the precise transformation matrix.

[0006] Preferably, the hand-eye transformation matrix is ​​obtained by: According to the motion transformation of the end of the robotic arm and the motion transformation observed by the sensor, the hand-eye transformation matrix is ​​obtained.

[0007] Preferably, the step of obtaining the hand-eye transformation matrix based on the motion transformation of the end of the robotic arm and the motion transformation observed by the sensor includes: Perform camera intrinsic calibration, extract corner points and refine them to sub-pixel accuracy, and use Zhang's calibration algorithm to obtain camera internal parameters and distortion coefficients as the camera calibration results; Perform image distortion correction, use the camera calibration result to dedistort the image, and re-extract sub-pixel precision corner point coordinates; Define the world coordinate system on the calibration plate and obtain the 3D coordinates of the corner points; Use the PnP algorithm to build a perspective projection model, solve the rotation and translation matrices from the world coordinate system to the camera coordinate system and use them as the camera pose; Based on the robot arm pose and camera pose, the hand-eye transformation matrix is ​​obtained by solving the equation.

[0008] Preferably, obtaining the hand-eye transformation matrix by solving equations based on the robot arm posture and the camera posture includes: Based on the robot arm pose and camera pose, the hand-eye calibration matrix is ​​obtained through the conversion relationship; Evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error, and using the hand-eye calibration matrix as the hand-eye transformation matrix when the accuracy meets the preset accuracy requirement; The step of evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error includes: Using two sets of pose data, one set is used as the true value to estimate the other set of camera poses. The poses and 3D point sets are reprojected to obtain the coordinates of the 2D point set on the image plane. The mean square error between the projected point coordinates and the corner point coordinates is calculated and used as the accuracy value.

[0009] Preferably, when the sensor is fixed at the end of the robotic arm, the hand-eye transformation matrix is ​​obtained by eye-on-hand calibration, and the conversion relationship is as follows: ; When the sensor holder is fixed outside the robotic arm, the hand-eye transformation matrix is ​​obtained by calibrating the eye outside the hand. The conversion relationship is as follows:

[0010] in, is the transformation matrix from the target coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the robot end coordinate system, is the transformation matrix from the manipulator end coordinate system to the manipulator base coordinate system, is the transformation matrix from the sensor coordinate system to the robot base coordinate system, is the transformation matrix from the robot base coordinate system to the robot end coordinate system; T 0 and T 1 represents the transformation matrix in two different states.

[0011] Preferably, the calculation of the transformation relationship between the point cloud posture of the target object and the tool coordinate system by teaching based on the pre-acquired hand-eye transformation matrix includes: When the eyes are on the hand, the transformation relationship between the point cloud pose of the target object and the tool coordinate system is calculated by teaching the following formula: ; When the eye is outside the hand, the following formula is used to teach and calculate the transformation relationship between the target object's point cloud posture and the tool coordinate system: ; in, is the transformation matrix from the target point cloud coordinate system to the tool coordinate system, is the transformation matrix from the target point cloud coordinate system to the robot arm base coordinate system, is the transformation matrix from the robot base coordinate system to the tool coordinate system, is the transformation matrix from the target cloud coordinate system to the sensor coordinate system, The transformation matrix from the end coordinate system to the robot base coordinate system when taking pictures; The method of calculating the grasping position coordinates of the target object in the robot arm base coordinate system according to the conversion relationship and the precise transformation matrix includes: When the eye is on the hand, the grasping position is calculated using the following formula: ; When the eye is outside the hand, the grasping position is calculated using the following formula: ; in, is the transformation matrix from the base coordinate system of the robot arm to the end coordinate system when taking pictures, is the transformation matrix from the robot end coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the target point cloud coordinate system, is the transformation matrix from the robot base coordinate system to the sensor coordinate system.

[0012] Preferably, a method for acquiring the target point cloud of the target object includes: The original point cloud data is obtained by using a sensor. When the eye is on the hand, the sensor is mounted on the end of the robotic arm. When the eye is outside the hand, the sensor is mounted at a fixed position outside the robotic arm. The target object is extracted from the original point cloud data to obtain a target object point cloud.

[0013] Preferably, extracting the target object from the original point cloud data to obtain the target object point cloud includes: Filter the point cloud data according to the preset XYZ direction maximum value, filter out the redundant points in the scene, and retain the point cloud data in the area of ​​interest; Get the estimated normal vector value of each retained point in its neighborhood, and use the normal vector information to filter the point cloud; For the filtered point cloud data, points within a preset distance range are divided into multiple connected clusters; from the multiple connected clusters obtained by cluster segmentation, point clouds that meet the characteristics of the target object are screened and extracted as the target object point cloud; The method of dividing the points within a preset distance range into multiple connected clusters for the filtered point cloud data includes: S131, select a point as the starting point; S132, mark the starting point as visited and add it to a separate target cluster; S133, taking the point as the center, selecting points within a preset range as candidate points, screening out unvisited points and adding them to the target cluster; S134, for each point in the target cluster, repeat S133 until all points have been visited; S135 , selecting a remaining unvisited point as a new starting point, and repeating S132 until all points are clustered to form multiple connected clusters.

[0014] Preferably, performing point cloud matching on the target object point cloud using a point cloud registration model based on a deep learning network to obtain an initial matching position includes: Build deep learning networks; Acquire a data set, wherein the data set is obtained by collecting a preset number of sample target object point clouds through a sensor device, and then traverse the obtained target object point clouds and perform denoising and downsampling; The data set is used as a training set to train the deep learning network to obtain a point cloud registration model; the target object point cloud is input into the point cloud registration model to obtain an initial matching position.

[0015] In a second aspect, the present application also provides an object grasping system based on point cloud deep learning, comprising an electronic device and a point cloud acquisition device, the electronic device comprising a memory and at least one processor, the memory storing a computer program, the at least one processor being configured to implement a method as described in any one of the first aspects when executing the computer program; the point cloud acquisition device is used to obtain a target object point cloud and transmit it to the electronic device.

[0016] In combination with the above-mentioned technical solutions and the technical problems solved, the present application provides an object grasping method based on point cloud deep learning. First, the 3D data (point cloud) of the target object and the coordinate transformation relationship (hand-eye transformation matrix) between the sensor and the end of the robotic arm are obtained. Next, the point cloud of the target object is matched using a deep learning network to obtain an initial matching position, and the position is optimized through the latest iteration algorithm to obtain the precise transformation matrix of the target object relative to the template in the camera coordinate system. Then, based on the hand-eye transformation matrix, the transformation relationship of the point cloud posture of the target object in the tool coordinate system is calculated through teaching. Finally, based on the transformation relationship and the precise transformation matrix, the specific grasping position coordinates of the target object in the base coordinate system of the robotic arm are calculated to guide the robotic arm to move to that position to complete the grasping.

[0017] The advantages and positive effects of the technical solution to be protected are as follows: On the one hand, the introduction of the hand-eye transformation matrix solves the coordinate conversion problem between visual information and robotic arm motion control. At the same time, through point cloud registration through a deep learning network, the complex features of the target object can be learned, and the position of the target object can be determined more accurately than traditional methods. The subsequent optimization of the latest iterative algorithm further improves the matching accuracy, thereby obtaining more accurate target position and posture information, effectively reducing the grasping deviation caused by inaccurate matching, and improving the success rate and accuracy of grasping. On the other hand, the posture of the target object in the camera coordinate system is converted to the posture in the robotic arm base coordinate system, so that the robotic arm can understand and use this information to plan the grasping action. The entire process, from data acquisition, point cloud registration, position solution to subsequent robotic arm grasping guidance, realizes an automated grasping process, which not only reduces manual intervention and complex parameter settings, improves operating efficiency and reliability, but also enables this method to adapt to grasping tasks of different types of objects and scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present application is further described below with reference to the accompanying drawings and implementation methods.

[0019] Figure 1 This is a flow chart of an object grasping method based on point cloud deep learning provided in an embodiment of the present application.

[0020] Figure 2 This is a flowchart of obtaining a hand-eye transformation matrix provided in an embodiment of the present application.

[0021] Figure 3 This is a flow chart of obtaining a hand-eye transformation matrix provided in an embodiment of the present application.

[0022] Figure 4 This is a flow chart of obtaining a point cloud of a target object provided in an embodiment of the present application.

[0023] Figure 5 This is a flow chart of obtaining a connected cluster provided in an embodiment of the present application.

[0024] Figure 6 This is a flow chart of obtaining an initial matching position provided in an embodiment of the present application.

[0025] Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0026] Figure 8 This is a schematic diagram of the calibration plate images and acquisition postures of the 21 sets of calibration data provided in the embodiment of the present application.

[0027] Figure 9a and Figure 9b This is a coordinate system distribution diagram for the cases where the eye is on the hand and the eye is outside the hand, provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments described below or the various technical features can be arbitrarily combined to form new embodiments. The implementation procedures of the present application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation procedures, and the various details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for the purpose of illustrating the present application, and are not intended to limit the scope of protection of the present application.

[0029] The following briefly describes the technical field and related terms of the embodiments of the present application to facilitate understanding by those skilled in the art.

[0030] An object grasping system is an automated technology device that combines multiple technologies, including robotic arms, sensors, and intelligent algorithms, to automatically identify, locate, and grasp target objects. It uses 3D sensors to acquire point cloud data of objects, analyzes and processes the point cloud to determine the object's position and posture.

[0031] The more common technical solutions are grasping systems based on 2D vision and grasping systems based on traditional 3D point clouds. The existing 2D grasping method is suitable for grasping planar objects. Due to the lack of depth information, the posture in the RX and RY directions cannot be obtained. The obtained object coordinates are used for grasping, which has low efficiency and accuracy. Compared with 2D vision technology, the 3D point cloud method in the related technology can obtain the posture in the RX and RY directions through depth information, but it relies more on the integrity of the target object point cloud. When changes in ambient lighting, etc. will cause imaging loss, the obtained object coordinates are used for grasping, and the error is large.

[0032] To address the aforementioned technical issues, this paper proposes an object grasping method based on point cloud deep learning. This method uses a 3D sensor to acquire scene data and performs point cloud matching based on deep learning methods. The acquired object coordinates can be used to achieve high-precision and stable grasping. The following describes the method first, followed by a description of the system.

[0033] Method embodiment.

[0034] See also Figure 1 , Figure 1 This is a flow chart of an object grasping method based on point cloud deep learning provided in an embodiment of the present application.

[0035] The embodiment of the present application provides an object grasping method based on point cloud deep learning, the method comprising S100 to S300: S100, obtaining a target point cloud of the target object, performing point cloud matching on the target point cloud using a point cloud registration model based on a deep learning network to obtain an initial matching position, and then optimizing the matching position using a nearest iteration algorithm to obtain an accurate transformation matrix of the target object relative to the template in the camera coordinate system; In some embodiments, a method for acquiring the target point cloud of the target object includes: The original point cloud data is obtained by using a sensor. When the eye is on the hand, the sensor is mounted on the end of the robotic arm. When the eye is outside the hand, the sensor is mounted at a fixed position outside the robotic arm. The target object is extracted from the original point cloud data to obtain a target object point cloud.

[0036] Depending on the application requirements and the robotic arm structure, the sensor (such as a 3D camera) can be mounted at the end of the robotic arm (eyes on the hand) or in a fixed position outside the robotic arm (eyes off the hand). When using the "eyes on the hand" method, the sensor moves with the robotic arm end, capable of acquiring point cloud data of objects near the robotic arm end in real time. This is suitable for scenarios that require the robotic arm to flexibly grasp objects in different positions. In the "eyes off the hand" method, the sensor is fixed, providing a stable field of view, capable of collecting point cloud data for the entire work area, and is suitable for tasks involving grasping objects within a fixed work area.

[0037] See also Figure 4 , Figure 4 This is a flow chart of obtaining a point cloud of a target object provided in an embodiment of the present application.

[0038] In some embodiments, extracting the target object from the original point cloud data to obtain the target object point cloud includes: S110, filtering the point cloud data according to the preset XYZ direction maximum value, filtering out redundant points in the scene, and retaining the point cloud data within the area of ​​interest; S120, obtaining a normal vector estimate value of each retained point in its neighborhood, and filtering the point cloud using the normal vector information; S130 , for the filtered point cloud data, dividing the points within a preset distance range into multiple connected clusters; screening and extracting point clouds that meet the characteristics of the target object from the multiple connected clusters obtained by clustering and segmentation as the target object point cloud.

[0039] Coordinate filtering (S110) refers to filtering point cloud data according to the set XYZ direction maximum value, filtering out redundant points in the scene, and retaining the point cloud within the area of ​​interest. In the normal filtering process (S120), the normal vector is understood as a vector perpendicular to the point cloud surface, which can represent the local characteristics of the point cloud surface; estimating the normal vector of each point in its neighborhood and using the normal vector information to filter the current point cloud can solve the point cloud adhesion problem caused by the shooting angle. In the clustering segmentation process (S130), points within a certain distance range are divided into multiple connected clusters. The specific process is as follows: select a point as the starting point, mark the starting point as visited, and add it to a cluster; with this point as the center, select points within a certain range as candidate points, filter out unvisited points and add them to the cluster; repeat the above operation for each point in the cluster until all points have been visited; select unvisited points and repeat the above steps until all points are clustered.

[0040] The advantage of this embodiment is that through target object extraction steps such as coordinate filtering, normal filtering and cluster segmentation, the target object point cloud can be quickly and effectively screened out from a large amount of original point cloud data, reducing the data processing volume and calculation time, so that the robotic arm can continuously and stably complete the grasping task of multiple targets.

[0041] See also Figure 5 , Figure 5 This is a flow chart of obtaining a connected cluster provided in an embodiment of the present application.

[0042] In some embodiments, the step of dividing the points within a preset distance range from the filtered point cloud data into a plurality of connected clusters ( S130 ) includes: S131, select a point as the starting point; S132, mark the starting point as visited and add it to a separate target cluster; S133, taking the point as the center, selecting points within a preset range as candidate points, screening out unvisited points and adding them to the target cluster; S134, for each point in the target cluster, repeat S133 until all points have been visited; S135 , selecting a remaining unvisited point as a new starting point, and repeating S132 until all points are clustered to form multiple connected clusters.

[0043] Only when the distance between two points is less than or equal to the preset range are they considered to be in the neighborhood and classified into the same connected cluster. The preset range can be determined based on the size of the target object, the resolution of the point cloud data, and the object spacing factors in the actual application scenario to ensure that points belonging to the same object can be reasonably clustered together, while avoiding the mistaken merging of points from different objects into the same cluster.

[0044] The advantage of this embodiment is that by dividing the filtered point cloud data into multiple connected clusters, the specific location and shape of the target object in the point cloud can be accurately identified. Each connected cluster represents a possible object or a local area of ​​an object, allowing the robot arm to accurately locate the target object based on the geometric characteristics and position information of the connected clusters.

[0045] Since each connected cluster is formed based on the neighborhood relationship and distance threshold between points, background point clouds or noise point clouds that do not match the characteristics of the target object will be divided into different clusters or filtered out individually, ensuring that the robotic arm only performs grasping operations on connected clusters that meet the characteristics of the target object, avoiding misgrasping and empty grasping caused by incorrect identification of background or other objects, and improving grasping efficiency.

[0046] See also Figure 6 , Figure 6 This is a flow chart of obtaining an initial matching position provided in an embodiment of the present application.

[0047] In some embodiments, performing point cloud matching on the target object point cloud using a point cloud registration model based on a deep learning network to obtain an initial matching position includes: S140, building a deep learning network; S150, acquiring a data set, wherein the data set is obtained by collecting a preset number of sample target object point clouds through a sensor device, and then traversing the obtained target object point clouds and performing denoising and downsampling; S160: Using the data set as a training set, the deep learning network is trained to obtain a point cloud registration model; inputting the target object point cloud into the point cloud registration model to obtain an initial matching position.

[0048] The deep learning network can be a REGTR deep learning network, and the trained deep learning network can be used to match the actual point cloud with the template point cloud for registration. During the training process, sensor equipment collects a certain amount of initial target object point cloud data from a certain scene. The acquired initial target object point cloud data is traversed, denoised and downsampled to remove environmental noise, and preprocessed to form a dataset. The dataset is then trained in the REGTR deep learning network to obtain a trained model file.

[0049] Extract downsampled key points from the input point cloud and , and its corresponding features are , These key points and features are passed to the transformer cross-coding layer. The transformer layer contains multi-head self-attention and cross-attention to facilitate global information aggregation. The transformer encoder outputs features , Output the transformed position of the keypoint , The corresponding relationship can be obtained from and Obtained from , the same for the other direction. Output the overlap score at the same time , , represents the probability of each key point being located in the overlapping region. The required transformation matrix is ​​estimated directly from the correspondences within the overlapping region.

[0050] The template point cloud (i.e., template) refers to predefined and saved point cloud data that describes the shape, structure, and features of the target object, acting as a blueprint for the target object. When an object needs to be grasped, the actual point cloud data is compared with the template point cloud to identify the portion that best matches the template, thereby determining the object's position and posture in space. The template point cloud is created based on a precise scan of the target object, ensuring that it accurately reflects the object's geometric characteristics. In practical applications, by matching the template point cloud with the actual point cloud data, the target object can be quickly and accurately identified.

[0051] Therefore, the trained point cloud registration model is used to perform point cloud matching on the target object point cloud to obtain the initial matching position of the point cloud. The precise position can be calculated by matching using the nearest iteration (ICP) algorithm. By using the ICP algorithm, the precise transformation matrix can be quickly obtained while ensuring the matching accuracy, so that the grasping position coordinates can be solved in time.

[0052] S200, based on the pre-acquired hand-eye transformation matrix, calculates the transformation relationship of the target object's point cloud posture relative to the tool coordinate system through teaching; the hand-eye transformation matrix is ​​used to indicate the transformation relationship between the sensor coordinate system and the robot arm end coordinate system; the tool coordinate system is the coordinate system for Cam description and definition of the posture of the actuator at the end of the robot arm (such as the robot arm's clamp and other tools).

[0053] The method for obtaining the hand-eye transformation matrix includes: solving and obtaining the hand-eye transformation matrix according to the motion transformation of the end of the robotic arm and the motion transformation observed by the sensor.

[0054] See also Figure 2 , Figure 2 This is a flowchart of obtaining a hand-eye transformation matrix provided in an embodiment of the present application.

[0055] In some embodiments, the hand-eye transformation matrix is ​​obtained by: S210, performing camera intrinsic calibration, extracting corner points and refining them to sub-pixel accuracy, and using Zhang's calibration algorithm to obtain camera intrinsic parameters and distortion coefficients as camera calibration results; The camera is calibrated using the Zhang calibration algorithm. The calibration plate image is then captured, corner points are extracted, and the image is refined to sub-pixel accuracy to obtain the camera's internal parameters (such as focal length) and distortion coefficients.

[0056] S220, performing image distortion correction, dedistorting the image using the camera calibration result, and re-extracting sub-pixel precision corner point coordinates; The captured image is dedistorted using the camera intrinsic parameters and distortion coefficients obtained through calibration, and the sub-pixel precision corner coordinates of the rectified image can be re-extracted.

[0057] S230, defining a world coordinate system on the calibration plate and obtaining 3D coordinates of corner points; Define the world coordinate system on the calibration plate to obtain the precise 3D coordinates of the corner points. This step establishes the physical coordinate system through the calibration plate.

[0058] S240, using the PnP algorithm to establish a perspective projection model, solving the rotation and translation matrices from the world coordinate system to the camera coordinate system and using them as the camera pose; Specifically, a perspective projection model can be constructed using the PnP algorithm based on the acquired 2D image coordinates of the corner points and their corresponding 3D world coordinates. The perspective projection model is used to solve the rotation and translation matrices from the world coordinate system to the camera coordinate system, thereby obtaining the camera pose. The camera pose reflects the camera's position and orientation in space and is used for subsequent hand-eye calibration.

[0059] S250 obtains the hand-eye transformation matrix by solving the equation based on the robot arm pose and the camera pose.

[0060] Obtaining motion transformation data for the end-of-arm at different poses, along with the corresponding camera pose data, can be considered to encompass the relative motion relationship between the end-of-arm and camera in space. Substituting the end-of-arm and camera pose data into the hand-eye calibration equation (e.g., AX = XB), mathematical solutions are used to obtain the hand-eye transformation matrix. This matrix describes the geometric transformation between the sensor coordinate system and the end-of-arm coordinate system, enabling the integration of visual information with end-of-arm motion control.

[0061] The advantage of this embodiment is that the accuracy and reliability of visual information are ensured through camera intrinsic calibration and distortion correction, as well as PnP algorithm calculation of camera pose.

[0062] See also Figure 3 , Figure 3 This is a flow chart of obtaining a hand-eye transformation matrix provided in an embodiment of the present application.

[0063] In some embodiments, obtaining the hand-eye transformation matrix by solving equations based on the robot arm posture and the camera posture (S250) includes: S251, based on the robot arm posture and the camera posture, obtain the hand-eye calibration matrix through the conversion relationship; S252, evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error, and using the hand-eye calibration matrix as the hand-eye transformation matrix when the accuracy meets the preset accuracy requirement.

[0064] This can be understood as constructing a hand-eye calibration equation (e.g., AX = XB) based on the motion transformation of the robotic arm endpiece (A) and the motion transformation observed by the sensor (B). Here, X is the hand-eye transformation matrix to be solved. Substituting the robotic arm pose data (describing the coordinate information of the robotic arm endpiece at different positions and postures) and the camera pose data (the rotation and translation matrices from the world coordinate system to the camera coordinate system, obtained by the PnP algorithm) into the above equation, the hand-eye calibration matrix is ​​obtained through mathematical solutions.

[0065] In some embodiments, evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error includes: Using two sets of pose data, one set is used as the true value to estimate the other set of camera poses. The poses and 3D point sets are reprojected to obtain the coordinates of the 2D point set on the image plane. The mean square error (MSE error value) between the projected point coordinates and the corner point coordinates is calculated and used as the accuracy value.

[0066] Two sets of pose data are selected, one as the ground truth reference and the other for estimating the camera pose. Both sets of data contain coordinate information for the robotic arm and camera at different positions and postures. Using the hand-eye calibration matrix, the pose data and 3D point set are converted from one coordinate system to another and reprojected onto the image plane to obtain 2D point set coordinates, completing the reprojection. The difference between the reprojected 2D point set coordinates and the actual corner point coordinates is compared, and the mean square error (MSE) method is used to calculate the error between the projected point coordinates and the corner point coordinates, i.e., the reprojection error. This error reflects the accuracy of the hand-eye calibration matrix; the smaller the error, the more accurately the matrix describes the geometric relationship between the sensor and the end of the robotic arm.

[0067] The benefit of this embodiment is that by constructing equations and combining the pose data of the robotic arm and camera, a hand-eye calibration matrix can be solved relatively accurately. The reprojection error evaluation and optimization steps further ensure the high precision of the matrix, enabling the robotic arm to precisely adjust its posture based on visual information, achieving high-precision grasping of objects and effectively reducing grasping deviations caused by hand-eye coordinate system conversion errors. Furthermore, using reprojection error as an accuracy evaluation metric directly reflects the accuracy of the hand-eye transformation matrix in predicting object position and posture in practical applications.

[0068] In some embodiments, when the sensor is fixed to the end of the robotic arm, the hand-eye transformation matrix is ​​obtained by eye-on-hand calibration, and the conversion relationship is as follows: ; When the sensor holder is fixed outside the robotic arm, the hand-eye transformation matrix is ​​obtained by calibrating the eye outside the hand. The conversion relationship is as follows: ; in, is the transformation matrix from the target coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the robot end coordinate system, is the transformation matrix from the manipulator end coordinate system to the manipulator base coordinate system, is the transformation matrix from the sensor coordinate system to the robot base coordinate system, is the transformation matrix from the robot base coordinate system to the robot end coordinate system. T 0 and T1 represents the transformation matrix of (the end of the manipulator or the target) in two different states (position and posture), is the transformation matrix from the target coordinate system to the sensor coordinate system in the first state; is the inverse transformation matrix from the target coordinate system to the sensor coordinate system in the second state; is the inverse transformation matrix from the manipulator end coordinate system to the manipulator base coordinate system in the first state; is the transformation matrix from the robot end coordinate system to the robot base coordinate system in the second state; is the inverse transformation matrix from the robot base coordinate system to the robot end coordinate system in the first state; is the transformation matrix from the robot base coordinate system to the robot end coordinate system in the second state.

[0069] The hand-eye calibration problem can be expressed as the equation AX = XB, where A is the kinematic transformation of the end-arm, B is the kinematic transformation observed by the sensor, and X is the hand-eye transformation matrix to be solved. The transformation relationship differs for the "eye-in-hand" and "eye-out-of-hand" approaches and needs to be addressed separately.

[0070] For the case where the eyes are on the hands, the coordinate system is distributed as follows Figure 9a As shown, the coordinate system here is only a schematic diagram and does not represent the actual coordinate system direction. The sensor is fixed to the end of the robotic arm and moves with the robotic arm. The calibration plate is fixed to the object plane, and the calibration image is obtained by moving the robotic arm. The goal is to solve the relationship between the sensor coordinate system (cam) and the end coordinate system (end) The conversion relationship is .

[0071] is the transformation matrix from the target coordinate system (tar) to the robot base coordinate system (base); is the transformation matrix from the target coordinate system to the sensor coordinate system; is the transformation matrix from the sensor coordinate system to the robot end coordinate system; is the transformation matrix from the robot end coordinate system to the robot base coordinate system.

[0072] The relative position of the calibration object (tar, i.e. the target object) to the base of the robot arm remains unchanged, so is fixed, so it can be eliminated by two sets of posture relations, namely , sorted ,Right now ,in is the conversion from the target coordinate system to the camera coordinate system under different poses, It is the transformation from the end coordinate system of the robot arm to the base coordinate system of the robot arm under different postures.

[0073] For the case where the eye is outside the hand, the coordinate system is distributed as follows Figure 9b As shown, the coordinate system here is only a schematic diagram and does not represent the actual coordinate system direction. The sensor is fixed at a fixed position not at the end of the manipulator and does not move with the manipulator. The calibration plate is fixed to the end of the manipulator, and the calibration image is obtained by moving the manipulator. The goal is to solve the transformation matrix from the manipulator base coordinate system to the camera The conversion relationship is .

[0074] is the transformation matrix from the target coordinate system to the robot end coordinate system; is the transformation matrix from the target coordinate system to the sensor coordinate system; is the transformation matrix from the sensor coordinate system to the robot base coordinate system; is the transformation matrix from the robot base coordinate system to the robot end coordinate system.

[0075] The relative position of the calibration object to the end of the robotic arm remains unchanged, so is fixed, so it can be eliminated by two sets of posture relations, namely . Arranged ,Right now ,in is the conversion from the target coordinate system to the camera coordinate system under different poses, It is the transformation from the end coordinate system of the robot arm to the base coordinate system of the robot arm under different postures.

[0076] Therefore, whether the eye is outside the hand or inside the hand, the hand-eye calibration problem is transformed into ,right (Hand-eye transformation matrix) solution.

[0077] In some embodiments, the method of calculating the transformation relationship between the point cloud pose of the target object and the tool coordinate system through teaching based on the pre-acquired hand-eye transformation matrix includes: When the eye is on the hand, the transformation relationship between the target object's point cloud pose and the tool coordinate system is calculated using the following formula: ; When the eye is outside the hand, the following formula is used to teach and calculate the transformation relationship between the target object's point cloud posture and the tool coordinate system: .

[0078] in, is the transformation matrix from the target point cloud coordinate system to the tool coordinate system, is the transformation matrix from the target point cloud coordinate system to the robot arm base coordinate system, is the transformation matrix from the robot base coordinate system to the tool coordinate system, is the transformation matrix from the target point cloud coordinate system to the sensor coordinate system, It is the transformation matrix from the end coordinate system to the robot base coordinate system when taking pictures. S300, calculating the grasping position coordinates of the target object in the robot arm base coordinate system according to the conversion relationship and the precise transformation matrix; the grasping position coordinates are used to guide the robot arm to move to the grasping position to complete the grasping.

[0079] In some embodiments, the step of calculating the grasping position coordinates of the target object in the robot arm base coordinate system according to the conversion relationship and the precise transformation matrix (S300) includes: When the eye is on the hand, the grasping position is calculated using the following formula:

[0080] When the eye is outside the hand, the grasping position is calculated using the following formula: ; In the above formula, is the transformation matrix from the base coordinate system of the robot arm to the end coordinate system when taking pictures, is the transformation matrix from the robot end coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the target point cloud coordinate system, is the transformation matrix from the robot base coordinate system to the sensor coordinate system.

[0081] As an example, taking a workpiece grasping as an example, the above object grasping method is used to grasp the workpiece, including: Determine the hand-eye calibration results. Take the eye on the hand as an example to illustrate. 21 sets of calibration data are collected based on the calibration object to collect the calibration plate image and collection pose. Figure 8 , Figure 8 This is a schematic diagram of the calibration plate images and acquisition postures of the 21 sets of calibration data provided in the embodiment of the present application.

[0082] Calibration results for:

[0083] Teach gripping matrix for:

[0084] Teaching photo matrix for:

[0085] Calculated teaching result matrix for:

[0086] Get the target object matching transformation matrix :

[0087] Get the final crawl position: pass Calculate and get the final grasping position as

[0088] In specific applications, the above technical solution can be applied to industrial production lines to identify and grasp parts of different shapes and sizes to meet the needs of different industrial automation production lines.

[0089] The technical solution provided by this method embodiment first obtains the 3D data (point cloud) of the target object and the coordinate transformation relationship (hand-eye transformation matrix) between the sensor and the end of the robotic arm. Next, a deep learning network is used to match the point cloud of the target object to obtain an initial matching position. This position is optimized using a recent iteration algorithm to obtain the precise transformation matrix of the target object relative to the template in the camera coordinate system. Based on the hand-eye transformation matrix, the transformation relationship of the target object's point cloud posture in the tool coordinate system is calculated through teaching. Finally, based on the transformation relationship and the precise transformation matrix, the specific grasping position coordinates of the target object in the robotic arm's base coordinate system are calculated, which are used to guide the robotic arm to move to that position to complete the grasp.

[0090] The beneficial effects of the embodiment of this method are: on the one hand, the introduction of the hand-eye transformation matrix solves the coordinate conversion problem between visual information and robotic arm motion control, and at the same time, through the deep learning network for point cloud registration, it is possible to learn the complex features of the target object, and the position of the target object can be determined more accurately than the traditional method. The subsequent optimization of the latest iterative algorithm further improves the matching accuracy, thereby obtaining more accurate target position and posture information, effectively reducing the grasping deviation caused by inaccurate matching, and improving the success rate and accuracy of grasping. On the other hand, the posture of the target in the camera coordinate system is converted to the posture in the robotic arm base coordinate system, so that the robotic arm can understand and use this information to plan the grasping action. The entire process from data acquisition, point cloud registration, position solution to subsequent robotic arm grasping guidance realizes an automated grasping process, which not only reduces manual intervention and complex parameter settings, improves operating efficiency and reliability, but also enables the method to adapt to grasping tasks of different types of objects and scenes.

[0091] System embodiment.

[0092] This embodiment provides an object grasping system based on point cloud deep learning. Its specific embodiments are consistent with the embodiments described in the above method embodiments and the technical effects achieved, and some contents will not be repeated here.

[0093] The object grasping system includes an electronic device and a point cloud acquisition device. The electronic device includes a memory and at least one processor. The memory stores a computer program. The at least one processor is configured to implement any of the method embodiments when executing the computer program. The point cloud acquisition device is used to acquire a point cloud of a target object and transmit it to the electronic device. The point cloud acquisition device, for example, includes a 3D camera and a bracket for fixing the 3D camera.

[0094] See also Figure 7 , Figure 7 The electronic device 10 may include at least one memory 11, at least one processor 12, and a bus 13 connecting different platform systems.

[0095] The memory 11 may include (computer) readable media in the form of volatile memory, such as a random access memory (RAM) 111 and / or a cache memory 112 , and may further include a read-only memory (ROM) 113 .

[0096] The memory 11 also stores a computer program, which can be executed by the processor 12 so that the processor 12 implements the steps of any of the above methods.

[0097] The memory 11 may also include a utility 114 having at least one program module 115, such program module 115 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0098] Accordingly, the processor 12 may execute the aforementioned computer program and may execute the utility 114 .

[0099] The processor 12 may be implemented as one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0100] Bus 13 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures.

[0101] The electronic device 10 may also communicate with one or more external devices, such as a keyboard, a pointing device, a Bluetooth device, or the like, as well as one or more devices capable of interacting with the electronic device 10, and / or any device that enables the electronic device 10 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output interface 14. Furthermore, the electronic device 10 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 15. The network adapter 15 may communicate with other modules of the electronic device 10 via the bus 13. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 10 in actual applications, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0102] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more items".

[0103] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are configured to distinguish similar objects and are not necessarily configured to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0104] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has complied with the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A method for grasping objects based on point cloud deep learning, characterized in that: include: Obtaining a target point cloud of the target object, performing point cloud matching on the target point cloud using a point cloud registration model based on a deep learning network to obtain an initial matching position, and then optimizing the matching position using a nearest-iterative algorithm to obtain an accurate transformation matrix of the target object relative to the template in the camera coordinate system; Based on the pre-acquired hand-eye transformation matrix, the transformation relationship between the target object's point cloud posture and the tool coordinate system is calculated through teaching; the hand-eye transformation matrix is ​​used to indicate the transformation relationship between the sensor coordinate system and the robot arm end coordinate system; The grasping position coordinates of the target object in the robot arm base coordinate system are calculated based on the conversion relationship and the precise transformation matrix.

2. The object grasping method according to claim 1, characterized in that: The method of obtaining the hand-eye transformation matrix includes: According to the motion transformation of the end of the robotic arm and the motion transformation observed by the sensor, the hand-eye transformation matrix is ​​obtained.

3. The object grasping method according to claim 2, characterized in that: The method of obtaining the hand-eye transformation matrix based on the motion transformation of the end of the robotic arm and the motion transformation observed by the sensor includes: Perform camera intrinsic calibration, extract corner points and refine them to sub-pixel accuracy, and use Zhang's calibration algorithm to obtain camera internal parameters and distortion coefficients as the camera calibration results; Perform image distortion correction, use the camera calibration result to dedistort the image, and re-extract sub-pixel precision corner point coordinates; Define the world coordinate system on the calibration plate and obtain the 3D coordinates of the corner points; Use the PnP algorithm to build a perspective projection model, solve the rotation and translation matrices from the world coordinate system to the camera coordinate system and use them as the camera pose; Based on the robot arm pose and camera pose, the hand-eye transformation matrix is ​​obtained by solving the equation.

4. The object grasping method according to claim 3, characterized in that: The hand-eye transformation matrix is ​​obtained by solving equations based on the robot arm posture and the camera posture, including: Based on the robot arm pose and camera pose, the hand-eye calibration matrix is ​​obtained through the conversion relationship; Evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error, and using the hand-eye calibration matrix as the hand-eye transformation matrix when the accuracy meets the preset accuracy requirement; The step of evaluating the accuracy of the hand-eye calibration matrix by calculating the reprojection error includes: Using two sets of pose data, one set is used as the true value to estimate the other set of camera poses. The poses and 3D point sets are reprojected to obtain the coordinates of the 2D point set on the image plane. The mean square error between the projected point coordinates and the corner point coordinates is calculated and used as the accuracy value.

5. The object grasping method according to claim 3, characterized in that: When the sensor is fixed at the end of the robotic arm, the hand-eye transformation matrix is ​​obtained by eye-on-hand calibration. The conversion relationship is as follows: When the sensor holder is fixed outside the robotic arm, the hand-eye transformation matrix is ​​obtained by calibrating the eye outside the hand. The conversion relationship is as follows: in, is the transformation matrix from the target coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the robot end coordinate system, is the transformation matrix from the manipulator end coordinate system to the manipulator base coordinate system, is the transformation matrix from the sensor coordinate system to the robot base coordinate system, is the transformation matrix from the robot base coordinate system to the robot end coordinate system; T 0 and T 1 represents the transformation matrix in two different states.

6. The object grasping method according to claim 5, characterized in that: The method of calculating the transformation relationship between the point cloud posture of the target object and the tool coordinate system through teaching based on the pre-acquired hand-eye transformation matrix includes: When the eyes are on the hand, the transformation relationship between the point cloud pose of the target object and the tool coordinate system is calculated by teaching the following formula: ； When the eye is outside the hand, the following formula is used to teach and calculate the transformation relationship between the target object's point cloud posture and the tool coordinate system: ; in, is the transformation matrix from the target point cloud coordinate system to the tool coordinate system, is the transformation matrix from the target point cloud coordinate system to the robot arm base coordinate system, is the transformation matrix from the robot base coordinate system to the tool coordinate system, is the transformation matrix from the target point cloud coordinate system to the sensor coordinate system, The transformation matrix from the end coordinate system to the robot base coordinate system when taking pictures; The method of calculating the grasping position coordinates of the target object in the robot arm base coordinate system according to the conversion relationship and the precise transformation matrix includes: When the eye is on the hand, the grasping position is calculated using the following formula: ； When the eye is outside the hand, the grasping position is calculated using the following formula: ； in, is the transformation matrix from the base coordinate system of the robot arm to the end coordinate system when taking pictures, is the transformation matrix from the robot end coordinate system to the sensor coordinate system, is the transformation matrix from the sensor coordinate system to the target point cloud coordinate system, is the transformation matrix from the robot base coordinate system to the sensor coordinate system.

7. The object grasping method according to claim 1, characterized in that: The method of obtaining the target point cloud of the target object includes: The original point cloud data is obtained by using a sensor. When the eye is on the hand, the sensor is set at the end of the robotic arm. When the eye is outside the hand, the sensor is set at a fixed position outside the robotic arm. The target object is extracted from the original point cloud data to obtain a target object point cloud.

8. The object grasping method according to claim 7, characterized in that: The extracting the target object from the original point cloud data to obtain the target object point cloud includes: Filter the point cloud data according to the preset XYZ direction maximum value, filter out the redundant points in the scene, and retain the point cloud data in the area of ​​interest; Get the estimated normal vector value of each retained point in its neighborhood, and use the normal vector information to filter the point cloud; For the filtered point cloud data, points within a preset distance range are divided into multiple connected clusters; from the multiple connected clusters obtained by cluster segmentation, point clouds that meet the characteristics of the target object are screened and extracted as the target object point cloud; The method of dividing the points within a preset distance range into multiple connected clusters for the filtered point cloud data includes: S131, select a point as the starting point; S132, mark the starting point as visited and add it to a separate target cluster; S133, taking the point as the center, selecting points within a preset range as candidate points, screening out unvisited points and adding them to the target cluster; S134, for each point in the target cluster, repeat S133 until all points have been visited; S135 , selecting a remaining unvisited point as a new starting point, and repeating S132 until all points are clustered to form multiple connected clusters.

9. The object grasping method according to claim 8, characterized in that: The point cloud registration model based on the deep learning network is used to perform point cloud matching on the target object point cloud to obtain an initial matching position, including: Build deep learning networks; Acquire a data set, wherein the data set is obtained by collecting a preset number of sample target object point clouds through a sensor device, and then traverse the obtained target object point clouds and perform denoising and downsampling; The data set is used as a training set to train the deep learning network to obtain a point cloud registration model; the target object point cloud is input into the point cloud registration model to obtain an initial matching position.

10. An object grasping system based on point cloud deep learning, characterized in that: The invention comprises an electronic device and a point cloud acquisition device, wherein the electronic device comprises a memory and at least one processor, the memory stores a computer program, and the at least one processor is configured to implement the method according to any one of claims 1 to 9 when executing the computer program; the point cloud acquisition device is used to obtain a target point cloud of a target object and transmit it to the electronic device.

Images