Mechanical arm grabbing method, system and equipment under disordered screening scene and storage medium

Abstract

The invention discloses a mechanical arm grabbing method in a disordered screening scene, which comprises the following steps: acquiring point pair feature information of a workpiece model based on a color image and a point cloud picture, and storing the point pair feature information in a hash table; acquiring a color image and a point cloud picture of the scene instance, and realizing point cloud instance segmentation through a target recognition model algorithm; appearance edge detection is carried out on the scene instance point cloud, point pair features are established, and point pair feature descriptors are generated; performing point cloud registration on the point pair feature descriptor and point pair feature information of a workpiece model to obtain 6D pose information of a plurality of instances of the target workpiece; and predefining the reference direction of the target workpiece in the coordinate system and setting the pose specified axis as a Z axis, and determining the effective pose in the 6D pose information of the multiple instances of the target workpiece. Correspondingly, the invention further provides a mechanical arm grabbing system, electronic equipment and a storage medium. According to the invention, the accuracy and efficiency of picking up the target workpiece from the scattered and shielded industrial parts in a disordered screening scene can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a robot arm grasping method, system, electronic device and storage medium in an unordered screening scenario. Background Art

[0002] Robotic grasping is the most basic and difficult task in industrial automated manufacturing. Its purpose is to replace humans in repeatedly completing tedious tasks such as adding, assembling and sorting industrial parts. The picking task refers to grabbing a single part from a pile of messy parts in a transport or box. To grab a part, its position and orientation must be determined first. At present, most methods focus on studying the pose estimation problem in disordered and occluded scattered scenes, and few studies have clearly pointed out the identification of key objects for work needs. In a typical robotic grasping task, the robot needs to identify the required object in a messy scene containing multiple objects, pick it up and place it in a specific position. Accurate pose estimation is considered to be one of the key technologies to solve this problem. In this complex scene, the robot needs to accurately understand the position, orientation and posture of industrial parts in order to perform efficient and reliable picking operations. The purpose of pose estimation is to estimate the six degrees of freedom of the rigid transformation (i.e., translation and rotation) from the object coordinate system to the camera reference coordinate system.

[0003] In recent years, convolutional neural networks (CNNs) have demonstrated excellent performance in target detection. Most research on target recognition focuses on deep learning techniques, using learning-based methods to extract features for matching, and using convolution weights instead of feature extraction to improve the efficiency and performance of the algorithm. Traditional pose estimation relies on feature matching between an image and its corresponding model. Usually, hand-coded templates or features optimized by proxy targets are used for corresponding grouping and hypothesis verification. Recent data-driven methods attempt to extract initial poses from color images and improve them using ICP or MCN algorithms. However, these methods are computationally expensive and cannot achieve end-to-end optimization.

[0004] The latest methods adopt a different strategy. They introduce visible appearance information on color images as auxiliary information for geometric reasoning in point cloud image processing. Compared with color images, point clouds are an intuitive scene representation and are almost unaffected by lighting conditions, which helps to identify textureless objects. For industrial objects, the surface of the object does not contain rich texture information. Therefore, point clouds are an efficient and convenient format to represent the surface of industrial objects without texture. Point clouds are more robust to changing lighting conditions or scene contexts. Because of their permutation invariance, they need to be converted into other formats through planar projection or voxel grids and then directly processed by CNN. In some grasping scenarios, there are occlusions between objects, which will reduce the performance of image-based pose estimation methods. These new methods pay more attention to the auxiliary role of geometric information in image representation learning. However, they ignore the perception of pitch and roll of object poses by point cloud images, while traditional 2D CNNs perform poorly in continuous geometric reasoning. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a robot arm grasping method in a disordered screening scenario to improve the accuracy and efficiency of picking up target workpieces from scattered and obscured industrial parts in a disordered screening scenario. Correspondingly, the present invention also provides a robot arm grasping system, an electronic device and a storage medium.

[0006] According to one aspect of the present invention, a robot arm grasping method in an unordered screening scenario is provided, comprising the following steps:

[0007] Obtaining point pair feature information of the workpiece model based on the color image and the point cloud image, and storing it in a hash table;

[0008] Obtain color images and point cloud images of scene instances, and implement point cloud instance segmentation through target recognition model algorithms;

[0009] Perform appearance edge detection on the scene instance point cloud, establish point pair features and generate point pair feature descriptors;

[0010] Performing point cloud registration on the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece;

[0011] Predefine a reference direction of the target workpiece in the coordinate system and set the pose specifying axis as the Z axis, and determine the valid poses in the 6D pose information of the multiple instances of the target workpiece according to the angle between the pose specifying axis and the reference direction;

[0012] Sorting the valid poses and selecting the first N as candidate poses;

[0013] The target workpiece is grasped based on the candidate pose.

[0014] In one embodiment of the present invention, the step of acquiring point pair feature information of a workpiece model based on a color image and a point cloud image and storing the information in a hash table includes:

[0015] The PPF-MEAM method in PCL is used to extract the edge appearance points of the workpiece model point cloud;

[0016] Identify boundary points by using the x, y, z gradients and normal directions of the points, and use the farthest point sampling algorithm to select the first N key points;

[0017] The features of all point pairs for each keypoint are calculated, descriptors are generated, and stored in a hash table.

[0018] In one embodiment of the present invention, the step of acquiring a color image and a point cloud image of a scene instance and implementing point cloud instance segmentation through a target recognition model algorithm includes:

[0019] Perform point filtering on color images and point cloud images to eliminate outliers and obtain processed point cloud information;

[0020] The processed point cloud information and color image are input into the model, and features are extracted through encoding and decoding technology to obtain semantic segmentation results;

[0021] Instance segmentation is performed based on semantic perception to obtain instance segmentation results of the target artifact.

[0022] Furthermore, the point clouds generated by the color image and the depth image are input into the ResNet34 and PointNet networks respectively, the convolutional neural network is used to extract the appearance features of the color image, and the PointNet network is used to extract the geometric features from the point cloud;

[0023] In the feature branches of the ResNet34 and PointNet networks, a fusion module for mutual mapping between 3D feature points and 2D feature points is added to each layer as a bidirectional communication support of the fusion network;

[0024] The fusion module introduces a fusion network composed of a multi-layer perceptron and maximum pooling in the encoding and decoding process. The fusion network uses the 3D point cloud as a link to connect pixel-by-pixel and point-by-point features.

[0025] Furthermore, according to the correspondence between the aligned color image and the depth image, paired dense appearance features and geometric features are obtained, which are spliced ​​through a multi-layer perceptron, and finally the embedded point-by-point depth features are input into the feature branches of the ResNet34 and PointNet networks, and then instance features and semantic features are output;

[0026] The instance features are cached through a fully connected layer and then concatenated with the semantic features. The fused semantic features are passed through a multi-layer perceptron to assign semantic labels to each point, generate the category of each point, and further fuse the semantic features into the instance features to generate the final instance embedding.

[0027] Furthermore, a K-nearest neighbor search is performed on each point in the generated instance embedding space, and outliers are filtered out;

[0028] Embed each point close to the mean embedding to supervise instance embedding learning;

[0029] According to the kNN search, the semantic features of each point in the instance embedding are grouped into a Shape feature tensor, is the index matrix output by the kNN search, and each group formed corresponds to the local area adjacent to the centroid of each instance in the instance embedding space;

[0030] The semantic features of each group are fused together through the channel-wise max aggregation operation as the semantic features of its centroid point, that is, for , the semantic features of instance fusion can be obtained as follows:

[0031]

[0032] In the formula, Represents the semantic features of K neighboring points centered on point i. After instance fusion, the output becomes Feature Matrix ,Finally, a semantic classifier is used to classify different objects in the same instance and complete the point cloud instance segmentation.

[0033] In one embodiment of the present invention, the performing appearance edge detection on the scene instance point cloud to establish a point pair feature descriptor includes:

[0034] Use the algorithm in PCL to perform appearance edge detection on the scene instance point cloud and generate B2B-DL point pair feature descriptors;

[0035] The expression of the B2B-DL point pair feature descriptor is as follows:

[0036]

[0037] In the formula, is the vector from the reference point to the referenced point, and Represents the angle between two point vectors. The first component , represents the Euclidean distance between two point clouds; the second and third components and They are vector With passing point and Point The acute angle between the tangents of is the acute angle between these two tangents; and and The distance and angle to sample as steps on the border.

[0038] According to another aspect of the present invention, there is provided a robotic arm grasping system, comprising:

[0039] An acquisition module is used to acquire point pair feature information of a workpiece model based on a color image and a point cloud image, and store the information in a hash table;

[0040] The instance segmentation module is used to obtain color images and point cloud images of scene instances, and implement point cloud instance segmentation through the target recognition model algorithm;

[0041] Point pair feature generation module, used to perform appearance edge detection on scene instance point cloud, establish point pair features and generate point pair feature descriptors;

[0042] A feature registration module is used to perform point cloud registration on the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece;

[0043] A posture screening module, used to predefine a reference direction of the target workpiece in the coordinate system and set the posture specified axis as the Z axis, and determine the valid postures in the 6D posture information of multiple instances of the target workpiece according to the angle between the posture specified axis and the reference direction;

[0044] A posture sorting module, used to sort the valid postures and select the first N as candidate postures;

[0045] The grasping module grasps the target workpiece based on the candidate pose.

[0046] According to another aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0047] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0048] processor;

[0049] A storage medium stores a computer program thereon, wherein the computer program executes the steps of the method described above when executed by the processor.

[0050] The present invention can improve the accuracy and efficiency of picking up target workpieces from scattered and obscured industrial parts in disordered screening scenarios. Specifically:

[0051] 1. The present invention can accurately estimate the 6D pose of the target workpiece through comprehensive processing and feature extraction of color images and point cloud images, thereby improving the accuracy of grasping and effectively avoiding the grasping failure phenomenon caused by inaccurate pose estimation in traditional methods.

[0052] 2. The present invention can process industrial parts of multiple types and instances, has strong adaptability, and can be applied to workpieces of different shapes and sizes, thereby improving the versatility and flexibility of the system and being suitable for application in a variety of industrial environments.

[0053] 3. The present invention can ensure the availability and accuracy of the retained posture in actual operation by screening effective postures, reduce errors and failure rates in the grasping process, and thus improve the overall performance of the system.

[0054] 4. The present invention improves the stability and adaptability of the system in complex environments by sorting the postures and selecting the top N candidate postures, and can cope with the uncertainties and environmental changes that may be encountered during the grasping process, thereby ensuring the smooth completion of the grasping task. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0056] Figure 1 It is a flow chart of a robotic arm grasping method in an unordered screening scenario in one embodiment of the present invention.

[0057] Figure 2 It is a module diagram of a robotic arm grasping system in one embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention.

[0059] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium in one embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the description of the present invention more detailed and complete, reference may be made to the accompanying drawings and various embodiments described below, in which the same numbers represent the same or similar components. On the other hand, well-known components and steps are not described in the embodiments to avoid unnecessary limitations on the present invention. In the embodiments and claims, the description of "connection" may generally refer to a component being indirectly coupled to another component through other components, or a component being directly connected to another component without passing through other components. In the embodiments and claims, the description of "online" may generally refer to a component being indirectly connected to another component through other components for wired and / or wireless communication, or a component being physically connected to another component without passing through other components. In the embodiments and claims, unless the article is specifically limited in the text, "a" and "the" may generally refer to a single or multiple. "About", "approximately" or "roughly" as used herein is used to modify any quantity that can be slightly changed, but such slight changes do not change its essence. In the embodiments, if there is no special explanation, the error range of the numerical value modified by "about", "approximately" or "substantially" is generally allowed within 20%, preferably within 10%, and more preferably within 5%.

[0061] According to one aspect of the present invention, a method for grasping by a robot arm in an unordered screening scenario is provided. Figure 1 FIG. 1 is a flow chart of a robot arm grasping method according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0062] S1, obtains the point pair feature information of the workpiece model based on the color image and the point cloud image, and stores it in the hash table.

[0063] Specifically, multiple parts of different types can be pushed onto the vibration plate through the push bin, and photographed by a 3D industrial camera fixed above the vibration plate to obtain scene point cloud and color image; then, the point cloud of each different part model is subjected to appearance edge detection by the algorithm in PCL to obtain appearance point pair information, which is then stored in a hash table. This step may specifically include:

[0064] The PPF-MEAM method in PCL is used to extract the edge appearance points of the workpiece model point cloud;

[0065] Identify boundary points by using the x, y, z gradients and normal directions of the points, and use the farthest point sampling algorithm to select the first N key points;

[0066] The features of all point pairs for each keypoint are calculated, descriptors are generated, and stored in a hash table.

[0067] S2, obtains the color image and point cloud map of the scene instance, and implements point cloud instance segmentation through the target recognition model algorithm.

[0068] Specifically, this step may include:

[0069] Perform point filtering on color images and point cloud images to eliminate outliers and obtain processed point cloud information;

[0070] The processed point cloud information and color image are input into the model, and features are extracted through encoding and decoding technology to obtain semantic segmentation results;

[0071] Instance segmentation is performed based on semantic perception to obtain instance segmentation results of the target artifact.

[0072] Furthermore, the point clouds generated by the color image and the depth image are input into the ResNet34 and PointNet networks respectively, the convolutional neural network is used to extract the appearance features of the color image, and the PointNet network is used to extract the geometric features from the point cloud;

[0073] In the feature branches of the ResNet34 and PointNet networks, a fusion module for mutual mapping between 3D feature points and 2D feature points is added to each layer as a bidirectional communication support for the fusion network; in this way, the two branches can use the local and full features from the other branch as auxiliary information to promote their own feature representation learning, thereby improving the point-by-point features of the image. The fusion module introduces a fusion network composed of a shared perceptron MLP and maximum pooling in the encoding and decoding process. The network uses 3D point clouds as a link to connect pixel-by-pixel and point-by-point features. Given the pixels of a single color image , its coordinates on the two-dimensional plane are ( ). The depth of each pixel i is mapped to the corresponding 3D point by using the camera intrinsic parameter matrix ( ), and then obtain the XYZ image aligned with the RGB image. The point features aligned with the 3D point coordinates , find its The nearest neighbor points are found from the appearance feature map, and the corresponding pixels and appearance features of these points are found. These appearance features are integrated using multi-layer MLP and maxpooling:

[0074]

[0075] In the formula, is the i-th nearest neighbor in RGB, is the feature of the integrated pixel after fusion. and Generate fused point features using shared MLP concatenation :

[0076]

[0077] Here, ⊕ is the concatenation operator.

[0078] Similarly, given a point j in the point cloud, we get the 3D coordinates of each pixel in XYZ according to the RGB image, and find the corresponding point in the point cloud. The nearest point is obtained and its geometric features are found, which are then input into a multi-layer MLP and maxpooling to be mapped into a feature map of the same channel size as the RGB feature map:

[0079]

[0080] In the formula, is the jth nearest neighbor point in the point cloud, is the feature of the integrated point after fusion. and Generate fused point features using shared MLP concatenation :

[0081]

[0082] Because at each layer of feature encoding and decoding, since the feature map is scanned by the convolution kernel, its height and width become smaller as the network goes deeper. In order to ensure that each feature pixel can be mapped to its corresponding 3D coordinate, the present invention uses a kernel of the same size to calculate the average value of the 3D coordinates, thereby generating a new 3D coordinate of the pixel point.

[0083] Furthermore, the instance segmentation network obtains paired dense appearance features and geometric features (ci, cj) according to the correspondence of the aligned RGBD images, concatenates them through the shared MLP, and finally embeds the point-by-point deep features (Each point has C-dimensional features) is input into two branches, and instance features and semantic features are output in the two branches through multiple multi-layer perceptrons (MLPs). The instance features are cached through a fully connected layer, and then concatenated with the semantic features. The fused semantic features are passed through multi-layer MLPs to assign semantic labels to each point and generate the category of each point. Inspired by similarity metric learning, points belonging to the same object instance should have very similar features. Therefore, for points of different categories, their distances in the instance feature space are different, so the semantic features are Further integration into instance features middle.

[0084]

[0085] In the formula, It is the influence of categories on instance features that points belonging to instances of different categories are further excluded, while instances of the same category are rarely affected. It is the instance feature after fusing the semantic features and generates the final instance embedding.

[0086] In order to realize that the instance features of different objects in the category are different, the present invention performs a K-nearest neighbor search on each point in the generated instance embedding space. In order to ensure that the K sample points belong to the same instance, outliers are filtered according to the margin used in the formula. Each point is embedded in a position close to the mean embedding to supervise the instance embedding learning. According to the kNN search, the semantic features of each point in the instance embedding are grouped into a Shape feature tensor, is the index matrix of the kNN search output. Each group formed corresponds to the local area adjacent to the centroid of each instance in the instance embedding space. The semantic features of each group are fused together through the channel-wise max aggregation operation as the semantic features of its centroid, that is, for , the semantic features of instance fusion can be obtained as follows:

[0087]

[0088] In the formula, Represents the semantic features of K neighboring points centered on point i. After instance fusion, the output becomes Feature Matrix Finally, a semantic classifier is used to classify different objects in the same instance, completing the instance segmentation of the point cloud. The instance segmentation network proposed by the present invention can include semantic perception of a large number of point clouds, and can improve the performance of multi-instance segmentation when the number of input point clouds is large.

[0089] Furthermore, in order to achieve accurate multi-instance segmentation effect, the loss function of the semantic branch in the present invention is to train the network by cross entropy loss:

[0090]

[0091] Where N is the total number of pixels in the scene image; is the pixel predicted by the model The probability of belonging to category M; 1 is the indicator function, when the pixel When the ground truth label is M, the value of this function is 1, otherwise it is 0. For the instance branch, because the strength embeddings of different semantic classes are learned separately, this means that semantic classification and instance segmentation are not learned in parallel. Incorrect semantic segmentation will lead to incorrect instance segmentation. Therefore, the variance term is used to pull the instance embedding toward its average embedding, while the distance term is used to separate objects in the same category. The discriminant loss function trains the network and obtains multiple instance labels in the same category. The expression of this loss function is:

[0092]

[0093] In the formula, is the average embedding that pulls the embedding towards the instance, while Keep different instances of the same class separate, is a regularization that ensures that the instance embedding value is bounded.

[0094]

[0095]

[0096]

[0097] In the formula, is the number of instances contained in the ground truth labels, is the number of points in each instance point cloud, is the instance embedding of a point, is the average embedding of each instance, and is the margin, and Indicates a hinge.

[0098] Furthermore, in order to achieve the effect of instance segmentation, the present invention uses mean shift clustering for each point to assign the pattern of semantic labels of points in the same instance to its final category.

[0099] S3, performs appearance edge detection on the scene instance point cloud, establishes point pair features and generates point pair feature descriptors.

[0100] Specifically, this step may include:

[0101] Use the algorithm in PCL to perform appearance edge detection on the scene instance point cloud and generate B2B-DL point pair feature descriptors;

[0102] Furthermore, based on the instance point cloud of a given scene, the pose of the instance is calculated online. The bidirectional mapping between the depth map and the point cloud can be used for edge detection, and then the curvature information of the point cloud is added to the feature description to calculate the B2B-DL descriptor of the edge.

[0103] The B2B-DL point pair feature descriptor refers to finding the corresponding point pairs between the scene and the model. Using these corresponding point pairs, the 6D posture can be calculated. For some industrial parts, the points along the boundary of the object have more essential information than the points on the surface. And the number of boundary points of an object is far less than the number of surface points. Therefore, the present invention selects the boundary points of the target to calculate the posture information in a shorter time. Because the normal direction of these points has countless corresponding points in the model and the scene, it is difficult to ensure that the same direction is selected, so the present invention uses the tangent direction passing through these points instead of the normal direction. The expression of the B2B-DL point pair feature descriptor is as follows:

[0104]

[0105] In the formula, is the vector from the reference point to the referenced point, and Represents the angle between two point vectors. The first component , represents the Euclidean distance between two point clouds. The second and third components and They are vector With passing point and Point The acute angle between the tangent lines of is the acute angle between the two tangents. and As the step sampling distance and angle on the boundary, the local coordinate system of the point is established in two-dimensional space to vote to obtain multiple rough pose candidate results.

[0106] Furthermore, in order to verify the effectiveness of the pose estimation network and obtain accurate translation and rotation parameters of pose estimation, the indicators ADD and ADD-S are used for evaluation during the training process. Average distance (ADD) method: Given the ground truth pose , predict pose ,ADD is composed of and The average distance between the transformed 3D model points is considered to be correct when it is less than 10% of the diameter of the 3D model.

[0107]

[0108] In the formula, It is a 3D model The vertices in the point cloud, is the number of points in the point cloud of the 3D model M, , is the model predicted pose, , is the ground truth pose.

[0109] The ADD-S metric represents the average closest point distance of symmetric objects, given the ground truth pose. , predict pose , then the average closest point distance is given by the predicted pose Each 3D point obtained by transformation and the ground truth pose The average distance between the closest points on the transformed 3D model. The present invention also provides the area under the ADD-S curve (AUC).

[0110]

[0111] In the formula, is the closest point between two poses.

[0112] S4, performing point cloud registration on the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece.

[0113] Specifically, the B2B-DL point pair feature descriptor can be aligned with the model point pair information stored offline in the hash table, and Hough voting can be performed to obtain the translation and rotation parameters of the final predicted pose. The optimal pose parameters can be obtained by optimizing the minimum distance between points in the model point cloud and the instance point cloud, thereby improving the optimal pose ranking of each instance of the target.

[0114] S5, predefine a reference direction of the target workpiece in the coordinate system and set the pose specifying axis as the Z axis, and determine the valid poses in the 6D pose information of the multiple instances of the target workpiece according to the angle between the pose specifying axis and the reference direction.

[0115] Specifically, an angle threshold can be set to retain valid poses by calculating the angle between the specified axis of the pose and the reference direction. The poses with an angle within the set maximum angle difference are defined as valid poses and retained; poses exceeding the maximum angle difference will be removed.

[0116] Further evaluation of the pose. For the retained pose, further evaluate its reachability and reliability. Reachability refers to whether the robot can use the pose for grasping in the actual scene; reliability refers to the grasping error, which is mainly affected by the pose estimation accuracy. The higher the estimation accuracy, the higher the reliability.

[0117] S6, sorting the valid postures and selecting the first N as candidate postures.

[0118] Specifically, to ensure the robustness of the system, the multiple retained postures can be sorted, a sorting rule can be set according to the application scenario, and the top N candidate postures can be selected to cope with the uncertainties and complex environmental changes that may be encountered during the grasping process.

[0119] S7: Grasp the target workpiece based on the candidate posture.

[0120] For a given set of candidate poses, the robot arm only needs to grab one instance and place it at the designated workstation.

[0121] The present invention can improve the accuracy and efficiency of picking up target workpieces from scattered and obscured industrial parts in disordered screening scenarios. Specifically:

[0122] 1. The present invention can accurately estimate the 6D pose of the target workpiece through comprehensive processing and feature extraction of color images and point cloud images, thereby improving the accuracy of grasping and effectively avoiding the grasping failure phenomenon caused by inaccurate pose estimation in traditional methods.

[0123] 2. The present invention can process industrial parts of multiple types and instances, has strong adaptability, and can be applied to workpieces of different shapes and sizes, thereby improving the versatility and flexibility of the system and being suitable for application in a variety of industrial environments.

[0124] 3. The present invention can ensure the availability and accuracy of the retained posture in actual operation by screening effective postures, reduce errors and failure rates in the grasping process, and thus improve the overall performance of the system.

[0125] 4. The present invention improves the stability and adaptability of the system in complex environments by sorting the postures and selecting the top N candidate postures, and can cope with the uncertainties and environmental changes that may be encountered during the grasping process, thereby ensuring the smooth completion of the grasping task.

[0126] According to another aspect of the present invention, a system robot arm grasping system is provided. Figure 2 is a block diagram of a robotic arm grasping system in one embodiment of the present invention. Figure 2The robot gripping system 300 in this embodiment includes: an acquisition module 310, an instance segmentation module 320, a point pair feature generation module 330, a feature registration module 340, a pose screening module 350, a pose sorting module 360 ​​and a gripping module 370. The acquisition module 310 is used to acquire the point pair feature information of the workpiece model based on the color image and the point cloud map, and store it in a hash table. The instance segmentation module 320 is used to acquire the color image and point cloud map of the scene instance, and implement point cloud instance segmentation through the target recognition model algorithm. The point pair feature generation module 330 is used to perform appearance edge detection on the scene instance point cloud, establish point pair features and generate point pair feature descriptors. The feature registration module 340 is used to perform point cloud registration between the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece. The pose screening module 350 is used to predefine the reference direction of the target workpiece in the coordinate system and set the pose designation axis to the Z axis, and determine the valid poses in the 6D pose information of multiple instances of the target workpiece according to the angle between the pose designation axis and the reference direction. The pose sorting module 360 ​​is used to sort the valid poses and select the first N as candidate poses. The grasping module 370 grasps the target workpiece based on the candidate poses.

[0127] The present invention can improve the accuracy and efficiency of picking up target workpieces from scattered and blocked industrial parts in a disordered screening scenario.

[0128] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0129] processor;

[0130] A storage medium stores a computer program thereon, wherein the computer program executes the steps of the method described above when executed by the processor.

[0131] Refer to the following Figure 3 To describe an electronic device 600 according to an embodiment of the present application. Figure 3 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0132] like Figure 3 As shown, the electronic device 600 is in the form of a general computing device. The components of the device 600 may include, but are not limited to: at least one processor 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processor 610), a display unit 640, etc.

[0133] The storage unit stores program codes, which can be executed by the processor 610, so that the processor 610 executes the steps of various exemplary embodiments of the present application described in the electronic prescription circulation processing method section of this specification. For example, the processor 610 can execute the steps in the above method.

[0134] The storage unit 620 may include a storage medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0135] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0136] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0137] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable tenants to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. In addition, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network 102 adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] The present invention can improve the accuracy and efficiency of picking up target workpieces from scattered and blocked industrial parts in a disordered screening scenario.

[0139] According to another aspect of the present invention, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0140] refer to Figure 4 In one embodiment, the program product 800 for implementing the above method may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a server. However, those skilled in the art should understand that the program product referred to in this application is not limited thereto, and the computer storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0141] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0142] In addition, a part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can be called or provided through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer storage media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer storage medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0143] Communication media include media by which communication signals containing, for example, computer readable instructions, data structures, program modules or other data are transmitted from one system to another. Communication media may include guided transmission media (such as cables and wires, for example, fiber optic, coaxial, etc.) and wireless (unguided transmission) media that can propagate energy waves, such as acoustic, electromagnetic, RF, microwave and infrared. Computer readable instructions, data structures, program modules or other data may be embodied as a modulated data signal in, for example, a wireless medium (such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in such a manner as to encode information in the signal. Modulation may be analog, digital, or a hybrid modulation technique.

[0144] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media include (but are not limited to) volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, magnetic tapes, CDs, DVDs); or other media now known or later developed that can store computer-readable information or data for use by a computer system.

[0145] In summary, the present invention can improve the accuracy and efficiency of picking up target workpieces from scattered and occluded industrial parts in disordered screening scenarios.

[0146] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A robotic arm grasping method in an unordered screening scenario, characterized in that: include: Obtaining point pair feature information of the workpiece model based on the color image and the point cloud image, and storing it in a hash table; Obtain color images and point cloud images of scene instances, and implement point cloud instance segmentation through target recognition model algorithms; Perform appearance edge detection on the scene instance point cloud, establish point pair features and generate point pair feature descriptors; Performing point cloud registration on the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece; Predefine a reference direction of the target workpiece in the coordinate system and set the pose specifying axis as the Z axis, and determine the valid poses in the 6D pose information of the multiple instances of the target workpiece according to the angle between the pose specifying axis and the reference direction; Sorting the valid poses and selecting the first N as candidate poses; The target workpiece is grasped based on the candidate pose.

2. The method according to claim 1, characterized in that: The step of obtaining point pair feature information of the workpiece model based on the color image and the point cloud image and storing the information in a hash table includes: The PPF-MEAM method in PCL is used to extract the edge appearance points of the workpiece model point cloud; Identify boundary points by using the x, y, z gradients and normal directions of the points, and use the farthest point sampling algorithm to select the first N key points; The features of all point pairs for each keypoint are calculated, descriptors are generated, and stored in a hash table.

3. The method according to claim 1, characterized in that: The method of obtaining a color image and a point cloud image of a scene instance and implementing point cloud instance segmentation through a target recognition model algorithm includes: Perform point filtering on color images and point cloud images to eliminate outliers and obtain processed point cloud information; The processed point cloud information and color image are input into the model, and features are extracted through encoding and decoding technology to obtain semantic segmentation results; Instance segmentation is performed based on semantic perception to obtain instance segmentation results of the target artifact.

4. The method according to claim 3, characterized in that: include: The point clouds generated by the color image and the depth image are input into the ResNet34 and PointNet networks respectively. The appearance features of the color image are extracted using the convolutional neural network, and the geometric features are extracted from the point cloud using the PointNet network. In the feature branches of the ResNet34 and PointNet networks, a fusion module for mutual mapping between 3D feature points and 2D feature points is added to each layer as a bidirectional communication support of the fusion network; The fusion module introduces a fusion network composed of a multi-layer perceptron and maximum pooling in the encoding and decoding process. The fusion network uses the 3D point cloud as a link to connect pixel-by-pixel and point-by-point features.

5. The method according to claim 4, characterized in that: include: According to the correspondence between the aligned color image and the depth image, paired dense appearance features and geometric features are obtained, which are spliced ​​through a multi-layer perceptron, and finally the embedded point-by-point depth features are input into the feature branches of the ResNet34 and PointNet networks, and then instance features and semantic features are output; The instance features are cached through a fully connected layer and then concatenated with the semantic features. The fused semantic features are passed through a multi-layer perceptron to assign semantic labels to each point, generate the category of each point, and further fuse the semantic features into the instance features to generate the final instance embedding.

6. The method according to claim 5, characterized in that: include: Perform a K-nearest neighbor search for each point in the generated instance embedding space and filter outliers; Embed each point close to the mean embedding to supervise instance embedding learning; According to the kNN search, the semantic features of each point in the instance embedding are grouped into a Shape feature tensor, is the index matrix output by the kNN search, where each group formed corresponds to the local area adjacent to the centroid of each instance in the instance embedding space; The semantic features of each group are fused together through the channel-wise max aggregation operation as the semantic features of its centroid point, that is, for , the semantic features of instance fusion can be obtained as follows: In the formula, Represents the semantic features of K neighboring points centered on point i. After instance fusion, the output becomes Feature Matrix ,Finally, a semantic classifier is used to obtain the classification of different objects in the same instance and complete the point cloud instance segmentation.

7. The method according to claim 1, characterized in that: The performing appearance edge detection on the scene instance point cloud to establish a point pair feature descriptor includes: Use the algorithm in PCL to perform appearance edge detection on the scene instance point cloud and generate B2B-DL point pair feature descriptors; The expression of the B2B-DL point pair feature descriptor is as follows: In the formula, is the vector from the reference point to the referenced point, and Represents the angle between two point vectors. The first component , represents the Euclidean distance between two point clouds; the second and third components and They are vector With passing point and Point The acute angle between the tangents of is the acute angle between these two tangents; and and The distance and angle to sample as steps on the border.

8. A robotic arm grasping system, characterized in that: include: An acquisition module is used to acquire point pair feature information of a workpiece model based on a color image and a point cloud image, and store the information in a hash table; The instance segmentation module is used to obtain color images and point cloud images of scene instances, and implement point cloud instance segmentation through the target recognition model algorithm; Point pair feature generation module, used to perform appearance edge detection on scene instance point cloud, establish point pair features and generate point pair feature descriptors; A feature registration module is used to perform point cloud registration on the point pair feature descriptor and the point pair feature information of the workpiece model to obtain 6D pose information of multiple instances of the target workpiece; A posture screening module, used to predefine a reference direction of the target workpiece in the coordinate system and set the posture specified axis as the Z axis, and determine the valid postures in the 6D posture information of multiple instances of the target workpiece according to the angle between the posture specified axis and the reference direction; A posture sorting module, used to sort the valid postures and select the first N as candidate postures; The grasping module grasps the target workpiece based on the candidate pose.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

10. An electronic device, characterized in that: The electronic device comprises: processor; A storage medium having a computer program stored thereon, wherein the computer program, when executed by the processor, performs the steps of the method according to any one of claims 1 to 7.

Images