Article grabbing method and device, equipment, storage medium and product

Abstract

The invention discloses an article grabbing method and device, equipment, a storage medium and a product, and relates to the technical field of computers, and the article grabbing method comprises the following steps: obtaining a 3D point cloud of a to-be-grabbed article; on the basis of the 3D point cloud, through a preset instance segmentation model, feature information extraction of a high-order Taylor function is carried out on the 3D point cloud, an article feature map is obtained, and the article feature map is a feature map with high-order feature information; based on the article feature map and the 3D point cloud, an instance segmentation mask is determined, and the exact boundary of the to-be-grabbed article is defined by the instance mask; and based on the instance segmentation mask, the to-be-grabbed article is grabbed. According to the object grabbing method and device, object grabbing in the disordered environment can be more accurately achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object grabbing method, device, equipment, storage medium and product. Background Art

[0002] With the popularization of intelligence, robots with object grasping capabilities are widely used in various complex scenarios, such as homes or factories.

[0003] At present, the existing object grasping method first uses the semantic segmentation network of low-level information to segment the object, and then uses the grasping generator to generate the grasping posture of the segmented object to achieve the grasping of the object. However, when the existing grasping generator with instance segmentation grasps in a cluttered environment, for stacked objects, the low-level information is not conducive to the object instance segmentation, which affects the object grasping results and reduces the grasping accuracy. Summary of the invention

[0004] The main purpose of this application is to provide an object grasping method, device, equipment, storage medium and product, aiming to solve the technical problem of how to improve the accuracy of object grasping in a cluttered environment.

[0005] To achieve the above purpose, the present application proposes a method for grabbing an object, wherein the object grabbing comprises:

[0006] Get the 3D point cloud of the object to be grasped;

[0007] Based on the 3D point cloud, extract feature information of the 3D point cloud using a high-order Taylor function through a preset instance segmentation model to obtain an object feature map, wherein the object feature map is a feature map with high-order feature information;

[0008] Determining an instance segmentation mask based on the object feature map and the 3D point cloud;

[0009] The object to be grasped is grasped based on the instance segmentation mask.

[0010] In one embodiment, the step of extracting feature information of a high-order Taylor function from the 3D point cloud through a preset instance segmentation model based on the 3D point cloud to obtain an object feature map includes:

[0011] Performing local feature extraction on the 3D point cloud by using a local Taylor neural network in a preset instance segmentation model to obtain a local feature map;

[0012] High-order Taylor feature information is extracted from the local feature map to obtain an object feature map.

[0013] In one embodiment, the step of determining an instance segmentation mask based on the object feature map and the 3D point cloud comprises:

[0014] Determine a semantic mask and a center offset of the object feature map, wherein the semantic mask is used to mark the object to be grasped, and the center offset is used to locate the center point of the object to be grasped;

[0015] Combining the 3D point cloud with the center offset to obtain a combined graph;

[0016] Based on the semantic mask, mean shift clustering calculation is performed on the combination graph to obtain an instance segmentation mask.

[0017] In one embodiment, the step of grasping the object to be grasped based on the instance segmentation mask includes:

[0018] Classifying the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects;

[0019] The unobstructed objects, the overlapping objects and the intersecting objects are grasped in sequence respectively.

[0020] In one embodiment, the step of classifying the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects, comprises:

[0021] Based on the instance segmentation mask, determining the peripheral contour information and area of ​​the object to be grasped;

[0022] Deleting the peripheral contour information of each of the objects to be grasped respectively to obtain remaining peripheral contour information;

[0023] Comparing the remaining outer contour information with the outer contour information, if the number of contours in the remaining outer contour information is less than the number of contours in the outer contour information, the object to be grasped is an unobstructed object; if the number of contours in the remaining outer contour information is equal to or greater than the number of contours in the outer contour information, the object to be grasped is an obstructed object;

[0024] Performing classification calculation based on the area of ​​the obstructing objects to determine overlapping objects and intersecting objects;

[0025] In one embodiment, the step of performing classification calculation based on the area of ​​the obstructing objects to determine the overlapping objects and the intersecting objects includes:

[0026] Adding the area of ​​the obstructing object and the area of ​​an object adjacent to the obstructing object to obtain an addition result;

[0027] The addition result is compared with the area of ​​the blocking object. If the addition result is equal to the area of ​​the blocking object, the object to be grasped is an overlapping object. If the addition result is not equal to the area of ​​the blocking object, the object to be grasped is an intersecting object.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes an article grabbing device, the article grabbing device comprising:

[0029] The acquisition module is used to obtain the 3D point cloud of the object to be grasped;

[0030] An instance module is used to extract the second-order Taylor function feature information based on the 3D point cloud of the object to be grasped through a local Taylor neural network in a preset instance segmentation model to obtain a local feature map, and extract high-order Taylor function feature information from the local feature map to obtain an object feature map;

[0031] A conversion module, configured to obtain an instance segmentation mask based on the object feature map;

[0032] The grasping module is used to grasp the object to be grasped according to the instance segmentation mask.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes an object grasping device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the object grasping method as described in any one of claims 1 to 6.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the object grasping method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a product, which includes a computer program, and when the computer program is executed by a processor, the steps of the object grabbing method described above are implemented.

[0036] This application uses a Taylor neural network to perform high-order Taylor function feature extraction on the 3D point cloud of the grasped object. The features extracted by the high-order Taylor function are conducive to the segmentation of the object boundary, and the objects are grasped in layers. Unobstructed objects are grasped first, which can more accurately realize the grasping of objects in cluttered environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 This is a schematic diagram of the process of the first embodiment of the object grabbing method of the present application;

[0040] Figure 2 A schematic diagram of an instance segmentation model involved in the object grasping method of this application;

[0041] Figure 3 A schematic diagram of the device structure of the hardware operating environment involved in the object grabbing method of this application;

[0042] Figure 4 A schematic diagram of a Taylor neural network involved in the object grasping method of this application;

[0043] Figure 5 A schematic diagram showing the extraction of kernels of different convolutional layers involved in the object grasping method of the present application on high-frequency information;

[0044] Figure 6 A schematic diagram of generating a grabbing rectangle for an unobstructed object involved in the object grabbing method of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The main solution of the embodiment of the present application is: obtain a 3D point cloud of the object to be grasped; extract high-order Taylor function feature information of the 3D point cloud through a preset instance segmentation model to obtain an object feature map; obtain an instance segmentation mask based on the object feature map and the 3D point cloud; and grasp the object to be grasped according to the instance segmentation mask.

[0049] At present, the existing object grasping method first uses the semantic segmentation network of low-level information to segment the object, and then uses the grasping generator to generate the grasping posture of the object. After the segmentation, the grasping posture of the object is used to grasp the object. However, when the existing grasping generator with instance segmentation grasps in a cluttered environment, for stacked objects, the low-level information is not conducive to the object instance segmentation, which affects the object grasping results and reduces the grasping accuracy.

[0050] This application uses a Taylor neural network to perform high-order Taylor function feature extraction on the 3D point cloud of the grasped object. The features extracted by the high-order Taylor function are conducive to the segmentation of the object boundary, and the objects are grasped in layers. Unobstructed objects are grasped first, which can more accurately realize the grasping of objects in cluttered environments.

[0051] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the object grabbing method of the present application.

[0052] In a first embodiment, the object grabbing method includes the following steps:

[0053] Step S100, obtaining a 3D point cloud of the object to be grasped;

[0054] It should be noted that the executor of this embodiment is an item grabbing device. Preferably, the item grabbing device is a server to implement the item grabbing method. It can also be other terminals with data transmission and data processing functions, and no specific restrictions are made here.

[0055] It is understandable that the objects to be grasped refer to multiple objects stacked together in the grasping scene. The 3D point cloud of the objects to be grasped refers to the 3D point cloud P generated by reversely projecting the object depth map and the internal parameters of the grasping device.

[0056] The device first uses a preset gripping device to shoot the object to be grasped, and then uses the left and right infrared cameras to achieve depth measurement using a triangulation method based on the camera's internal parameters, thereby obtaining a depth map of the photographed object, and generates a 3D point cloud P by reversely projecting the object's depth map through the internal parameters of the gripping device. The gripping device can be a computer device with a shooting function, which has its own internal parameters and can convert the captured image into a depth map.

[0057] Step S200, based on the 3D point cloud, extract feature information of the 3D point cloud using a preset instance segmentation model using a high-order Taylor function to obtain an object feature map, wherein the object feature map is a feature map having high-order feature information;

[0058] It should be noted that the preset instance segmentation model refers to a model that uses U-Net as the network architecture and uses a Taylor neural network to extract the features of the object to be grasped; the high-order Taylor function refers to the expansion of the Taylor function of the second order or above; the feature information extraction refers to the feature extraction of the high-order information that is conducive to boundary segmentation; the object feature map refers to the feature map after the high-order information is extracted. The device inputs the 3D point cloud P of the object to be grasped into the preset instance segmentation model, performs high-order Taylor function feature extraction, and obtains an object feature map with high-order feature information. Among them, the preset instance segmentation model is a model existing in a computer device with a camera function, and the 3D point cloud P is processed by the instance segmentation model in the computer device to obtain a processed feature map.

[0059] This application first examines the relationship between fully connected neural networks and Taylor expansion. Then, the convolutional neural network is redesigned as a Taylor neural network using Taylor expansion. Let Φ(X) be a mapping consisting of several stacked layers, where Φ(X) may not be the entire network, and X represents the input of the first layer of these layers. Further, consider a fully connected neural network with L hidden layers, W l represents the weight matrix of the l = {1, 2, …, L}th hidden layer.

[0060] Among them, the mapping Φ fitted by the L-layer fully connected neural network fc (X) can be expressed as:

[0061] F fc (X) = σ(W L (…σ(W1X)))

[0062] Where σ is the ReLU activation function, and the bias is ignored to simplify the notation. Fully connected neural networks usually use nonlinear activation functions such as ReLU and Tanh to approximate complex functions. ReLU is widely used as an activation function due to its own advantages. However, fully connected neural networks that only use ReLU can only perform first-order modeling. As a result, the network can only use multiple linear functions to approximate complex functions. And Tanh usually has poor derivatives when implementing second-order and higher data modeling.

[0063] Among them, the multivariate function is The Taylor expansion at can be expressed as:

[0064]

[0065] in, All items in X0 are 0. fc The expansion of (X) and multivariate functions are From the Taylor expansion at , we can see that each part of the fully connected neural network can be regarded as using only the first-order term of the Taylor expansion to represent it. Assuming that the stacked nonlinear layer, whose activation is ReLU, can asymptotically approximate complex functions, a first-order Taylor expansion network can be used for approximation. In addition, designing the network with a high-order Taylor expansion can make up for its learning of high-order information.

[0066] Each convolution operation in a convolutional neural network can be viewed as a fully connected network. Similarly, convolutional neural networks using ReLU can only achieve first-order modeling. First, consider a network consisting of two convolutional layers, whose output Φ c (X) can be expressed as:

[0067] F c (X) = W c2 *(W c1 *X)

[0068] in, and Represent the weight matrices of the two convolutional layers respectively, and * represents the convolution operation. It should be noted that M is usually an odd number in convolutional neural networks. c (X) Expanding with second-order or higher Taylor expansion may help improve the performance of the network. The second-order Taylor expansion at the Taylor expansion is similar to the form of the second-order Taylor expansion, and Φ c (X) Redesign as follows:

[0069]

[0070] in, W cen It indicates that a virtual weight matrix of the present invention is used to keep consistent with the structure of the Taylor neural network designed later, and the value of the center position is 1, and the values ​​of other positions are 0. Denotes the Hadamard product. Finally, Φ c (X) The constructed Taylor neural network is called the Minimal Taylor Neural Network (M-TNN), such as Figure 4 shown.

[0071] For objects placed in a disorderly manner, the segmentation of boundaries affects the segmentation of different instance objects. When using neural networks for image segmentation, the high-frequency features of objects may be beneficial to segment the boundaries of different objects. To this end, the present application hopes to perform high-order modeling of the main high-frequency information in the image. For a network with several stacked convolutional layers, different layers can further extract high-frequency features from their respective inputs, such as textures and edges. Only a few kernels in each convolutional layer can be used to achieve the extraction of main high-frequency features. The present application tests and statistics the performance of the kernels of the first three convolutional layers of UOIS-Net-3D in outputting high-frequency information. For simplicity, the second and third layers only tested half of their number of channels (32) and half of their number of kernels (64), respectively. As to whether a kernel can achieve the extraction of main low-frequency information, the present application checks whether it satisfies the characteristics of a low-pass filter, for example, each weight matrix is ​​positive or negative, and visualizes its filtering results for judgment. Figure 5 The statistical results of the number of kernels that can output more high-frequency information in different convolutional layers are shown. The horizontal axis and vertical axis represent the number of channels (3 / 32 / 64) and the number of kernels (64 / 64 / 64) corresponding to different convolutional layers, respectively. The experimental results show that a small number of kernels in the same convolutional layer can be used to further extract the main high-frequency information from its input.

[0072] Based on the above considerations and findings, this application is designed Figure 4 The feature difference-Taylor Neural Network (FD-TNN) shown in Figure 1 is a feature difference-Taylor Neural Network (FD-TNN). FD (X) can be expressed as:

[0073]

[0074] in for for for P FD (X) Further analysis shows that Φ FD The first term of (X) extracts mainly low-frequency features compared to the second term. On the contrary, Φ FD The second term of (X) extracts more high-frequency features.

[0075] In Φ c (X) and Φ FD In (X), the present application constructs a virtual matrix W cenSince only the center position of the virtual matrix has a value of 1, this is equivalent to only considering the association between the convolution center and other positions, while ignoring the association between other positions except the convolution center. This application further considers the association between all positions of the entire convolution kernel. This application can be replaced with a real convolution layer. This application will Figure 4 The minimum Taylor neural network and feature difference-Taylor neural network in the network with convolutional layers added are called Standard Taylor Neural Network (S-TNN) and Feature Difference-Standard Taylor Neural Network (FDS-TNN). The redesigned S-TNN and FDS-TNN are as follows: Figure 4 Another motivation behind the design of S-TNN and FDS-TNN comes from the degradation problem of the model. The present invention assumes that it is easier to optimize the first-order Taylor expansion than the second-order Taylor expansion. In extreme cases, if the first-order Taylor expansion is optimal, the model will push the second-order Taylor expansion to zero. Specifically, when the model degenerates to the first-order Taylor expansion, Figure 4 The weights of the C2 convolutional layer of the Standard Taylor Neural Network (S-TNN) and Feature Difference-Standard Taylor Neural Network (FDS-TNN) are pushed to zero and the same weight as the C1 convolutional layer, respectively. Therefore, the performance of S-TNN and FDS-TNN should at least remain the same as when higher-order Taylor expansions are not used. Experiments show that although such a solution exists, it is difficult for current solvers to find a solution that is comparable or better than the constructed solution.

[0076] Step S300, determining an instance segmentation mask based on the object feature map and the 3D point cloud, wherein the instance mask defines the exact boundary of the object to be grasped;

[0077] It is understood that the instance mask refers to information used to segment a single object. The object feature map and the 3D point cloud P are processed to obtain the instance mask. The device processes the object feature map and the 3D point cloud P through a program in a computer device to obtain the object instance mask.

[0078] In a feasible implementation manner, step S300 further includes the following steps:

[0079] Determine a semantic mask and a center offset of the object feature map, wherein the semantic mask is used to mark the object to be grasped, and the center offset is used to locate the center point of the object to be grasped; combine the 3D point cloud with the center offset to obtain a combined map; and perform mean shift clustering calculation on the combined map based on the semantic mask to obtain an instance segmentation mask.

[0080] It should be noted that the semantic mask refers to the three semantic categories of background, desktop and desktop objects; the combination map refers to the image of the predicted object center of each pixel obtained by adding the 3D point cloud P and the center offset O. The device combines the 3D point cloud P with the center offset O to obtain a combination map; the device performs mean shift clustering calculation on the combination map based on the semantic mask S to obtain an instance segmentation mask. The device sends the object feature map to two convolutional layers to obtain the semantic mask S and the center offset O, and visualizes the center offset O using the optical flow method. The device processes the object feature map through the preset instance segmentation model in the computer device to obtain the semantic mask S∈R H×W×3 and the center offset O∈R H×W×3 , S contains three semantic categories: background, desktop and desktop object. The preset instance segmentation model adds the 3D point cloud P and the center offset O to obtain the center vote V, and calculates the center vote V to obtain the instance segmentation mask.

[0081] This application uses a loss function to calibrate the object grasping method. The loss function used includes semantic segmentation loss (L SS ), center shift loss (L CO ), clustering loss (L C ) and separation loss (L Sep ). The final loss function is expressed as L Total =λ SS L SS +λ CO L CO +λ C L C +λ Sep L Sep The hyperparameter weight λ SS , L CO , L C and L Sep They are set to 3, 5, 1 and 1 respectively.

[0082] Among them, the semantic segmentation loss uses weighted cross entropy, which can be expressed as:

[0083]

[0084] Among them, S i and Represent the predicted category and the true category respectively, L CE represents the cross entropy. The weight w i With label equal to S i is inversely proportional to the number of pixels.

[0085] The center shift loss uses a weighted smooth L1 loss (Huber loss), which can be calculated as:

[0086]

[0087] Among them, O i and Represent the predicted center offset and the actual center offset respectively, and the weight w i With semantic segmentation loss, the label is equal to S i is inversely proportional to the number of pixels, L H Represents Huber loss.

[0088] Further, following the above loss function, this application adopts clustering loss to expand GMS for several iterations and apply a loss on the cluster points. The mean shift algorithm with Gaussian kernel iteratively moves a set of Ω 3D seed points, Use a gradient ascent style approach to reach higher density regions of the kernel density estimate. Let Z (k) represents the k-th iteration point of mean shift. (k) and V use the following loss function with the corresponding object instance label Y:

[0089]

[0090] Among them, Ο represents the set of pixels belonging to the object category, w ij represents the weight inversely proportional to the category size, d(·,·) represents the Euclidean distance, [·] + = max(·,0), I{·} represents the indicator function, which returns 1 or 0 when it is true or false according to the specified condition. This loss function affects the kernel density estimation mode to be close to its points and at least δ away from the points that do not belong to the cluster. δ is set to 0.1 in the experiment.

[0091] In the specific implementation, using it for all object points will result in excessive memory usage, so the use of this loss function is changed. This application randomly extracts an index set and set Z (0) =V D , and run mean shift clustering only on these points. |D| is set to 150 and 200 in training and testing, respectively. This application performs mean shift clustering for K iterations and applies In each iteration, K is set to 5 and 10 in training and testing respectively. The final clustering loss is expressed as

[0092] It is important to note that the separation loss encourages the center vote V not necessarily to be located at the center of an object, as long as it is far away from other object centers.

[0093]

[0094] Among them, c j is the jth true object center, i∈O. τ is a hyperparameter, which is set to 15. The above formula is just the distance from the center vote V to all object centers, scaled by τ, and softmax is applied. The separation loss is expressed using cross entropy as:

[0095] L Sep (M ij )=-∑ j＝1 I{y i =j}log(M ij )

[0096] In a feasible implementation manner, step S300 may further include the following steps:

[0097] Perform mean shift clustering calculation on the combined graph to obtain the instance segmentation mask;

[0098] It is understood that in order to generate the instance segmentation mask, the device uses mean shift clustering on the center vote V. Mean shift clustering is a non-parametric, iterative pattern search algorithm that does not require a priori knowledge of the number of clusters and does not restrict the shape of the clusters. Mean shift clustering is used to find distribution patterns approximated by kernel density estimation (KDE). This application uses a Gaussian kernel as The mean shift of is called Gaussian mean shift (GMS). σ is a hyperparameter that affects the number of clusters. After the center vote V is clustered, each pixel is assigned to the nearest cluster center. The computer device calculates the instance segmentation mask on the combined graph.

[0099] Step S400: grab the object to be grabbed according to the instance segmentation mask.

[0100] The device generates the object grasping posture according to the instance segmentation mask through the grasping generation algorithm, and generates a rectangular frame for the object to be grasped for grasping.

[0101] This embodiment uses a Taylor neural network to perform high-order Taylor function feature extraction on the 3D point cloud P of the object to be grasped. The features extracted by the high-order Taylor function are conducive to the segmentation of the object boundary. The objects are then grasped in layers, first grasping the unobstructed objects, and then grasping the overlapping objects and intersecting objects in turn. This can more accurately realize the grasping of objects in a cluttered environment.

[0102] In a feasible implementation, a high-order Taylor function feature information is extracted from the 3D point cloud by using a preset instance segmentation model to obtain an object feature map; step S200 may include the following steps:

[0103] Through the local Taylor neural network in the preset instance segmentation model, the second-order Taylor function feature extraction is performed on the local feature map to obtain a local feature map, and high-order Taylor feature information is extracted based on the local feature map to obtain an object feature map.

[0104] It should be noted that the local Taylor neural network is the Taylor neural network in the downsampling block in the preset instance segmentation model, and the local feature map refers to the feature map obtained by capturing the feature information of the second-order Taylor function of the object to be grasped. The network proposed in the object grasping method uses U-Net as the basic architecture, which is an encoder-decoder network. The encoder consists of 1 CGR and 4 downsampling blocks with Taylor neural networks. Each downsampling block D i ∈{D1, D2, D3, D4} is composed of a stack of three components: a maximum pooling layer (MaxPool), a Taylor neural network (TNN), and a CGR. 3D point cloud It is fed into the encoder to generate feature maps E of different scales i ∈{E1, E2, E3, E4, E5}, where H = 480, W = 640. Specifically, 64 feature maps E1 are extracted from the depth data using 1 CGR. The size of each feature map in E1 is the same as the input size. Subsequently, 4 downsampling blocks with Taylor neural networks are used in sequence to generate 4 feature maps of different scales {E2, E3, E4, E5}. Among them, the sizes of E2 to E5 are reduced by 2 times respectively. Each downsampling block D i Except that the number of feature maps output by CGR is doubled as the number of input feature maps, the number of feature maps output by other components remains unchanged.

[0105] The decoder comes from UOIS-Net-3D, which consists of an efficient spatial pyramid (ESP), 4 upsampling modules, 1 CGR and 1 convolution. i ∈{U1, U2, U3, U4} are all stacked by 1 CGR, 1 bilinear interpolation upsampling, 1 feature cascade and 1 CGR. The size of the feature map output from U1 to U4 is doubled and the number is halved. The feature map output by U4 passes through 1 CGR and 1 convolution layer in turn and is sent to 2 convolution layers to obtain the semantic mask S∈R H ×W×3 and the center offset O∈R H×W×3 S contains three semantic categories: background, desktop, and desktop object. P+O represents the predicted object center of each pixel, which is called the center vote V.

[0106] In a feasible implementation manner, the object to be grasped is grasped according to the instance segmentation mask; step S400 may include the following steps:

[0107] Classifying the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects;

[0108] The unobstructed objects, the overlapping objects and the intersecting objects are grasped in sequence respectively.

[0109] It is understandable that unobstructed objects refer to objects that are separated from each other; overlapping objects refer to objects that completely overlap each other; and intersecting objects refer to objects that partially overlap each other. The computer device classifies the relationship between the objects to be grasped according to the instance mask.

[0110] In a feasible implementation, the following steps are also included:

[0111] Based on the instance segmentation mask, the outer contour information and area of ​​the object to be grasped are determined; the outer contour information of each object to be grasped is deleted respectively to obtain the remaining outer contour information; the remaining outer contour information is compared with the outer contour information, if the number of contours in the remaining outer contour information is less than the number of contours in the outer contour information, the object to be grasped is an unobstructed object, if the number of contours in the remaining outer contour information is equal to or greater than the number of contours in the outer contour information, the object to be grasped is an obstructed object; classification calculation is performed based on the area of ​​the obstructed object to determine overlapping objects and intersecting objects.

[0112] In a feasible implementation, the following steps are also included:

[0113] Adding the area of ​​the obstructing object and the area of ​​an object adjacent to the obstructing object to obtain an addition result;

[0114] The addition result is compared with the area of ​​the blocking object. If the addition result is equal to the area of ​​the blocking object, the object to be grasped is an overlapping object. If the addition result is not equal to the area of ​​the blocking object, the object to be grasped is an intersecting object.

[0115] It should be noted that the device adopts the strategy of grabbing unobstructed objects first and then occluded objects for grasping in cluttered object scenes. Algorithm 1 describes the outline of inferring the occlusion relationship between different objects. The device simply divides the occlusion relationship between objects into separation, overlap and intersection. Given an instance mask set In the first stage, the device sequentially calculates whether each instance mask is separated from other instance masks. Represents the instance mask set Delete This application calculates and Number of outer contours and when Less than When , the instance mask to be judged is a separated mask.

[0116] When no separation mask is detected in the first stage, the device recursively detects in the second stage Specifically, the device first calculates whether different masks overlap. The number of outer contours and area Secondly, calculate Number of outer contours and area as well as Area in Instance Mask and The overlap between is calculated as follows:

[0117]

[0118] in, Equivalent to 1 or 0 indicates that there is overlap or intersection between the two objects. In order to reduce the possible collision between objects caused by grasping, the device gives priority to grasping objects closer to the camera. and When there is overlap between equal or Simply judge the distance between two objects and the camera. equal Get closer to the camera. Otherwise, Closer to the camera. After the above two stages fail to detect occlusion relationships, Each instance mask in has an intersection relationship with other instance masks.

[0119] Specifically, the content of Algorithm 1 is as follows:

[0120] Algorithm 1 Occlusion Derivation Input: Instance Mask Set:

[0121] 1: function OCCLUSION DEDUCTION

[0122] 2: Separate mask sets Overlapping Mask Set Intersection Mask Set

[0123] 3: Calculate for

[0124] 4: for do#Searching for isolated masks.

[0125] 5: Calculate for

[0126] 6: if then#Calculate the number of masks

[0127] 7: R S .append

[0128] 8: end if

[0129] 9: end for

[0130] 10: if then#Find overlapping mask sets

[0131] 11: for do# Check if the mask overlaps

[0132] 12: for do

[0133] 13: Calculate for Calculate for

[0134] 14: Calculate and for

[0135] 15: if and do

[0136] 16: .append

[0137] 17: end if

[0138] 18: end for

[0139] 19: end for

[0140] 20: else return R S

[0141] 21: end if

[0142] 22: if then

[0143] 23: R I .append

[0144] 24: else return

[0145] 25: end if

[0146] 26: return R I

[0147] 27: end function

[0148] like Figure 6 As shown, the grasping generation algorithm used by the device is GR-ConvNet-RGB-D. Since the grasping generation algorithm is customized for single-object scenes, the performance of the algorithm will be affected if it is directly used in cluttered object scenes. Therefore, grasping generation algorithms are designed for objects in three different occlusion conditions: separation, overlap, and intersection. It should be noted that the device preferentially grasps separated objects, followed by overlapping objects, and finally intersecting objects. If the minimum depth (distance from the camera) of the two finger areas corresponding to the generated rectangle is greater than the minimum depth in the rectangle, the rectangular frame is considered feasible. In the grasping experiment, the minimum depth was set to 15mm. After all generated grasping rectangles are not feasible, this application directly uses GR-Conv-RGB-D to generate grasping rectangles. This application uses the same grasping generation strategy as for separated objects to generate grasping rectangles for overlapping objects. For the case where all objects intersect, this application directly uses GR-Conv-RGB-D to generate grasping rectangles.

[0149] The present application also provides an article grabbing device, the article grabbing device comprising:

[0150] The acquisition module is used to obtain the 3D point cloud of the object to be grasped;

[0151] An instance module, configured to extract feature information of a high-order Taylor function from the 3D point cloud through a preset instance segmentation model based on the 3D point cloud, so as to obtain an object feature map, wherein the object feature map is a feature map having high-order feature information;

[0152] An information module, configured to determine an instance segmentation mask based on the object feature map and the 3D point cloud;

[0153] The grasping module is used to grasp the object to be grasped according to the instance segmentation mask.

[0154] Optionally, the instance module includes:

[0155] A Taylor module, used for extracting local features of the 3D point cloud through a local Taylor neural network in a preset instance segmentation model to obtain a local feature map;

[0156] High-order Taylor feature information is extracted from the local feature map to obtain an object feature map.

[0157] Optionally, the information module includes:

[0158] A conversion module is used to determine the semantic mask and center offset of the object feature map, wherein the semantic mask is used to mark the object to be grasped, and the center offset is used to locate the center point of the object to be grasped; the 3D point cloud is combined with the center offset to obtain a combined map; based on the semantic mask, the combined map is subjected to mean shift clustering calculation to obtain an instance segmentation mask.

[0159] Optionally, the capture module includes:

[0160] A classification module is used to classify the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects; and grasp the unobstructed objects, the overlapping objects and the intersecting objects in sequence.

[0161] The analysis module is used to determine the outer contour information and area of ​​the object to be grasped based on the instance segmentation mask; delete the outer contour information of each object to be grasped respectively to obtain the remaining outer contour information; compare the remaining outer contour information with the outer contour information, if the number of contours in the remaining outer contour information is less than the number of contours in the outer contour information, the object to be grasped is an unobstructed object, if the number of contours in the remaining outer contour information is equal to or greater than the number of contours in the outer contour information, the object to be grasped is an obstructed object; perform classification calculation based on the area of ​​the obstructed object to determine overlapping objects and intersecting objects.

[0162] The present application also provides an object grabbing device, the object grabbing device comprising: a memory, a processor, and a program stored in the memory for implementing the object grabbing method.

[0163] The memory is used to store a program for implementing the object grabbing method;

[0164] The processor is used to execute a program for implementing the object grabbing method to implement the steps of the object grabbing method.

[0165] The specific implementation of the object grabbing device of the present application is basically the same as the embodiments of the above-mentioned object grabbing method, and will not be repeated here.

[0166] An embodiment of the present application provides a storage medium, and the storage medium stores one or more programs, and the one or more programs can also be executed by one or more processors to implement the steps of any of the above-mentioned object grabbing methods.

[0167] The specific implementation method of the storage medium of the present application is basically the same as the above-mentioned embodiments of the object grabbing method, and will not be repeated here.

[0168] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0169] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus hardware platform, or by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes several instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0171] It can be seen from the above embodiments that the present application overcomes the technical problem of affecting the accuracy of object grasping in a cluttered environment by adopting layered grasping, thereby realizing object grasping in a cluttered environment.

[0172] The above are only some embodiments of the present application, and the scope of implementation of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the present application specification, or directly or indirectly applied in other related technical fields, are also included in the protection scope of the present application.

Claims

1. A method for grabbing an object, characterized in that: The object grabbing method comprises: Get the 3D point cloud of the object to be grasped; Based on the 3D point cloud, extract feature information of the 3D point cloud using a high-order Taylor function through a preset instance segmentation model to obtain an object feature map, wherein the object feature map is a feature map with high-order feature information; Determine an instance segmentation mask based on the object feature map and the 3D point cloud, wherein the instance mask defines the exact boundary of the object to be grasped; The object to be grasped is grasped based on the instance segmentation mask.

2. The object grabbing method according to claim 1, characterized in that: The step of extracting feature information of a high-order Taylor function from the 3D point cloud through a preset instance segmentation model based on the 3D point cloud to obtain an object feature map comprises: Performing local feature extraction on the 3D point cloud by using a local Taylor neural network in a preset instance segmentation model to obtain a local feature map; High-order Taylor feature information is extracted from the local feature map to obtain an object feature map.

3. The object grabbing method according to claim 1, characterized in that: The step of determining an instance segmentation mask based on the object feature map and the 3D point cloud comprises: Determine a semantic mask and a center offset of the object feature map, wherein the semantic mask is used to mark the object to be grasped, and the center offset is used to locate the center point of the object to be grasped; Combining the 3D point cloud with the center offset to obtain a combined graph; Based on the semantic mask, mean shift clustering calculation is performed on the combination graph to obtain an instance segmentation mask.

4. The object grabbing method according to claim 1, characterized in that: The step of grasping the object to be grasped based on the instance segmentation mask comprises: Classifying the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects; The unobstructed objects, the overlapping objects and the intersecting objects are grasped in sequence respectively.

5. The object grabbing method according to claim 4, characterized in that: The step of classifying the objects to be grasped according to the instance segmentation mask to obtain unobstructed objects and obstructed objects, wherein the obstructed objects include overlapping objects and intersecting objects, comprises: Based on the instance segmentation mask, determining the peripheral contour information and area of ​​the object to be grasped; Deleting the peripheral contour information of each of the objects to be grasped respectively to obtain remaining peripheral contour information; Comparing the remaining outer contour information with the outer contour information, if the number of contours in the remaining outer contour information is less than the number of contours in the outer contour information, the object to be grasped is an unobstructed object; if the number of contours in the remaining outer contour information is equal to or greater than the number of contours in the outer contour information, the object to be grasped is an obstructed object; Classification calculation is performed based on the areas of the obstructing objects to determine overlapping objects and intersecting objects.

6. The object grabbing method according to claim 5, characterized in that: The step of performing classification calculation based on the area of ​​the blocking objects to determine overlapping objects and intersecting objects comprises: Adding the area of ​​the obstructing object and the area of ​​an object adjacent to the obstructing object to obtain an addition result; The addition result is compared with the area of ​​the blocking object. If the addition result is equal to the area of ​​the blocking object, the object to be grasped is an overlapping object. If the addition result is not equal to the area of ​​the blocking object, the object to be grasped is an intersecting object.

7. An object grabbing device, characterized in that: The device comprises: The acquisition module is used to obtain the 3D point cloud of the object to be grasped; An instance module, configured to extract feature information of a high-order Taylor function from the 3D point cloud through a preset instance segmentation model based on the 3D point cloud, so as to obtain an object feature map, wherein the object feature map is a feature map having high-order feature information; An information module, configured to determine an instance segmentation mask based on the object feature map and the 3D point cloud; The grasping module is used to grasp the object to be grasped according to the instance segmentation mask.

8. An object grabbing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the object grabbing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the object grabbing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the object grabbing method according to any one of claims 1 to 6 are implemented.

Images