A method for predicting object grasping order from images based on deep learning

Abstract

The present invention discloses a method for predicting the order of grasping objects from images based on deep learning, comprising the following steps: 1) collecting disordered grasping scene pictures; 2) using a deep segmentation network to detect the detection frames and segmentation masks of all objects to be grasped from the pictures; 3) pooling the feature map areas corresponding to different segmentation masks to make them feature vectors of equal length. At the same time, the global feature map is feature pooled to become a global feature vector; 4) the feature vectors corresponding to all feature masks are connected to the global feature vector to become the feature vector of each object; and they are sent to a special recurrent neural network in disorder, and the recurrent neural network outputs the grasping order of the objects. The present invention can predict a reasonable grasping order in a complex stacked object scene, can speed up the grasping speed of the robot arm to the object and reduce collisions. It is crucial to grasp objects in a reasonable grasping order in industrial scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and specifically relates to a method for predicting the order of object grasping from images based on deep learning. Background Art

[0002] As the manufacturing industry transforms to intelligent manufacturing, how to use AI technology to assist in building intelligent functions has gradually attracted the attention of the industry and computer vision research. In order to gradually use AI technology to replace the cumbersome and customized links in traditional manufacturing, many companies have turned to computer vision for help. In the industrial manufacturing process, vision plus AI has gradually been used to solve some static task links, such as defect detection and volume measurement. In some industrial tasks involving robot grasping of objects, such as disordered grasping and loading and unloading, there are already some methods that try to use AI to restore the grasping posture of the object as the input of the robot for object grasping.

[0003] However, in real-world scenarios, the robot's interaction with environmental objects does not only depend on a single target object, but is also affected by other target instances and other scene objects. For example, in a stacked scene, it is best to grab the upper object first, and then grab the lower object. In order to reduce robot arm collisions, existing methods will try to use robot arm collision avoidance algorithms to calculate collision-avoiding paths. Using visual technology to infer a reasonable object grabbing order from scene images can not only reduce collisions during the actual grabbing process, but also speed up the entire grabbing process.

[0004] In the prior art "Grasp Planning Based On Scene Grasp Ability In Unstructured Environment", the grasping of the scene of basic geometric bodies is planned so that the grasping can avoid collision with scene objects. The grasping planning is carried out by sorting the objects by their graspable scores, which has limitations and is not applicable to any object. Summary of the invention

[0005] In order to solve the problem of how to predict a reasonable grasping order of objects when objects are stacked, the present invention proposes a method for predicting the grasping order of objects in a stacked object grasping scenario, and also proposes a means for generating training data to support end-to-end training of the algorithm.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] A method for predicting the order of grasping objects from images based on deep learning, comprising the following steps:

[0008] Step 1: Use the segmentation network to detect all foreground objects in the image, and output segmentation masks for all foreground objects, retaining the global feature map of the image and the object feature map before outputting the mask;

[0009] Step 2: Use the segmentation mask of the object to cut out the feature map of the mask position from the object feature map and perform pooling to obtain the local feature vector of the object, perform pooling on the global feature map to obtain the global feature vector, and connect the global feature vector to the local feature vectors of each object to obtain the object feature of each object;

[0010] Step 3: Use a recurrent neural network as an encoder to feed the object feature vector sequences of all objects into the encoder in sequence, and finally obtain a feature vector of fixed length;

[0011] Step 4: Use the feature vector encoded in step 3 as the hidden feature, randomly generate an input vector, and input the hidden feature vector into the grasping order predictor. The grasping order predictor receives a fixed-length input vector and a hidden feature obtained in the previous step at each step, and outputs an index. The index points to a feature in the object feature sequence. The object corresponding to the feature is the grasped object predicted in the current step. The number of steps in the loop prediction is the number of objects detected. The final predicted index sequence is the grasping order of the objects.

[0012] Furthermore, the segmentation network includes a binary classifier for separating foreground objects and background objects.

[0013] Furthermore, the specific steps of step 2 are as follows:

[0014] 21. Use the segmentation network to detect the masks of all foreground objects i ,i∈1,2,…N, where N is the number of objects detected by the segmentation network in the current image, and the mask of the foreground object is used to mask the previous feature layer of the predicted object mask, and then feature pooling is performed after masking, and then a linear network is used to convert the number of feature channels to a fixed-length local feature of the object To generate local feature vectors for each object;

[0015] 22. Perform feature pooling directly on the global feature layer with the most complete resolution, and use another linear network to convert the global features to a fixed length to become the global features of the scene f global ;

[0016] 23. Connect the local features and global features of the object to form the features of the object

[0017] Furthermore, each cycle of the encoder uses an object feature as input and outputs a corresponding hidden feature The last encoded hidden feature Feature encoding as a sequence of object features in, is the hidden feature of the last encoding output, is the feature of the object, N is the total number of objects, This is the result after encoding the features of all objects.

[0018] Further, step 4 includes the following steps:

[0019] 41. Use the LSTM recurrent neural network as the crawling order predictor and use the feature vector encoded in step 3 as the first hidden feature Randomly generate the first input vector m is a fixed input feature length, and the hidden features and input vector are fed into the crawl order predictor;

[0020] The fetch order predictor accepts a hidden feature at each step and an input vector And output an output vector Where j represents the current position in the jth cycle, that is, the current prediction of the jth grasp target, is the feature corresponding to the grasp target predicted in the previous cycle. When j=1, it means the first step of prediction. That is, generate a random vector as input;

[0021] 42. For the feature vector output by the grasping order predictor at each step, use the mechanism in PointerNet to calculate an index from the feature sequence of the object, and use the object corresponding to the index as the grasped object in this step;

[0022] 43. Step 41 to step 42 are looped h times, where h is the number of objects detected, thereby obtaining an index sequence of length h, which is the object grasping order.

[0023] Furthermore, the labeled data is a large amount of data with grabbing order labels that are automatically generated by simulation plus rendering. The generation method of grabbing order labels uses a heuristic algorithm. The specific steps are as follows:

[0024] 51. Start building a scene, randomly import n objects into the simulator, and copy m instances of each object. The number of objects and instance data are randomly generated in each scene build;

[0025] 52. Take the world center of the simulator as the origin and divide it into p×p grids. The size of each square in the grid is the average diameter of the imported object plus a fixed constant d.

[0026] 53. Place objects at the edge of the grid, randomly select an instance from the object instances each time, place it at the center of the corresponding grid, lift it up along the z-axis, and then randomly translate it in the xy plane, and randomly assign a texture to the instance each time;

[0027] 54. Repeat step 53 until the grid is full, and then continue to divide the grid into (p-2)×(p-2) grids with the same center and the same size;

[0028] 56. Repeat steps 53 and 54 until three layers are placed. If the number of instances is insufficient or other conditions are not met, the scene construction is terminated and the next stage is entered. The final scene is in the shape of a suspended pyramid.

[0029] 57. Take the world coordinate origin as the origin of the sphere, generate a hemispherical surface in the positive direction of the z-axis, and uniformly sample o positions on the spherical surface to assume a virtual camera;

[0030] 58. Render the sampled camera positions one by one, each time the camera faces the world origin. At the same time, each rendering randomly perturbs the light and the material of the object surface, and then renders the image;

[0031] 59. Filter out objects whose occlusion exceeds the set ratio.

[0032] Furthermore, the annotated data is obtained by using a camera to collect RGB images of stacked scenes at multiple angles, using a trained segmentation network to detect object frames, and annotating the grasping order of objects through manual annotation.

[0033] Furthermore, the annotated data uses images synthesized by using a simulation rendering method as data, and the data synthesized by using a simulation rendering method is automatically annotated, and no additional manual annotation is required.

[0034] Furthermore, the segmentation network can generate a feature map of the image while obtaining the segmentation mask of the object in the image. At the same time, the segmentation network can detect all foreground objects in the image. At the same time, the PointerNet network used in the grasping order prediction stage performs cyclic prediction on any number of objects, not limited to a fixed number of objects. Therefore, the present invention can effectively predict the grasping order for any number of objects to be grasped in the scene.

[0035] Furthermore, when extracting features of an object, the image features of the area where the object is located are used as the features of the object, and a global feature is also connected. The features obtained in this way have both local and global information, providing sufficient information for subsequent prediction of the capture order.

[0036] Compared with the prior art, the beneficial effects achieved by the present invention include:

[0037] The present invention is based on a deep learning method, using global information and local information to build a connection between objects and scenes, and inferring a reasonable object grabbing order from it. Using this grabbing order to grab objects in stacked scenes can reduce collisions and speed up the grabbing process. The present invention is not limited to simple basic geometric bodies, but is applicable to any object. At the same time, the present invention uses an end-to-end algorithm, which does not require more searches and simplifies the entire grabbing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a method for predicting the order of grasping objects from images based on deep learning in the present invention;

[0039] Figure 2 This is a diagram of the architecture of a method for predicting the order of grasping objects from images based on deep learning in the present invention;

[0040] Figure 3 A side view of the scene constructed for this embodiment;

[0041] Figure 4 A schematic diagram of a picture generated for this embodiment. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] Example 1

[0044] A method for predicting the order of grasping objects from images based on deep learning, comprising the following steps:

[0045] Step 1: Foreground object detection and segmentation mask prediction: treat all target objects, that is, foreground objects, as one type of objects, and background-independent objects as another type of objects. Use the segmentation network to detect all foreground objects in the image, and output segmentation masks for all foreground objects, retaining the global feature map of the image. global And the object feature map before outputting the mask

[0046] 7. The use of the segmentation network is used to separate foreground objects and background objects. The segmentation network here is modified based on the existing segmentation network. The main change is the training method. Since the present invention does not need to distinguish specific object categories, it only needs to detect all foreground objects of interest. Therefore, the classification head of the segmentation network is transformed into a binary classifier to separate foreground and background objects. The modified segmentation network can be trained on the existing segmentation data set. The difference is that all similar foreground objects in the segmentation data set are classified as foreground classes, and similar background objects are classified as background classes. That is, the number of object categories in the data set used for training is 2.

[0047] Step 2: Generation of foreground object feature vectors: After obtaining the segmentation masks of all foreground objects and the feature map of the entire image, use the segmentation mask of the object to extract the feature vector from the object feature map. Cut out the feature map of the mask position and perform pooling to obtain the local feature vector of the object, and map the global feature map global Perform feature pooling to obtain a global feature vector, and connect the global feature vector to the local feature vectors of each object to obtain the object features of each object.

[0048] The feature vector generation of the foreground object combines both local features and global features. The specific steps are as follows:

[0049] 21. Use the segmentation network to detect the masks of all foreground objects i , i∈1,2,…N, and use the mask to mask the previous feature layer of the predicted object mask, perform feature pooling after masking, and then use a layer of linear network to convert the number of feature channels to a fixed length of local features of the object In this way, a local feature vector is generated for each object; here N refers to the number of objects detected by the segmentation network from the image.

[0050] 22. Perform feature pooling directly on the global feature layer with the most complete resolution, and use another linear network to convert the global features to a fixed length to become the global features of the scene f global .

[0051] 23. Connect the local features and global features of the object to form the features of the object By combining the local features of the object with the global features of the image, the grasping order prediction network can better capture the relative position of the object relative to the scene and other objects.

[0052] Step 3, feature encoding: Use a recurrent neural network as an encoder to feed the object feature vector sequences of all objects into the encoding network in sequence, and finally obtain a feature vector of fixed length. Here, the feature sequences of objects are unordered, so it does not matter what order is chosen to input the encoding network. The reason for choosing a recurrent neural network as an encoder here is that different scenes contain different numbers of foreground object instances. Using a recurrent neural network can adapt to different numbers of objects.

[0053] The feature encoder in step 3 uses a recurrent neural network to encode the object feature sequence. The object feature sequence obtained in step 2 That is, the object feature sequence represents the sequence of object features corresponding to all objects. The feature encoder uses an object feature as input at each step and outputs a hidden feature The last encoded hidden feature Feature encoding as a sequence of object features in, is the hidden feature of the last encoding output. Feature encoding as a sequence of object features Where N is the total number of objects, therefore, This is the result after encoding the features of all objects.

[0054] Step 4, grasping order prediction: Take the feature vector encoded in step 3 as the first hidden feature, randomly generate the first input vector, input the first hidden feature and the first input vector into the grasping order predictor, which is also a recurrent neural network. Each step receives a fixed-length input feature vector and a hidden feature vector obtained in the previous step, and selects one from the object feature sequence and outputs its index as the object selected in the current step. The number of steps of the recurrent prediction is the number of objects detected. Finally, the index sequence of the object sequence is predicted, which is the grasping order of the objects. The specific steps are as follows:

[0055] 41. Use a recurrent neural network as a grab order predictor. The predictor accepts a hidden feature at each step and an input vector Where j represents the current position in the jth loop, that is, the jth grasp target is currently being predicted. is the feature corresponding to the grasped target predicted in the previous cycle. When j=1, it is the first step of prediction. That is, a random vector is generated as input. At the same time, an output vector is output More specifically, in order to be able to gradually output an index from an object feature sequence, this embodiment uses PointerNet as a grasping order predictor.

[0056] 42. Using the sequence feature after feature encoding in step 4 of claim 1 as the first hidden feature of the object grasping sequence predictor At the same time, a simple vector is generated as the input feature of the first step m is the fixed input feature length.

[0057] 43. For the feature vector output by the predictor at each step, use the mechanism in PointerNet to calculate an index from the feature sequence of the object in a manner similar to the attention mechanism. The object corresponding to the index is used as the grasping object of this step. The reason for using PointerNet as the grasping order prediction network is that 1) the number of objects is variable, and different scene images contain different numbers of objects. 2) The output of the network is a discrete value, each value represents the object target that should be grasped at this step.

[0058] 44. Repeat the above steps h times, where h is the number of objects detected. This results in an index sequence of length h, which is the object grabbing order.

[0059] Step 5, use the automatically synthesized annotated data to train the segmentation network and the recurrent neural network: use the data synthesized by simulation and rendering to train a large number of data and use real data for optimization: in order to reduce the cost of manual annotation, the present invention also provides a method for automatically generating a large number of training data with grasping order annotations using simulation-rendering synthetic data. Use the simulation engine to build a virtual disordered grasping object stacking scene, in which the objects are placed in a step-by-step stacking manner, so that the grasping order annotations can be automatically obtained; then use the renderer to render a large amount of training data with grasping order at multiple angles and randomization of parameters such as lighting and materials. The other data is collected from the real scene using an RGB camera, and manually annotated to obtain a grasping order that conforms to human intuition. At the same time, for the trained basic network, the real training data obtained by the manual annotation method is optimized so that the network can be better generalized to real scene pictures.

[0060] The acquisition of training data comes from three aspects. One is to use simulation and rendering to automatically generate a large number of data with grabbing order annotations. The method for generating the grabbing order annotations here uses a simple heuristic algorithm. The specific steps are as follows:

[0061] 51. Start building a scene

[0062] 52. Import n random objects into the simulator, and copy each object into m instances. The number of objects and instance data here are randomly generated in each scene construction.

[0063] 53. Take the world center of the simulator as the origin and divide it into a p×p grid. The size of each square in the grid is the average diameter of the imported object plus a fixed constant d.

[0064] 54. Place objects from left to right and from top to bottom. Each time, randomly select an instance from the object instance, place it at the center of the corresponding grid, and lift it up along the z-axis by a certain distance, such as 5cm. This distance is a parameter that needs to be adjusted. Generally, a distance can be randomly sampled from 5-10cm, and then randomly translated within a certain radius in the xy plane. Each time, a texture is randomly assigned to the instance;

[0065] 55. Repeat step 4 until the grid is full. Then continue to divide the grid into (p-2)×(p-2) grids with the same center and the same size.

[0066] 56. Repeat steps 4-5 until three layers are placed. If the number of instances is insufficient or other conditions are not met, the scene construction is terminated and the next stage is entered. The resulting scene presents a suspended pyramid-like shape. Figure 3 A side view of a construction scene in this embodiment is given in FIG.

[0067] 57. The origin of the world coordinate system is used as the origin of the sphere, and a certain distance is used as the radius. The radius is generally related to the distance between the camera and the target scene in the actual scene. It can be set to the distance between the camera position and the center of the scene in the application scene, and take it as the center point, with a positive and negative extension of 0.5 meters. A distance is randomly sampled within this range as the final radius. A hemispherical surface is generated in the positive direction of the z-axis, and o positions are uniformly sampled on the spherical surface to assume a virtual camera.

[0068] 58. Render the sampled camera positions one by one, with the camera facing the world origin each time. At the same time, make some random disturbances to the light and the material of the object surface each time, and then render the image. For example, before each rendering, sample a random number of 0-1. If the number is less than 0.5, it is replaced with a probability of half. At this time, randomly select a material from the material package and paste it to the surface of the object through the simulator. The material package can be downloaded directly from the Internet, such as the cctexture material package.

[0069] 59. Filter out objects that are blocked by more than a certain proportion. By projecting the object separately, we can get the complete segmentation mask of the object. full , calculate the area ratio of the segmentation mask and the complete mask of the object in the actual rendering: Where Area(·) refers to the area occupied by the segmentation mask, Mask visib It is the visual segmentation mask of the object in the actual rendering, after removing the objects occluded by other objects. The occlusion threshold can be set to 0.5. mask When it is less than 0.5, the object is removed because it is too much occluded by other objects.

[0070] Since the objects are placed in a free-falling manner, the objects placed later can be considered to be caught first, so the order of grabbing the objects is automatically marked. At the same time, the bounding box and segmentation mask of the object can be calculated using the renderer generation method. Therefore, the data obtained using this method can be used to train the entire neural network in the method of the present invention.

[0071] Another way to obtain training data is to use a real camera to collect RGB images of stacked scenes at multiple angles, use a pre-trained segmentation network to detect the object boxes of objects, and annotate the grasping order of objects through manual annotation.

[0072] Example 2

[0073] like Figure 1 As shown, the method of the present invention for predicting the object grasping order from an image based on deep learning comprises the following steps:

[0074] S1. Foreground object detection and segmentation mask generation: This step detects the masks of all foreground objects from the image, and in this process obtains the feature map of the object and the global feature map of the scene image.

[0075] Specifically, the segmentation mask can be generated using any object segmentation neural network in the prior art. This embodiment uses MaskRCNN as the segmentation network to predict the segmentation mask and generate the feature map. Figure 2 The segmentation network in is illustrated using the MaskRCNN framework.

[0076] The special thing is that this method does not classify objects specifically, but divides all objects in the image into scene object class and background object class. The background object class mainly includes desktop, ground and background frame for placing objects. Therefore, using the existing segmentation neural network in this method only requires setting the output category of the segmentation network to 2. At the same time, the category ID of the object during training is adjusted to foreground (1) or background (0).

[0077] S2. Generate local features and global features of the object, and connect the two to generate object features: In order to enable the grasping order prediction network to perceive the relative position relationship of each object with respect to other objects and the scene, the present invention integrates the global features and the local features of the object in the process of generating object features.

[0078] Specifically, for the local features of the object, the feature map before the output object segmentation mask is used as the feature source, and the segmentation mask of the object is covered to extract the mask-covered part. Figure 2 The feature area covered by the mask is pooled, and then a one-dimensional convolution or linear layer is used to map the pooled feature vector into a feature space of fixed length. In this embodiment, the feature is mapped to a feature space of dimension 256, and the local features of the object are Feature spaces of different dimensions can be used in scenarios of different complexity, so 256 is not a strict requirement here. It can be other numbers.

[0079] For the global features of the image, Figure 2 The figure is shown in the annotation 2 in FIG. Global pooling is performed directly from the feature map with the most complete resolution of the image. Common pooling operations such as average pooling and maximum pooling can be used. This embodiment uses maximum pooling. The pooled global features are also feature mapped to a fixed-length feature space. In this embodiment, the global features are still mapped to a feature space with a dimension of 256. That is, f global ∈R 256 .

[0080] Each object's local features Connect the global features to get the object features In this embodiment, the object feature is a feature vector with a length of 512. global is the global feature vector obtained in the previous step.

[0081] S3. The object feature sequence is encoded by the encoder to generate the synthetic features of the object feature sequence: the object feature sequence is obtained from the previous step Where N is the number of detected foreground objects. The feature sequence is encoded using an encoder to obtain a feature vector that combines local and global features.

[0082] Furthermore, the encoder here uses a recurrent neural network. The recurrent neural network is used to be able to adapt to different numbers of objects. At the same time, the recurrent neural network can better encode content with sequence properties. The encoder here can use various variants of recurrent neural networks. In this embodiment, a single-layer LSTM network is used as an encoder. The object feature sequence here is actually disordered. In each cycle step of the encoder, a feature vector that has not been selected is randomly selected from the feature sequence as the current step input of the encoder. This cycle is repeated N times, where N is the number of objects. The hidden features output in the Nth step are used as the synthetic features of the object feature sequence.

[0083] S4, object grasping order prediction: In order to gradually select the objects to be grasped from the feature sequence of the objects and finally restore an object grasping order sequence, the grasping order predictor uses a special recurrent neural network. More specifically, this embodiment uses PointerNet as the order predictor. The predictor receives a hidden feature at each cycle step. and an input feature And use the output features of each step Object feature sequence Calculate the object selection probability distribution vector and select the index with the highest probability as the object output of this step. The probability distribution vector is calculated as follows:

[0084] 1.

[0085] 2.p j =softmax(u j ),p j ∈R N

[0086] In the above calculation, v T , W 1 , W 2 These are all learnable parameters of the network. It is the correlation coefficient of the feature output of each step of the predictor to the object feature sequence. Each step output of the predictor can be correlated with all object features, that is, the entire object feature sequence to calculate the correlation coefficient into a correlation coefficient vector. Among them, p j is the probability distribution calculated from the correlation coefficient vector using the softmax function. It means that in p j The i-th value in represents the predicted probability of grasping object i in the jth cycle step. N is the number of objects in the image, so p j is a vector of length N, that is, p j ∈R N.From p j Select the index with the highest probability As the object selected in this step. At the same time, add the index to the selected index pool so that the index will not be selected repeatedly in the next loop.

[0087] S5. Use automatically synthesized annotated data to perform network training on large-scale data: The existing data set containing the object grasping order is small, so how to obtain sufficient training data is also part of the present invention. Using manual annotation is a common way to obtain data. In this embodiment, a step of obtaining data through manual annotation is as follows:

[0088] a) Randomly place 5-20 instances of 2-3 objects in a working area, usually a material box or a workbench. Assume that the camera is capturing images from different angles above. Each scene can capture 10-30 images.

[0089] b) Use the segmentation network trained on other datasets to detect and segment foreground and background objects.

[0090] c) Manually select more accurate detection frames and manually optimize the segmentation mask

[0091] d) Use human expert knowledge to label the grasping order of the detected objects.

[0092] Furthermore, this method uses a simulation and rendering method to generate larger-scale training data. The specific steps are as follows:

[0093] 1) Start building a scene

[0094] 2) Randomly import n objects into the simulator, and copy m instances of each object. The number of objects and instance data here are randomly generated in each scene construction. In this embodiment, n is sampled from 2-5, and m is sampled from 1-10

[0095] 3) With the world center of the simulator as the origin, a p×p grid is divided. In this embodiment, p is randomly selected from 3 to 7 each time. The size of each square of the grid is the average diameter of the imported object plus a fixed constant d. In this embodiment, d is set to 5 cm.

[0096] 4) Place objects from left to right and from top to bottom. Each time, randomly select an instance from the object instance, place it at the center of the corresponding grid, and lift it up a certain distance along the z axis. In this example, the lifting distance is randomly sampled from 3-8 cm. Then, randomly translate it within a certain radius in the xy plane. Each time, a texture is randomly assigned to the instance.

[0097] 5) Repeat step 4) until the grid is full. Then continue to divide the grid into (p-2)×(p-2) grids with the same center and the same size.

[0098] 6) Repeat steps 4)-5) until three layers are placed. If the number of instances is insufficient or other conditions are not met, the scene construction is terminated and the next stage is entered. The resulting scene presents a suspended pyramid-like shape. Figure 3 A side view of a construction scene in this embodiment is given in FIG.

[0099] 7) With the world coordinate origin as the origin of the sphere and a certain distance as the radius, a hemispherical surface is generated in the positive direction of the z-axis, and o positions are uniformly sampled on the spherical surface to assume the virtual camera. Here, o is not a fixed value, and can usually be 50-200.

[0100] 8) The sampled camera positions are rendered one by one, with the camera facing the world origin each time. At the same time, each rendering makes a certain random disturbance to the light and the material of the object surface, and then renders the image.

[0101] 9) Objects that are blocked by more than a certain percentage are filtered out. In this embodiment, objects that are blocked by more than 50% are filtered out. Figure 4 An example of a generated image is given. The number on each box represents the order in which the objects in the box are grasped.

[0102] 10) Since the objects are placed in a free-falling manner, the objects placed later can be considered to be caught first, so the order of grabbing the objects is automatically marked. At the same time, the bounding box and segmentation mask of the object can be calculated using the renderer generation method. Therefore, the data obtained using this method can be used to train the entire neural network in the method of the present invention.

[0103] Example 3

[0104] like Figure 1 As shown, the method of the present invention for predicting the object grasping order from an image based on deep learning comprises the following steps:

[0105] S1. Foreground object detection and segmentation mask generation: This step detects the masks of all foreground objects from the image, and in this process obtains the feature map of the object and the global feature map of the scene image.

[0106] Specifically, the segmentation mask can be generated using any object segmentation neural network in the prior art. This embodiment uses the MaskRCNN object segmentation method. Figure 2The segmentation network in the example uses the MaskRCNN framework. S2. Set up cameras and build multiple real scenes. For each scene, capture images from different camera positions. Build about 10-30 scenes, and collect 25-50 real images from different angles for each scene. Use annotation tools such as LabelMe for annotation. Annotation types include: foreground and background object categories, foreground and background object detection boxes, foreground and background object segmentation masks, and foreground object capture order.

[0107] Use the MaskRCNN network pre-trained on other datasets, such as the MaskRCNN network trained on the COCO dataset, fix its backbone network parameters, and change the network's category output to 2 categories. Use the labeled data to retrain the segmentation network.

[0108] S3. Generate local features and global features of the object, and connect the two to generate object features: In order to enable the grasping order prediction network to perceive the relative position relationship of each object with respect to other objects and the scene, the present invention integrates the global features and the local features of the object in the process of generating object features.

[0109] Specifically, for the local features of the object, the feature map before the output object segmentation mask is used as the feature source, and the segmentation mask of the object is covered to extract the mask-covered part. Figure 2 The feature area covered by the mask is pooled, and then a one-dimensional convolution or linear layer is used to map the pooled feature vector into a feature space of fixed length. In this embodiment, the feature is mapped to a feature space of dimension 256, and the local features of the object are Feature spaces of different dimensions can be used in scenarios of different complexity, so 256 is not a strict requirement here. It can be other numbers.

[0110] For the global features of the image, Figure 2 The figure is shown in the annotation 2 in FIG. Global pooling is performed directly from the feature map with the most complete resolution of the image. Common pooling operations such as average pooling and maximum pooling can be used. This embodiment uses maximum pooling. The pooled global features are also feature mapped to a fixed-length feature space. In this embodiment, the global features are still mapped to a feature space with a dimension of 256. That is, f global ∈R 256 .

[0111] Each object's local features Connect the global features to get the object features In this embodiment, the object feature is a feature vector with a length of 512. globalis the global feature vector obtained in the previous step.

[0112] S4. The object feature sequence is encoded by the encoder to generate the synthetic features of the object feature sequence: the object feature sequence is obtained from the previous step Where N is the number of detected foreground objects. The feature sequence is encoded using an encoder to obtain a feature vector that combines local and global features.

[0113] Furthermore, the encoder here uses a recurrent neural network. The recurrent neural network is used to be able to adapt to different numbers of objects. At the same time, the recurrent neural network can better encode content with sequence properties. The encoder here can use various variants of recurrent neural networks. In this embodiment, a single-layer LSTM recurrent neural network is used as an encoder. The object feature sequence here is actually disordered. In each cycle step of the encoder, a feature vector that has not been selected is randomly selected from the feature sequence as the current step input of the encoder. This cycle is repeated N times, where N is the number of objects. The hidden features output in the Nth step are used as the synthetic features of the object feature sequence.

[0114] S5. Object grasping order prediction: In order to gradually select the objects to be grasped from the feature sequence of the objects and finally restore an object grasping order sequence, the grasping order predictor uses a special recurrent neural network. More specifically, this embodiment uses PointerNet as the order predictor. The predictor receives a hidden feature at each cycle step. and an input feature And use the output features of each step Object feature sequence Calculate the object selection probability distribution vector and select the index with the highest probability as the object output of this step. The probability distribution vector is calculated as follows:

[0115]

[0116] p j =softmax(u j ),p j ∈R N

[0117] In the above calculation, v T , W 1 , W 2 These are all learnable parameters of the network. It is the correlation coefficient of the feature output of each step of the predictor to the object feature sequence. Each step output of the predictor can be correlated with all object features, that is, the entire object feature sequence to calculate the correlation coefficient into a correlation coefficient vector. Among them, p jis the probability distribution calculated from the correlation coefficient vector using the softmax function. It means that in p j The i-th value in represents the predicted probability of grasping object i in the jth cycle step. N is the number of objects in the image, so p j is a vector of length N, that is, p j ∈R N .From p j Select the index with the highest probability As the object selected in this step. At the same time, add the index to the selected index pool so that the index will not be selected repeatedly in the next loop.

[0118] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for predicting object grasping order from images based on deep learning, characterized by: The following steps are involved: Step 1: Use the segmentation network to detect all foreground objects in the image, and output segmentation masks for all foreground objects, retaining the global feature map of the image and the object feature map before outputting the mask; Step 2: Use the segmentation mask of the object to cut out the feature map of the mask position from the object feature map and perform pooling to obtain the local feature vector of the object, perform pooling on the global feature map to obtain the global feature vector, and connect the global feature vector to the local feature vectors of each object to obtain the object feature of each object; Step 3: Use a recurrent neural network as an encoder to feed the object feature vector sequences of all objects into the encoder in sequence, and finally obtain a feature vector of fixed length; Step 4: Use the feature vector encoded in step 3 as the hidden feature, randomly generate an input vector, and input the hidden feature vector into the grasping order predictor. The grasping order predictor is PointerNet. The grasping order predictor receives a fixed-length input vector and a hidden feature obtained in the previous step at each step, and outputs an index. The index points to a feature in the object feature sequence. The object corresponding to the feature is the grasped object predicted in the current step. The number of steps in the loop prediction is the number of objects detected. The final predicted index sequence is the grasping order of the objects.

2. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: The segmentation network includes a binary classifier for separating foreground objects and background objects.

3. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: Step 2 The specific steps are as follows:

21. Use the segmentation network to detect the masks of all foreground objects i , i∈1,2,...N, where N is the number of objects detected by the segmentation network in the current image, and the mask of the foreground object is used to mask the previous feature layer of the predicted object mask, and then feature pooling is performed after masking, and then a linear network is used to convert the number of feature channels to local features of objects with a fixed length To generate local feature vectors for each object; 22. Perform feature pooling directly on the global feature layer with the most complete resolution, and use another linear network to convert the global features to a fixed length to become the global features of the scene f global ; 23. Connect the local features and global features of the object to form the features of the object 4. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: Each cycle of the encoder takes an object feature as input and outputs a corresponding hidden feature. The last encoded hidden feature As the feature encoding of the object feature sequence, in, is the hidden feature of the last encoding output, is the feature of the object, N is the total number of objects, This is the result after encoding the features of all objects.

5. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: Step 4 includes the following steps:

41. Use the LSTM recurrent neural network as the crawling order predictor and use the feature vector encoded in step 3 as the first hidden feature Randomly generate the first input vector m is a fixed input feature length, and the hidden features and input vector are fed into the crawl order predictor; The fetch order predictor accepts a hidden feature at each step and an input vector And output an output vector Where j represents the current position in the jth cycle, that is, the current prediction of the jth grasp target, is the feature corresponding to the grasped target predicted in the previous cycle. When j=1, it means the first step of prediction. That is, generate a random vector as input; 42. For the feature vector output by the grasping order predictor at each step, use the mechanism in PointerNet to calculate an index from the feature sequence of the object, and use the object corresponding to the index as the grasped object in this step; 43. Step 41 to step 42 are looped h times, where h is the number of objects detected, thereby obtaining an index sequence of length h, which is the object grasping order.

6. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: The labeled data is automatically generated in batches with crawling order annotations by using simulation and rendering. The generation method of crawling order annotations uses a heuristic algorithm. The specific steps are as follows:

51. Start building a scene, randomly import n objects into the simulator, and copy m instances of each object. The number of objects and instance data are randomly generated in each scene build; 52. Take the world center of the simulator as the origin and divide it into p×p grids. The size of each square in the grid is the average diameter of the imported object plus a fixed constant d.

53. Place objects at the edge of the grid, randomly select an instance from the object instances each time, place it at the center of the corresponding grid, lift it up along the z-axis, and then randomly translate it in the xy plane, and randomly assign a texture to the instance each time; 54. Repeat step 53 until the grid is full, and then continue to divide the grid into (p-2)×(p-2) grids with the same center and the same size; 56. Repeat steps 53 and 54 until three layers are placed; if the number of instances is insufficient or other conditions are not met, the scene construction is terminated and the next stage is entered. The final scene is in the shape of a suspended pyramid; 57. Take the world coordinate origin as the origin of the sphere, generate a hemispherical surface in the positive direction of the z axis, and uniformly sample o positions on the spherical surface to assume a virtual camera; 58. Render the sampled camera positions one by one, each time the camera faces the world origin. At the same time, each rendering randomly perturbs the light and the material of the object surface, and then renders the image; 59. Filter out objects whose occlusion exceeds the set ratio.

7. The method for predicting object grasping order from images based on deep learning according to claim 1, characterized in that: The labeled data is obtained by using a camera to collect RGB images of stacked scenes at multiple angles, using a trained segmentation network to detect the object boxes of the objects, and annotating the order in which the objects are grasped through manual labeling.

8. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: The labeled data uses images synthesized using a simulation rendering method as data, and the data synthesized using the simulation rendering method is automatically labeled.

9. The method for predicting the object grasping order from an image based on deep learning according to claim 1, characterized in that: The segmentation network can generate a feature map of the image while obtaining the segmentation mask of the objects in the image. At the same time, the segmentation network can detect all foreground objects in the image. At the same time, the PointerNet network used in the grasping order prediction stage performs cyclic prediction on any number of objects.

10. The method for predicting object grasping order from images based on deep learning according to claim 1, characterized in that: When extracting features of an object, the image features of the area where the object is located are used as the features of the object.

Images