Object stacking scene-oriented robot vision grabbing detection method and system and medium

Abstract

The invention belongs to the related technical field of robot visual grabbing, and particularly discloses a robot visual grabbing detection method and system for an object stacking scene and a medium, and the method comprises the steps: extracting the image features of an object RGB image in a working scene; based on the image features, an object detection frame and a grabbing rectangle are determined, the object detection frame represents the category and position of an object in the operation scene, and the grabbing rectangle represents the grabbing pose of the object in the operation scene; the object detection frame is matched with the grabbing rectangle; reflecting the matched object detection frame and the grabbed rectangle into image features, and further performing operation relation detection on objects in the operation scene to obtain an object operation relation tree; and the robot grabs the object in the operation scene according to the object operation relation tree, the object detection frame and the grabbing rectangle. Accurate grabbing of the robot under the object stacking scene can be achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to robot visual grasping, and more specifically, relates to a robot visual grasping detection method, system and medium for object stacking scenes. Background Art

[0002] Robots have gradually become indispensable key equipment in industries such as industrial manufacturing, logistics warehousing, and domestic services. They play an important role in improving production efficiency, reducing costs, and providing safety services. Grasping objects is the first step for robots to achieve complex operations. Compared with traditional robot grasping methods, robot grasping detection methods based on deep learning and machine vision reduce the requirements for scenes and the reliance on multi-dimensional perception information. Only the RGB or depth image of the object is needed to generate a usable object grasping posture. These advantages have made it widely used in robot grasping tasks in recent years.

[0003] However, as the application scenarios of robots tend to be unstructured and complex, the objects in the scene have interactive relationships in spatial position, that is, operational relationships, which will have a significant impact on the robot's grasping. To achieve robot grasping in complex stacking scenarios, it is necessary not only to implement basic grasping detection, but also to judge the operational relationship between objects to generate a reasonable grasping order. However, existing grasping detection methods still have certain limitations when dealing with these scenarios: 1) Most methods connect various functional modules, such as target detection and grasping detection, into a multi-stage network, resulting in reduced detection performance; 2) The lack of subdivision of object operational relationships makes it impossible for robots to perform efficient grasping planning similar to humans.

[0004] Therefore, it is of far-reaching significance to refine the operational relationship of objects and effectively integrate it into the grasping detection network, and to design an integrated multifunctional, end-to-end grasping detection network to achieve efficient stacked object grasping. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a robot visual grasping detection method, system and medium for object stacking scenarios, the purpose of which is to enable the robot to achieve efficient and robust grasping in object stacking scenarios.

[0006] To achieve the above object, according to a first aspect of the present invention, a robot visual grasping and detection method for object stacking scenes is proposed, comprising the following steps:

[0007] S1, extracting image features of RGB images of objects in the working scene;

[0008] S2. Based on the image features, determine the object detection frame and the grasping rectangle respectively. The object detection frame represents the category and position of the object in the working scene, and the grasping rectangle represents the grasping posture of the object in the working scene;

[0009] S3, matching the object detection frame and the grab rectangle;

[0010] S4. The matched object detection frame and the grab rectangle are reflected in the image features, and then the operation relationship detection is performed on the objects in the operation scene to obtain an object operation relationship tree; wherein, for any two objects, the rule for the operation relationship detection is:

[0011] If two objects are in a parent-child relationship, the child object must be grabbed before the parent object;

[0012] If two objects are in a special parent-child relationship, it means that the parent object provides stable support for the child object at the physical level, so the child object is skipped and the parent object is directly grabbed;

[0013] If the two objects are not related, the order of grasping is random;

[0014] S5. The robot grasps objects in the working scene according to the object operation relationship tree, object detection box and grasping rectangle.

[0015] As a further preferred embodiment, robot visual grasping detection is performed based on a trained neural network model, and the neural network model includes a feature extractor, a target detector, a grasping detector, a grasping matcher and an object operation relationship detector, wherein the feature detector is used to extract image features, the target detector is used to determine the object detection frame, the grasping detector is used to determine the grasping rectangle, the grasping matcher is used to match the object detection frame and the grasping rectangle, and the object operation relationship detector is used to determine the object operation relationship tree.

[0016] As a further preference, the object operation relationship detector includes three convolution channels and a fully connected network; when determining the object operation relationship tree, the image features and the matched object detection frame and the grab rectangle are respectively input into the three convolution channels to extract features, and their feature maps are connected together, and then the operation relationship detection is performed through the fully connected network to finally obtain the object operation relationship tree.

[0017] As a further preference, the grab matcher matches the object detection frame and the grab rectangle based on an intersection over union (IoU) ratio between the object detection frame and the grab rectangle.

[0018] As a further preference, when the grasp matcher matches the object detection frame and the grasp rectangle, for any grasp rectangle: the intersection over union (IoU) between the grasp rectangle and each object detection frame is calculated respectively; if the IoU is greater than a preset threshold, the object detection frame is matched with the grasp rectangle; but if there are multiple IoUs greater than the preset threshold, only the object detection frame with the largest IoU is selected to match the grasp rectangle.

[0019] As a further preference, the target detector adopts the structure of the target detection algorithm SSD.

[0020] As a further preference, the grasping detector adopts a convolution-residual-transposed convolutional network structure.

[0021] As a further preference, the feature detector adopts the ResNet50 network structure.

[0022] According to a second aspect of the present invention, a robot visual grasping and detection system for object stacking scenes is provided, comprising a processor, wherein the processor is used to execute the above-mentioned robot visual grasping and detection method for object stacking scenes.

[0023] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the robot visual grasping and detection method for object stacking scenarios is implemented.

[0024] In general, the above technical solution conceived by the present invention has the following technical advantages compared with the prior art:

[0025] 1. The present invention expands the expression of object relationships in the scene and proposes special parent-child relationships, thereby providing more sophisticated object operation relationship detection, simplifying the robot's grasping steps in complex stacking scenes, and enabling the robot to achieve object grasping that is more in line with human habits. The present invention can detect special stacking relationships in complex operating scenes, and simultaneously output object detection frames, grasping rectangles, and refined object operation relationship trees, enabling the robot to achieve efficient and robust grasping in object stacking scenes.

[0026] 2. The target detection, operation relationship detection and grasping detection functions are integrated into a neural network framework, and an end-to-end neural network model suitable for grasping detection in object stacking scenarios is constructed, which improves the grasping detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a robot visual grasping and detection method for object stacking scenarios according to an embodiment of the present invention;

[0028] Figure 2Schematic diagram of the neural network model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] The embodiment of the present invention provides a robot visual grasping and detection method for object stacking scenes, such as Figure 1 As shown, the following steps are included:

[0031] S1. Extract the image features F of the RGB image of the object in the working scene.

[0032] Specifically, the RGB image of the object in the working scene is obtained through a visual sensor, such as using a Realsense camera or a Kinect camera to obtain a scene image containing the object.

[0033] Specifically, a feature extractor is used to obtain image features, and a feature detector uses a backbone network based on a convolutional neural network. The process of extracting input image features is as follows: the RGB image collected by the visual sensor is input into a pre-trained backbone network such as ResNet, MobileNet or VGG, and the output features of a certain convolution layer or the final output are extracted as the initial features F of the image. Preferably, the feature extractor uses a ResNet50 network, and the RGB image is input. After forward propagation of the first four residual convolution blocks, the feature map output by the fourth layer is obtained as the initial features of the image, which is recorded as F.

[0034] S2. Input the extracted image feature F into the target detector to realize the recognition of the object in the working scene and obtain the object detection frame B; input the extracted image feature F into the grasping detector to obtain the grasping posture representation of the object in the working scene, that is, the grasping rectangle G.

[0035] Specifically, the process of realizing object recognition in the working scene is as follows: Since different detectors require different input dimensions, convolution, upsampling or downsampling operations are required before the image feature F is input into the target detector to ensure that the input dimension is consistent with the input dimension required by the detector. After the image feature F is input into the target detector, it is subjected to operations such as convolution, pooling, and non-maximum suppression to output a detection frame that reflects the confidence and position of the object category. Preferably, the target detector adopts the structure of the target detection algorithm SSD, and F sequentially passes through 5 layers of additional convolution layers including 2 convolutions, 2 regularization operations, and 2 Relu activation functions to extract the features of the image at different scales. These features reflect the semantic information and detail information of the image respectively, and then the output features of the 5 different scales are integrated and non-maximum suppression operations are performed to obtain the object detection frame B in the scene.

[0036] Specifically, the process of grasping detection of scene objects is as follows: input the image feature F into the grasping detector, and obtain the grasping posture representation of the object in the scene through a series of convolution operations, which is an oriented rectangle. The center point of the rectangle reflects the grasping center point of the object, the side length of the rectangle reflects the grasping width of the object, and the angle between the rectangle and the horizontal line reflects the grasping angle of the object. Preferably, the grasping detector adopts a lightweight structure of convolution-residual-transposed convolution. F first passes through a series of convolution layers to further extract features, and then passes through 5 residual layers to prevent gradient disappearance and dimensionality error. Finally, transposed convolution is performed to ensure that the input and output feature map sizes are consistent, and the grasping posture representation of the object in the picture is output, that is, the grasping rectangle G, including the grasping center coordinates, grasping angle, and grasping width.

[0037] S3. The grasp matcher matches the object detection box with the grasp rectangle according to the intersection over union (IoU) of the object detection box B and the grasp rectangle G, thereby assigning the grasp rectangle to the corresponding object so that each type of object identified corresponds to its grasp rectangle one by one.

[0038] Specifically, the process of matching the detection box and grab rectangle of the scene object is as follows: There will be multiple object detection boxes and multiple grab rectangles in the stacked scene. At this time, the IoU of all object detection boxes and grab rectangles must be calculated by traversal. The object detection box is called obox, the grab rectangle is called gbox, and the intersection of obox and gbox is used as the criterion for matching. The IoU is defined as:

[0039]

[0040] When the IoU is greater than a preset threshold (set to 0.9 in this embodiment), the grab rectangle is assigned to the class. If multiple detection boxes exceed the threshold, the grab rectangle is matched with the one with the largest IoU.

[0041] S4. Input the matched object detection box B, the grab rectangle G and the image feature F into the object operation relationship detector that considers the special stacking relationship to obtain the object operation relationship tree, thereby realizing the refined construction of the object operation relationship in the scene.

[0042] Specifically, the process of obtaining the object operation relationship tree through the object operation relationship detector is as follows: B, G and F are respectively input into three convolution channels to extract features, where each channel consists of two depth-wise separable convolutions, and then their feature maps are connected together so that the matched B and G are reflected on the initial image feature F, and then the operation relationship is detected through a fully connected network, and finally the object operation relationship tree T in the image is obtained.

[0043] Furthermore, for scene objects, the operational relationship between any two objects a and b is defined as the following five types:

[0044] 1) a is the parent class of b; 2) a is the child class of b; 3) a is the special parent class of b; 4) a is the special child class of b; 5) a has no relationship with b.

[0045] Among them, the "parent-child relationship" of 1) and 2) means that if you want to grab the parent object, you must first grab all its child objects. The "special parent-child relationship" of 3) and 4) means that the parent object provides stable support for the child object at the physical level, so you can skip the child object and directly grab the parent object. For example, if an apple is placed in a bowl, there is a special parent-child relationship between them. In this case, if you want to grab the bowl, you don’t need to grab the apple first, you can grab it directly. This greatly simplifies the grabbing steps, especially when dealing with complex stacking scenes. 5) When there is no relationship, there is no requirement for the order of grabbing operations.

[0046] S5. Integrate the matched object detection box B, the grasping rectangle G, and the object operation relationship tree T to obtain the output of the entire neural network model. The robot grasps the object according to the output of the neural network model.

[0047] Specifically, Figure 2 As shown in the figure, the neural network model includes the above-mentioned feature extractor, target detector, grasp detector, matcher and object operation relationship detector; when outputting the results, the integration process is as follows: first, indexing is performed based on the object recognition results to obtain the grasping rectangle corresponding to each object class; second, the object grasping order is determined according to the object operation relationship tree; finally, the model will output an RGB image, in which each identified object has its corresponding detection box, grasping rectangle and grasping sequence number.

[0048] In summary, the present invention takes the RGB image of the scene as input, uses a backbone network based on a convolutional neural network to extract features of the image, uses a target detector and a grasping detector based on a deep convolutional neural network to identify objects in the scene and generate their grasping postures, and uses the output features of the backbone network, object detection boxes, and grasping rectangles as inputs of an object operation relationship detector that considers special stacking relationships to achieve refined prediction of object relationships, and finally outputs the object detection box, grasping rectangle, and grasping order at the same time.

[0049] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A robot visual grasping and detection method for object stacking scenes, characterized in that: The steps include: S1, extracting image features of RGB images of objects in the working scene; S2. Based on the image features, determine the object detection frame and the grasping rectangle respectively. The object detection frame represents the category and position of the object in the working scene, and the grasping rectangle represents the grasping posture of the object in the working scene; S3, matching the object detection frame and the grab rectangle; S4. The matched object detection frame and the grab rectangle are reflected in the image features, and then the operation relationship detection is performed on the objects in the operation scene to obtain an object operation relationship tree; wherein, for any two objects, the rule for the operation relationship detection is: If two objects are in a parent-child relationship, the child object must be grabbed before the parent object; If two objects are in a special parent-child relationship, it means that the parent object provides stable support for the child object at the physical level, so the child object is skipped and the parent object is directly grabbed; If the two objects are not related, the order of grasping is random; S5. The robot grasps objects in the working scene according to the object operation relationship tree, object detection box and grasping rectangle.

2. The robot visual grasping and detection method for object stacking scenes according to claim 1, characterized in that: Robot visual grasping detection is performed based on a trained neural network model, and the neural network model includes a feature extractor, a target detector, a grasping detector, a grasping matcher and an object operation relationship detector, wherein the feature detector is used to extract image features, the target detector is used to determine the object detection frame, the grasping detector is used to determine the grasping rectangle, the grasping matcher is used to match the object detection frame and the grasping rectangle, and the object operation relationship detector is used to determine the object operation relationship tree.

3. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The object operation relationship detector includes three convolution channels and a fully connected network. When determining the object operation relationship tree, the image features and the matched object detection box and the grab rectangle are respectively input into the three convolution channels to extract features, and their feature maps are connected together. The operation relationship detection is then performed through the fully connected network to finally obtain the object operation relationship tree.

4. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The grasp matcher matches the object detection box and the grasp rectangle based on the intersection over union (IoU) between the object detection box and the grasp rectangle.

5. The robot visual grasping and detection method for object stacking scenes according to claim 4, characterized in that: When the grasp matcher matches the object detection frame and the grasp rectangle, for any grasp rectangle: the intersection over union (IoU) between the grasp rectangle and each object detection frame is calculated respectively. If the IoU is greater than the preset threshold, the object detection frame is matched with the grasp rectangle; but if there are multiple IoUs greater than the preset threshold, only the object detection frame with the largest IoU is selected to match with the grasp rectangle.

6. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The target detector adopts the structure of the target detection algorithm SSD.

7. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The grasp detector adopts a convolution-residual-transposed convolutional network structure.

8. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The feature detector adopts the ResNet50 network structure.

9. A robot visual grasping and detection system for object stacking scenes, characterized in that: It includes a processor, and the processor is used to execute the robot visual grasping and detection method for object stacking scenes according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot visual grasping and detection method for object stacking scenes as described in any one of claims 1 to 8 is implemented.

Images