A robot visual grasping and detection method, system and medium for object stacking scenes

Abstract

The present invention belongs to the technical field related to robot visual grasping and specifically discloses a robot visual grasping detection method, system, and medium for object stacking scenarios. The method comprises: extracting image features from an RGB image of an object in a work scene; determining an object detection frame and a grasping rectangle based on the image features, wherein the object detection frame represents the category and position of the object in the work scene, and the grasping rectangle represents the grasping posture of the object in the work scene; matching the object detection frame and the grasping rectangle; reflecting the matched object detection frame and grasping rectangle into the image features, and then performing operation relationship detection on the objects in the work scene to obtain an object operation relationship tree; and a robot grasping the objects in the work scene based on the object operation relationship tree, the object detection frame, and the grasping rectangle. The present invention can achieve accurate grasping of the robot in a stacked object scenario.

Description

Technical Field

[0001] The present invention belongs to the technical field related to robot visual grasping, and more specifically, relates to a robot visual grasping detection method, system and medium for object stacking scenarios. Background Art

[0002] Robots have gradually become essential equipment in industries such as industrial manufacturing, logistics and warehousing, and domestic services, playing a vital role in improving production efficiency, reducing costs, and providing safe services. Grasping objects is the first step for robots to perform complex operations. Compared to traditional robotic grasping methods, robotic grasping detection methods based on deep learning and machine vision have reduced scene requirements and reliance on multi-dimensional perceptual information. They only require RGB or depth images of the object to generate usable grasping poses. These advantages have led to their widespread application in robotic grasping tasks in recent years.

[0003] However, as robotic application scenarios become increasingly unstructured and complex, objects in the scene interact with each other in spatial position, known as operational relationships, which significantly impact robotic grasping. Implementing robotic grasping in complex stacked scenarios requires not only basic grasp detection but also the ability to determine the operational relationships between objects to generate a reasonable grasping sequence. However, existing grasp detection methods have limitations when handling these scenarios: 1) Most methods string together various functional modules, such as object detection and grasp detection, into a multi-stage network, resulting in reduced detection performance; 2) The lack of detailed understanding of object operational relationships prevents robots from performing efficient grasp planning similar to humans.

[0004] Therefore, it is of far-reaching significance to refine the operational relationship of objects and effectively integrate it into the grasping detection network, and design an integrated multifunctional, end-to-end grasping detection network to achieve efficient stacked object grasping. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a robot visual grasping detection method, system and medium for object stacking scenarios, the purpose of which is to enable the robot to achieve efficient and robust grasping in object stacking scenarios.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a robot visual grasping and detection method for object stacking scenarios is proposed, comprising the following steps:

[0007] S1, extracting image features of RGB images of objects in the working scene;

[0008] S2. Based on the image features, determine the object detection frame and grasping rectangle respectively. The object detection frame represents the object category and position in the work scene, and the grasping rectangle represents the grasping posture of the object in the work scene;

[0009] S3, matching the object detection frame and the grasping rectangle;

[0010] S4. The matched object detection frame and grab rectangle are reflected in the image features, and then the operation relationship detection of the objects in the work scene is performed to obtain the object operation relationship tree; wherein, the rule for the operation relationship detection of any two objects is:

[0011] If two objects are in a parent-child relationship, the child object must be grabbed before the parent object;

[0012] If two objects have a special parent-child relationship, it means that the parent object provides stable support for the child object at the physical level, so the child object is skipped and the parent object is directly grabbed;

[0013] If the two objects are unrelated, the order of grasping is random;

[0014] S5. The robot grasps objects in the operation scene according to the object operation relationship tree, object detection box and grasping rectangle.

[0015] As a further preferred embodiment, robot visual grasping detection is performed based on a trained neural network model, wherein the neural network model includes a feature extractor, a target detector, a grasping detector, a grasping matcher and an object operation relationship detector, wherein the feature detector is used to extract image features, the target detector is used to determine the object detection frame, the grasping detector is used to determine the grasping rectangle, the grasping matcher is used to match the object detection frame and the grasping rectangle, and the object operation relationship detector is used to determine the object operation relationship tree.

[0016] As a further preference, the object operation relationship detector includes three convolution channels and a fully connected network; when determining the object operation relationship tree, the image features and the matched object detection frame and the grab rectangle are respectively input into the three convolution channels to extract features, and their feature maps are connected together, and then the operation relationship detection is performed through the fully connected network to finally obtain the object operation relationship tree.

[0017] As a further preferred embodiment, the grab matcher matches the object detection box and the grab rectangle based on the intersection over union (IoU) ratio between the object detection box and the grab rectangle.

[0018] As a further preference, when the grab matcher matches the object detection frame and the grab rectangle, for any grab rectangle: the intersection over union (IoU) of the grab rectangle and each object detection frame is calculated respectively; if the IoU is greater than a preset threshold, the object detection frame is matched with the grab rectangle; but if there are multiple IoUs greater than the preset threshold, only the object detection frame with the largest IoU is selected to match with the grab rectangle.

[0019] As a further preference, the target detector adopts the structure of the target detection algorithm SSD.

[0020] As a further preference, the grasping detector adopts a convolution-residual-transposed convolutional network structure.

[0021] As a further preference, the feature detector adopts the ResNet50 network structure.

[0022] According to a second aspect of the present invention, a robot visual grasping and detection system for object stacking scenarios is provided, comprising a processor configured to execute the above-mentioned robot visual grasping and detection method for object stacking scenarios.

[0023] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the robot visual grasping and detection method for object stacking scenarios is implemented.

[0024] In general, the above technical solutions conceived by the present invention have the following technical advantages compared with the existing technology:

[0025] 1. This invention expands the representation of object relationships within a scene, proposing special parent-child relationships. This provides more refined detection of object manipulation relationships, simplifies robot grasping procedures in complex stacking scenarios, and enables more human-like object grasping. This invention can detect special stacking relationships in complex work scenarios and simultaneously output object detection boxes, grasping rectangles, and a refined object manipulation relationship tree, enabling efficient and robust grasping in stacked object scenarios.

[0026] 2. Integrate target detection, operation relationship detection, and grasp detection functions into a neural network framework, and build an end-to-end neural network model suitable for grasp detection in object stacking scenarios, thereby improving grasp detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a robot visual grasping and detection method for object stacking scenarios according to an embodiment of the present invention;

[0028] Figure 2Schematic diagram of the neural network model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0030] The embodiment of the present invention provides a robot visual grasping and detection method for object stacking scenes, such as Figure 1 As shown, the following steps are included:

[0031] S1. Extract the image features F of the RGB image of the object in the working scene.

[0032] Specifically, the RGB image of the object in the working scene is obtained through a visual sensor, such as a Realsense camera or a Kinect camera, to obtain a scene image containing the object.

[0033] Specifically, a feature extractor is used to obtain image features, and a feature detector uses a backbone network based on a convolutional neural network. The process of extracting input image features is as follows: the RGB image collected by the visual sensor is input into a pre-trained backbone network such as ResNet, MobileNet, or VGG, and the output features of a certain convolutional layer or the final output are extracted as the initial features F of the image. Preferably, the feature extractor uses a ResNet50 network. The RGB image is input and passed through the first four residual convolution blocks through the forward propagation. The feature map output by the fourth layer is obtained as the initial features of the image, denoted as F.

[0034] S2. Input the extracted image feature F into the target detector to recognize the object in the work scene and obtain the object detection frame B; input the extracted image feature F into the grasping detector to obtain the grasping posture representation of the object in the work scene, that is, the grasping rectangle G.

[0035] Specifically, the process of realizing object recognition in the working scene is as follows: Since different detectors require different input dimensions, convolution, upsampling or downsampling operations are required before the image feature F is input into the target detector to ensure that the input dimension is consistent with the input dimension required by the detector. After the image feature F is input into the target detector, it undergoes operations such as convolution, pooling, and non-maximum suppression to output a detection frame that reflects the confidence and position of the object category. Preferably, the target detector adopts the structure of the target detection algorithm SSD, and F sequentially passes through 5 layers of additional convolution layers containing 2 convolutions, 2 regularization operations, and 2 Relu activation functions to extract features of the image at different scales. These features reflect the semantic information and detail information of the image respectively. Then, the 5 output features of different scales are integrated and non-maximum suppression operations are performed to obtain the object detection frame B in the scene.

[0036] Specifically, the process of grasping detection of scene objects is as follows: the image feature F is input into the grasping detector, and after a series of convolution operations, the grasping posture representation of the object in the scene is obtained, which is an oriented rectangle. The center point of the rectangle reflects the grasping center point of the object, the side length of the rectangle reflects the grasping width of the object, and the angle between the rectangle and the horizontal line reflects the grasping angle of the object. Preferably, the grasping detector adopts a lightweight structure of convolution-residual-transposed convolution. F first passes through a series of convolution layers to further extract features, and then passes through 5 residual layers to prevent gradient disappearance and dimensionality error. Finally, a transposed convolution is performed to ensure that the input and output feature maps are consistent in size, and the grasping posture representation of the object in the image is output, that is, the grasping rectangle G, including the grasping center coordinates, grasping angle, and grasping width.

[0037] S3. The grasp matcher matches the object detection box with the grasp rectangle based on the intersection-over-union (IoU) of the object detection box B and the grasp rectangle G, thereby assigning the grasp rectangle to the corresponding object, so that each identified object type corresponds to its grasp rectangle one-to-one.

[0038] Specifically, the process of matching the detection box and the grab rectangle of the scene object is as follows: In a stacked scene, there will be multiple object detection boxes and multiple grab rectangles. At this time, the IoU of all object detection boxes and grab rectangles must be calculated by traversing. The object detection box is called obox and the grab rectangle is called gbox. The intersection of obox and gbox is used as the criterion for matching. The IoU is defined as:

[0039]

[0040] When the IoU is greater than a preset threshold (set to 0.9 in this example), the grab rectangle is assigned to the class. If multiple detection boxes exceed the threshold, the grab rectangle is matched with the one with the largest IoU.

[0041] S4. Input the matched object detection box B, grab rectangle G and image feature F into the object operation relationship detector that considers the special stacking relationship to obtain the object operation relationship tree, thereby realizing the refined construction of the object operation relationship in the scene.

[0042] Specifically, the process of obtaining the object operation relationship tree through the object operation relationship detector is as follows: B, G and F are respectively input into three convolution channels to extract features, where each channel consists of two depth-wise separable convolutions, and then their feature maps are connected together so that the matched B and G are reflected on the initial image feature F. Then, the operation relationship is detected through a fully connected network, and finally the object operation relationship tree T in the image is obtained.

[0043] Furthermore, for scene objects, the operational relationship between any two objects a and b is defined as the following five types:

[0044] 1) a is the parent class of b; 2) a is the child class of b; 3) a is the special parent class of b; 4) a is the special child class of b; 5) a and b have no relationship.

[0045] Among them, the "parent-child relationship" in 1) and 2) means that if you want to grab the parent object, you must first grab all its child objects. The "special parent-child relationship" in 3) and 4) means that the parent object provides stable support for the child objects at the physical level, so you can skip the child objects and directly grab the parent object. For example, if an apple is placed in a bowl, there is a special parent-child relationship between them. In this case, if you want to grab the bowl, you don't need to grab the apple first, you can grab it directly. This greatly simplifies the grabbing steps, especially when dealing with complex stacking scenes. 5) When there is no relationship, there is no requirement for the order of grabbing operations.

[0046] S5. Integrate the matched object detection box B, grasping rectangle G, and object operation relationship tree T to obtain the output of the entire neural network model. The robot grasps the object based on the output of the neural network model.

[0047] Specifically, such as Figure 2 As shown in the figure, the neural network model includes the above-mentioned feature extractor, target detector, grasp detector, matcher and object operation relationship detector; when outputting the results, the integration process is as follows: first, indexing is performed based on the object recognition results to obtain the grasping rectangle corresponding to each object class; second, the object grasping order is determined according to the object operation relationship tree; finally, the model will output an RGB image, in which each identified object has its corresponding detection box, grasping rectangle and grasping sequence number.

[0048] In summary, the present invention takes the RGB image of the scene as input, uses a backbone network based on a convolutional neural network to extract features of the image, uses a target detector and a grasping detector based on a deep convolutional neural network to identify objects in the scene and generate their grasping postures, and uses the backbone network output features, object detection frames, and grasping rectangles as inputs of an object operation relationship detector that considers special stacking relationships to achieve refined prediction of object relationships, and finally outputs the object detection frame, grasping rectangle, and grasping order at the same time.

[0049] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A robot visual grasping and detection method for object stacking scenes, characterized in that: The steps include: S1, extracting image features of RGB images of objects in the working scene; S2. Based on the image features, determine the object detection frame and grasping rectangle respectively. The object detection frame represents the object category and position in the work scene, and the grasping rectangle represents the grasping posture of the object in the work scene; S3, matching the object detection frame and the grasping rectangle; S4. The matched object detection frame and grab rectangle are reflected in the image features, and then the operation relationship detection of the objects in the work scene is performed to obtain the object operation relationship tree; wherein, the rule for the operation relationship detection of any two objects is: If two objects are in a parent-child relationship, the child object must be grabbed before the parent object; If two objects have a special parent-child relationship, it means that the parent object provides stable support for the child object at the physical level, so the child object is skipped and the parent object is directly grabbed; If the two objects are unrelated, the order of grasping is random; S5. The robot grasps objects in the operation scene according to the object operation relationship tree, object detection box and grasping rectangle.

2. The robot visual grasping and detection method for object stacking scenes according to claim 1, characterized in that: Robot visual grasping detection is performed based on a trained neural network model, and the neural network model includes a feature extractor, a target detector, a grasp detector, a grasp matcher and an object operation relationship detector, wherein the feature detector is used to extract image features, the target detector is used to determine the object detection frame, the grasp detector is used to determine the grasp rectangle, the grasp matcher is used to match the object detection frame and the grasp rectangle, and the object operation relationship detector is used to determine the object operation relationship tree.

3. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The object operation relationship detector includes three convolution channels and a fully connected network. When determining the object operation relationship tree, the image features and the matched object detection box and grab rectangle are respectively input into the three convolution channels to extract features, and their feature maps are connected together. The operation relationship is then detected through the fully connected network, and the object operation relationship tree is finally obtained.

4. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The grasp matcher matches the object detection box and the grasp rectangle based on the intersection over union (IoU) between the object detection box and the grasp rectangle.

5. The robot visual grasping and detection method for object stacking scenes according to claim 4, characterized in that: When the grab matcher matches the object detection box and the grab rectangle, for any grab rectangle: the intersection over union (IoU) between the grab rectangle and each object detection box is calculated respectively. If the IoU is greater than the preset threshold, the object detection box is matched with the grab rectangle; however, if there are multiple IoUs greater than the preset threshold, only the object detection box with the largest IoU is selected to match with the grab rectangle.

6. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The target detector adopts the structure of the target detection algorithm SSD.

7. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The grasp detector adopts a convolution-residual-transposed convolutional network structure.

8. The robot visual grasping and detection method for object stacking scenes according to claim 2, characterized in that: The feature detector adopts the ResNet50 network structure.

9. A robot visual grasping and detection system for object stacking scenes, characterized in that: The method comprises a processor configured to execute the robot visual grasping and detection method for object stacking scenarios according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot visual grasping and detection method for object stacking scenarios according to any one of claims 1 to 8 is implemented.

Images