A visual relationship understanding method and system for grasping tasks

Abstract

The present disclosure relates to a visual relationship understanding method and system for grasping tasks, comprising: obtaining a scene image containing multiple objects; performing feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; inputting the basic feature layer into a target detection network to obtain classification information and position information of each object; mapping the classification information and position information onto the basic feature layer to obtain a region of interest for each object, eliminating irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs; inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix; determining an object grasping order based on the grasping task and the relationship matrix, and grasping the target objects according to the object grasping order. The disclosed method obtains a reasonable grasping order by understanding the visual relationship of multiple target objects in a complex scene, thereby ensuring the safety and reliability of the robot's grasping operation.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision application technology, and in particular to a visual relationship understanding method and system for grasping tasks. Background Art

[0002] Intelligent robots, with their high degree of automation and strong adaptability, play an irreplaceable role in a variety of fields, including industrial manufacturing, logistics and warehousing, and domestic services. Grasping is one of their fundamental capabilities. Before robots can perform grasping tasks, a deep understanding of the scene and accurate perception of the environment are crucial. Traditional robotic grasping methods rely on point cloud data, integrating information from multiple sensors to create a three-dimensional model of the target object. They then apply analysis and optimization techniques to understand the relationship between the target object and its surroundings or other objects, providing a basis for grasping actions.

[0003] However, with the increasing diversity of grasped object types and the complexity of production environments, grasping tasks are becoming increasingly difficult. Traditional 3D reconstruction methods are not only difficult but also require a large amount of time and computing resources, making them difficult to meet the needs of real-time operations. In contrast, deep learning-based grasping methods can quickly generate effective grasping strategies using only RGB image or depth image information. However, deep learning-based grasping detection methods still have some limitations, mainly reflected in the following two aspects: First, most existing methods focus only on identifying the grasping position, while ignoring the in-depth understanding of the semantic information of objects in the scene; second, the grasping order between objects is crucial to the success of the task. When the target object cannot be grasped directly, it is necessary to prioritize grasping other objects that may have an impact on the target object. These limitations indicate that although deep learning methods have made certain progress in the field of grasping, further optimization and improvement are still needed in practical engineering applications. Summary of the Invention

[0004] To address the problems that most existing methods focus solely on grasping location recognition while neglecting a deep understanding of the semantic information of objects in the scene, and fail to consider the need to prioritize grasping other objects that may have an impact on the target object when the target object cannot be grasped directly, this paper proposes a visual relationship understanding method for grasping tasks to address the above issues.

[0005] According to one aspect of the present disclosure, a visual relationship understanding method for grasping tasks is provided, comprising:

[0006] S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects captured by a visual sensor;

[0007] S20, performing feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0008] S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object;

[0009] S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and removing irrelevant object feature pairs through a relationship filter based on the region of interest of each object to obtain relevant object feature pairs;

[0010] S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix;

[0011] S60 , determining an object grasping sequence based on the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0012] Preferably, the basic feature layer is input into the object detection network to obtain the classification information and location information of each object, including:

[0013] The target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoding result into the encoder-decoder structure, and obtains the classification information and position information of each object through a predefined number of target queries.

[0014] Preferably, the acquired classification information and location information are mapped onto the basic feature layer to obtain the region of interest of each object, including:

[0015] The position information of each object is mapped to the basic feature layer, the region of interest features of each object are extracted, and the region of interest features of any two objects are spliced ​​to obtain the joint features of the two objects; by traversing all possibly related objects in the scene graph, the features of each object and its joint features are obtained, and the features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object.

[0016] Preferably, irrelevant object feature pairs are eliminated through a relationship filter to obtain relevant object feature pairs, including:

[0017] The relationship between the distance between the two object detection frames and the set threshold is judged by the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as:

[0018]

[0019] In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, h min Indicates the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting object detection boxes.

[0020] Preferably, the object feature pairs are input into a visual relationship detection network to obtain a relationship matrix, including:

[0021] By inputting object feature pairs into the visual relationship detection network, the semantic information of the object and its feature small batch data are input into the feature extraction part of the visual relationship detection network, and the input of the visual relationship classification network is obtained through the average pooling, splicing and flattening operations of the deep convolutional network. The input is processed through the fully connected layer to obtain the probability distribution of the visual relationship between object features.

[0022] Preferably, the object feature pairs are input into a visual relationship detection network to obtain a relationship matrix, further comprising:

[0023] According to the probability distribution of visual relationships between object features, the visual relationship between any pair of objects in the scene is determined, and all visual relationships in the scene are represented by a relationship matrix M, which is expressed as:

[0024] And M ij ={0, 1, 2, 3, 4}

[0025] Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

[0026] According to one aspect of the present disclosure, a visual relationship understanding system for grasping tasks is provided, comprising:

[0027] A scene image acquisition module, which acquires a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by the visual sensor;

[0028] A basic feature layer acquisition module extracts features from the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0029] A classification information and location information acquisition module inputs the basic feature layer into the object detection network to obtain the classification information and location information of each object;

[0030] The object feature pair acquisition module maps the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object. Based on the region of interest of each object, irrelevant object feature pairs are eliminated through the relationship filter to obtain relevant object feature pairs;

[0031] The relationship matrix acquisition module inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix;

[0032] The target object grasping module combines the grasping task and the relationship matrix to determine the object grasping order, and grasps the target object according to the object grasping order.

[0033] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-mentioned visual relationship understanding method for grasping tasks.

[0034] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned visual relationship understanding method for grasping tasks is implemented.

[0035] Compared with the prior art, the beneficial effects of the present disclosure are:

[0036] 1) The Transformer-based target detection model disclosed in this paper uses an encoder-decoder architecture to replace the traditional convolutional neural network, eliminating the necessity of the non-maximum suppression (NMS) post-processing step and prior knowledge and constraints such as anchors, thereby simplifying the transmission path of target detection.

[0037] 2) The present invention adopts five ways of expressing the order of grasping objects, replacing the complex description of the spatial relationship of objects, so that the robot can perform the grasping task of the target object in a way that is closer to the grasping habits of humans, thereby significantly improving the grasping efficiency and the naturalness of the operation.

[0038] 3) This disclosure utilizes a relationship filter to pre-screen and eliminate irrelevant object feature pairs before performing grasp relationship detection. This effectively reduces the amount of computation required in complex scenes, especially those with a large number of objects, optimizes processing speed, and improves overall system efficiency.

[0039] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0040] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0042] Figure 1 A flowchart of a visual relationship understanding method for grasping tasks is shown;

[0043] Figure 2 Shows the overall network structure diagram in the embodiment of the present disclosure;

[0044] Figure 3 Shows the structural block diagram of a visual relationship understanding system for grasping tasks. DETAILED DESCRIPTION

[0045] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0046] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0047] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0048] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Example 1

[0051] Based on the above ideas, this paper proposes a visual relationship understanding method for grasping tasks. Figure 1 A flowchart of a visual relationship understanding method for grasping tasks is shown. The method includes:

[0052] S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects captured by a visual sensor;

[0053] S20, performing feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0054] S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object;

[0055] S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and removing irrelevant object feature pairs through a relationship filter based on the region of interest of each object to obtain relevant object feature pairs;

[0056] S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix;

[0057] S60 , determining an object grasping sequence based on the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0058] The present disclosure provides a method for understanding visual relationships for grasping tasks, including the following steps:

[0059] S10: Acquire a scene image containing multiple objects, where the scene image is obtained by processing images of the multiple objects collected by a visual sensor.

[0060] In this embodiment, a visual sensor is used to collect RGB images including multiple objects and pre-process the images; wherein the visual sensor is installed at the end of the robotic arm in the manner of an eye on a hand.

[0061] Specifically, step S10 includes:

[0062] S101, install the camera at the end of the robotic arm in an "eye-on-hand" configuration, obtain the camera's intrinsic parameters through camera calibration, and obtain the hand-eye matrix representing the conversion relationship between the camera coordinate system and the robotic arm end coordinate system through hand-eye calibration.

[0063] S102, moving the robotic arm to a set photographing position, collecting RGB images including multiple objects through a visual sensor, and preprocessing the RGB images.

[0064] In this embodiment, when training the target detection network, each epoch of training includes a training mode phase and a verification mode phase. The training mode is used to train the target detection network, and the verification mode is used to evaluate the performance of the target detection network.

[0065] To enhance the performance of the object detection network in training mode, random image processing is typically performed to increase the diversity of the training data. This preprocessing includes cropping, resizing, random color adjustments, random horizontal flipping, and / or image normalization. Verification mode, on the other hand, only requires ensuring the consistency of the image input. This preprocessing includes resizing and / or normalization. Therefore, the two modes use different image preprocessing methods. Verification mode is only used to evaluate the performance of the object detection network and does not modify the network.

[0066] S20. Perform feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image, providing key information for subsequent object detection and classification.

[0067] In this embodiment, the overall network structure diagram is as follows: Figure 2 As shown, the process is as follows: the preprocessed scene image is input into the improved EfficientNet-B7 deep convolutional network model. The network framework of the EfficientNet-B7 model is divided into 9 stages in total, of which the 2nd to 8th stages are composed of repeated stacking of MBConv structures. The disclosed embodiment extracts the first 5 stages of the model as the backbone network, performs forward propagation processing to extract the key features of the scene image, and generates the output feature layer of the backbone network, which is defined as the basic feature layer B, which will provide important basic information for subsequent object detection and classification tasks. In this way, the EfficientNet-B7 network can effectively extract rich features from the scene image. These features are crucial for understanding the relationship between objects in the scene image, thereby providing strong support for the visual relationship understanding of the grasping task.

[0068] S30: Input the basic feature layer into the target detection network to obtain classification information and location information of each object.

[0069] In this embodiment, the extracted basic feature layer is input into the target detection network built by the Transformer-based target detection model DETR to identify all objects in the scene and obtain their target detection frames and corresponding classification information.

[0070] The basic feature layer is input into the target detection network to obtain the classification information and position information of each object, including: the target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoded result into the encoder-decoder structure, and identifies the classification information and position information of each object through a predefined number of target queries.

[0071] Specifically, a target detection network is constructed based on the Transformer-based target detection model DETR. This network implements end-to-end target detection and adopts an encoder-decoder architecture to replace the traditional convolutional neural network, thereby eliminating the necessity of the non-maximum suppression NMS post-processing step and prior knowledge and constraints such as anchors, simplifying the target detection process. The backbone network in the embodiment of the present disclosure is used to replace the first image encoder in the DETR architecture, and then each pixel of the output basic feature layer B is mapped to a vector form and positionally encoded, and input to the following example: Figure 2 In the Transformer encoder layer shown, the Transformer encoder uses a multi-head self-attention mechanism to extract global image features. The encoded results are then input into the Transformer decoder layer, which uses a predefined number of object queries to generate predictions. These predictions are then input into the prediction head, which accurately identifies the location information L and semantic information C of each object. This model achieves precise detection and classification of objects in the scene.

[0072] S40 , mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and removing irrelevant object feature pairs through a relationship filter based on the region of interest of each object to obtain relevant object feature pairs.

[0073] In this embodiment, the acquired classification information and position information are mapped to the basic feature layer to obtain the region of interest of each object, including: mapping the position information of each object to the basic feature layer, extracting the region of interest features of each object, and splicing the region of interest features of any two objects to obtain the joint features of the two objects; by traversing all potentially related objects in the scene graph, the features of each object and its joint features are obtained, and the features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object.

[0074] Specifically, the object’s location information L is mapped onto the basic feature layer B to extract the region of interest (ROI) features of each object. Furthermore, the ROIs of any two objects are concatenated to form their joint features. These features are then converted to a fixed size and packaged into a set of data, denoted as (F i , F j , U ij ), where i, j∈n and i≠j. By traversing all objects that may be related in the scene graph, the features of each object and its joint features are obtained to form a feature graph, and these data are aggregated into a mini-batch, denoted as mini-batch (F i , F j , U ij ).

[0075] Furthermore, an ROI-Align operation is used to accurately extract the region of interest of each object in the feature map. Compared to the ROI-Pooling operation used in the prior art, the disclosed embodiment uses the ROI-Align operation to extract feature values ​​from the feature map using a bilinear interpolation method, ensuring that detailed information is not lost when extracting features.

[0076] In order to further improve the processing efficiency, a relation filter is introduced when traversing all objects in the scene graph. When there are n objects in the graph, the mini-batch size of a graph is As the number of objects increases, the number of features of irrelevant object feature pairs also increases rapidly. In order to optimize this process, a relation filter is used to pre-eliminate some irrelevant object feature pairs.

[0077] Specifically, irrelevant object feature pairs are removed through the relationship filter to obtain relevant object feature pairs, including:

[0078] The relationship between the distance between the two object detection frames and the set threshold is judged by the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as:

[0079]

[0080] In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, h min Indicates the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting target detection boxes. h min With h max The ratio of is used to normalize the minimum distance between two target detection frames so that it is between 0 and 1. Compared with the existing related technologies using h min By normalizing the minimum distance between two target detection frames by a ratio with a fixed constant, the embodiment of the present disclosure can better reflect the relative relationship between two specific target detection frames.

[0081] To ensure the accuracy and adaptability of the relationship filter, the disclosed embodiments further dynamically adjust the threshold δ based on the network output. Compared to the fixed threshold δ used in existing research, this dynamic adjustment allows the filter to adaptively optimize its performance based on actual detection results and scene complexity, thereby maintaining efficient and accurate filtering in different application scenarios and conditions.

[0082] S50: Input the object feature pairs into the visual relationship detection network to obtain a relationship matrix.

[0083] In this embodiment, the object feature pairs are input into the visual relationship detection network to obtain the relationship matrix, which includes the following steps:

[0084] S501, by inputting object feature pairs into the visual relationship detection network, inputting the semantic information of the object and its feature small batch data into the feature extraction part of the visual relationship detection network, and obtaining the input of the visual relationship classification network through the average pooling, splicing and flattening operations of the deep convolutional network, and processing the input through the fully connected layer to obtain the probability distribution of the visual relationship of the object feature pairs.

[0085] Specifically, the semantic information C of the object in the image and its feature mini-batch (F i , F j , U ij ) is input into the feature extraction part of the visual relationship detection network. In this process, continue to extract (F i , F j , U ij) features. Among them, the feature extraction part is constructed by the 6th to 8th stages of the EfficientNet-B7 deep convolutional network model, which ensures the continuity and consistency of the target feature extraction. Furthermore, the embodiment of the present disclosure uses three independent feature extraction models to respectively extract the features of two objects and their joint features, ensuring that the weights of each channel are different during the training process. Compared with the existing technology, the use of independent weights can effectively analyze the spatial relationship between objects to better adapt to different inputs and enhance the expressive power of the model.

[0086] At step S502, the three feature layers are average pooled, concatenated, and flattened to obtain the input for the subsequent visual relationship classification network. This input is then processed through a fully connected layer to obtain the probability distribution of object features over visual relationships. In the disclosed embodiment, the visual relationship classification network consists of an input layer, an intermediate layer, and an output layer. The input layer nodes are consistent with the output of the feature extraction part. The output layer consists of five nodes, which are used to represent the five classification results of the visual relationship.

[0087] The disclosed embodiment uses five simple visual relationships to replace the complex spatial relationship expression in visual reasoning, and only considers the order of grasping objects, without considering the positional relationship between objects. For any two objects A and B in the scene, the five visual relationships in the disclosed embodiment are defined as: 1) A and B are unrelated, 2) A is weak support for B, 3) B is weak support for A, 4) A is strong support for B, and 5) B is strong support for A. Among them, weak support means that the supported object must be grasped first before grasping the supporting object; strong support means that the supporting object provides stable support, allowing the supported object to be skipped and the supporting object to be grasped directly. At this time, the two objects may be grasped together. In the case of a strong support relationship, the grasping efficiency can be improved, and there is no need to grasp objects one by one until the target object can be grasped.

[0088] The detection of visual relationships is essentially a five-category classification problem. Based on the probability distribution of visual relationships based on object features, the visual relationship between any pair of objects in the scene is determined. All visual relationships in the scene are represented by a relationship matrix, M, which is expressed as:

[0089] And M ij ={0, 1, 2, 3, 4}

[0090] Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

[0091] S60 , determining an object grasping sequence based on the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0092] In this embodiment, the grasping task and the relationship matrix M are combined, and the relationship matrix M is queried according to the task goal to obtain the support relationship related to the target object, obtain the grasping order of the task goal, and grasp the objects in sequence until the task goal is successfully grasped.

[0093] The disclosed embodiments provide a visual relationship understanding method for grasping tasks, the purpose of which is to obtain a reasonable grasping sequence by understanding the visual relationships of multiple target objects in complex scenes, thereby ensuring the safety and reliability of the robot during subsequent grasping operations.

[0094] Example 2

[0095] As another aspect of the embodiment of the present disclosure, a visual relationship understanding system 100 for grasping tasks is also provided. Figure 3 As shown, including:

[0096] A scene image acquisition module 1 is configured to acquire a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects captured by a visual sensor;

[0097] A basic feature layer acquisition module 2 extracts features from the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0098] Classification information and location information acquisition module 3, inputs the basic feature layer into the object detection network to obtain classification information and location information of each object;

[0099] Object feature pair acquisition module 4 maps the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object. Based on the region of interest of each object, irrelevant object feature pairs are eliminated through a relationship filter to obtain relevant object feature pairs;

[0100] Relationship matrix acquisition module 5, which inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix;

[0101] The target object grasping module 6 determines the object grasping order based on the grasping task and the relationship matrix, and grasps the target object according to the object grasping order.

[0102] In the absence of any contradiction, the above modules in the system of the embodiment of the present disclosure can implement any implementation method of the method in the above embodiment 1.

[0103] Based on the description of the above embodiments, it can be seen that the embodiments of the present disclosure can achieve the following technical effects:

[0104] 1) The Transformer-based target detection model disclosed in this paper uses an encoder-decoder architecture to replace the traditional convolutional neural network, eliminating the necessity of the non-maximum suppression (NMS) post-processing step and prior knowledge and constraints such as anchors, thereby simplifying the transmission path of target detection.

[0105] 2) The present invention adopts five ways of expressing the order of grasping objects, replacing the complex description of the spatial relationship of objects, so that the robot can perform the grasping task of the target object in a way that is closer to the grasping habits of humans, thereby significantly improving the grasping efficiency and the naturalness of the operation.

[0106] 3) This disclosure utilizes a relationship filter to pre-screen and eliminate irrelevant object feature pairs before performing grasp relationship detection. This effectively reduces the amount of computation required in complex scenes, especially those with a large number of objects, optimizes processing speed, and improves overall system efficiency.

[0107] The present disclosure also provides an electronic device comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the aforementioned visual relationship understanding method for grasping tasks. The electronic device may be provided as a terminal, server, or other device.

[0108] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the above-mentioned visual relationship understanding method for grasping tasks. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0109] Those skilled in the art will understand that in the above-mentioned visual relationship understanding method and system for grasping tasks in the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0110] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0111] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual relationship understanding method for grasping tasks, characterized by: The steps include: S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects captured by a visual sensor; S20, performing feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object; S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and removing irrelevant object feature pairs through a relationship filter based on the region of interest of each object to obtain relevant object feature pairs; Map the acquired classification and location information onto the basic feature layer to obtain the region of interest for each object, including: The position information of each object is mapped to the basic feature layer, and the features of the region of interest of each object are extracted. The features of the region of interest of any two objects are spliced ​​together to obtain the joint features of the two objects. By traversing all possible related objects in the scene graph, the features of each object and its joint features are obtained. The features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object. Irrelevant object feature pairs are removed through the relationship filter to obtain relevant object feature pairs, including: The relationship between the distance between the two object detection frames and the set threshold is judged by the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as: In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, h min Indicates the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting target detection boxes; S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix; S60 , determining an object grasping sequence based on the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

2. The method according to claim 1, characterized in that Input the basic feature layer into the object detection network to obtain the classification and location information of each object, including: The target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoding result into the encoder-decoder structure, and obtains the classification information and position information of each object through a predefined number of target queries.

3. The method according to claim 1, characterized in that Input the object feature pairs into the visual relationship detection network to obtain the relationship matrix, including: By inputting object feature pairs into the visual relationship detection network, the semantic information of the object and its feature small batch data are input into the feature extraction part of the visual relationship detection network, and the input of the visual relationship classification network is obtained through the average pooling, splicing and flattening operations of the deep convolutional network. The input is processed through the fully connected layer to obtain the probability distribution of the visual relationship between object features.

4. The method according to claim 3, characterized in that Input the object feature pairs into the visual relationship detection network to obtain the relationship matrix, which also includes: According to the probability distribution of visual relationships between object features, the visual relationship between any pair of objects in the scene is determined, and all visual relationships in the scene are represented by a relationship matrix M, which is expressed as: And M ij ={0,1,2,3,4} Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

5. A visual relationship understanding system for grasping tasks, characterized by: include: A scene image acquisition module, which acquires a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by the visual sensor; A basic feature layer acquisition module extracts features from the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; A classification information and location information acquisition module inputs the basic feature layer into the object detection network to obtain the classification information and location information of each object; The object feature pair acquisition module maps the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object. Based on the region of interest of each object, irrelevant object feature pairs are eliminated through the relationship filter to obtain relevant object feature pairs; Map the acquired classification and location information onto the basic feature layer to obtain the region of interest for each object, including: The position information of each object is mapped to the basic feature layer, and the features of the region of interest of each object are extracted. The features of the region of interest of any two objects are spliced ​​together to obtain the joint features of the two objects. By traversing all possible related objects in the scene graph, the features of each object and its joint features are obtained. The features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object. Irrelevant object feature pairs are removed through the relationship filter to obtain relevant object feature pairs, including: The relationship between the distance between the two object detection frames and the set threshold is judged by the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as: In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, h min Indicates the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting target detection boxes; The relationship matrix acquisition module inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix; The target object grasping module combines the grasping task and the relationship matrix to determine the object grasping order, and grasps the target object according to the object grasping order.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the visual relationship understanding method for grasping tasks according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the visual relationship understanding method for grasping tasks according to any one of claims 1 to 4 is implemented.

Images