Grabbing task-oriented visual relationship understanding method and system

Abstract

The invention relates to a grabbing task-oriented visual relationship understanding method and a grabbing task-oriented visual relationship understanding system. The grabbing task-oriented visual relationship understanding method comprises the following steps of: acquiring a scene image containing a plurality of objects; performing feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; inputting the basic feature layer into a target detection network, and obtaining classification information and position information of each object; the classification information and the position information are mapped to a basic feature layer to obtain a region of interest of each object, irrelevant object feature pairs are removed through a relation filter, and relevant object feature pairs are obtained; inputting the object feature pairs into a visual relation detection network to obtain a relation matrix; and the object grabbing sequence is determined by combining the grabbing task and the relation matrix, and the target object is grabbed according to the object grabbing sequence. According to the method, the reasonable grabbing sequence is obtained by understanding the visual relation of the multiple target objects in the complex scene, and the safety and reliability of the robot in the grabbing operation process are guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine vision applications, and in particular to a visual relationship understanding method and system for grasping tasks. Background Art

[0002] Intelligent robots, with their high degree of automation and strong adaptability, play an irreplaceable role in many fields such as industrial manufacturing, logistics warehousing, and domestic services. Among them, grasping function is one of their basic capabilities. Before the robot performs the grasping task, a deep understanding of the scene and accurate perception of the environment are crucial. Traditional robot grasping methods rely on point cloud data, integrating information from multiple sensors to achieve three-dimensional modeling of the target object, and using analysis and optimization technology to grasp the relationship between the target object and the surrounding environment or objects, thereby providing a basis for the grasping action.

[0003] However, with the increasing diversity of the types of grasped objects and the complexity of the production environment, the difficulty of grasping tasks is also increasing. Traditional 3D reconstruction methods are not only difficult, but also consume a lot of time and computing resources, and it is difficult to meet the needs of real-time operations. In contrast, grasping methods based on deep learning can quickly generate effective grasping strategies only through RGB images or depth image information. However, there are still some limitations in grasping detection methods based on deep learning, which are mainly reflected in the following two aspects: First, most existing methods only focus on the recognition of grasping positions, but ignore the in-depth understanding of the semantic information of objects in the scene; second, the grasping order between objects is crucial to the success of the task. When the target object cannot be grasped directly, it is necessary to give priority to grasping other objects that may have an impact on the target object. These limitations show that although deep learning methods have made certain progress in the field of grasping, they still need further optimization and improvement in practical engineering applications. Summary of the invention

[0004] In order to solve the problem that most existing methods only focus on the recognition of grasping positions, but ignore the in-depth understanding of the semantic information of objects in the scene, and do not consider the need to prioritize grasping other objects that may have an impact on the target object when the target object cannot be grasped directly, this disclosure proposes a visual relationship understanding method for grasping tasks to solve the above problems.

[0005] According to one aspect of the present disclosure, a visual relationship understanding method for a grasping task is provided, comprising:

[0006] S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by a visual sensor;

[0007] S20, extracting features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0008] S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object;

[0009] S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and according to the region of interest of each object, removing irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs;

[0010] S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix;

[0011] S60 , determining an object grasping sequence in combination with the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0012] Preferably, the basic feature layer is input into the object detection network to obtain the classification information and location information of each object, including:

[0013] The target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoded result into the encoder-decoder structure, and identifies the classification information and position information of each object through a predefined number of target queries.

[0014] Preferably, the acquired classification information and location information are mapped to the basic feature layer to obtain the region of interest of each object, including:

[0015] The position information of each object is mapped to the basic feature layer, the features of the region of interest of each object are extracted, and the features of the region of interest of any two objects are spliced ​​to obtain the joint features of the two objects; by traversing all possible related objects in the scene graph, the features of each object and its joint features are obtained, and the features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object.

[0016] Preferably, irrelevant object feature pairs are eliminated through a relationship filter to obtain relevant object feature pairs, including:

[0017] The relationship between the distance between the two object detection frames and the set threshold is determined through the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as:

[0018]

[0019] In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, and h min represents the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting object detection boxes.

[0020] Preferably, the object feature pairs are input into a visual relationship detection network to obtain a relationship matrix, including:

[0021] By inputting object feature pairs into the visual relationship detection network, the semantic information of the object and its feature small batch data are input into the feature extraction part of the visual relationship detection network, and the input of the visual relationship classification network is obtained through the average pooling, splicing and flattening operations of the deep convolutional network. The input is processed through the fully connected layer to obtain the probability distribution of the visual relationship between object features.

[0022] Preferably, the object feature pairs are input into a visual relationship detection network to obtain a relationship matrix, further comprising:

[0023] According to the probability distribution of the visual relationship of the object features, the visual relationship of any pair of objects in the scene is determined, and the relationship matrix is ​​used to represent all the visual relationships in the scene. The relationship matrix M is expressed as:

[0024] And M ij ={0, 1, 2, 3, 4}

[0025] Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

[0026] According to one aspect of the present disclosure, a visual relationship understanding system for grasping tasks is provided, comprising:

[0027] A scene image acquisition module, which acquires a scene image containing multiple objects, wherein the scene image is obtained by processing images of multiple objects collected by a visual sensor;

[0028] A basic feature layer acquisition module extracts features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0029] A classification information and position information acquisition module inputs the basic feature layer into the target detection network to obtain the classification information and position information of each object;

[0030] The object feature pair acquisition module maps the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object. According to the region of interest of each object, irrelevant object feature pairs are eliminated through the relationship filter to obtain relevant object feature pairs.

[0031] The relationship matrix acquisition module inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix;

[0032] The target object grasping module combines the grasping task and the relationship matrix to determine the object grasping order, and grasps the target object according to the object grasping order.

[0033] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: execute the above-mentioned visual relationship understanding method for grasping tasks.

[0034] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the visual relationship understanding method for grasping tasks is implemented.

[0035] Compared with the prior art, the beneficial effects of the present disclosure are:

[0036] 1) The Transformer-based target detection model disclosed in this paper uses an encoder-decoder architecture to replace the traditional convolutional neural network, eliminating the necessity of the non-maximum suppression (NMS) post-processing step and prior knowledge and constraints such as anchors, thereby simplifying the transmission path of target detection.

[0037] 2) The present invention adopts five ways of expressing the order of grasping objects, replacing the complex description of the spatial relationship of objects, so that the robot can perform the grasping task of the target object in a way that is closer to the grasping habits of humans, thereby significantly improving the grasping efficiency and the naturalness of the operation.

[0038] 3) The present invention uses a relationship filter to pre-screen and remove irrelevant object feature pairs before performing the grasping relationship detection, which effectively reduces the amount of calculation in complex scenes, that is, when there are a large number of objects in the scene, optimizes the processing speed, and improves the efficiency of the overall system.

[0039] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0040] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0042] Figure 1 A flow chart of a visual relationship understanding method for grasping tasks is shown;

[0043] Figure 2 The overall network structure diagram in the embodiment of the present disclosure is shown;

[0044] Figure 3 Shows a block diagram of the structure of a visual relationship understanding system for grasping tasks. DETAILED DESCRIPTION

[0045] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0046] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0047] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0048] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] Example 1

[0051] Based on the above ideas, the present invention proposes a visual relationship understanding method for grasping tasks. Figure 1 A flowchart of a visual relationship understanding method for grasping tasks is shown. The method comprises:

[0052] S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by a visual sensor;

[0053] S20, extracting features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0054] S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object;

[0055] S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and according to the region of interest of each object, removing irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs;

[0056] S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix;

[0057] S60 , determining an object grasping sequence in combination with the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0058] The present disclosure provides a visual relationship understanding method for a grasping task, comprising the following steps:

[0059] S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by a visual sensor.

[0060] In this embodiment, a visual sensor is used to collect RGB images including multiple objects and pre-process the images; wherein the visual sensor is installed at the end of the robotic arm in the manner of an eye on a hand.

[0061] Specifically, step S10 includes:

[0062] S101, installing a camera at the end of a robotic arm in an "eye-on-hand" configuration, obtaining the camera's intrinsic parameters through camera calibration, and obtaining a hand-eye matrix representing the transformation relationship between the camera coordinate system and the robotic arm's end coordinate system through hand-eye calibration.

[0063] S102, moving the robot arm to a set photographing position, collecting RGB images including multiple objects through a visual sensor, and preprocessing the RGB images.

[0064] In this embodiment, when training the target detection network, each epoch of training includes a training mode phase and a verification mode phase, the training mode is used to train the target detection network, and the verification mode is used to evaluate the performance of the target detection network;

[0065] In order to enhance the performance of the target detection network in the training mode, it is usually necessary to perform some random processing on the image to increase the diversity of the training data. In the training mode, the preprocessing includes: cropping the image, adjusting the image size, randomly adjusting the image color information, randomly flipping the image horizontally, and / or normalizing the image; while in the verification mode, it is only necessary to ensure the consistency of the image input. In the verification mode, the preprocessing includes: adjusting the image size and / or normalizing the image. Therefore, the two modes have different preprocessing methods for images; the verification mode is only used to evaluate the performance of the target detection network and will not modify the target detection network.

[0066] S20. Perform feature extraction on the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image, providing key information for subsequent object detection and classification.

[0067] In this embodiment, the overall network structure diagram is as follows: Figure 2 As shown, the process is as follows: the preprocessed scene image is input into the improved EfficientNet-B7 deep convolutional network model. The network framework of the EfficientNet-B7 model is divided into 9 stages in total, of which the 2nd to 8th stages are formed by repeated stacking of MBConv structures. The disclosed embodiment extracts the first 5 stages of the model as the backbone network, performs forward propagation processing to extract the key features of the scene image, and generates the output feature layer of the backbone network, which is defined as the basic feature layer B, which will provide important basic information for subsequent object detection and classification tasks. In this way, the EfficientNet-B7 network can effectively extract rich features from the scene image, which are crucial for understanding the relationship between objects in the scene image, thereby providing strong support for the visual relationship understanding of the grasping task.

[0068] S30: Input the basic feature layer into the target detection network to obtain classification information and location information of each object.

[0069] In this embodiment, the extracted basic feature layer is input into the target detection network constructed by the Transformer-based target detection model DETR to identify all objects in the scene and obtain their target detection frames and corresponding classification information.

[0070] The basic feature layer is input into the target detection network to obtain the classification information and position information of each object, including: the target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoded result into the encoder-decoder structure, and identifies the classification information and position information of each object through a predefined number of target queries.

[0071] Specifically, a target detection network is constructed based on the Transformer-based target detection model DETR. The network implements end-to-end target detection and uses an encoder-decoder architecture to replace the traditional convolutional neural network, thereby eliminating the necessity of the non-maximum suppression NMS post-processing step and prior knowledge and constraints such as anchors, simplifying the target detection process. The backbone network in the disclosed embodiment is used to replace the first image encoder in the DETR architecture, and then each pixel of the output basic feature layer B is mapped to a vector form and position encoded, and input into the following Figure 2 In the Transformer encoder layer shown in the figure, the Transformer encoder uses a multi-head self-attention mechanism to extract global features of the image. The encoding result is then input into the Transformer decoder layer, and the prediction result is obtained through a predefined number of object queries, and the prediction result is input into the prediction head to accurately identify the location information L and semantic information C of each object. The model realizes accurate detection and classification of objects in the scene.

[0072] S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and according to the region of interest of each object, removing irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs.

[0073] In this embodiment, the acquired classification information and position information are mapped to the basic feature layer to obtain the region of interest of each object, including: mapping the position information of each object to the basic feature layer, extracting the region of interest features of each object, splicing the region of interest features of any two objects to obtain the joint features of the two objects; by traversing all possibly related objects in the scene graph, the features of each object and its joint features are obtained, and the features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object.

[0074] Specifically, the location information L of the object is mapped onto the basic feature layer B to extract the region of interest (ROI) features of each object. Furthermore, the ROIs of any two objects are concatenated to form their joint features. These features are then converted to a fixed size and packaged into a set of data, denoted as (F i , F j , U ij ), where i, j∈n and i≠j. By traversing all objects that may be related in the scene graph, the features of each object and their joint features are obtained to form a feature graph, and these data are aggregated into a mini-batch, denoted as mini-batch (F i , F j , U ij ).

[0075] Furthermore, the ROI-Align operation is used to accurately extract the region of interest of each object in the feature map. Compared with the ROI-Pooling operation used in the prior art, the embodiment of the present disclosure uses the ROI-Align operation to extract feature values ​​from the feature map through a bilinear interpolation method, ensuring that detail information is not lost when extracting features.

[0076] To further improve processing efficiency, a relation filter is introduced when traversing all objects in the scene graph. When there are n objects in the graph, the mini-batch size of a graph is As the number of objects increases, the features of irrelevant object feature pairs also increase rapidly. In order to optimize this process, a relation filter is used to pre-eliminate some irrelevant object feature pairs.

[0077] Specifically, irrelevant object feature pairs are removed through the relationship filter to obtain relevant object feature pairs, including:

[0078] The relationship between the distance between the two object detection frames and the set threshold is determined through the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as:

[0079]

[0080] In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, and h min represents the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting target detection boxes. h min With h max The ratio of is used to normalize the minimum distance between two target detection frames to be between 0 and 1. min By normalizing the minimum distance between two target detection frames by a ratio with a fixed constant, the embodiment of the present disclosure can better reflect the relative relationship between two specific target detection frames.

[0081] To ensure the accuracy and adaptability of the relationship filter, the disclosed embodiment further dynamically adjusts the threshold δ according to the network output results. Compared with the fixed threshold δ in existing related research, this dynamic adjustment allows the filter to adaptively optimize its performance according to the actual detection results and scene complexity, thereby maintaining efficient and accurate filtering effects in different application scenarios and conditions.

[0082] S50, inputting the object feature pairs into the visual relationship detection network to obtain a relationship matrix.

[0083] In this embodiment, the object feature pairs are input into the visual relationship detection network to obtain a relationship matrix, which includes the following steps:

[0084] S501, by inputting the object feature pair into the visual relationship detection network, the semantic information of the object and its feature small batch data are input into the feature extraction part of the visual relationship detection network, and the input of the visual relationship classification network is obtained through the average pooling, splicing and flattening operations of the deep convolutional network, and the input is processed through the fully connected layer to obtain the probability distribution of the visual relationship between the object feature pair.

[0085] Specifically, the semantic information C of the object in the image and its feature mini-batch (F i , F j , U ij ) is input into the feature extraction part of the visual relationship detection network. In this process, continue to extract (F i , F j , U ij) features. Among them, the feature extraction part is constructed by the 6th to 8th stages of the EfficientNet-B7 deep convolutional network model, which ensures the continuity and consistency of the target feature extraction. Furthermore, the disclosed embodiment uses three independent feature extraction models to respectively extract the features of two objects and their joint features, ensuring that the weights of each channel are different during the training process. Compared with the prior art, the use of independent weights can effectively analyze the spatial relationship between objects to better adapt to different inputs and enhance the expression ability of the model.

[0086] S502, the three feature layers of the output are subjected to average pooling, concatenation, and flattening operations to obtain the input of the subsequent visual relationship classification network. The input is then processed through a fully connected layer to obtain the probability distribution of object features to visual relationships. In the disclosed embodiment, the visual relationship classification network consists of an input layer, an intermediate layer, and an output layer. The input layer nodes are consistent with the output of the feature extraction part. The output layer consists of five nodes for representing the five classification results in the visual relationship.

[0087] The disclosed embodiment uses five simple visual relationships to replace the complex spatial relationship expression in visual reasoning, and only considers the order of grabbing objects, without considering the positional relationship between objects. For any two objects A and B in the scene, the five visual relationships in the disclosed embodiment are defined as: 1) A and B are unrelated, 2) A is a weak support for B, 3) B is a weak support for A, 4) A is a strong support for B, and 5) B is a strong support for A. Among them, weak support means that the supported object must be grabbed first before grabbing the supporting object; strong support means that the supporting object provides stable support, allowing the supported object to be skipped and the supporting object to be directly grabbed. At this time, the two objects may be grabbed together. In the case of a strong support relationship, the grabbing efficiency can be improved, and there is no need to grab objects one by one until the target object can be grabbed.

[0088] The detection of visual relationships is essentially a five-class classification problem. Based on the probability distribution of visual relationships based on object features, the visual relationship between any pair of objects in the scene is determined, and a relationship matrix is ​​used to represent all visual relationships in the scene. The relationship matrix M is expressed as:

[0089] And M ij ={0, 1, 2, 3, 4}

[0090] Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

[0091] S60 , determining an object grasping sequence in combination with the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

[0092] In this embodiment, the grasping task and the relationship matrix M are combined, the relationship matrix M is queried according to the task target, the support relationship related to the target object is obtained, the grasping order of the task target is obtained, and the objects are grasped in sequence until the task target is successfully grasped.

[0093] The disclosed embodiments provide a visual relationship understanding method for grasping tasks, the purpose of which is to obtain a reasonable grasping sequence by understanding the visual relationship of multiple target objects in complex scenes, thereby ensuring the safety and reliability of the robot in subsequent grasping operations.

[0094] Example 2

[0095] As another aspect of the embodiment of the present disclosure, a visual relationship understanding system 100 for grasping tasks is also provided. Figure 3 As shown, including:

[0096] A scene image acquisition module 1 is used to acquire a scene image containing multiple objects, wherein the scene image is obtained by processing images of multiple objects collected by a visual sensor;

[0097] A basic feature layer acquisition module 2 extracts features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image;

[0098] The classification information and position information acquisition module 3 inputs the basic feature layer into the target detection network to obtain the classification information and position information of each object;

[0099] The object feature pair acquisition module 4 maps the acquired classification information and position information to the basic feature layer to obtain the region of interest of each object, and according to the region of interest of each object, removes irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs;

[0100] The relationship matrix acquisition module 5 inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix;

[0101] The target object grasping module 6 determines the object grasping sequence in combination with the grasping task and the relationship matrix, and grasps the target object according to the object grasping sequence.

[0102] In the absence of any contradiction, the above modules in the system of the embodiment of the present disclosure can implement any implementation method of the method in the above embodiment 1.

[0103] Based on the description of the above embodiments, it can be seen that the embodiments of the present disclosure can achieve the following technical effects:

[0104] 1) The Transformer-based target detection model disclosed in this paper uses an encoder-decoder architecture to replace the traditional convolutional neural network, eliminating the necessity of the non-maximum suppression (NMS) post-processing step and prior knowledge and constraints such as anchors, thereby simplifying the transmission path of target detection.

[0105] 2) The present invention adopts five ways of expressing the order of grasping objects, replacing the complex description of the spatial relationship of objects, so that the robot can perform the grasping task of the target object in a way that is closer to the grasping habits of humans, thereby significantly improving the grasping efficiency and the naturalness of the operation.

[0106] 3) The present invention uses a relationship filter to pre-screen and remove irrelevant object feature pairs before performing the grasping relationship detection, which effectively reduces the amount of calculation in complex scenes, that is, when there are a large number of objects in the scene, optimizes the processing speed, and improves the efficiency of the overall system.

[0107] The disclosed embodiment also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned visual relationship understanding method for grasping tasks. The electronic device can be provided as a terminal, a server, or other forms of devices.

[0108] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the visual relationship understanding method for grasping tasks described above is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0109] Those skilled in the art will understand that in the specific implementation manner of the above-mentioned visual relationship understanding method and system for grasping tasks, the writing order of the steps does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0110] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0111] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A visual relationship understanding method for grasping tasks, characterized in that: The steps include: S10, acquiring a scene image containing multiple objects, wherein the scene image is obtained by processing images of the multiple objects collected by a visual sensor; S20, extracting features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; S30, inputting the basic feature layer into the target detection network to obtain classification information and location information of each object; S40, mapping the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, and according to the region of interest of each object, removing irrelevant object feature pairs through a relationship filter to obtain relevant object feature pairs; S50, inputting the object feature pairs into a visual relationship detection network to obtain a relationship matrix; S60 , determining an object grasping sequence in combination with the grasping task and the relationship matrix, and grasping the target object according to the object grasping sequence.

2. The method according to claim 1, characterized in that Input the basic feature layer into the object detection network to obtain the classification and location information of each object, including: The target detection network adopts an encoder-decoder structure, maps each pixel of the basic feature into a vector form for position encoding, inputs the position encoded result into the encoder-decoder structure, and identifies the classification information and position information of each object through a predefined number of target queries.

3. The method according to claim 1, characterized in that Map the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object, including: The position information of each object is mapped to the basic feature layer, the features of the region of interest of each object are extracted, and the features of the region of interest of any two objects are spliced ​​to obtain the joint features of the two objects; by traversing all possible related objects in the scene graph, the features of each object and its joint features are obtained, and the features of each object and its joint features are converted into features of the same size through the region of interest alignment operation, thereby extracting the region of interest of each object.

4. The method according to claim 1, characterized in that: Irrelevant object feature pairs are removed through relationship filters to obtain relevant object feature pairs, including: The relationship between the distance between the two object detection frames and the set threshold is determined through the relationship filter to determine whether the two object feature pairs are filtered. If F = 0, the object feature pair is filtered. If F = 1, the object feature pair is retained, which is expressed as: In the formula, (o i , o j ) represents the target detection frame of two objects, D is the distance between the two target detection frames, S represents the intersection area between the two target detection frames, F is a filter function, δ is the set threshold, and h min represents the minimum distance between two non-intersecting target detection boxes, h max Indicates the maximum distance between two non-intersecting object detection boxes.

5. The method according to claim 4, characterized in that Input the object feature pairs into the visual relationship detection network to obtain the relationship matrix, including: By inputting object feature pairs into the visual relationship detection network, the semantic information of the object and its feature small batch data are input into the feature extraction part of the visual relationship detection network, and the input of the visual relationship classification network is obtained through the average pooling, splicing and flattening operations of the deep convolutional network. The input is processed through the fully connected layer to obtain the probability distribution of the visual relationship between object features.

6. The method according to claim 5, characterized in that Input the object feature pairs into the visual relationship detection network to obtain the relationship matrix, which also includes: According to the probability distribution of the visual relationship of the object features, the visual relationship of any pair of objects in the scene is determined, and the relationship matrix is ​​used to represent all the visual relationships in the scene. The relationship matrix M is expressed as: And M ij ={0,1,2,3,4} Where n represents the number of objects in the scene, M ij Represents the elements of the relationship matrix, 0, 1, 2, 3, and 4 represent five types of visual relationships.

7. A visual relationship understanding system for grasping tasks, characterized in that: include: A scene image acquisition module, which acquires a scene image containing multiple objects, wherein the scene image is obtained by processing images of multiple objects collected by a visual sensor; A basic feature layer acquisition module extracts features of the scene image based on a deep convolutional network to obtain a basic feature layer of the scene image; A classification information and position information acquisition module inputs the basic feature layer into the target detection network to obtain the classification information and position information of each object; The object feature pair acquisition module maps the acquired classification information and location information to the basic feature layer to obtain the region of interest of each object. According to the region of interest of each object, irrelevant object feature pairs are eliminated through the relationship filter to obtain relevant object feature pairs; The relationship matrix acquisition module inputs the object feature pairs into the visual relationship detection network to obtain the relationship matrix; The target object grasping module combines the grasping task and the relationship matrix to determine the object grasping order, and grasps the target object according to the object grasping order.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the visual relationship understanding method for grasping tasks described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the visual relationship understanding method for grasping tasks described in any one of claims 1 to 6 is implemented.

Images