Method for detecting task relationship between articles based on deep learning

Abstract

The invention belongs to the technical field of robot cognition, and particularly relates to an inter-article task relation detection method based on deep learning. An adopted task relation detection model comprises a pre-training feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module and a reasoning module. The pre-training feature extraction network performs feature extraction on an input image, the RPN network generates a candidate frame of an object, and a shared sub-graph is obtained through the candidate frame; performing region-of-interest pooling operation on the candidate frame and the shared sub-graph of the object to obtain object features and sub-graph features; performing feature extraction on two object features and one subgraph feature corresponding to the task relationship through an EMA attention module to obtain subject object features, relationship features and object object features; inputting the subject object features, the relation features and the object object features into a three-feature fusion module for fusion to obtain fused relation features; the subject object features, the object object features and the fused relation features are subjected to task relation reasoning through a reasoning module, and a predicted task relation is obtained; and performing category reasoning on the subject and object features through a reasoning module to obtain a prediction category of the object. The method can effectively detect the potential task relationship between the articles, and is used for guiding the task operation of the robot.

Description

Technical Field

[0001] The present invention belongs to the field of robot cognition technology, and specifically relates to a method for detecting task relationships between objects based on deep learning. Background Art

[0002] The premise for a robot to perform complex tasks is to have a good cognitive ability of the surrounding environment, that is, the robot can identify the operational task relationship between objects in the scene. For example, there is a writing task relationship between a pen and paper. This operational task relationship is the key information to guide the robot to perform tasks. However, in the field of robot cognition, more means such as knowledge graphs and prior knowledge are used, which are extremely limited in complex environments. In the field of visual cognition, objects in visual images and their spatial positions can be detected by means such as neural networks, but the task relationship between objects cannot be identified, so that the robot cannot be guided to perform task operations. Therefore, the present invention proposes a method for detecting task relationships between objects based on deep learning. Summary of the invention

[0003] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to provide a method for detecting task relationships between objects based on deep learning.

[0004] The present invention solves the technical problem by adopting the following technical solution: A method for detecting task relationships between objects based on deep learning, comprising the following steps: Step 1: Build a dataset consisting of several images. Each image contains at least two objects and is labeled. The task relationship between objects is structured in the form of triples. Step 2: Build a task relationship detection model, including a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module, and an inference module; The pre-trained feature extraction network extracts features from the input image, and the RPN network generates candidate boxes of the object, and the shared sub-graph is obtained from the candidate boxes; the candidate boxes of the object and the shared sub-graph are respectively subjected to the region of interest pooling operation to obtain object features and sub-graph features; the two object features and one sub-graph feature corresponding to the task relationship are respectively subjected to feature extraction by the three parameter-sharing EMA attention modules to obtain the subject object feature X, the relationship feature Z and the object object feature Y; the subject object feature X, the relationship feature Z and the object object feature Y are input into the three-feature fusion module for fusion to obtain the fused relationship feature; the subject object feature X, the object object feature Y and the fused relationship feature are subjected to task relationship reasoning by the reasoning module to obtain the predicted task relationship; the subject object feature X and the object object feature Y are subjected to category reasoning by the reasoning module to obtain the predicted category of the object; The feature fusion process of the three feature fusion modules is expressed as: (1) (2) (3) (4) (5) In the formula, W represents the fused relational features, M1, M2, M3 and M4 represent intermediate features, MS represents the MS-CAM module, Represents element-by-element addition operation; Step 3: Train the task relationship detection model and use the trained task relationship detection model for task relationship detection between objects.

[0005] Furthermore, assuming that there is a task relationship between any two objects, the union of the candidate boxes of any two objects forms a joint candidate box. The NMS algorithm is used to process all the joint candidate boxes in the image, and the joint candidate boxes with confidence greater than or equal to the confidence threshold are retained. Each retained joint candidate box is a shared subgraph.

[0006] Furthermore, the task relationship reasoning process of the reasoning module is expressed as: (8) In the formula, represents the task relationship prediction score, represents the fully connected layer, Relu represents the ReLU activation function, Represents the convolution operation; The category reasoning process is expressed as: (9) In the formula, Represents the category prediction score, and O represents the features of the object extracted by the EMA attention module.

[0007] Compared with the prior art, the present invention has the following beneficial effects: The present invention combines target detection with task relationship detection. By performing target detection on objects in an image and further detecting the task relationship between objects, the present invention can effectively detect the potential task relationship between objects, which is used to guide the robot to perform corresponding operations, thereby improving the robot's recognition and cognitive capabilities. The EMA attention module realizes further information extraction and produces a clearer feature representation, so that the model can better understand the association and dependency between objects in the scene, thereby improving the model's processing capabilities in multi-task scenarios. Using shared subgraphs to describe the task relationship between objects can reduce the computational complexity of the model without losing information, thereby speeding up the reasoning speed. The three-feature fusion module realizes feature fusion in a layered form, making the information more concentrated, improving the model's understanding of task relationships, and thus improving the accuracy of task relationship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Flowchart for constructing the dataset; Figure 2 A structural diagram of the task relationship detection model; Figure 3 It is the structural diagram of the three-feature fusion module; Figure 4 A visualization of the detection results. DETAILED DESCRIPTION

[0009] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the protection scope of the present application.

[0010] The present invention provides a method for detecting task relationships between objects based on deep learning (hereinafter referred to as method, see Figures 1 to 4 ), including the following steps: Step 1: Use an RGBD camera to collect RGB images, each image contains at least two objects, and obtain more images by changing the spatial position and stacking effect between objects under different backgrounds to form a data set; in this embodiment, a total of 4042 images are collected, of which 3647 images are used for model training and 395 images are used for model testing. Use LABELME software to annotate the images, establish task relationships between objects, and structure the task relationships in the form of triples, such as [pen, write, paper], which indicates that there is a writing task relationship between pen and paper.

[0011] Step 2: Build a task relationship detection model; Figure 2As shown in the figure, the task relationship detection model includes a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module and an inference module; the pre-trained feature extraction network extracts features from the input image, and the RPN network generates a candidate box of the object, which is further processed to obtain a shared sub-image; the candidate box of the object and the shared sub-image are respectively subjected to ROI-Pooling (region of interest pooling) operations to obtain object features and sub-image features; two object features and one sub-image feature corresponding to a task relationship are respectively subjected to feature extraction by the three-parameter shared EMA attention module to obtain subject object features X, relationship features Z and object object features Y; the subject object features X, relationship features Z and object object features Y are input into the three-feature fusion module for fusion to obtain fused relationship features; the subject object features X, object object features Y and the fused relationship features are subjected to task relationship reasoning by the inference module to obtain the predicted task relationship; the object features are subjected to feature extraction by another EMA attention module, and then subjected to category reasoning by the inference module to obtain the predicted category of the object.

[0012] The input image is extracted by the pre-trained feature extraction network, and the features are generated by the RPN network to generate candidate boxes, and then a candidate box is generated for each object; assuming that there is a task relationship between any two objects, a joint candidate box is formed between any two objects, and the joint candidate box is the union of the candidate boxes of the two objects; the NMS algorithm is used to process all the joint candidate boxes in the image to remove overlapping joint candidate boxes and retain joint candidate boxes with confidence greater than or equal to the confidence threshold. Each retained joint candidate box is a shared subgraph, which is used to uniformly describe the task relationship between objects; the candidate box and shared subgraph of the object are respectively extracted by ROI-Pooling (region of interest pooling) operation to obtain object features and subgraph features, and then all object features and subgraph features are obtained; all object features and subgraph features are used to construct a directed fully connected graph to describe the task relationship between objects in the form of graph data. Object features and subgraph features are nodes and edges of the directed fully connected graph, respectively. Therefore, a task relationship is fully described by two object features and one subgraph feature. The operation object of the task is the subject object, and the operated object is the object object. In this embodiment, the feature extraction network adopts VGG16.

[0013] like Figure 3As shown, the three-feature fusion module includes multiple MS-CAM (multi-scale channel attention) modules; the subject object feature X and the relationship feature Z are added element by element and then passed through the first MS-CAM module for feature extraction, and the object object feature Y and the relationship feature Z are added element by element and then passed through the second MS-CAM module for feature extraction; the subject object feature X and the output feature of the first MS-CAM module are added element by element, and then the feature after the element-by-element addition of the relationship feature Z and the output feature of the first MS-CAM module is added element by element to obtain feature M1, and feature M1 is input to the third MS-CAM module for feature extraction; the object object feature Y and the output feature of the second MS-CAM module are added element by element, and then the feature after the element-by-element addition of the relationship feature Z and the output feature of the second MS-CAM module is added element by element to obtain feature M2, and feature M2 is input to the fourth Feature extraction is performed in the MS-CAM module; the output feature of the third MS-CAM module is added element by element to the main object feature X, and then the feature after element by element addition of the output feature of the third MS-CAM module and the relationship feature Z is added element by element to obtain feature M3; the output feature of the fourth MS-CAM module is added element by element to the object feature Y, and then the feature after element by element addition of the output feature of the fourth MS-CAM module and the relationship feature Z is added element by element to obtain feature M4; feature M4 is added element by element to M3, and then input into the fifth MS-CAM module for feature extraction, the output feature of the fifth MS-CAM module is added element by element to feature M3, and then the feature after element by element addition of the output feature of the fifth MS-CAM module and feature M4 is added element by element to obtain the output feature W of the three-feature fusion module, that is, the fused relationship feature. The formula of the three-feature fusion module is expressed as follows: (1) (2) (3) (4) (5) Where, MS represents the MS-CAM module, Represents an element-wise addition operation.

[0014] The structures of the above five MS-CAM modules are the same. In the MS-CAM module, the input feature N1 is sequentially added element by element with the feature extracted by another group point convolution module after passing through the global average pooling and a group point convolution (Group PWConv) module. The feature obtained by element-by-element addition passes through the Sigmoid activation function and is then multiplied element by element with the input feature N1 to obtain the output feature N2 of the MS-CAM module. The formula of the MS-CAM module is expressed as follows: (6) In the formula, represents an element-wise multiplication operation, represents the Sigmoid activation function, represents the cluster point convolution module, Represents a global average pooling operation.

[0015] The cluster point convolution module aggregates channel information through cluster point convolution. In the cluster point convolution module, the input feature N1 is sequentially subjected to cluster point convolution, batch normalization, ReLU activation, cluster point convolution and batch normalization operations to obtain the output feature N3 of the cluster point convolution module; the formula of the cluster point convolution module is expressed as follows: (7) In the formula, PWConv represents the point-group convolution operation, and the convolution kernel size is , r represents the channel reduction ratio, C represents the number of channels; BN represents the batch normalization operation, and Relu represents the ReLU activation function.

[0016] The task relationship detection model considers two detection tasks, one is object detection and the other is task relationship detection. Therefore, the reasoning module includes task relationship reasoning and category reasoning. For task relationship reasoning, the subject object features, the object object features and the fused relationship features are added element by element. The features obtained by element-by-element addition are sequentially passed through 3×3 convolution and fully connected layers for relationship reasoning to obtain the task relationship prediction score, and then the predicted task relationship is obtained; the formula for task relationship reasoning is expressed as follows: (8) In the formula, represents the task relationship prediction score, represents the fully connected layer, Represents a convolution operation.

[0017] For category reasoning, the features extracted by the EMA attention module are sequentially passed through 3×3 convolution and fully connected layers to obtain the category prediction score, and then the predicted category of the object is obtained; the formula for category reasoning is expressed as follows: (9) In the formula, Represents the category prediction score, and O represents the features of the object extracted by the EMA attention module.

[0018] Step 3: Train the task relationship detection model and calculate the loss using the following formula; (10) In the formula, Represents the regression loss of the RPN network, represents the object category prediction loss, represents the task relationship prediction loss. Both the object category prediction loss and the task relationship prediction loss use cross entropy loss. All represent weights; The trained task relationship detection model is used for task relationship detection between objects. Figure 4 A visualization of the detection results.

[0019] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A method for detecting task relationships between objects based on deep learning, characterized in that: The method comprises the following steps: Step 1: Build a dataset consisting of several images. Each image contains at least two objects and is labeled. The task relationship between objects is structured in the form of triples. Step 2: Build a task relationship detection model, including a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module, and an inference module; The pre-trained feature extraction network extracts features from the input image, and the RPN network generates candidate boxes of the object, and the shared sub-graph is obtained from the candidate boxes; the candidate boxes of the object and the shared sub-graph are respectively subjected to the region of interest pooling operation to obtain object features and sub-graph features; the two object features and one sub-graph feature corresponding to the task relationship are respectively subjected to feature extraction by the three parameter-sharing EMA attention modules to obtain the subject object feature X, the relationship feature Z and the object object feature Y; the subject object feature X, the relationship feature Z and the object object feature Y are input into the three-feature fusion module for fusion to obtain the fused relationship feature; the subject object feature X, the object object feature Y and the fused relationship feature are subjected to task relationship reasoning by the reasoning module to obtain the predicted task relationship; the subject object feature X and the object object feature Y are subjected to category reasoning by the reasoning module to obtain the predicted category of the object; The feature fusion process of the three feature fusion modules is expressed as: （1） （2） （3） （4） （5） In the formula, W represents the fused relational features, M1, M2, M3 and M4 represent intermediate features, MS represents the MS-CAM module, Represents element-by-element addition operation; Step 3: Train the task relationship detection model and use the trained task relationship detection model for task relationship detection between objects.

2. The method for detecting task relationships between objects based on deep learning according to claim 1, characterized in that: Assuming that there is a task relationship between any two objects, the union of the candidate boxes of any two objects forms a joint candidate box. The NMS algorithm is used to process all the joint candidate boxes in the image, and the joint candidate boxes with confidence greater than or equal to the confidence threshold are retained. Each retained joint candidate box is a shared subgraph.

3. The method for detecting task relationships between objects based on deep learning according to claim 1 or 2, characterized in that: The task relationship reasoning process of the reasoning module is expressed as: （8） In the formula, represents the task relationship prediction score, represents the fully connected layer, Relu represents the ReLU activation function, Represents the convolution operation; The category reasoning process is expressed as: （9） In the formula, Represents the category prediction score, and O represents the features of the object extracted by the EMA attention module.

4. The method for detecting task relationships between objects based on deep learning according to claim 1, characterized in that: During model training, the loss is calculated using the following formula; （10） In the formula, Represents the regression loss of the RPN network, represents the object category prediction loss, represents the task relationship prediction loss, Both represent weights.

Images