A method for detecting task relationships between objects based on deep learning

Abstract

The present invention belongs to the field of robot cognitive technology, and specifically is a method for detecting task relationships between objects based on deep learning. The task relationship detection model used includes a pre-trained feature extraction network, a RPN network, an EMA attention module, a three-feature fusion module, and an inference module; the pre-trained feature extraction network extracts features from the input image, the RPN network generates a candidate frame of the object, and a shared subgraph is obtained from the candidate frame; the candidate frame of the object and the shared subgraph are respectively subjected to a region of interest pooling operation to obtain object features and subgraph features; the two object features and one subgraph feature corresponding to the task relationship are respectively subjected to feature extraction by the EMA attention module to obtain subject object features, relationship features, and object object features; the subject object features, relationship features, and object object features are input into the three-feature fusion module for fusion to obtain fused relationship features; the subject object features, object object features, and fused relationship features are subjected to task relationship reasoning by the inference module to obtain a predicted task relationship; the subject and object object features are subjected to category reasoning by the inference module to obtain the predicted category of the object. This method can effectively detect potential task relationships between objects and is used to guide the robot's task operations.

Description

Technical Field

[0001] The present invention belongs to the field of robot cognition technology, and specifically relates to a method for detecting task relationships between objects based on deep learning. Background Art

[0002] A prerequisite for a robot to perform complex tasks is a good cognitive ability of its surrounding environment, that is, the robot can identify the operational task relationships between objects in the scene, for example, the writing task relationship between a pen and paper. This operational task relationship is the key information to guide the robot to perform tasks. However, in the field of robot cognition, knowledge graphs and prior knowledge are more commonly used, and these methods are extremely limited in complex environments. In the field of visual cognition, neural networks and other means can detect objects and their spatial positions in visual images, but they cannot identify the task relationships between objects, making it impossible to guide the robot to perform task operations. Therefore, the present invention proposes a method for detecting task relationships between objects based on deep learning. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the technical problem that the present invention intends to solve is to provide a method for detecting task relationships between objects based on deep learning.

[0004] The present invention solves the technical problem by adopting the following technical solutions:

[0005] A method for detecting task relationships between objects based on deep learning, comprising the following steps:

[0006] Step 1: Build a dataset consisting of several images. Each image contains at least two objects and is labeled. The task relationships between objects are structured in the form of triplets.

[0007] Step 2: Build a task relationship detection model, including a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module, and an inference module;

[0008] The pre-trained feature extraction network extracts features from the input image, and the RPN network generates candidate boxes for the object, from which a shared subgraph is obtained. The candidate boxes and shared subgraphs are pooled through the region of interest to obtain object features and subgraph features. The two object features and one subgraph feature corresponding to the task relationship are extracted through the three parameter-sharing EMA attention modules to obtain subject object features X, relationship features Z, and object object features Y. The subject object features X, relationship features Z, and object object features Y are input into the three-feature fusion module for fusion to obtain the fused relationship features. The subject object features X, object object features Y, and the fused relationship features are subjected to task relationship reasoning in the reasoning module to obtain the predicted task relationship. The subject object features X and object object features Y are subjected to category reasoning in the reasoning module to obtain the predicted category of the object.

[0009] The feature fusion process of the three feature fusion modules is expressed as:

[0010] (1)

[0011] (2)

[0012] (3)

[0013] (4)

[0014] (5)

[0015] Where W represents the fused relational features, M1, M2, M3 and M4 represent intermediate features, MS represents the MS-CAM module, Represents element-by-element addition operation;

[0016] Step 3: Train the task relationship detection model and use the trained task relationship detection model for task relationship detection between items.

[0017] Furthermore, assuming that there is a task relationship between any two objects, the union of the candidate boxes of any two objects forms a joint candidate box. The NMS algorithm is used to process all joint candidate boxes in the image, and the joint candidate boxes with confidence greater than or equal to the confidence threshold are retained. Each retained joint candidate box is a shared subgraph.

[0018] Furthermore, the task relationship reasoning process of the reasoning module is expressed as:

[0019] (8)

[0020] Where, represents the task relationship prediction score, Represents the fully connected layer, Relu represents the ReLU activation function, Represents the convolution operation;

[0021] The category reasoning process is expressed as:

[0022] (9)

[0023] Where, Represents the category prediction score, and O represents the object feature extracted by the EMA attention module.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention combines target detection with task relationship detection. By performing target detection on objects in an image and further detecting the task relationship between objects, it can effectively detect the potential task relationship between objects, which is used to guide the robot to perform corresponding operations, thereby improving the robot's recognition and cognitive capabilities. The EMA attention module realizes further information extraction and produces a clearer feature representation, so that the model can better understand the association and dependency between each object in the scene, thereby improving the model's processing capabilities in multi-task scenarios. Using shared subgraphs to describe the task relationship between objects can reduce the computational complexity of the model without losing information, thereby speeding up the reasoning speed. The three-feature fusion module realizes feature fusion in a layered form, making the information more concentrated, improving the model's understanding of task relationships, and thus improving the accuracy of task relationship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart for constructing the dataset;

[0027] Figure 2 A structural diagram of the task relationship detection model;

[0028] Figure 3 This is the structural diagram of the three-feature fusion module;

[0029] Figure 4 A visualization of the detection results. DETAILED DESCRIPTION

[0030] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to introduce the technical solutions of the present invention in detail and are not intended to limit the scope of protection of the present application.

[0031] The present invention provides a method for detecting task relationships between objects based on deep learning (hereinafter referred to as the method, see Figures 1 to 4 ), including the following steps:

[0032] Step 1: Use an RGBD camera to capture RGB images, with each image containing at least two objects. By varying the spatial position and stacking of objects under different backgrounds, more images are acquired to form a dataset. In this example, a total of 4042 images were collected, of which 3647 were used for model training and 395 for model testing. LABELME software was used to annotate the images, establishing task relationships between objects. These task relationships were structured as triples, such as [pen, writing, paper], which represents the writing task relationship between pen and paper.

[0033] Step 2: Build a task relationship detection model; Figure 2 As shown in the figure, the task relationship detection model includes a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module and an inference module; the pre-trained feature extraction network extracts features from the input image, and the RPN network generates candidate boxes of the object. The candidate boxes are further processed to obtain shared sub-graphs; the candidate boxes of the object and the shared sub-graphs are respectively subjected to ROI-Pooling (region of interest pooling) operations to obtain object features and sub-graph features; the two object features and one sub-graph feature corresponding to a task relationship are respectively subjected to feature extraction by the three-parameter shared EMA attention module to obtain the subject object feature X, relationship feature Z and object object feature Y; the subject object feature X, relationship feature Z and object object feature Y are input into the three-feature fusion module for fusion to obtain the fused relationship feature; the subject object feature X, object object feature Y and the fused relationship feature are subjected to task relationship reasoning by the inference module to obtain the predicted task relationship; the object feature is subjected to feature extraction by another EMA attention module, and then subjected to category reasoning by the inference module to obtain the predicted category of the object.

[0034] The input image is extracted through a pre-trained feature extraction network. These features are then passed through the RPN network to generate candidate boxes, which in turn generate a candidate box for each object. Assuming a task relationship exists between any two objects, a joint candidate box is formed between them. This joint candidate box is the union of the candidate boxes of the two objects. The NMS algorithm is used to process all joint candidate boxes in the image, removing overlapping joint candidate boxes and retaining those with a confidence level greater than or equal to a confidence threshold. Each retained joint candidate box is a shared subgraph, which is used to uniformly describe the task relationship between objects. The object candidate boxes and shared subgraphs are each subjected to feature extraction using a region of interest (ROI) pooling operation, resulting in object features and subgraph features, which in turn yield all object features and subgraph features. A directed fully connected graph is constructed using all object features and subgraph features, describing the task relationship between objects in the form of graph data. Object features and subgraph features are nodes and edges of the directed fully connected graph, respectively. Therefore, a task relationship is fully described by two object features and one subgraph feature. The task operation object is the subject object, and the operated object is the object object. In this embodiment, the feature extraction network uses VGG16.

[0035] like Figure 3As shown, the three-feature fusion module includes multiple MS-CAM (multi-scale channel attention) modules; the subject object feature X and the relationship feature Z are added element by element and then passed through the first MS-CAM module for feature extraction, and the object object feature Y and the relationship feature Z are added element by element and then passed through the second MS-CAM module for feature extraction; the subject object feature X is added element by element to the output feature of the first MS-CAM module, and then the feature obtained by element-by-element addition of the relationship feature Z and the output feature of the first MS-CAM module is added element by element to obtain feature M1, and feature M1 is input to the third MS-CAM module for feature extraction; the object object feature Y is added element by element to the output feature of the second MS-CAM module, and then the feature obtained by element-by-element addition of the relationship feature Z and the output feature of the second MS-CAM module is added element by element to obtain feature M2, and feature M2 is input to the fourth Feature extraction is performed in the MS-CAM module; the output feature of the third MS-CAM module is added element-by-element to the subject object feature X, and then the feature obtained by adding the output feature of the third MS-CAM module and the relationship feature Z is added element-by-element to obtain feature M3; the output feature of the fourth MS-CAM module is added element-by-element to the object object feature Y, and then the feature obtained by adding the output feature of the fourth MS-CAM module and the relationship feature Z is added element-by-element to obtain feature M4; feature M4 is added element-by-element to M3, and then input into the fifth MS-CAM module for feature extraction, the output feature of the fifth MS-CAM module is added element-by-element to feature M3, and then the feature obtained by adding the output feature of the fifth MS-CAM module and feature M4 is added element-by-element to obtain the output feature W of the three-feature fusion module, that is, the fused relationship feature. The formula of the three-feature fusion module is expressed as follows:

[0036] (1)

[0037] (2)

[0038] (3)

[0039] (4)

[0040] (5)

[0041] Where, MS represents the MS-CAM module, Represents an element-wise addition operation.

[0042] The five MS-CAM modules described above share the same structure. In each MS-CAM module, the input feature N1 undergoes global average pooling and a group pointwise convolution (Group PWConv) module. The feature is then element-wise added to the feature extracted from the input feature N1 by another group pointwise convolution module. The resulting element-wise addition passes through a sigmoid activation function and is then element-wise multiplied with the input feature N1 to produce the output feature N2 of the MS-CAM module. The formula for the MS-CAM module is as follows:

[0043] (6)

[0044] Where, represents the element-wise multiplication operation, represents the Sigmoid activation function, represents the cluster convolution module, Represents a global average pooling operation.

[0045] The cluster convolution module aggregates channel information through cluster convolution. In the cluster convolution module, the input feature N1 undergoes cluster convolution, batch normalization, ReLU activation, cluster convolution, and batch normalization operations in sequence to obtain the output feature N3 of the cluster convolution module. The formula of the cluster convolution module is expressed as follows:

[0046] (7)

[0047] In the formula, PWConv represents the point convolution operation, and the convolution kernel size is , r represents the channel reduction ratio, C represents the number of channels; BN represents the batch normalization operation, and Relu represents the ReLU activation function.

[0048] The task relationship detection model considers two detection tasks: object detection and task relationship detection. Therefore, the reasoning module includes task relationship reasoning and category reasoning. For task relationship reasoning, the subject object features, object object features, and fused relationship features are added element by element. The resulting element-by-element features are then passed through a 3×3 convolution and a fully connected layer for relationship reasoning to obtain the task relationship prediction score, and then the predicted task relationship. The formula for task relationship reasoning is expressed as follows:

[0049] (8)

[0050] Where, represents the task relationship prediction score, represents the fully connected layer, Represents a convolution operation.

[0051] For category reasoning, the object features extracted by the EMA attention module are sequentially passed through 3×3 convolution and fully connected layers to obtain the category prediction score, and then the predicted category of the object is obtained; the formula for category reasoning is expressed as follows:

[0052] (9)

[0053] Where, Represents the category prediction score, and O represents the object feature extracted by the EMA attention module.

[0054] Step 3: Train the task relationship detection model and calculate the loss using the following formula;

[0055] (10)

[0056] Where, Represents the regression loss of the RPN network, represents the object category prediction loss, Represents the task relationship prediction loss. Both the object category prediction loss and the task relationship prediction loss use cross entropy loss. All represent weights;

[0057] The trained task relationship detection model is used for task relationship detection between items. Figure 4 A visualization of the detection results.

[0058] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A method for detecting task relationships between objects based on deep learning, characterized in that: The method comprises the following steps: Step 1: Build a dataset consisting of several images. Each image contains at least two objects and is labeled. The task relationships between objects are structured in the form of triplets. Step 2: Build a task relationship detection model, including a pre-trained feature extraction network, an RPN network, an EMA attention module, a three-feature fusion module, and an inference module; The pre-trained feature extraction network extracts features from the input image, and the RPN network generates candidate boxes for the object, from which a shared subgraph is obtained. The candidate boxes and shared subgraphs are pooled through the region of interest to obtain object features and subgraph features. The two object features and one subgraph feature corresponding to the task relationship are extracted through the three parameter-sharing EMA attention modules to obtain subject object features X, relationship features Z, and object object features Y. The subject object features X, relationship features Z, and object object features Y are input into the three-feature fusion module for fusion to obtain the fused relationship features. The subject object features X, object object features Y, and the fused relationship features are subjected to task relationship reasoning in the reasoning module to obtain the predicted task relationship. The subject object features X and object object features Y are subjected to category reasoning in the reasoning module to obtain the predicted category of the object. The feature fusion process of the three feature fusion modules is expressed as: （1） （2） （3） （4） （5） Where W represents the fused relational features, M1, M2, M3 and M4 represent intermediate features, MS represents the MS-CAM module, Represents element-by-element addition operation; Step 3: Train the task relationship detection model and use the trained task relationship detection model for task relationship detection between items.

2. The method for detecting task relationships between objects based on deep learning according to claim 1, characterized in that: Assuming that there is a task relationship between any two objects, the union of the candidate boxes of any two objects forms a joint candidate box. The NMS algorithm is used to process all joint candidate boxes in the image, and the joint candidate boxes with confidence greater than or equal to the confidence threshold are retained. Each retained joint candidate box is a shared subgraph.

3. The method for detecting task relationships between objects based on deep learning according to claim 1 or 2, characterized in that: The task relationship reasoning process of the reasoning module is expressed as: （8） Where, represents the task relationship prediction score, Represents the fully connected layer, Relu represents the ReLU activation function, Represents the convolution operation; The category reasoning process is expressed as: （9） Where, Represents the category prediction score, and O represents the object feature extracted by the EMA attention module.

4. The method for detecting task relationships between objects based on deep learning according to claim 1, characterized in that: During model training, the loss is calculated using the following formula; （10） Where, Represents the regression loss of the RPN network, represents the object category prediction loss, represents the task relationship prediction loss, Both represent weights.

Images