Fine-grained tool recommendation method and device based on interactive task knowledge graph
Through the fine-grained tool recommendation method based on the interactive task knowledge graph and combined with the interactive tool recommendation network, the problem of not considering the object being operated in the existing technology is solved, and the precise recommendation of fine-grained task tools and the effective retrieval of alternative tools is realized, which improves the improvement of the improved The task execution effect.
Patent Information
- Application Number
- CN202111310620.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-11-04
AI Technical Summary
The prior art fails to fully consider the objects to be operated during task execution when recommending interactive tools, resulting in poor recommendation of fine-grained task tools and affecting the task execution effect.
Using a fine-grained tool recommendation method based on the interactive task knowledge graph, we can optimize the embedded feature distance between the tools and the operated object by establishing an interactive task knowledge graph and interactive tool recommendation network, learn common and proprietary features of the tools and the operated object, and optimize the embedded feature distance of the tools and the operated object. Ensure the adaptability of tools and tasks.
It effectively improves the accuracy of fine-grained task tool recommendations, and can retrieve alternative tools with similar functions when the optimal tool does not exist, improving task execution effect.
Smart Images

Figure CN114077692B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning and computer vision, and in particular to a fine-grained tool recommendation method based on an interactive task knowledge graph, and a fine-grained tool recommendation device based on an interactive task knowledge graph. Background Art
[0002] At present, the recommendation of interactive tools for specific interactive tasks is mainly completed by analyzing the affordance of the tools. This type of method does not consider the operated objects in the task execution. For example, the affordance of "bread knife" and "fruit knife" is "cut", so they can both be used as candidate tools for the "cutting" task. However, for the "cutting bread" task, "bread knife" is a more suitable choice because "bread knife" has a serrated blade and is more suitable for cutting soft bread without squashing the bread compared to "fruit knife". Choosing "fruitknife" will affect the execution effect of the task. Therefore, only analyzing the affordance of the tool without considering the operated object cannot guarantee the execution effect of the recommended tool after it is used for the task. The defect of the prior art is that the method for recommending tools for fine-grained tasks does not consider the operated objects in the task execution, resulting in the recommended tools not being well adapted to the fine-grained tasks, affecting the execution effect of the task. Summary of the invention
[0003] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a fine-grained tool recommendation method based on an interactive task knowledge graph, which can well recommend tools for fine-grained tasks and can effectively retrieve alternative tools when the optimal tool does not exist.
[0004] The technical solution of the present invention is: a fine-grained tool recommendation method based on an interactive task knowledge graph, the method comprising the following steps:
[0005] (1) Establish an interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools, and operated objects;
[0006] (2) Recommend fine-grained task-adaptive tools through the interactive tool recommendation network IT-Net;
[0007] (3) By constraining the coarse-grained semantic prediction loss of the tool and the operated object, and using the fine-grained semantic prediction loss, IT-Net can learn the common and unique features of the tool and the operated object;
[0008] (4) By constraining the embedded feature distance between tools that are suitable for fine-grained tasks and the operated objects to be smaller than the embedded feature distance between tools that are not suitable for fine-grained tasks and the operated objects, IT-Net learns the feature relationship between tools that are suitable for fine-grained tasks and the operated objects.
[0009] The present invention establishes an interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools and operated objects, recommends tools adapted to fine-grained tasks through an interactive tool recommendation network IT-Net, constrains the coarse-grained semantic prediction loss of tools and operated objects, and enables IT-Net to learn the common features and proprietary features of tools and operated objects through fine-grained semantic prediction loss. By constraining the embedded feature distance between tools adapted to fine-grained tasks and operated objects to be less than the embedded feature distance between tools not adapted to fine-grained tasks and operated objects, IT-Net learns the feature relationship between tools adapted to fine-grained tasks and operated objects, and takes into account the operated objects in task execution. Therefore, it is possible to recommend tools for fine-grained tasks well, and when the optimal tool does not exist, it is possible to effectively retrieve alternative tools.
[0010] A fine-grained tool recommendation device based on an interactive task knowledge graph is also provided, the device comprising:
[0011] Establish a module, which is configured to build the interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools and operated objects;
[0012] The recommendation module is configured to recommend fine-grained task-adapted tools through the interactive tool recommendation network IT-Net;
[0013] The prediction module is configured to predict the loss by constraining the coarse-grained semantics of the tool and the operated object, and predict the loss by fine-grained semantics, so that IT-Net can learn the common and unique features of the tool and the operated object;
[0014] The learning module is configured to enable IT-Net to learn the feature relationship between the tool adapted to the fine-grained task and the operated object by constraining the embedded feature distance between the tool adapted to the fine-grained task and the operated object to be smaller than the embedded feature distance between the tool not adapted to the fine-grained task and the operated object. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 The conceptual layer representation of the interactive task knowledge graph is shown.
[0016] Figure 2 The positive samples, neutral samples and negative samples in the dataset are shown.
[0017] Figure 3An interactive tool recommendation network is shown.
[0018] Figure 4 Shows the vector
[0019] Figure 5 Three test units for the "cutting bread" task are shown.
[0020] Figure 6 A flowchart of a fine-grained tool recommendation method based on an interactive task knowledge graph according to the present invention is shown. DETAILED DESCRIPTION
[0021] like Figure 6 As shown in FIG. 1 , this fine-grained tool recommendation method based on the interactive task knowledge graph includes the following steps:
[0022] (1) Establish an interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools, and operated objects;
[0023] (2) Recommend fine-grained task-adaptive tools through the interactive tool recommendation network IT-Net;
[0024] (3) By constraining the coarse-grained semantic prediction loss of the tool and the operated object, and using the fine-grained semantic prediction loss, IT-Net can learn the common and unique features of the tool and the operated object;
[0025] (4) By constraining the embedded feature distance between tools that are suitable for fine-grained tasks and the operated objects to be smaller than the embedded feature distance between tools that are not suitable for fine-grained tasks and the operated objects, IT-Net learns the feature relationship between tools that are suitable for fine-grained tasks and the operated objects.
[0026] The present invention establishes an interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools and operated objects, recommends tools adapted to fine-grained tasks through an interactive tool recommendation network IT-Net, constrains the coarse-grained semantic prediction loss of tools and operated objects, and enables IT-Net to learn the common features and proprietary features of tools and operated objects through fine-grained semantic prediction loss. By constraining the embedded feature distance between tools adapted to fine-grained tasks and operated objects to be less than the embedded feature distance between tools not adapted to fine-grained tasks and operated objects, IT-Net learns the feature relationship between tools adapted to fine-grained tasks and operated objects, and takes into account the operated objects in task execution. Therefore, it is possible to recommend tools for fine-grained tasks well, and when the optimal tool does not exist, it is possible to effectively retrieve alternative tools.
[0027] Preferably, in the step (1), for the subclass tools and subclass operated objects in the knowledge graph, semantically consistent image data are crawled from the Internet and manually screened, and the screened image data are annotated with multi-granular semantic labels; the data format of the semantic label is a binary vector with a length of 37, and the index with a component value of 1 is the semantic category of the annotated object; in the semantic label, components with indexes of 0-11 correspond to subclass tools, components with indexes of 12-26 correspond to subclass operated objects, components with indexes of 27-31 correspond to parent class tools, and components with indexes of 32-36 correspond to parent class operated objects.
[0028] Preferably, in step (2), the fine-grained interactive task dataset includes: positive samples, neutral samples and negative samples; each sample is composed of two images from the ITKG instance layer model: one is a subclass tool image, one of the components indexed 0-11 in the semantic label has a value of 1, corresponding to the subclass tool node in the ITKG concept model; the other is a subclass operated object image, one of the components indexed 12-26 in the semantic label has a value of 1, corresponding to the subclass operated object node in the ITKG concept layer; if a sample has If there is a directed edge between the subclass tool node and the subclass operated object node associated with two images in the ITKG conceptual model, then the sample is a positive sample, otherwise it is a neutral sample or a negative sample; the difference between a neutral sample and a negative sample is whether the subclass tool node and the subclass operated object node associated with the two images constituting the sample have a common grandparent node in the ITKG conceptual model. If they have a common grandparent node, then the sample is a neutral sample, otherwise the sample is a negative sample; the tool in the positive sample is defined as the optimal tool, and the tool in the neutral sample is defined as an alternative tool.
[0029] Preferably, in step (2), the data set contains 5,750,000 samples, of which there are 330,000 positive samples, 5,354,000 negative samples, and 66,000 neutral samples.
[0030] Preferably, in step (3), the interactive tool recommendation network includes two parts: (a) feature extraction; and (b) node classification and relationship prediction.
[0031] Preferably, in (a) feature extraction, each batch of the input model contains positive samples and negative samples, and the images corresponding to the head node and the tail node of the positive sample are input into ResNet-50 to extract the corresponding visual features respectively. The visual features corresponding to the head node and the tail node in the positive sample are recorded as: Where i represents the index of the positive sample in the batch, h and t represent the head node and tail node respectively, and the visual features corresponding to the head node and tail node in the negative sample are recorded as: Where j represents the index of the negative sample in the batch.
[0032] Preferably, in (b) node classification and relationship prediction, the features obtained after the visual features of the head node and the tail node of the same sample are input into the multi-layer perceptron are normalized by the sigmoid activation function to obtain the coarse-grained semantics and fine-grained semantic predictions of the node:
[0033]
[0034] in, is the visual feature of the head node of the i-th sample, W h represents the parameters of the multilayer perceptron, g(·) represents the sigmoid activation function, The first S components represent the fine-grained semantic prediction of the image, S is the sum of the number of all sub-category tools and sub-category operated object categories, and the last C components represent the coarse-grained image prediction of the image, C is the sum of the number of all parent-category tools and parent-category operated object categories. represents the probability that IT-Net predicts the head node of the i-th sample belongs to the k-th category of the subclass tool or the subclass operated object, represents the probability that IT-Net predicts that the head node of the i-th sample belongs to the d-th category of the parent class tool or the parent class operated object, is the value of the dth category in the ground-truth vector of the parent tool and parent operated object categories, For the value of the k-th category in the ground-truth vector of the sub-category tool and sub-category manipulated object category, calculate the binary cross entropy loss of the network's prediction for the i-th sample:
[0035]
[0036]
[0037] The classification loss of the head node is and The linear weighted sum of s is the weight coefficient:
[0038]
[0039] Calculate the head nodes of all negative samples The average classification loss of the head node of all samples in a batch is recorded as loss h , the classification loss of the tail nodes of all samples in a batch is averaged and recorded as loss t ;
[0040] Calculate the feature embedding distance for the visual features of the head node and the tail node in the i-th positive sample
[0041]
[0042] Calculate the feature embedding distance of the visual features of the head node and the tail node in the jth negative sample
[0043]
[0044] Among them, W e and W p They represent the parameters of the feature embedding layer respectively. IT-Net defines the feature embedding distance loss based on MarginLoss. e :
[0045]
[0046] Where M is the number of positive samples in the batch, N is the number of negative samples in the batch, and margin is a hyperparameter.
[0047] IT-Net total loss all for:
[0048] loss all =λ h loss h +λ t loss t +λ e loss e (8)
[0049] λ h ,λ t ,λ e is the weight coefficient in the loss function, and the model updates the parameters through back propagation.
[0050] A fine-grained tool recommendation device based on an interactive task knowledge graph is also provided, the device comprising:
[0051] Establish a module, which is configured to build the interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools and operated objects;
[0052] The recommendation module is configured to recommend fine-grained task-adapted tools through the interactive tool recommendation network IT-Net;
[0053] The prediction module is configured to predict the loss by constraining the coarse-grained semantics of the tool and the operated object, and predict the loss by fine-grained semantics, so that IT-Net can learn the common and unique features of the tool and the operated object;
[0054] The learning module is configured to enable IT-Net to learn the feature relationship between the tool adapted to the fine-grained task and the operated object by constraining the embedded feature distance between the tool adapted to the fine-grained task and the operated object to be smaller than the embedded feature distance between the tool not adapted to the fine-grained task and the operated object.
[0055] Preferably, the establishment module performs: for subclass tools and subclass operated objects in the knowledge graph, crawling semantically consistent image data from the Internet and manually screening, and labeling the screened image data with multi-granularity semantic labels; the data form of the semantic label is a binary vector with a length of 37, and the index with a component value of 1 is the semantic category of the labeled object; in the semantic label, components with indexes of 0-11 correspond to subclass tools, components with indexes of 12-26 correspond to subclass operated objects, components with indexes of 27-31 correspond to parent class tools, and components with indexes of 32-36 correspond to parent class operated objects.
[0056] Preferably, in the recommendation module,
[0057] The fine-grained interactive task dataset includes: positive samples, neutral samples and negative samples; each sample consists of two images from the ITKG instance layer model: one is a subclass tool image, one of the components indexed 0-11 in the semantic label has a value of 1, corresponding to the subclass tool node in the ITKG concept model; the other is a subclass operated object image, one of the components indexed 12-26 in the semantic label has a value of 1, corresponding to the subclass operated object node in the ITKG concept layer; if the two images of a sample are associated If there is a directed edge between the subclass tool node and the subclass operated object node in the ITKG conceptual model, then the sample is a positive sample, otherwise it is a neutral sample or a negative sample; the difference between a neutral sample and a negative sample is whether the subclass tool node and the subclass operated object node associated with the two images constituting the sample have a common grandparent node in the ITKG conceptual model. If they have a common grandparent node, then the sample is a neutral sample, otherwise it is a negative sample; the tool in the positive sample is defined as the optimal tool, and the tool in the neutral sample is defined as an alternative tool.
[0058] The present invention is described in more detail below.
[0059] 1. Interactive Task Knowledge Graph
[0060] ITKG includes a concept layer model and an instance layer model. The concept layer model of ITKG is defined at the coarse-grained semantics (parent class) and fine-grained semantics (subclass) levels. Table 1 shows the various tasks, tools, and categories of operated objects at different semantic levels. Table 2 shows the relationships between the nodes in the graph. The relationships in Table 2 are represented by directed edges in the graph. The ITKG concept layer model is shown in Table 2. Figure 1 shown.
[0061] Table 1
[0062]
[0063] Table 2
[0064]
[0065]
[0066] The ITKG instance layer model is built based on the concept layer model. Specifically, for the subclass tools and subclass operated objects in the knowledge graph, semantically consistent image data is crawled from the Internet and manually screened, and multi-granular semantic labels are annotated on the screened image data. The data form of the semantic label is a binary vector with a length of 37, and the index with a component value of 1 is the semantic category of the annotated object. In the semantic label, components with indexes 0-11 correspond to subclass tools, components with indexes 12-26 correspond to subclass operated objects, components with indexes 27-31 correspond to parent class tools, and components with indexes 32-36 correspond to parent class operated objects. Table 3 gives the correspondence between semantic label indexes and semantic categories, as well as the number of images annotated with different semantic categories:
[0067] Table 3
[0068]
[0069]
[0070] As shown in Table 3, the number of images in each subcategory is no less than 1500, and the number of images of all subcategory tools and subcategory operated objects is 75275.
[0071] 2. Fine-grained interactive task dataset
[0072] The fine-grained interactive task dataset includes three types of samples: positive samples, neutral samples, and negative samples. Each sample consists of two images from the ITKG instance layer model: one is a subclass tool image (one of the components indexed 0-11 in the semantic label has a value of 1), corresponding to the subclass tool node in the ITKG concept model; the other is a subclass operated object image (one of the components indexed 12-26 in the semantic label has a value of 1), corresponding to the subclass operated object node in the ITKG concept layer. If the subclass tool node and the subclass operated object node associated with the two images of a sample have a directed edge in the ITKG concept model, then the sample is a positive sample, otherwise it is a neutral sample or a negative sample. The difference between a neutral sample and a negative sample is whether the subclass tool node and the subclass operated object node associated with the two images constituting the sample have a common grandparent node in the ITKG concept model. If they have a common grandparent node, then the sample is a neutral sample, otherwise it is a negative sample.
[0073] Figure 2 For example, three types of samples are Figure 2 As shown in (a), there is a direct relationship “cut” between the tool “bread knife” and the operated object “bread”, so the sample consisting of the tool “bread knife” and the operated object “bread” is a positive sample; Figure 2 In (b), there is no direct connection between the tool "fruit knife" and the operated object "bread", but the tool "fruit knife" and the operated object "bread" have a common grandparent node "cutting task", so the sample composed of "fruit knife" and "bread" is a neutral sample; Figure 2 The image pair in (c) does not belong to the above two cases and is a negative sample. The tool in the positive sample is defined as the optimal tool; the tool in the neutral sample is an alternative tool. The purpose of the neutral sample is to test whether IT-Net can select an alternative tool with similar functions to the optimal tool to complete the task when the optimal tool does not exist.
[0074] According to the above principles, the final data set contains 5,750,000 samples, including 330,000 positive samples, 5,354,000 negative samples, and 66,000 neutral samples.
[0075] 3. Interactive Tool Recommendation Network
[0076] The structure of the interactive tool recommendation network is as follows Figure 3 As shown in Figure 3, the interactive tool recommendation network consists of two parts: (a) feature extraction; (b) node classification and relationship prediction.
[0077] (a) Feature extraction
[0078] Each batch of the input model contains positive samples and negative samples. Here, the positive sample is used as an example to illustrate the feature extraction process. The images corresponding to the head node and tail node of the positive sample are input into ResNet-50 to extract the corresponding visual features. The visual features corresponding to the head node and tail node in the positive sample are recorded as: Where i represents the index of the positive sample in the batch, h and t represent the head node and tail node respectively, and the visual features corresponding to the head node and tail node in the negative sample are recorded as: Where j represents the index of the negative sample in the batch.
[0079] (b) Node classification and relationship prediction
[0080] The visual features of the head node and the tail node of the same sample are input into the multi-layer perceptron and the obtained features are normalized by the sigmoid activation function to obtain the coarse-grained semantic and fine-grained semantic predictions of the node:
[0081]
[0082] in, is the visual feature of the head node of the i-th sample, W h represents the parameters of the multilayer perceptron, and g(·) represents the sigmoid activation function. Figure 4 The vector shown The first S components represent the fine-grained semantic prediction of the image, where S is the sum of the number of all sub-category tools and sub-category operated object categories; the last C components represent the coarse-grained image prediction of the image, where C is the sum of the number of all parent-category tools and parent-category operated object categories. represents the probability that IT-Net predicts the head node of the i-th sample belongs to the k-th category of the subclass tool or the subclass operated object, It represents the probability predicted by IT-Net for the head node of the i-th sample that it belongs to the d-th category of the parent class tool or the parent class operated object. is the value of the dth category in the ground-truth vector of the parent tool and parent operated object categories, For the value of the k-th category in the ground-truth vector of the sub-category tool and sub-category manipulated object category, calculate the binary cross entropy loss of the network's prediction for the i-th sample:
[0083]
[0084]
[0085] The classification loss of the head node is and The linear weighted sum of s is the weight coefficient:
[0086]
[0087] Similarly, the head nodes of all negative samples are calculated The average classification loss of the head node of all samples in a batch is recorded as loss h Similarly, the classification loss of the tail nodes of all samples in a batch is averaged and recorded as loss t .
[0088] Calculate the feature embedding distance for the visual features of the head node and the tail node in the i-th positive sample
[0089]
[0090] Calculate the feature embedding distance of the visual features of the head node and the tail node in the jth negative sample
[0091]
[0092] Among them, W e and W p They represent the parameters of the feature embedding layer respectively. IT-Net defines feature embedding distance loss based on MarginLoss. e :
[0093]
[0094] Where M is the number of positive samples in the batch, N is the number of negative samples in the batch, and margin is a hyperparameter. all for:
[0095] loss all =λ h loss h +λ t loss t +λ e loss e (8)
[0096] λ h ,λ t ,λ e is the weight coefficient in the loss function, and the model updates the parameters through back propagation.
[0097] In order to test the performance of IT-Net's recommendation tools for the 15 fine-grained tasks (subclass tasks) in Table 1, the present invention sets up three test sets (Test-A, Test-B, Test-C). Test-A and Test-B are used to evaluate the performance of recommending optimal tools for fine-grained tasks, and Test-C is used to evaluate the performance of recommending alternative tools for fine-grained tasks when the optimal tool does not exist. For fine-grained tasks, the present invention sets up corresponding test units, each test unit contains multiple samples, and samples belonging to the same test unit share the same image of the operated object. Each test unit in Test-A contains 1 positive sample, multiple negative samples, and multiple neutral samples; each test unit in Test-B contains 1 positive sample and multiple neutral samples; each test unit in Test-C contains 1 neutral sample and multiple negative samples. Test-A contains 15,000 test units, with 15,000 positive samples, 22,000 neutral samples, and 143,000 negative samples; Test-B contains 15,000 test units, with 15,000 positive samples and 22,000 neutral samples; Test-C contains 22,000 test units, with 22,000 neutral samples and 206,000 negative samples.
[0098] Figure 5 Examples of test units from three datasets for testing the "cutting bread" task are given. Figure 5 (a) is the test unit of Test-A, which includes the optimal tool “breadknife” that can form a positive sample with “bread”, the alternative tools “slicing knife” and “fruit knife” that can form a neutral sample with “bread”, and the tools used in other parent tasks that can form a negative sample with “bread”. Figure 5 (b) is the test unit of Test-B, which includes the optimal tool “bread knife” that can form a positive sample with “bread”, and the alternative tools “slicing knife” and “fruit knife” that can form a neutral sample with “bread”. Figure 5 (c) is the test unit of Test-C, which includes the alternative tool "fruit knife" that can form a neutral sample with "bread", as well as tools used in other parent tasks that can form negative samples with "bread". Each test unit of Test-C can only contain one alternative tool.
[0099] This paper uses Hit@N as an indicator to measure the effectiveness of tool recommendation. Hit@N represents the proportion of correct tools (optimal tools in Test-A and Test-B, and alternative tools in Test-C) that appear in the first N positions of the tool recommendation ranking. For the three test sets, N is set to "1", and the Hit@N results of IT-Net on each subclass task and the average results of all subclass tasks are counted. The experimental results are shown in Tables 4, 5, and 6.
[0100] Table 4
[0101]
[0102] Table 5
[0103]
[0104] Table 6
[0105]
[0106] Table 4 shows the test results of IT-Net on Test-A. When N is equal to 1, each sub-category of "cutting task" has achieved an accuracy rate of more than 80%. Except for the "cropping paper" sub-category, the accuracy rates of other sub-category tasks are all above 50%, and all sub-category tasks have an average accuracy rate of 69.1%. Table 5 shows the test results of IT-Net on Test-B. When N is equal to 1, IT-Net has achieved an average accuracy rate of 70.7%. Table 6 shows the test results of IT-Net on Test-C. When N is equal to 1, IT-Net has achieved an average accuracy rate of 81.7%. The experimental results show that IT-Net can well recommend tools for fine-grained tasks, and can effectively retrieve alternative tools when the optimal tool does not exist.
[0107] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.
Claims
1. A fine-grained tool recommendation method based on interactive task knowledge graph, characterized by: The method comprises the following steps: (1) Establish an interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools, and operated objects; (2) Recommend fine-grained task-adaptive tools through the interactive tool recommendation network IT-Net; (3) By constraining the coarse-grained semantic prediction loss of the tool and the operated object, and using the fine-grained semantic prediction loss, IT-Net can learn the common and unique features of the tool and the operated object; (4) By constraining the embedding feature distance between tools that are suitable for fine-grained tasks and the operated objects to be smaller than the embedding feature distance between tools that are not suitable for fine-grained tasks and the operated objects, IT-Net learns the feature relationship between tools that are suitable for fine-grained tasks and the operated objects; In the step (2), the fine-grained interactive task dataset includes: positive samples, neutral samples and negative samples; each sample is composed of two images from the ITKG instance layer model: one is a subclass tool image, and one of the components indexed from 0 to 11 in the semantic label has a value of 1, corresponding to the subclass tool node in the ITKG concept model; the other is a subclass operated object image, and one of the components indexed from 12 to 26 in the semantic label has a value of 1, corresponding to the subclass operated object node in the ITKG concept layer; if the two images of a sample are If there is a directed edge between the subclass tool node and the subclass operated object node associated with the image in the ITKG conceptual model, then the sample is a positive sample, otherwise it is a neutral sample or a negative sample; the difference between a neutral sample and a negative sample is whether the subclass tool node and the subclass operated object node associated with the two images constituting the sample have a common grandparent node in the ITKG conceptual model. If they have a common grandparent node, then the sample is a neutral sample, otherwise it is a negative sample; the tool in the positive sample is defined as the optimal tool, and the tool in the neutral sample is defined as an alternative tool; In step (2), the interactive tool recommendation network includes two parts: (a) feature extraction; (b) node classification and relationship prediction; (a) In feature extraction, each batch of the input model contains positive samples and negative samples. The images corresponding to the head node and tail node of the positive sample are input into ResNet-50 to extract the corresponding visual features. The visual features corresponding to the head node and tail node in the positive sample are recorded as: Where i represents the index of the positive sample in the batch, h and t represent the head node and tail node respectively, and the visual features corresponding to the head node and tail node in the negative sample are recorded as: Where j represents the index of the negative sample in the batch; (b) In node classification and relationship prediction, the visual features of the head node and tail node of the same sample are input into the multi-layer perceptron and the obtained features are normalized by the sigmoid activation function to obtain the coarse-grained semantic and fine-grained semantic predictions of the node: in, is the visual feature of the head node of the i-th sample, W h represents the parameters of the multilayer perceptron, g(·) represents the sigmoid activation function, The first S components represent the fine-grained semantic prediction of the image, S is the sum of the number of all sub-category tools and sub-category operated object categories, and the last C components represent the coarse-grained image prediction of the image, C is the sum of the number of all parent-category tools and parent-category operated object categories. represents the probability that IT-Net predicts the head node of the i-th sample belongs to the k-th category of the subclass tool or the subclass operated object, represents the probability that IT-Net predicts that the head node of the i-th sample belongs to the d-th category of the parent class tool or the parent class operated object, is the value of the dth category in the ground-truth vector of the parent tool and parent operated object categories, For the value of the k-th category in the ground-truth vector of the sub-category tool and sub-category manipulated object category, calculate the binary cross entropy loss of the network's prediction for the i-th sample: The classification loss of the head node is and The linear weighted sum of s is the weight coefficient: Calculate the head nodes of all negative samples The average classification loss of the head node of all samples in a batch is recorded as loss h , the classification loss of the tail nodes of all samples in a batch is averaged and recorded as loss t ; Calculate the feature embedding distance for the visual features of the head node and the tail node in the i-th positive sample Calculate the feature embedding distance of the visual features of the head node and the tail node in the jth negative sample Among them, W e and W p They represent the parameters of the feature embedding layer respectively. IT-Net defines the feature embedding distance loss based on MarginLoss. e : Where M is the number of positive samples in the batch, N is the number of negative samples in the batch, and margin is a hyperparameter. IT-Net total loss all for: loss all =λ h loss h +λ t loss t +λ e loss e (8) λ h ,λ t ,λ e is the weight coefficient in the loss function, and the model updates the parameters through back propagation.
2. The fine-grained tool recommendation method based on interactive task knowledge graph according to claim 1 is characterized in that: In the step (1), for the subclass tools and subclass operated objects in the knowledge graph, semantically consistent image data are crawled from the Internet and manually screened, and multi-granularity semantic labels are annotated on the screened image data; the data format of the semantic label is a binary vector with a length of 37, and the index with a component value of 1 is the semantic category of the annotated object; in the semantic label, components with indexes of 0-11 correspond to subclass tools, components with indexes of 12-26 correspond to subclass operated objects, components with indexes of 27-31 correspond to parent class tools, and components with indexes of 32-36 correspond to parent class operated objects.
3. The fine-grained tool recommendation method based on interactive task knowledge graph according to claim 2 is characterized in that: In step (2), the data set contains 5,750,000 samples, of which 330,000 are positive samples, 5,354,000 are negative samples, and 66,000 are neutral samples.
4. A fine-grained tool recommendation device based on an interactive task knowledge graph, characterized in that: The device executes the method of claim 1 and comprises: Establish a module, which is configured to build the interactive task knowledge graph ITKG to define the multi-granular semantics of interactive tasks, tools and operated objects; The recommendation module is configured to recommend fine-grained task-adapted tools through the interactive tool recommendation network IT-Net; The prediction module is configured to predict the loss by constraining the coarse-grained semantics of the tool and the operated object, and predict the loss by fine-grained semantics, so that IT-Net can learn the common and unique features of the tool and the operated object; The learning module is configured to enable IT-Net to learn the feature relationship between the tool adapted to the fine-grained task and the operated object by constraining the embedded feature distance between the tool adapted to the fine-grained task and the operated object to be smaller than the embedded feature distance between the tool not adapted to the fine-grained task and the operated object.
5. The fine-grained tool recommendation device based on interactive task knowledge graph according to claim 4 is characterized in that: The establishment module executes: for subclass tools and subclass operated objects in the knowledge graph, crawling semantically consistent image data from the Internet and manually screening, and marking the screened image data with multi-granularity semantic labels; the data form of the semantic label is a binary vector with a length of 37, and the index with a component value of 1 is the semantic category of the labeled object; in the semantic label, components with indexes of 0-11 correspond to subclass tools, components with indexes of 12-26 correspond to subclass operated objects, components with indexes of 27-31 correspond to parent class tools, and components with indexes of 32-36 correspond to parent class operated objects.
