Multimodal knowledge graph multi-hop reasoning method, system, terminal and storage medium

Abstract

The present invention discloses a multi-hop reasoning method, system, terminal and storage medium of a multimodal knowledge graph, wherein the method comprises: generating structural embedding, category embedding and description embedding for entities and relationships in the multimodal knowledge graph, and fusing the three embeddings to obtain a multimodal embedding; constructing a structural data agent, a category data agent and a description data agent; using the multimodal knowledge graph as an environment for agent interaction, using a MAPF-based reinforcement learning framework to make three agents interact with the environment for iterative learning, and obtaining a policy network of the three trained agents; and using the policy network of the three trained agents to perform reasoning and prediction in the multimodal knowledge graph based on a query request. The present invention can quickly make reasonable reasoning decisions in complex scenarios and find the best reasoning path by fusing data of different modalities and using multiple agents to collaborate and explore different reasoning paths in the knowledge graph respectively.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-modal knowledge graph multi-hop reasoning method, system, terminal and storage medium. Background Art

[0002] With the continuous development of artificial intelligence, knowledge graphs have been widely used in intelligent question-answering systems, information retrieval and other fields as a tool for representing knowledge structured information. In recent years, researchers have proposed many methods to use existing information to infer facts that do not exist in knowledge graphs. However, most traditional reasoning methods lack interpretability and it is difficult to obtain reasoning paths. Therefore, multi-hop path reasoning methods of knowledge graphs have gradually become a hot topic. Multi-hop reasoning refers to the process of gradually finding answers from the initial question through multiple intermediate reasoning steps. Unlike simple single-hop reasoning, multi-hop reasoning requires the system to cross multiple nodes in the reasoning chain and connect multiple entities and relationships to obtain the final conclusion. This process simulates the logical thinking process of humans under complex problems and realizes the traceability and interpretability of the reasoning path. Although multi-hop reasoning has broad application prospects, it still lacks the ability to capture complex multi-hop reasoning paths and the ability to fuse multimodal data.

[0003] Existing reasoning methods mainly include reasoning methods based on embedding models and reasoning methods based on reinforcement learning. These two types of methods represent the development paths of knowledge graph reasoning technology, but there are still many challenges in dealing with reasoning tasks in complex scenarios.

[0004] 1. Reasoning method based on embedding model.

[0005] Models such as TransE and TransH use embedding-based reasoning methods to map entities and relations in knowledge graphs to vector spaces and use geometric structures to represent semantic relations between entities. The TransE model considers relations as translation vectors between two entities, i.e. (in and are the vector representations of the head entity and the tail entity respectively, is the relationship vector). TransE performs well in simple relationship reasoning and single-hop query, has high computational efficiency, and is suitable for processing one-to-one relationships. The TransH model improves on TransE by mapping each relationship to a hyperplane to solve the problem that TransE is difficult to handle one-to-many and many-to-many relationships.

[0006] Although these embedding models perform well in simple single-hop reasoning tasks, they have obvious shortcomings when faced with complex multi-hop reasoning and queries with strong path dependencies:

[0007] (1) Unable to handle complex relationship types: The embedding model embeds entities and relationships into the same low-dimensional vector space and assumes that relationships can be represented by simple vector addition. However, in multi-hop reasoning, it involves a combination of multiple complex relationships, which may be of different types (such as one-to-many, many-to-many, etc.). Simple vector addition cannot effectively capture and represent these complex relationship combinations, thus affecting the accuracy of multi-hop reasoning.

[0008] (2) Error accumulation problem: In multi-hop reasoning, reasoning needs to be performed along multiple relationship paths, and each step of reasoning will have a certain amount of error. However, in the process of step-by-step reasoning, errors will gradually accumulate in a linear model such as an embedding model, resulting in a decrease in the accuracy of the final reasoning result. As the number of hops increases, this error accumulation problem will become more serious.

[0009] (3) Lack of modeling of rich semantic information: Embedding models can only model simple distance relationships between entities and cannot effectively capture multimodal information (such as text descriptions, categories, etc.) and more complex features of entities. In multi-hop reasoning, more semantic information needs to be integrated to guide the selection of reasoning paths. Embedding models cannot effectively model this semantic information fusion, thus limiting their performance in complex reasoning tasks.

[0010] 2. Reasoning method based on reinforcement learning.

[0011] In recent years, reinforcement learning (RL), as a technology that can handle dynamic decisions and complex environments, has been gradually applied to reasoning tasks in knowledge graphs. DeepPath is one of the representative methods of reinforcement learning in the field of knowledge reasoning. It regards knowledge reasoning as a sequential decision problem and gradually builds reasoning paths on the knowledge graph through reinforcement learning agents to achieve multi-hop reasoning tasks. The agents in DeepPath learn how to find logical reasoning paths in the graph through exploration and trial and error in the environment to maximize their cumulative rewards.

[0012] However, DeepPath's reasoning method based on reinforcement learning has obvious shortcomings under multimodal data:

[0013] (1) Lack of multimodal information fusion capability: DeepPath is used to find a reasoning path from the starting entity to the target entity in the knowledge graph. However, when processing multimodal data, it mainly relies on structured information (such as the structure of entities and relationships) and has difficulty in effectively integrating other modal information. Multimodal data usually contains rich contextual and semantic information. Using only structural information will lead to insufficient understanding of entities and relationships, affecting the accuracy of reasoning.

[0014] (2) Lack of collaboration mechanism: DeepPath is a single-agent algorithm and does not design a cross-modal collaboration mechanism to comprehensively utilize the complementary information of different modalities. Its reinforcement learning policy network is mainly based on structural relationships and ignores the interaction between multimodal information, which leads to its poor performance when it comes to cross-modal reasoning.

[0015] In summary, the existing technology mainly has the following problems:

[0016] 1. Existing technologies (such as TransE and DeepPath) are mainly based on simple linear embedding or path search strategies. These methods are usually good at modeling single-step relationships in knowledge graphs, but their representation and reasoning capabilities are insufficient for complex paths involving multiple relationship jumps. Since the information of multi-step relationships cannot be effectively combined, important semantic information is easily lost during the reasoning process, and the optimal reasoning path cannot be accurately found.

[0017] 2. Existing technologies usually only process structured entity and relationship information, lack effective integration of multimodal data (such as text descriptions, category labels, etc.), and ignore the rich semantic and contextual information provided by other modalities. This makes the reasoning model insufficient in the process of understanding entities and relationships, and the reasoning performance is limited. In addition, most existing methods lack a modeling mechanism for the interaction between different modal information, and cannot fully utilize the complementarity of multimodal data for more accurate reasoning. This results in poor reasoning performance of existing methods in scenarios involving multimodal information. Summary of the invention

[0018] In response to the problems existing in the prior art, the present invention provides a multi-modal knowledge graph multi-hop reasoning method, system, terminal and storage medium to solve the problems of insufficient capture of complex multi-hop paths and lack of multi-modal data fusion existing in the existing knowledge graph reasoning methods.

[0019] In the first aspect, a multi-modal knowledge graph multi-hop reasoning method is provided, comprising the following steps:

[0020] S1: Preprocessing a multimodal knowledge graph containing structural data, category data and description data: generating structural embedding, category embedding and description embedding for entities and relationships, and mapping the category embedding and description embedding to the structural embedding to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is knowledge related to question-answering systems and information retrieval;

[0021] S2: Construct three reinforcement learning agents, namely structured data agent, category data agent and description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions including multimodal embedding of all agents and evaluates the corresponding value.

[0022] S3: Using the preprocessed multimodal knowledge graph as the environment for agent interaction, the MAPF-based reinforcement learning framework is used to make the three reinforcement learning agents interact with the environment for iterative learning, and the policy networks of the three reinforcement learning agents are obtained after training.

[0023] S4: According to the query request, the policy network of the three trained reinforcement learning agents is used to perform reasoning and prediction in the multimodal knowledge graph to obtain the reasoning results corresponding to the query request.

[0024] Furthermore, in step S1, the structure of the triple is embedded The definition is as follows:

[0025]

[0026] In the formula, , , Respectively represent the head entity, relationship and tail entity in the knowledge graph triple, For relationship The corresponding hyperplane normal vector;

[0027] For an entity e, the category embedding It is expressed as follows:

[0028]

[0029] In the formula, represents the set of all category embedding vectors of entity e, c represents a category embedding vector of entity e, Represents the number of category embedding vectors of entity e; then the category embedding of the triple Defined as:

[0030]

[0031] In the formula, and Represent the category embeddings of the head entity and the tail entity respectively;

[0032] For an entity e, the description embedding It is expressed as follows:

[0033]

[0034] In the formula, is the weight matrix, is a pre-trained language representation model, Represents the description annotation of entity e; then the description embedding of the triple Defined as:

[0035]

[0036] In the formula, and Represent the description embedding of the head entity and the tail entity respectively.

[0037] Furthermore, in step S1, the category embedding and description embedding are mapped to the structure embedding by the following formula:

[0038]

[0039] In the formula, represents multimodal embedding, represents structural embedding, When it is C, it means category embedding, When it is D, it means description embedding. represents the weight parameter, Represents the normal vector of the hyperplane defined by the structural embedding.

[0040] Furthermore, in step S2, for the structured data agent, its corresponding state includes the current entity, the adjacent entity set, the candidate relationship set, the historical path information and the multimodal embedding, and its corresponding action is a tuple consisting of the relationship path selection and the adjacent entity selection;

[0041] For the category data agent, its corresponding states include the current entity, the current category, the candidate category set, the category constraint relationship and the multimodal embedding, and its corresponding actions are the binary group consisting of category verification and category adjustment;

[0042] For the description data agent, its corresponding states include the current entity, the current description information, the candidate entity or relationship set, the description semantic embedding and the multimodal embedding, and its corresponding actions are the tuples consisting of semantic matching and description verification.

[0043] Further,

[0044] Reward function for multimodal collaborative reasoning of structured data agents, categorical data agents, and description data agents It is expressed as follows:

[0045]

[0046] In the formula, , , , They represent path correctness reward, collaboration reward, reasoning efficiency reward, and multimodal fusion quality reward respectively; , , , They respectively represent the weights of path correctness reward, collaboration reward, reasoning efficiency reward and multimodal fusion quality reward.

[0047] Furthermore, the path correctness reward is expressed as follows:

[0048]

[0049] The path correctness reward is to give a positive reward when the current reasoning path conforms to the logical relationship of the knowledge graph; otherwise, a penalty is given;

[0050] The collaboration reward is expressed as follows:

[0051]

[0052] In the formula, represents the collaboration reward adjustment factor, , , Respectively represent the states corresponding to the structure data agent, the category data agent and the description data agent;

[0053] The inference efficiency reward is expressed as follows:

[0054]

[0055] In the formula, Represents the positive reward given to the agent when it completes reasoning within the optimal number of steps; represents the number of inference steps, Represents the negative reward given when the agent fails to complete reasoning within the optimal number of steps;

[0056] The multimodal fusion quality reward is expressed as follows:

[0057]

[0058] In the formula, Represents the multimodal fusion quality adjustment factor;

[0059]

[0060]

[0061] In the formula, Represents the structure embedding obtained after the structure data agent selects an action based on the current state, When it is C, it represents the category embedding obtained after the category data agent selects an action based on the current state. When it is D, it represents the description embedding obtained after the description data agent selects an action according to the current state. represents the weight parameter, Representation structure embedding The normal vector of the hyperplane defined by .

[0062] Furthermore, in the multimodal collaborative reasoning reward function of each agent, and is the local reward corresponding to the current agent’s reasoning result, and It is the global reward obtained based on the reasoning results of all agents.

[0063] In the second aspect, a multi-modal knowledge graph multi-hop reasoning system is provided, including:

[0064] A multimodal embedding module is used to preprocess a multimodal knowledge graph containing structural data, category data and description data: generate structural embedding, category embedding and description embedding for entities and relationships, and map the category embedding and description embedding to the structural embedding to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is related knowledge in question-answering system and information retrieval;

[0065] The agent building module is used to build three reinforcement learning agents, namely, the structured data agent, the category data agent, and the description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions of all agents including multimodal embedding and evaluates the corresponding value.

[0066] The agent training module is used to use the preprocessed multimodal knowledge graph as the environment for agent interaction, and adopt the MAPF-based reinforcement learning framework to make the three reinforcement learning agents interact with the environment for iterative learning, so as to obtain the policy network of the three reinforcement learning agents after training;

[0067] The reasoning module uses the policy network of the three trained reinforcement learning agents to perform reasoning and prediction in the multimodal knowledge graph according to the query request to obtain the reasoning result corresponding to the query request.

[0068] In a third aspect, an electronic terminal is provided, comprising:

[0069] Memory on which computer programs or instructions are stored;

[0070] A processor is used to load and execute the computer program or instructions to implement the multi-hop reasoning method of the multimodal knowledge graph as described above.

[0071] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the multi-hop reasoning method of the multimodal knowledge graph as described above is implemented.

[0072] The present invention proposes a multi-modal knowledge graph multi-hop reasoning method, system, terminal and storage medium, which have the following beneficial effects:

[0073] (1) The present invention designs three agents, namely, structured data agent, category data agent and description data agent. Multiple agents explore different reasoning paths in the knowledge graph respectively, and find the best reasoning path through collaboration and information sharing between agents;

[0074] (2) For multimodal knowledge graphs that include structural data, category data, and description data, by fusing data from different modalities, more context and clues can be provided in the reasoning process, and the deep relationships between entities can be inferred more accurately, thereby enhancing the accuracy and efficiency of reasoning, enabling the reasoning system to quickly make reasonable reasoning decisions in complex scenarios;

[0075] (3) The multimodal collaborative reasoning reward function is designed to integrate rewards from four dimensions: path correctness, collaboration, reasoning efficiency, and multimodal fusion quality. This ensures efficient collaboration among agents, enhances the accuracy and efficiency of reasoning, and encourages agents to explore the optimal reasoning path in the multimodal knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0077] Figure 1 It is a framework diagram of a multi-modal knowledge graph multi-hop reasoning method provided by an embodiment of the present invention;

[0078] Figure 2 It is a flow chart of a multi-modal knowledge graph multi-hop reasoning method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0080] Existing reasoning methods based on embedded models and reinforcement learning have the following disadvantages:

[0081] 1. Lack of ability to capture complex multi-hop reasoning paths.

[0082] (1) Limited representation capability based on simple linear embedding: Existing methods use simple linear embedding to map entities and relations into the same low-dimensional space and represent relations through vector operations. This simple representation method is suitable for single-step relation modeling, but in multi-hop reasoning scenarios, the relation path becomes more complex and the interaction of multiple relations cannot be effectively expressed through simple linear combinations.

[0083] (2) Lack of information integration of multi-step relationships: In knowledge graphs, multi-hop reasoning requires effective integration of information of multi-step relationships, but existing methods often only process each step of the relationship, ignoring the dependencies and contextual connections between different relationships. This lack of global information integration leads to the loss of important semantic information during the reasoning process, making the reasoning path selection inaccurate and difficult to find the optimal reasoning path.

[0084] 2. Lack of ability to integrate multimodal data.

[0085] (1) Lack of modeling mechanism for multimodal information fusion: There are complex relationships between different modalities in multimodal data. For example, text descriptions can supplement the semantic information of entities, and category labels can increase the contextual understanding of relationships. Most existing methods do not have a mechanism to handle the fusion and interaction of these different modal information, and cannot effectively utilize the complementarity of multimodal information to enhance the reasoning process. This makes existing reasoning methods perform poorly in scenarios that require the integration of multiple modal information.

[0086] (2) Lack of multimodal collaboration mechanism: In multimodal reasoning, information from different modalities needs to complement each other and work together to improve the accuracy of reasoning. Existing technologies lack such a multimodal collaboration mechanism in the reasoning process and cannot effectively realize cross-modal information interaction and comprehensive utilization, resulting in poor accuracy and effect of reasoning.

[0087] Based on this, the present invention proposes a multi-modal knowledge graph multi-hop reasoning method, system, terminal and storage medium, and combines the following innovative technologies:

[0088] 1. Multimodal data fusion: Map category embedding and description embedding to structure embedding to generate fused multimodal embedding to enhance the representation ability of the knowledge graph.

[0089] 2. Reinforcement learning framework based on MAPF (multi-agents path finding): This model designs three agents, namely structured data agent, category data agent and description data agent. Multiple agents explore different reasoning paths in the knowledge graph and find the best reasoning path through collaboration and information sharing between agents.

[0090] 3. Multimodal collaborative reasoning reward function: By integrating the collaborative information of structured data agents, category data agents, and description data agents, agents are encouraged to explore the optimal reasoning path in the multimodal knowledge graph. This mechanism combines rewards in four dimensions: path correctness, degree of collaboration, reasoning efficiency, and multimodal fusion quality, ensuring efficient collaboration among agents and enhancing the accuracy and efficiency of reasoning.

[0091] The technical solution of the present invention is described in detail below in conjunction with specific embodiments.

[0092] The embodiment of the present invention provides a multi-modal knowledge graph multi-hop reasoning method, such as Figure 2 As shown, the following steps are included:

[0093] S1: Preprocessing a multimodal knowledge graph containing structural data, category data and description data: generating structural embeddings, category embeddings and description embeddings for entities and relationships, and mapping the category embeddings and description embeddings to the structural embeddings to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is related knowledge in question-answering systems and information retrieval.

[0094] The existing technologies are mainly based on simple linear embedding or path search strategies. These methods are usually good at modeling single-step relationships in knowledge graphs, but their representation and reasoning capabilities are insufficient for complex paths involving multiple relationship jumps. Since the information of multi-step relationships cannot be effectively combined, important semantic information is easily lost during the reasoning process, and the optimal reasoning path cannot be accurately found. Therefore, when dealing with scenarios that require multimodal data fusion, the existing technologies cannot make full use of all information resources, limiting the comprehensiveness and accuracy of reasoning. The present invention proposes a method for multimodal data fusion, which fuses structural data, category data and description data to generate multimodal embedding. The core idea of ​​the method is to use structural embedding as the dominant mode, because structural embedding contains the core knowledge of the knowledge graph and describes the direct relationship between entities. By mapping the category embedding and the description embedding to the hyperplane defined by the structural embedding, the system can combine the category and description information for reasoning based on the structural information. Through this fusion of multimodal data, the system can make full use of various information sources in the multimodal knowledge graph, thereby improving the comprehensiveness and accuracy of reasoning.

[0095] In order to facilitate the understanding of multimodal knowledge graphs, a brief explanation is given by taking the multimodal knowledge graph in the medical field as an example. For the following knowledge: (a) Aspirin can be used to treat headaches, (b) Aspirin is an analgesic, (c) Aspirin is an anti-inflammatory drug, (d) Aspirin is an antipyretic and analgesic when used in large doses, and mainly acts as an antiplatelet when used in small doses. It has the following specific effects and indications. Analgesic and antipyretic: It can relieve mild or moderate pain, such as headaches, toothaches, neuralgia, muscle pain, and dysmenorrhea, and is also used to reduce fever for colds and flu. It should be noted that this product can only relieve symptoms and cannot treat the causes of pain and fever, so other drugs must be used at the same time to treat the causes. Anti-inflammatory and anti-rheumatic: It is a commonly used drug for the treatment of rheumatic fever. After taking the drug, it can reduce fever, improve joint symptoms, and reduce erythrocyte sedimentation rate, but it cannot eliminate the basic pathological changes of rheumatic fever, nor can it treat and prevent heart damage and other complications. In the above example, (a) is structural data, aspirin is the head entity, treatment is the relationship, and headache is the tail entity; (b) and (c) are categorical data, analgesics and anti-inflammatory drugs are categories of the entity aspirin; (d) is descriptive data, which is a description of the entity aspirin.

[0096] Specifically, the process of preprocessing a multimodal knowledge graph containing structural data, category data, and description data includes:

[0097] S11: Structural data embedding method.

[0098] This embodiment uses a translation embedding model to implement structural embedding. Traditional embedding methods believe that if there is a triple, its vector representation should meet the following requirements: , , , They represent the head entity, relation, and tail entity in the knowledge graph triple respectively. Although the traditional method is simple and efficient, it cannot handle non-one-to-one relations. On this basis, a translation embedding model is used to embed each relation. Introducing a hyperplane normal vector , so that each entity has a different vector representation when facing external relations. The structural embedding of triples The definition is as follows:

[0099]

[0100] In the formula, , , Respectively represent the head entity, relationship and tail entity in the knowledge graph triple, For relationship The corresponding hyperplane normal vector.

[0101] S12: Category data embedding method.

[0102] For category embedding, we first randomly initialize a category embedding matrix with the same dimension as the structural embedding, which will be learned together with the structural embedding. In the knowledge graph, each entity may have one or more category labels. In order to represent the overall characteristics of the entity in the category dimension, we take the entity The average of all category embedding vectors is taken as its category embedding and is expressed as follows:

[0103]

[0104] In the formula, represents the category embedding of entity e, Represents the set of all category embedding vectors of entity e, represents the number of category embedding vectors of entity e, and c represents a category embedding vector of entity e. Then the category embedding of the triple Defined as:

[0105]

[0106] In the formula, and Represent the category embeddings of the head entity and the tail entity respectively.

[0107] S13: Describe the data embedding method.

[0108] For description embedding, a pre-trained language representation model is used to convert description annotations into computable vectors. The description embedding of is as follows:

[0109]

[0110] In the formula, Represents the description embedding of entity e; is the weight matrix; It is a pre-trained language representation model. In this embodiment, the BERT model or its variant is preferred. For example, in the medical field, the pre-trained language representation model is preferably the BioBERT model. represents the description annotation of entity e. Then the description embedding of the triple Defined as:

[0111]

[0112] In the formula, and Represent the description embedding of the head entity and the tail entity respectively.

[0113] S14: The category embedding and description embedding are mapped to the hyperplane defined by the structure embedding through the fusion function.

[0114] Specifically, the fusion function is expressed as follows:

[0115]

[0116] In the formula, represents multimodal embedding, represents structural embedding, When it is C, it means category embedding, When it is D, it means description embedding. represents the weight parameter, Represents the normal vector of the hyperplane defined by the structural embedding.

[0117] S2: Construct three reinforcement learning agents, namely structured data agent, category data agent and description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions of all agents including multimodal embedding and evaluates the corresponding value.

[0118] In multi-hop reasoning on complex knowledge graphs, the existing technology lacks an effective collaborative mechanism to fully utilize multimodal information. In particular, when faced with multi-type data, the collaboration between various reasoning modules is poor, making it difficult to form a unified reasoning strategy. Due to the lack of an efficient collaboration mechanism between different modalities, existing methods often miss important information during the reasoning process, resulting in an inaccurate or inefficient reasoning path. Therefore, when dealing with multimodal and complex path reasoning tasks, the existing technology is unable to form a global optimal strategy, which limits the reasoning ability of the system. This embodiment proposes a multi-agent reinforcement learning framework that realizes multi-hop reasoning of multimodal data in knowledge graphs through collaboration between multiple agents. Figure 1 As shown, the core idea of ​​this embodiment is to introduce structural data agents, category data agents and description data agents into the system, among which the structural agent focuses on the structural reasoning of entity relationships, the category data agent focuses on extracting category constraints and category logic information, and the description data agent focuses on mining text or semantic detail information. Each agent makes full use of its own modal information by exploring different types of reasoning paths. The agents share state information and conduct joint training to collaboratively optimize reasoning strategies, thereby effectively combining multimodal data in the reasoning process to find the optimal reasoning path. Through this multi-agent reinforcement learning collaboration mechanism, the system can significantly improve the reasoning efficiency and accuracy in complex knowledge graphs.

[0119] Structural data agents reason based on entity relationship structures to explore the connections and path relationships between entities; category data agents reason based on category constraints and logic to ensure category consistency during the reasoning process; descriptive data agents reason based on text descriptions and semantic information to ensure that the reasoning results are consistent with the description information.

[0120] The structured data agent, the category data agent, and the description data agent all have independent policy networks and value networks. The policy network generates the action to be performed based on the current state of the current agent (local observation). Its input is the current state of the agent, and its output is the selected action. The value network estimates the value of a given state and action. Its input includes the states and actions of all agents, and its output is the value of the current action combination.

[0121] S3: Using the preprocessed multimodal knowledge graph as the environment for agent interaction, the MAPF-based reinforcement learning framework is used to enable the three reinforcement learning agents to interact with the environment for iterative learning, and the policy network of the three trained reinforcement learning agents is obtained.

[0122] The present invention proposes a multi-agent reinforcement learning framework based on MAPF, which realizes multi-hop reasoning of multimodal data in knowledge graphs through collaboration between multiple agents. Under the MAPF framework, each agent collaboratively optimizes the reasoning strategy by sharing state information and joint training, thereby effectively combining multimodal data in the reasoning process and finding the optimal reasoning path. Through this multi-agent reinforcement learning collaboration mechanism, the system can significantly improve the reasoning efficiency and accuracy in complex knowledge graphs. The following is the design process of the MAPF multi-agent reinforcement learning framework and its key components.

[0123] 1. Environment setup: First, we need to build an environment that supports multi-agent reinforcement learning. The environment contains three agents: structured data agent, category data agent, and description data agent. The structured data agent performs reasoning based on the entity relationship structure and explores the connection and path relationship between entities; the category data agent performs reasoning based on category constraints and logic to ensure that the reasoning process complies with category consistency; the description data agent performs reasoning based on text descriptions and semantic information to ensure that the reasoning results are consistent with the description information. In addition, the environment must also include the following four parts:

[0124] (1) State (s): represents the local observation of each agent in the environment. In the reasoning environment of this embodiment, the three agents have different states, as follows:

[0125] For structured data agents: their corresponding states include the current entity, adjacent entity set, candidate relationship set, historical path information, and multimodal embedding. Among them, the current entity is the entity at the current reasoning position of the current agent; the adjacent entity set is all entity nodes connected to the current entity, which is used for the next path selection; the candidate relationship set is the relationship set between the current entity and the adjacent entity; the historical path information is the path record so far; and the multimodal embedding is the fused multimodal embedding, which provides global information.

[0126] For category data agents: their corresponding states include the current entity, current category, candidate category set, category constraint relationship, and multimodal embedding. Among them, the current entity is the entity at the current reasoning position of the current agent; the current category is the category or type information to which the current entity belongs; the candidate category set is the candidate category set related to the current entity; the category constraint relationship is the category association constraint between entities; multimodal embedding: the fused multimodal embedding provides background support for global category information.

[0127] For the description data agent: its corresponding state includes the current entity, current description information, candidate entity or relationship set, description semantic embedding and multimodal embedding. Among them, the current entity is the entity at the current reasoning position of the current agent; the current description information is the text description or semantic information; the candidate entity or relationship set is the candidate entity or relationship set that matches the description information; the description semantic embedding is the semantic representation extracted from the text description; multimodal embedding: the fused multimodal embedding provides global context information.

[0128] (2) Action (a): The three agents in this reasoning environment have different action choices, as follows:

[0129] For structured data agents: the corresponding actions include: relationship path selection, selecting the possible relationship path between two entities; adjacent entity selection, selecting the next entity adjacent to the current entity.

[0130] For category data agents: the corresponding actions include: category verification, verifying whether the current entity or relationship satisfies a certain category constraint; category adjustment: selecting the next action that meets the category constraint based on the reasoning results.

[0131] For description data agents: the corresponding actions include: semantic matching, selecting entities and relationships that best match the description information; description verification, verifying whether the current path or decision matches the description information.

[0132] (3) Reward: This includes local rewards and global rewards. The environment returns rewards based on the actions of each agent.

[0133] The existing technology lacks an effective reward mechanism in multi-hop reasoning tasks, and it is difficult to fully motivate agents to find the best reasoning path in the multimodal knowledge graph. In particular, when faced with complex multimodal data, existing methods often find it difficult to achieve collaborative reasoning. Due to the lack of a comprehensive evaluation mechanism for information of different modalities, the collaboration between the various agents is insufficient, and it is difficult to give full play to the complementarity of multimodal information. This embodiment proposes a multimodal collaborative reasoning reward function, which aims to motivate multiple agents to collaborate with each other during the reasoning process and jointly explore the optimal reasoning path. The multimodal collaborative reasoning reward function combines rewards in four dimensions: path correctness, degree of collaboration, reasoning efficiency, and multimodal fusion quality, to ensure efficient collaboration among the agents and enhance the accuracy and efficiency of reasoning.

[0134] set up is the time step The reward value at that time, the agent needs to maximize the cumulative reward The multimodal collaborative reasoning reward function consists of the following four parts:

[0135] (3.1) Path Correctness Reward , which is a local reward, is expressed as follows:

[0136]

[0137] When the current reasoning path of the current agent conforms to the logical relationship of the knowledge graph, a positive reward is given; otherwise, a penalty is given.

[0138] (3.2) Collaboration reward, which is a global reward, is expressed as follows:

[0139]

[0140] In the formula, represents the collaboration reward adjustment factor, , , They represent the states of the structure data agent, the category data agent, and the description data agent. The more consistent and complementary the information shared by the structure data agent, the category data agent, and the description data agent is, the higher the collaboration reward will be.

[0141] (3.3) Reasoning efficiency reward, which is a local reward, is expressed as follows:

[0142]

[0143] In the formula, Represents the positive reward given to the agent when it completes reasoning within the optimal number of steps; represents the number of inference steps, Represents the negative reward given when the agent fails to complete reasoning within the optimal number of steps; and Controls the rewards and penalties for reasoning efficiency. Encourages the agent to find the shortest path or reduce ineffective exploration. Gives negative rewards for each step of reasoning. , and a positive reward is given when the reasoning is successfully completed within the optimal number of steps.

[0144] (3.4) Multimodal fusion quality reward, which is a global reward, is expressed as follows:

[0145]

[0146] In the formula, Represents the multimodal fusion quality adjustment factor;

[0147]

[0148]

[0149] In the formula, Represents the structure embedding obtained after the structure data agent selects an action based on the current state, When it is C, it represents the category embedding obtained after the category data agent selects an action based on the current state. When it is D, it represents the description embedding obtained after the description data agent selects an action according to the current state. represents the weight parameter, Representation structure embedding The normal vector of the hyperplane defined by . The multimodal fusion quality reward is used to evaluate whether the fused knowledge representation improves the reasoning performance.

[0150] (3.5) Final multimodal collaborative reasoning reward function It is expressed as follows:

[0151]

[0152] In the formula, , , , They represent the weights of path correctness reward, collaboration reward, reasoning efficiency reward and multimodal fusion quality reward, which are used to balance the contribution of different reward items to the overall goal. In the multimodal collaborative reasoning reward function of each agent, and is the local reward corresponding to the current agent’s reasoning result, and It is the global reward obtained based on the reasoning results of all agents.

[0153] 2. Experience replay buffer pool: The experience replay buffer pool is used to store the interaction data of the agent in the environment and conduct batch training to improve data utilization, break time correlation, and achieve more stable training. At each interaction, the following information is stored in the replay pool: state, action, reward, and next state.

[0154] 3. Centralized training and decentralized execution: During the training process, MAPF adopts a centralized value network, that is, the value network of each agent can access global information (the states and actions of all agents) to help better estimate the global value, thereby effectively guiding the policy update of the policy network.

[0155] (1) Centralized training: During the training phase, the value network can obtain global information, that is, inputs including the states and actions of all agents. This enables the value network to better capture the collaborative and competitive relationships among multiple agents and improve the accuracy of value estimation.

[0156] (2) Decentralized execution: During the actual execution phase, each agent can only access its own state (local observation) and does not rely on global information, thus ensuring flexibility and scalability in the real world or distributed environments.

[0157] 4. Value function update: In each training step, the MAPF-based reinforcement learning framework updates the value through the value network. The goal of value update is to minimize the Bellman error. The Bellman equation is expressed as follows:

[0158]

[0159] In the formula, is the value of the current state s and action a, is the immediate reward obtained after executing action a in state s (calculated by the multimodal collaborative reasoning reward function), is the discount factor , is the next state reached after the current state s executes action a, is in the next state The maximum expected value under .

[0160] The update goal of the value network is to minimize the difference between the current value and the target value, which is called the TD error. The current value is calculated by the value network based on the current state and action, and the target value is calculated by the target network as the weighted sum of the immediate reward and the maximum value of the next state. The target network is a delayed copy of the value network to ensure the stability of training.

[0161] The target Q value formula is as follows:

[0162]

[0163]

[0164] In the formula, is the target Q value, For intelligent agents The immediate reward (calculated by the multimodal collaborative reasoning reward function), For intelligent agents The target value network is used to calculate the value of the next state. is the next state of all agents, is the next state of the jth agent, is the target policy network of the jth agent in state The action selected is, N is the number of agents.

[0165] The TD error is expressed as follows:

[0166]

[0167]

[0168] In the formula, is the TD error, which is used to measure the difference between the current value and the target value. Q value (value) estimated for the current value network, is the current state of all agents, is the current state of the jth agent, is the policy network of the jth agent in the current state Next, select the action.

[0169] 5. Policy network update: The goal of the policy network is to improve the agent's policy by maximizing the value given by the value network. Specifically, the policy goal of the policy network is to maximize the value of each agent in the current state, that is, to maximize the output of the value network. This means that the update direction of the policy network is aimed at improving the value estimated by the value network. When updating the policy network, the value network remains fixed, the value is calculated based on the output of the policy network, and the parameters of the policy network are optimized through back propagation. Its update goal is to maximize the output of the value network, so the loss function of its policy update is for:

[0170]

[0171] In the formula, represents the expected value calculation of state s, and the distribution of state s is determined by the experience replay pool Decide, is the sample data in the experience replay pool, The estimated Q value for the value network.

[0172] By maximizing the value output of the value network, the policy network can be guided to learn better strategies.

[0173] 6. Soft update: In order to improve the stability of training, MAPF uses the target network to soft update the value network and the policy network. This means that at each update, the target network moves in the direction of the main network with a smaller step size instead of directly replacing the parameters of the target network. The soft update formula is as follows:

[0174]

[0175] in, are the parameters of the main network, are the parameters of the target network, is a small step size coefficient.

[0176] This approach can effectively reduce fluctuations during training and prevent drastic changes in value estimates.

[0177] S4: Based on the query request (such as whether aspirin can be used to treat stomach pain), the policy network of the three trained reinforcement learning agents is used to perform reasoning and prediction in the multimodal knowledge graph to obtain the reasoning result corresponding to the query request.

[0178] The multi-modal knowledge graph multi-hop reasoning method provided in the above embodiment has the following advantages:

[0179] (1) In view of the shortcomings of existing methods in capturing complex multi-hop paths, the embodiments of the present invention adopt the MAPF multi-agent reinforcement learning framework. Three types of agents are designed under this framework: structural data agents, category data agents, and description data agents. The structural data agent is responsible for processing the basic triple information (entity-relationship-entity) in the knowledge graph, the category data agent provides additional context for reasoning by capturing the category information of the entity, and the description data agent uses a pre-trained model to convert the text description of the entity into a vector representation, thereby providing fine-grained semantic information for the model. Multiple agents explore different reasoning paths in the knowledge graph respectively, and find the best reasoning path through collaboration and information sharing between agents. This multi-agent division of labor and collaboration mechanism enables the system to adaptively respond to complex queries and improve the efficiency and accuracy of reasoning.

[0180] (2) In view of the shortcomings of existing methods in multimodal data fusion, the embodiment of the present invention adopts a multimodal learning method of reverse hyperplane projection and a multimodal collaborative reasoning reward function to fuse structured data, category data and description data to generate multimodal embedding. First, through the reverse hyperplane projection method, the core idea is to use structural embedding as the dominant mode, because structural embedding contains the core knowledge in the knowledge graph and describes the direct relationship between entities. Then the category and description embeddings are mapped to the hyperplane defined by the structural embedding, and the system can further combine the category and description information for reasoning based on the structured information. In order to further improve the multimodal data feature fusion effect, a multimodal collaborative reasoning reward function is adopted. This function encourages the agent to explore the optimal reasoning path in the multimodal knowledge graph by fusing the collaborative information of the structure data, category data and description data agents. This mechanism integrates the rewards of four dimensions: path correctness, degree of collaboration, reasoning efficiency and multimodal fusion quality, ensuring efficient collaboration of each agent and enhancing the accuracy and efficiency of reasoning.

[0181] The embodiment of the present invention also provides a multi-modal knowledge graph multi-hop reasoning system, including:

[0182] A multimodal embedding module is used to preprocess a multimodal knowledge graph containing structural data, category data and description data: generate structural embedding, category embedding and description embedding for entities and relationships, and map the category embedding and description embedding to the structural embedding to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is related knowledge in question-answering system and information retrieval;

[0183] The agent building module is used to build three reinforcement learning agents, namely, the structured data agent, the category data agent, and the description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions of all agents including multimodal embedding and evaluates the corresponding value.

[0184] The agent training module is used to use the preprocessed multimodal knowledge graph as the environment for agent interaction, and adopt the MAPF-based reinforcement learning framework to make the three reinforcement learning agents interact with the environment for iterative learning, so as to obtain the policy network of the three reinforcement learning agents after training;

[0185] The reasoning module uses the policy network of the three trained reinforcement learning agents to perform reasoning and prediction in the multimodal knowledge graph according to the query request to obtain the reasoning result corresponding to the query request.

[0186] It should be understood that the functional unit modules in various embodiments of the present invention may be concentrated in one processing unit, or each unit module may exist physically separately, or two or more unit modules may be integrated in one unit module, and may be implemented in the form of hardware or software.

[0187] An embodiment of the present invention further provides an electronic terminal, including:

[0188] Memory on which computer programs or instructions are stored;

[0189] A processor is used to load and execute the computer program or instructions to implement the multi-modal knowledge graph multi-hop reasoning method as described in the above embodiment.

[0190] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the multi-modal knowledge graph multi-hop reasoning method as described in the above embodiment is implemented.

[0191] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0192] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0193] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-hop reasoning method for a multimodal knowledge graph, characterized in that: The steps include: S1: Preprocessing a multimodal knowledge graph containing structural data, category data and description data: generating structural embeddings, category embeddings and description embeddings for entities and relationships, and mapping the category embeddings and description embeddings to the structural embeddings to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is related knowledge in a question-answering system or information retrieval; S2: Construct three reinforcement learning agents, namely structured data agent, category data agent and description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions including multimodal embedding of all agents and evaluates the corresponding value. S3: Using the preprocessed multimodal knowledge graph as the environment for agent interaction, the MAPF-based reinforcement learning framework is used to make the three reinforcement learning agents interact with the environment for iterative learning, and the policy networks of the three reinforcement learning agents are obtained after training. S4: According to the query request, the policy network of the three trained reinforcement learning agents is used to perform reasoning and prediction in the multimodal knowledge graph to obtain the reasoning results corresponding to the query request.

2. The multi-hop reasoning method of multimodal knowledge graph according to claim 1 is characterized in that: In step S1, the structure of the triple is embedded The definition is as follows: ； In the formula, , , Respectively represent the head entity, relationship and tail entity in the knowledge graph triple, For relationship The corresponding hyperplane normal vector; For an entity e, the category embedding It is expressed as follows: ； In the formula, represents the set of all category embedding vectors of entity e, c represents a category embedding vector of entity e, Represents the number of category embedding vectors of entity e; then the category embedding of the triple Defined as: ； In the formula, and Represent the category embeddings of the head entity and the tail entity respectively; For an entity e, the description embedding It is expressed as follows: ； In the formula, is the weight matrix, is a pre-trained language representation model, Represents the description annotation of entity e; then the description embedding of the triple Defined as: ； In the formula, and Represent the description embedding of the head entity and the tail entity respectively.

3. The multi-hop reasoning method of multimodal knowledge graph according to claim 2 is characterized in that: In step S1, the category embedding and description embedding are mapped to the structure embedding through the following formula: ； In the formula, represents multimodal embedding, represents structural embedding, When it is C, it means category embedding, When it is D, it means description embedding. represents the weight parameter, Represents the normal vector of the hyperplane defined by the structural embedding.

4. The multi-hop reasoning method of multimodal knowledge graph according to claim 1, characterized in that: In step S2, for the structured data agent, its corresponding state includes the current entity, the adjacent entity set, the candidate relationship set, the historical path information and the multimodal embedding, and its corresponding action is a tuple consisting of the relationship path selection and the adjacent entity selection; For the category data agent, its corresponding states include the current entity, the current category, the candidate category set, the category constraint relationship and the multimodal embedding, and its corresponding actions are the binary group consisting of category verification and category adjustment; For the description data agent, its corresponding states include the current entity, the current description information, the candidate entity or relationship set, the description semantic embedding and the multimodal embedding, and its corresponding actions are the tuples consisting of semantic matching and description verification.

5. The multi-hop reasoning method of multimodal knowledge graph according to claim 1, characterized in that: Multimodal collaborative reasoning reward function for structured data agents, categorical data agents, and description data agents It is expressed as follows: ； In the formula, , , , They represent path correctness reward, collaboration reward, reasoning efficiency reward, and multimodal fusion quality reward respectively; , , , They respectively represent the weights of path correctness reward, collaboration reward, reasoning efficiency reward and multimodal fusion quality reward.

6. The multi-hop reasoning method of multimodal knowledge graph according to claim 5, characterized in that: The path correctness reward is expressed as follows: ； The path correctness reward is to give a positive reward when the current reasoning path conforms to the logical relationship of the knowledge graph; otherwise, a penalty is given; The collaboration reward is expressed as follows: ； In the formula, represents the collaboration reward adjustment factor, , , They respectively represent the states obtained after the structure data agent, the category data agent and the description data agent select actions based on their respective current states; The inference efficiency reward is expressed as follows: ； In the formula, Represents the positive reward given to the agent when it completes reasoning within the optimal number of steps; represents the number of inference steps, Represents the negative reward given when the agent fails to complete reasoning within the optimal number of steps; The multimodal fusion quality reward is expressed as follows: ； In the formula, Represents the multimodal fusion quality adjustment factor; ； ； In the formula, Represents the structure embedding obtained after the structure data agent selects an action based on the current state, When it is C, it represents the category embedding obtained after the category data agent selects an action based on the current state. When it is D, it represents the description embedding obtained after the description data agent selects an action according to the current state. represents the weight parameter, Representation structure embedding The normal vector of the hyperplane defined by .

7. The multi-hop reasoning method of multimodal knowledge graph according to claim 6, characterized in that: In the multimodal collaborative reasoning reward function of each agent, and is the local reward corresponding to the current agent’s reasoning result, and It is the global reward obtained based on the reasoning results of all agents.

8. A multi-modal knowledge graph multi-hop reasoning system, characterized in that: include: A multimodal embedding module is used to preprocess a multimodal knowledge graph containing structural data, category data and description data: generate structural embedding, category embedding and description embedding for entities and relationships, and map the category embedding and description embedding to the structural embedding to generate a fused multimodal embedding; the multimodal knowledge graph is obtained by extracting knowledge containing structural data, category data and description data, and the knowledge is related knowledge in question-answering system and information retrieval; The agent building module is used to build three reinforcement learning agents, namely, the structured data agent, the category data agent, and the description data agent. The policy network of each agent receives the state including multimodal embedding as input to generate actions. The value network of each agent receives the state and actions of all agents including multimodal embedding and evaluates the corresponding value. The agent training module is used to use the preprocessed multimodal knowledge graph as the environment for agent interaction, and adopt the MAPF-based reinforcement learning framework to make the three reinforcement learning agents interact with the environment for iterative learning, so as to obtain the policy network of the three reinforcement learning agents after training; The reasoning module uses the policy network of the three trained reinforcement learning agents to perform reasoning and prediction in the multimodal knowledge graph according to the query request to obtain the reasoning result corresponding to the query request.

9. An electronic terminal, characterized in that: include: Memory on which computer programs or instructions are stored; A processor, used to load and execute the computer program or instructions to implement the multi-hop reasoning method of the multimodal knowledge graph as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the multi-modal knowledge graph multi-hop reasoning method as described in any one of claims 1 to 7 is implemented.

Images