Reinforcement learning-based knowledge reasoning path selection and evaluation method, system, device and medium

Abstract

The application discloses a knowledge reasoning path selection and evaluation method, system, device and medium based on reinforcement learning, belongs to the technical field of path selection and evaluation, and comprises the following steps: representing a knowledge graph as a graph structure, adopting a deep reinforcement learning framework; generating node embedding vectors by using graph contrast learning and an adaptive enhancement mechanism; constructing a generation network and a discrimination network to obtain a candidate reasoning path; constructing a multi-objective reward function which fuses a topological connectivity reward and a semantic consistency reward, and performing reinforcement learning; calculating dynamic propagation weights of nodes in a reasoning process by using an information propagation model, and integrating the dynamic propagation weights into node feature representation; and adopting a deep reinforcement learning method to perform end-to-end training on an agent, and outputting a reasoning path and a path evaluation score. The application is suitable for multiple actual reasoning scenes, and provides a technical path for deep knowledge discovery and intelligent decision-making of a large-scale knowledge graph.

Description

Technical Field

[0001] This invention relates to the field of path selection and evaluation technology, specifically to methods, systems, devices, and media for knowledge reasoning path selection and evaluation based on reinforcement learning. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, knowledge graphs, as structured semantic knowledge bases, have become an important tool for representing and reasoning about complex relationships in the real world. Traditional knowledge reasoning methods can be mainly divided into rule-based methods, embedding-based methods, and path-based methods. However, when faced with large-scale, highly complex knowledge graphs, how to efficiently and accurately select reasoning paths and evaluate their rationality remains a challenge. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention provides a method, system, device and medium for knowledge reasoning path selection and evaluation based on reinforcement learning.

[0004] Therefore, the technical problem solved by this invention is: how to efficiently and accurately select reasoning paths and evaluate their rationality when facing large-scale, highly complex knowledge graphs.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a knowledge reasoning path selection and evaluation method based on reinforcement learning, comprising: representing the knowledge graph as a graph structure; employing a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which an agent selects a path along the edges of the graph structure; generating node embedding vectors based on the Markov decision process using graph contrastive learning and adaptive reinforcement mechanisms; constructing a generator network and a discriminator network based on the Markov decision process and the node embedding vectors, and optimizing the path generation strategy through adversarial training to obtain candidate reasoning paths; constructing a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards based on the candidate reasoning paths and the evaluation results of the discriminator network, and performing reinforcement learning; calculating the dynamic propagation weights of nodes in the reasoning process using an information propagation model based on the candidate reasoning paths, and incorporating the dynamic propagation weights into the node feature representations; and performing end-to-end training of the agent using a deep reinforcement learning method based on the multi-objective reward function and the node feature representations, and outputting the reasoning path and path evaluation score after training is completed.

[0006] As a preferred embodiment of the reinforcement learning-based knowledge reasoning path selection and evaluation method of the present invention, the following steps are included: representing the knowledge graph as a graph structure, employing a deep reinforcement learning framework, and modeling the reasoning process as a Markov decision process in which an agent selects a path along the edges of the graph structure. This includes: mapping entity relationships in the knowledge graph to nodes and edges in the graph structure to construct the graph structure corresponding to the knowledge graph; determining the state and actions of the agent during the reasoning process based on the graph structure, where the actions correspond to candidate relation edges of the current node; establishing state transition relationships and reward relationships based on the state and actions to construct the Markov decision process; and controlling the agent to select a path along the edges of the graph structure based on the Markov decision process.

[0007] This preferred solution constructs the knowledge graph reasoning task as a graph-based Markov decision process, providing a unified state space and action space for path selection.

[0008] As a preferred embodiment of the knowledge reasoning path selection and evaluation method based on reinforcement learning described in this invention, the step of generating node embedding vectors based on the Markov decision process and utilizing graph contrastive learning and adaptive enhancement mechanisms includes: determining an adaptive enhancement strategy based on the node attributes and structural features of the knowledge graph; perturbing the knowledge graph based on the adaptive enhancement strategy to generate an enhanced view; performing graph contrastive learning encoding on the enhanced view to obtain node embedding vectors; and enhancing the representation of complex relationships and long-distance dependencies in the knowledge graph based on high-order relation modeling and attention mechanisms.

[0009] This preferred approach, employing graph contrastive learning and adaptive enhancement mechanisms, significantly improves the model's tolerance to graph noise and missing information. Combined with higher-order modeling and attention mechanisms, it significantly enhances the feature extraction capabilities for complex relationships and long-range semantic dependencies.

[0010] As a preferred embodiment of the reinforcement learning-based knowledge reasoning path selection and evaluation method of the present invention, the step of constructing a generator network and a discriminator network based on the Markov decision process and the node embedding vector, and optimizing the path generation strategy through adversarial training to obtain candidate reasoning paths includes: constructing current state information based on the Markov decision process and the node embedding vector; generating candidate action distributions using the generator network based on the current state information; expanding the reasoning path based on the candidate action distributions; evaluating the expanded reasoning path using the discriminator network to obtain an evaluation result; and performing adversarial training on the generator network and the discriminator network based on the evaluation result to obtain the candidate reasoning path.

[0011] This preferred scheme's generative adversarial network framework allows the generated paths to naturally tend towards higher logical rationality and semantic consistency as they strive to improve their rationality to approximate real paths. The discriminator's evaluation, as part of the reward, provides direct optimization signals.

[0012] As a preferred embodiment of the reinforcement learning-based knowledge reasoning path selection and evaluation method of the present invention, the step of constructing a multi-objective reward function that integrates topological connectivity reward and semantic consistency reward based on the evaluation results of the candidate reasoning path and the discrimination network, and performing reinforcement learning, includes: constructing a topological connectivity reward based on the graph structure relationship between the path endpoint and the target entity of the candidate reasoning path; constructing a semantic consistency reward based on the semantic association relationship between the path endpoint and the target entity; constructing a logical consistency reward based on the matching relationship between the relation sequence in the candidate reasoning path and the predefined logical rules; constructing the multi-objective reward function based on the evaluation results of the topological connectivity reward, the semantic consistency reward, the logical consistency reward, and the discrimination network; and performing reinforcement learning on the path generation strategy of the candidate reasoning path based on the multi-objective reward function.

[0013] This preferred scheme's multi-objective reward function comprehensively considers the structural proximity, semantic relevance, and logical compliance of the path, overcoming the limitations of a single reward function and more accurately guiding policy learning towards high-quality inference.

[0014] As a preferred embodiment of the reinforcement learning-based knowledge reasoning path selection and evaluation method of the present invention, the step of calculating the dynamic propagation weight of a node in the reasoning process using an information propagation model based on the candidate reasoning path and integrating the dynamic propagation weight into the node feature representation includes: determining the propagation association relationship of a node in the current reasoning process based on the candidate reasoning path; calculating the dynamic propagation weight corresponding to each node based on the propagation association relationship and the information propagation model; performing weighted processing on the node feature representation based on the dynamic propagation weight, or concatenating the dynamic propagation weight with the node feature representation; and characterizing the dynamic association state of a node in the current reasoning process based on the node feature representation after integrating the dynamic propagation weight.

[0015] This preferred solution is based on the dynamic weight update mechanism of the propagation model, which enables the intelligent agent to adjust its judgment of the importance of nodes in the graph in real time according to the explored path, thereby achieving dynamic adaptation to the reasoning context and improving the flexibility and intelligence of the strategy.

[0016] As a preferred embodiment of the knowledge reasoning path selection and evaluation method based on reinforcement learning described in this invention, the following steps are performed: Based on the multi-objective reward function and the node feature representation, a deep reinforcement learning method is used to train the agent end-to-end. After training, the reasoning path and path evaluation score are output. This includes: acquiring interaction data corresponding to state, action, reward, and new state based on the interaction process between the agent and the knowledge graph; storing and sampling the interaction data based on the interaction data and an experience replay mechanism; updating the network parameters corresponding to the agent based on the multi-objective reward function and the sampled interaction data; stabilizing the update process of the network parameters based on a target network mechanism; and outputting the reasoning path and path evaluation score based on the initial entity and the target entity after completing end-to-end training.

[0017] The deep reinforcement learning training framework adopted in this preferred scheme is mature and stable, and the experience playback and target network mechanism ensure the smoothness of the learning process. The end-to-end training method enables all components such as feature extraction, path generation, evaluation and dynamic adjustment to be optimized in synergy, ultimately forming an efficient and self-consistent inference system.

[0018] This invention provides a knowledge reasoning path selection and evaluation system based on reinforcement learning.

[0019] To address the aforementioned technical problems, this invention provides the following technical solution: a knowledge reasoning path selection and evaluation system based on reinforcement learning, comprising: a graph representation and modeling module, a robust feature encoding module, a path generation and evaluation module, a multi-objective reward design module, a dynamic propagation weight fusion module, and an end-to-end training and inference module; the graph representation and modeling module is used to represent the knowledge graph as a graph structure, and employs a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which an agent selects a path along the edges of the graph structure; the robust feature encoding module is used to generate node embedding vectors based on the Markov decision process, utilizing graph contrastive learning and adaptive reinforcement mechanisms; the path generation and evaluation module is used to generate node embedding vectors based on the Markov decision process and the nodes. The algorithm embeds vectors to construct a generative network and a discriminative network, and optimizes the path generation strategy through adversarial training to obtain candidate inference paths. The multi-objective reward design module constructs a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards based on the candidate inference paths and the evaluation results of the discriminative network, and performs reinforcement learning. The dynamic propagation weight fusion module calculates the dynamic propagation weights of nodes during the inference process using an information propagation model based on the candidate inference paths, and integrates the dynamic propagation weights into the node feature representations. The end-to-end training and inference module performs end-to-end training on the agent using deep reinforcement learning methods based on the multi-objective reward function and the node feature representations. After training, it outputs the inference path and path evaluation score.

[0020] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the knowledge reasoning path selection and evaluation method based on reinforcement learning.

[0021] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the knowledge reasoning path selection and evaluation method based on reinforcement learning.

[0022] The beneficial effects of this invention are as follows: This invention is applicable to multiple practical reasoning scenarios: in intelligent question-answering systems, it can improve the accuracy of multi-hop question answering; in recommendation systems, it can uncover deep connections between users and products; in the field of financial risk control, it helps identify complex fraud patterns; and in the biomedical field, it can assist in drug target discovery. This invention provides a feasible technical path for deep knowledge discovery and intelligent decision-making in large-scale knowledge graphs, solving the problem of efficiently and accurately selecting reasoning paths and evaluating their rationality when facing large-scale, highly complex knowledge graphs. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The above is a flowchart of a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided as an embodiment of the present invention.

[0025] Figure 2 The above is a flowchart of S1 for a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided in an embodiment of the present invention.

[0026] Figure 3 The above is a flowchart of S2 for a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided in one embodiment of the present invention.

[0027] Figure 4 The above is a flowchart of S3 for a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided in one embodiment of the present invention.

[0028] Figure 5 The above is a flowchart of S4 for a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided in one embodiment of the present invention.

[0029] Figure 6The S5 flowchart illustrates a reinforcement learning-based knowledge reasoning path selection and evaluation method provided in one embodiment of the present invention.

[0030] Figure 7 The above is a flowchart of S6 for a knowledge reasoning path selection and evaluation method based on reinforcement learning, provided in one embodiment of the present invention. Detailed Implementation

[0031] To make the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0032] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a knowledge reasoning path selection and evaluation method based on reinforcement learning, including: S1. Represent the knowledge graph as a graph structure and use a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which an agent selects a path along the edge of the graph structure.

[0033] S2. Based on Markov decision process, node embedding vectors are generated using graph contrastive learning and adaptive reinforcement mechanisms.

[0034] S3. Based on Markov decision processes and node embedding vectors, a generative network and a discriminative network are constructed, and candidate inference paths are obtained by optimizing the path generation strategy through adversarial training.

[0035] S4. Based on the evaluation results of candidate inference paths and the discrimination network, construct a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards, and perform reinforcement learning.

[0036] S5. Based on the candidate reasoning path, the dynamic propagation weight of the node in the reasoning process is calculated using the information propagation model, and the dynamic propagation weight is integrated into the node feature representation.

[0037] S6. Based on the multi-objective reward function and node feature representation, a deep reinforcement learning method is used to train the agent end-to-end. After training, the inference path and path evaluation score are output.

[0038] Therefore, to address the problem of efficiently and accurately selecting reasoning paths and evaluating their rationality when facing large-scale, highly complex knowledge graphs, steps S1-S6 are used to achieve the selection and evaluation of knowledge reasoning paths.

[0039] Example 2, refer to Figures 2-7As one embodiment of the present invention, based on the previous embodiment, a knowledge reasoning path selection and evaluation method based on reinforcement learning is provided, including: Furthermore, in step S1, the knowledge graph is represented as a graph structure, and a deep reinforcement learning framework is used to model the reasoning process as a Markov decision process in which an agent selects a path along the edges of the graph structure, such as... Figure 2 As shown, the steps A1-A4 are included: A1. Map the entity relationships in the knowledge graph to nodes and edges in the graph structure to construct the graph structure corresponding to the knowledge graph.

[0040] A2. Based on the graph structure, determine the state and actions of the agent during the reasoning process. The actions correspond to the candidate relation edges of the current node.

[0041] A3. Based on states and actions, establish state transition relationships and reward relationships to construct a Markov decision process.

[0042] A4. Based on Markov decision process, control agents select paths along edges in graph structures.

[0043] Specifically, in this embodiment, each entity in the knowledge graph is mapped to a node in the graph structure, and the relationships between entities are mapped to directed edges, thus constructing a formal graph. ,in, For a set of nodes, For edge sets. To adapt to deep reinforcement learning frameworks, the knowledge reasoning task is defined as a Markov decision process: each state The action space is jointly represented by the current entity node, the visited path history, and the target entity. Defined as all possible relation edges starting from the current node; state transitions are achieved by executing a selected action, i.e., moving along the relation edge to the next entity; the agent gradually optimizes its decision-making strategy through learning, with the goal of constructing a reasoning path from the starting entity to the target entity. This transforms the multi-hop reasoning task into a sequential decision-making problem on a graph, enabling reinforcement learning frameworks to model and optimize it.

[0044] By using graph representation and Markov decision process modeling, the structure of knowledge graphs is formally defined. This transforms complex multi-step reasoning processes into a sequential decision problem, enabling reinforcement learning agents to learn and optimize within a state-action-reward framework.

[0045] Specifically, the formula for the Markov decision process is as follows: in, For state space; Indicates time The system status, including information such as current entity, path history, and target entity; For the action space, To start from the current entity node along the edge The decision to jump to an adjacent entity node; This represents the state transition probability distribution; For instant reward functions; Discount factor; For strategy The action value function; It's an instant reward. It is a counter that represents the time since the current time. Initially, the time offset for future rewards; Indicating in strategy Take the expected value under the probability distribution.

[0046] Furthermore, in step S2, based on the Markov decision process, a node embedding vector is generated using graph contrastive learning and adaptive reinforcement mechanisms, such as... Figure 3 As shown, it includes the following steps B1-B4: B1. Determine adaptive enhancement strategies based on node attributes and structural features of knowledge graphs.

[0047] B2. Based on an adaptive enhancement strategy, the knowledge graph is perturbed to generate an enhanced view.

[0048] B3. Perform graph contrastive learning encoding on the enhanced view to obtain node embedding vectors.

[0049] B4. Based on high-order relation modeling and attention mechanisms, enhance the representation of complex relations and long-distance dependencies in knowledge graphs.

[0050] By leveraging graph contrastive learning and adaptive augmentation mechanisms, robust encoding of node attributes and structural features in knowledge graphs is performed to generate node embedding vectors. Through robust feature encoding, graph contrastive learning and adaptive data augmentation techniques are used to learn node vector representations that are robust to graph noise and structural perturbations, thus solving the sparsity and noise problems that are common in knowledge graphs.

[0051] By combining high-order relation modeling and attention mechanisms, and through feature fusion or cascade processing, the ability to represent complex relations and long-distance dependencies is enhanced. Through high-order relation modeling and attention enhancement, the limitations of traditional single-hop neighborhood aggregation are broken through, and complex multi-hop semantic associations between entities, i.e., high-order relations, are explicitly captured. Attention mechanisms are used to dynamically focus on the local and global information most relevant to the current reasoning, thereby improving the ability to understand and represent complex logic and long-range dependencies.

[0052] Specifically, in this embodiment, the graph contrastive learning and adaptive enhancement mechanism includes: S2.1 Calculate the topological attribute indicators of the nodes.

[0053] In practice, the topological importance of each node in the knowledge graph is first calculated. For example, the degree centrality of each node (i.e., the number of directly connected neighbors) can be calculated, or the PageRank algorithm can be used to iteratively calculate the node's global influence score within the entire network. Alternatively, the K-shell decomposition method can be used to peel away low-degree nodes layer by layer from the network periphery to determine the core level of each node. In practical applications, a single or combined indicator can be selected based on the characteristics of the graph. For example, degree centrality can be emphasized in dense graphs; in hierarchical networks, K-shell decomposition values ​​better reflect the coreness of nodes.

[0054] In a specific embodiment, the comprehensive evaluation of node importance is achieved through a weighted fusion of multiple topological indicators. For example, weights can be set for degree centrality, PageRank value, and K-shell decomposition value, with the comprehensive importance score being a weighted sum of these indicators. Degree centrality reflects local connection density, PageRank value measures global influence, and K-shell decomposition value characterizes network coreness. For contrastive learning, the InfoNCE loss function is used, which calculates the loss by the ratio of the similarity between positive sample pairs and negative sample pairs, thereby bringing positive samples closer and pushing away negative samples. The encoder can choose either a graph convolutional network or a graph attention network for feature extraction. Both networks can effectively capture graph structure information. The graph convolutional network updates node representations through neighbor aggregation, while the graph attention network weights and aggregates neighbor information representations through an attention mechanism.

[0055] Specifically, the degree centrality formula is: in, For nodes out-degree or in-degree, The total number of nodes in the graph For degree centrality.

[0056] PageRank formula for: in, For nodes PageRank value; This is the damping coefficient, typically taken as 0.85; Pointing to a node The set of incoming neighbors; For nodes PageRank value; For nodes The degree of departure.

[0057] The K-shell decomposition value formula is: ‍: in, This is the K-shell decomposition value. For nodes The largest - Number of shell layers This indicates that the degree of peeling is less than or equal to The remaining subgraph nodes after the first node.

[0058] S2.2 Dynamically generate adaptive perturbation strategies.

[0059] Based on node importance, a perturbation probability is dynamically assigned to each edge and each node attribute in the graph. Edges connecting two highly important nodes are assigned a lower probability of being discarded; attributes of highly important nodes themselves are assigned a lower probability of being masked (partially zeroed). Key structural and semantic information is preserved while less important parts are perturbed, ensuring that the key structural and semantic information of the graph is retained as much as possible when generating the enhanced view.

[0060] S2.3, Generate multiple enhanced views.

[0061] Based on the established perturbation strategy, the original knowledge graph undergoes multiple independent random sampling operations. Each operation discards some edges and masks some node attributes according to probability, generating an enhanced view that differs slightly from the original graph but retains its core structure. Typically, two or more enhanced views are generated.

[0062] S2.4. Train a robust encoder through comparative learning.

[0063] Two or more enhanced views are generated and input into a graph neural network encoder with shared parameters, as shown in the figure convolutional network, to obtain the embedding vector of each node in different views. The training objective is to make the vector representations of the same node as similar as possible in different views, while making the vector representations of different nodes as different as possible. The training objective is usually optimized using the InfoNCE contrastive loss function. After contrastive learning optimization, the encoder can distinguish key structures from random noise, thereby extracting more discriminative node features. Experiments show that this method improves node classification accuracy by approximately 8.2% in noisy environments.

[0064] Specifically, in this embodiment, the combination of high-order relation modeling and attention mechanisms includes: S2.5 Construct higher-order tensors and capture multi-hop semantic associations.

[0065] In practice, the triplet data of the knowledge graph will be expanded into higher-order tensors, such as three-dimensional tensors. Representing entities and entity In a certain relational path pattern The strength of associations is assessed. Using graph convolutional networks based on tensor decomposition, techniques such as Tucker decomposition or CP decomposition are employed to decompose high-order tensors. Tucker decomposition decomposes a tensor into a product of a small core tensor and multiple factor matrices. The core tensor captures the essential patterns of high-order interactions, while the factor matrices correspond to low-dimensional representations of entities and relationships. The reconstructed tensor effectively reflects multi-hop relationship combinations. CP decomposition represents a tensor as the sum of multiple rank tensors. Each component can be viewed as a representation of a basic relationship path pattern, and their combination can also express complex associations, offering advantages in computation and storage. After decomposition, by specifically combining and reconstructing the core tensor and factor matrices, a new adjacency matrix is ​​obtained that directly reflects complex multi-step combinations of relationships between entities, explicitly capturing and enhancing implicit multi-hop semantic association patterns between nodes.

[0066] Multi-head self-attention mechanisms calculate the similarity between the query vector and the key vector using scaled dot product attention as the attention weights. The core of multi-head self-attention mechanisms is the calculation of scaled dot product attention: the input feature vector is mapped to the query vector, key vector, and value vector respectively through a linear transformation; the dot product of the query vector and all key vectors is calculated, and then divided by a scaling factor to stabilize the gradient. The scaling factor is usually the square root of the key vector dimension. The result is normalized to the attention weights using the Softmax function, and the attention weights determine the contribution of each value vector to the final output. Multiple self-attention mechanisms run in parallel (multi-head operation) to capture dependencies from different subspaces, resulting in a high-order semantic association feature matrix.

[0067] S2.6. Aggregate multi-level neighborhood information using a multi-head attention mechanism.

[0068] In practice, a multi-head self-attention mechanism is employed to handle the multi-order neighborhoods of nodes. The current feature representations of the node and its neighbors from first order to higher orders are taken as the input sequence. For each node, multiple sets of multi-head attention computations are performed simultaneously and in parallel. Within each head, the similarity between the current node's query vector and the key vectors of all neighboring nodes is calculated. This similarity is typically calculated using a scaled dot product and used as an attention weight to perform a weighted summation of the value vectors of neighboring nodes. Different attention heads can focus on neighbor dependencies of different types or at different distances. Finally, the outputs of all heads are concatenated to obtain a multi-order neighbor feature vector, achieving dynamic and differentiated aggregation of neighbor features of different orders.

[0069] S2.7 Feature fusion and enhanced node representation generation.

[0070] In practice, the high-order semantic association feature matrix obtained through tensor decomposition and reconstruction in step S2.5 is fused with the multi-order neighbor feature vector obtained through multi-head attention aggregation in step S2.6. The fusion method can be feature-level fusion, directly adding or element-wise multiplying the two types of feature vectors, or cascading processing, concatenating the two types of feature vectors into a longer vector. The fused features are then input into a multilayer perceptron or through a fully connected layer with a non-linear activation function such as ReLU, undergoing non-linear transformation and feature integration, outputting an enhanced node representation vector that contains rich, complex relational semantic information and long-range contextual dependency information.

[0071] Furthermore, in step S3, based on the Markov decision process and node embedding vectors, a generative network and a discriminative network are constructed. Candidate inference paths are obtained by optimizing the path generation strategy through adversarial training, such as... Figure 4 As shown, it includes the following steps C1-C4: C1. Construct current state information based on Markov decision process and node embedding vector.

[0072] C2. Generate candidate action distribution using the generative network based on the current state information, and expand the inference path based on the candidate action distribution.

[0073] C3. The expanded reasoning path is evaluated using a discriminative network to obtain the evaluation results.

[0074] C4. Based on the evaluation results, perform adversarial training on the generator network and the discriminator network to obtain candidate inference paths.

[0075] A generative network and a discriminative network are constructed. An agent is used as the generative network. The generative network generates candidate reasoning paths based on the current state information. The discriminative network evaluates the rationality of the generated paths and optimizes the path generation strategy through adversarial training.

[0076] Specifically, in this embodiment, the generator network generates candidate inference paths based on the current state information, including: S3.1 Construct a comprehensive state vector.

[0077] In practice, the generator network aggregates all contextual information relevant to the current decision. It combines the embedded representation of the current entity node, the aggregated representation of its neighboring nodes, and the historical encoding examples of the visited paths into a high-dimensional comprehensive state vector containing all contextual information. The embedded representation of the current entity node is also known as the node embedding vector. The aggregated representation of its neighboring nodes can be obtained by averaging or attention-weighting the features of all its first-order neighbors. The historical encoding of the visited paths can be obtained by concatenating the hidden states obtained by encoding the visited entity / relation sequence using a recurrent neural network or LSTM, along with the embedded representation of the target entity, along with the vector dimension.

[0078] S3.2 Calculate the initial probability of the action through the policy network.

[0079] In practice, the comprehensive state vector is input into the policy network. The policy network is implemented using a multilayer perceptron or a graph attention network. One approach is a multilayer fully connected layer (MLP) that performs a non-linear mapping of the input state, outputting a vector of length equal to the number of executable actions (outgoing edges) of the current node. Each element's value, after being normalized by the Softmax function, represents the initial probability of selecting the corresponding action. Another approach uses a graph attention layer, which, in addition to receiving state information, also inputs all outgoing edges of the current node and their target nodes as a graph structure. The graph attention mechanism then calculates the initial score for each candidate action in greater detail, converting it into a probability.

[0080] Specifically, a multilayer perceptron, a classic feedforward neural network containing multiple fully connected layers and nonlinear activation functions such as ReLU, is employed. The integrated state vector is used as input, and through inter-layer transformations, a vector equal to the number of candidate actions is output, which is then converted into a probability distribution using a Softmax function. Secondly, a graph attention network is used, forming a local subgraph with the current node, candidate actions (relationships), and associated neighbor nodes (tail entities). Attention coefficients between the current node and each candidate action / neighbor pair are calculated through an attention layer and normalized to the probability of selecting that action. The graph attention network more explicitly utilizes the local graph structure information.

[0081] The attention mechanism calculates weights based on the dot product or cosine similarity between the embedded representation of a candidate action and the current integrated state vector. In a specific embodiment, for the attention reweighting mechanism of action probabilities, the core of its weight calculation lies in evaluating the relevance between the candidate action and the current state. Specifically, it can calculate the dot product similarity, that is, directly performing a dot product operation between the action embedding vector and the state vector, or the cosine similarity, that is, first normalizing the two vectors and then calculating their dot product; the result is between -1 and 1, reflecting directional similarity. After the similarity score is processed by a Softmax function, the normalized attention weights are obtained, which are used to modulate the initial probability distribution of the policy network output.

[0082] S3.3, Reweighting action probabilities based on attention mechanism.

[0083] In practice, after obtaining the initial action probability distribution, an independent attention mechanism is introduced for secondary adjustment. This mechanism treats each candidate action as a query, which can be represented by an embedding vector of that relation type. The comprehensive state vector is used as the key and value in key-value pairs, and a relevance score between the action and the state is calculated. The relevance score reflects the immediate relevance of different candidate actions in the current specific context. This relevance score is used as a weight, multiplied by the initial probability value, or reweighted through other fusion methods to generate a final action probability distribution. This distribution simultaneously considers the historical experience of the policy network and the immediate attention preference of the current state.

[0084] S3.4, sample, execute actions, and iteratively expand the path.

[0085] In practice, based on the obtained action probability distribution, multinomial sampling or an ε-greedy strategy is used. Executing the action means moving along the selected relation edge to a new tail entity node and adding that entity to the visited path. The current entity node is updated to the target entity, and the process returns to step S3.1 to begin constructing the new state vector for the next time step. This process is repeated until the agent reaches a preset maximum number of steps, directly reaches the target entity, or enters a state with no available actions, satisfying one of the termination conditions. At this point, a complete candidate inference path has been generated.

[0086] Furthermore, in step S4, based on the evaluation results of the candidate inference paths and the discrimination network, a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards is constructed for reinforcement learning, such as... Figure 5 As shown, it includes the following steps D1-D5: D1. Construct a topological connectivity reward based on the graph structure relationship between the endpoint of the candidate reasoning path and the target entity.

[0087] D2. Construct a semantic consistency reward based on the semantic association between the path endpoint and the target entity.

[0088] D3. Construct a logical consistency reward based on the matching relationship between the relation sequence in the candidate reasoning path and the predefined logical rules.

[0089] D4. Based on the evaluation results of topological connectivity reward, semantic consistency reward, logical consistency reward, and discrimination network, construct a multi-objective reward function.

[0090] D5. Based on the multi-objective reward function, reinforcement learning is performed on the path generation strategy of candidate inference paths.

[0091] We design a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards, and combine this with the evaluation results of the discriminator network for reinforcement learning. Through reinforcement learning guided by multi-objective rewards, we design a comprehensive evaluation system, i.e., the reward function, which provides refined feedback signals for each step of the agent's decision based on multiple dimensions, including whether the destination can be reached (topological connectivity), whether the answer is correct (semantic consistency), and whether the path is smooth (discriminator score). This guides the agent to learn the comprehensive optimal path selection strategy.

[0092] Specifically, in this embodiment, the design of a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards includes: S4.1 Calculate the topological connectivity reward.

[0093] In practice, when the agent completes path generation or after each step, it calculates the shortest path distance from the current path's endpoint entity to the target entity in the objective task, based on the original graph structure information (adjacency relationships) of the knowledge graph. This can be achieved using graph search algorithms such as breadth-first search or Dijkstra's algorithm. Then, the reciprocal of this distance value, or its negative value, is used as a topological connectivity reward. The shorter and closer the distance, the higher the reward, thus incentivizing the agent to explore paths structurally closer to the target.

[0094] S4.2 Calculate semantic similarity reward.

[0095] In practice, a pre-trained embedding model on the knowledge graph, such as TransE or RotatE, is required. These models map entities and relations to a low-dimensional vector space, ensuring that semantically similar entities are also close to each other in the space. After obtaining the vectors of the path's endpoint entity and the target entity, the cosine similarity between them is calculated, ranging from -1 to 1, with larger values ​​indicating greater similarity. Alternatively, Euclidean distance is calculated, measuring the straight-line distance between two points, with smaller values ​​indicating closer proximity. The reciprocal of Euclidean distance is then taken as the semantic similarity reward. This reward encourages the agent to reason in a direction that is semantically closer to the answer.

[0096] S4.3, Rewards for Calculating Logical Consistency.

[0097] In practice, a predefined set of logical rules needs to be maintained. These rules can include transitivity, combinability, or mutual exclusion of relationships. For example, transitivity means that if A is B's colleague and B is C's colleague, then A may have some kind of connection with C. After the agent generates a path, it checks whether its sequence of relationships matches these rules. For instance, for each rule, if a sequence of relationships in the path matches its premise and conclusion, a pre-defined score is accumulated. Finally, a logical consistency reward is calculated based on the sum or weighted sum of the scores of all matching rules, thereby encouraging the agent to generate reasoning paths that conform to common-sense logic.

[0098] S4.4 Weighted fusion forms the final reward value.

[0099] In practice, the calculated topological connectivity reward, semantic similarity reward, and logical consistency reward are fused together. Weight coefficients can be set in three ways: first, by pre-setting equal weights; second, by obtaining fixed weights through optimization on a validation set; and third, by introducing a lightweight weight network to dynamically generate weights based on the current inference state. In actual deployment, the third method is more adaptable and can also be used to learn and dynamically adjust based on the current state through a small neural network. The final multi-objective reward value is calculated through a linearly weighted sum. This comprehensive reward value will serve as a key signal in the reinforcement learning algorithm for evaluating and optimizing the agent's policy.

[0100] The formula for the multi-objective reward function that combines topological connectivity rewards and semantic consistency rewards is as follows: in, For a multi-objective reward function, Here are the learnable weight coefficients for each sub-reward, and each sub-reward is defined as follows: Topological connectivity reward : in, End point of the current path Entity to Target Entity The shortest path distance, This is the adjustment coefficient.

[0101] The formula for semantic similarity reward is: : in, The embedding vector of the endpoint entity. The embedding vector of the target entity. This is a semantic similarity function.

[0102] Logical consistency reward : in, The length of the relation sequence in the path; As an indicator function, when the relation sequence The value is 1 when matching predefined logical rules; For logical consistency functions, For the first A sequence of relations.

[0103] Furthermore, in step S5, based on the candidate inference path, the dynamic propagation weight of the node during the inference process is calculated using the information propagation model, and the dynamic propagation weight is integrated into the node feature representation, such as... Figure 6 As shown, it includes the following steps E1-E4: E1. Based on candidate reasoning paths, determine the propagation relationship of nodes in the current reasoning process.

[0104] E2. Based on the propagation relationship and information propagation model, calculate the dynamic propagation weight corresponding to each node.

[0105] E3. Based on dynamic propagation weights, perform weighted processing on node feature representations, or concatenate dynamic propagation weights with node feature representations.

[0106] E4. Based on the node feature representation after incorporating dynamic propagation weights, the dynamic association state of the node in the current inference process is characterized.

[0107] The dynamic propagation weights of nodes during the reasoning process are calculated based on an information propagation model, and then incorporated into the node feature representation in a weighted or concatenated manner. This dynamic propagation weight fusion simulates the diffusion process of information or influence in a knowledge graph network. At different stages of reasoning, the importance weights of other nodes in the graph are dynamically adjusted based on the nodes already visited, and these weights are incorporated into the features. This step enables the agent's decision-making to be context-aware and dynamically adjusted based on the current exploration state.

[0108] Specifically, the formula for dynamic propagation weights is: in, For a moment The node propagation weight vector; It is the attenuation factor; The normalized adjacency matrix (random walk matrix); To point to the initial entity node The indicator vector. The propagation weights are calculated based on a personalized PageRank algorithm. This algorithm simulates a random walk process starting from an initial node, with the walk probability controlled by a decay factor. After multiple iterations, each node acquires a stable propagation weight, reflecting its relative importance in the current inference context.

[0109] Furthermore, in step S6, based on the multi-objective reward function and node feature representation, a deep reinforcement learning method is used to train the agent end-to-end. After training, the inference path and path evaluation score are output, such as... Figure 7 As shown, the steps F1-F5 are included: F1: Based on the interaction process between the agent and the knowledge graph, obtain the interaction data corresponding to the state, action, reward and new state.

[0110] F2. Based on interactive data and experience playback mechanism, the interactive data is stored and sampled.

[0111] F3. Based on the multi-objective reward function and sampled interaction data, update the network parameters corresponding to the agent.

[0112] F4. Based on the target network mechanism, the update process of network parameters is stably controlled.

[0113] F5. After completing end-to-end training, output the inference path and path evaluation score based on the initial entity and the target entity.

[0114] A deep reinforcement learning approach, combining experience replay and target network mechanisms, is employed for end-to-end training of the agent. After training, the agent autonomously selects the optimal inference path from the knowledge graph based on the initial and target entities, and outputs a path evaluation score. Through end-to-end training and inference execution, stabilization techniques such as experience replay and target networks are integrated to jointly and end-to-end optimize all the aforementioned modules. After training, the model parameters are solidified, forming a complete inference system capable of receiving initial and target entities, automatically planning and outputting the optimal inference path and its confidence evaluation.

[0115] Specifically, in this embodiment, end-to-end training using deep reinforcement learning methods includes: S6.1 Use the experience replay pool to store interaction data.

[0116] In practice, after each interaction between the agent and the environment (knowledge graph), a four-tuple is obtained: the current state, the action performed, the immediate reward received, and the new state reached after the action. The system stores these four-tuples in a data structure called the experience replay pool, which is typically a fixed-capacity first-in-first-out queue. Storing past experience allows subsequent training to randomly sample batches of data, breaking the temporal correlation between data points and improving training stability and data efficiency.

[0117] S6.2. Update network parameters using reinforcement learning algorithms.

[0118] In practical implementation, the Double DQN algorithm is preferred for selecting relationships in a discrete action space; if the action space is continuous or high-dimensional, the DDPG algorithm can be used. In actual knowledge graph reasoning, Double DQN is more commonly used due to the discrete and limited types of relationships. Double DQN consists of a policy network for selecting actions and a value network for evaluating the value of state-action pairs. The parameters of these two networks are updated using temporal difference error, i.e., the difference between the current value estimate and the target value based on the next state and reward. DDPG is more suitable for continuous action spaces. Its core is an actor-commentator architecture, where the actor network outputs deterministic actions, and the commentator network evaluates their value, updating the values ​​using a policy gradient method. During training, a batch of historical four-tuple data is randomly sampled from the experience replay pool to calculate the loss and perform backpropagation to update the network parameters.

[0119] S6.3. Use the target network for stable training.

[0120] In practice, to provide a more stable value estimation target, an independent target network is introduced, with the same structure as the main network and the online network. There are two main methods for updating the parameters of the target network. One is periodic hard updates, where the parameters of the main network are completely copied to the target network every fixed number of steps, such as C steps. The other is continuous soft updates, where the parameters of the main network and the target network are weighted and mixed at a very small ratio, such as 0.001, every step or every N steps, allowing the parameters of the target network to slowly track the changes in the main network. Both methods effectively alleviate the learning instability problem caused by frequent changes in the target value.

[0121] S6.4 Simultaneously optimize the generation network and the discrimination network.

[0122] In practice, the entire training framework is a hybrid system combining reinforcement learning and adversarial learning. The generator network, or policy network of the reinforcement learning agent, is responsible for generating inference paths; the discriminator network is an independent classifier responsible for judging the rationality of the generated paths. In each or every few rounds of training, the discriminator network is first trained using generated paths and real paths or negative sample paths to improve its ability to distinguish between true and false paths. Then, the discriminator network is fixed, and its output rationality score is used as part of the additional reward to train the generator network, or reinforcement learning agent, to incentivize it to generate high-quality inference paths that are closer to the real paths. This process is repeated alternately, allowing the path generation quality of the generator network and the discrimination accuracy of the discriminator network to mutually promote and improve each other in the adversarial process, achieving end-to-end collaborative optimization.

[0123] Example 3 is an embodiment of the present invention. This embodiment provides a knowledge reasoning path selection and evaluation system based on reinforcement learning, including a graph representation and modeling module, a robust feature encoding module, a path generation and evaluation module, a multi-objective reward design module, a dynamic propagation weight fusion module, and an end-to-end training and reasoning module.

[0124] The Graph Representation and Modeling module is used to represent knowledge graphs as graph structures. It employs a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which an agent selects a path along the edges of the graph structure.

[0125] The robust feature encoding module is used to generate node embedding vectors based on Markov decision processes, utilizing graph contrastive learning and adaptive enhancement mechanisms.

[0126] The path generation and evaluation module is used to construct a generative network and a discriminative network based on Markov decision processes and node embedding vectors, and to obtain candidate inference paths by optimizing the path generation strategy through adversarial training.

[0127] The multi-objective reward design module is used to construct a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards based on the evaluation results of candidate inference paths and the discrimination network, and then perform reinforcement learning.

[0128] The dynamic propagation weight fusion module is used to calculate the dynamic propagation weight of a node in the reasoning process based on the candidate reasoning path and the information propagation model, and then integrate the dynamic propagation weight into the node feature representation.

[0129] The end-to-end training and inference module is used to train the agent end-to-end based on a multi-objective reward function and node feature representation using deep reinforcement learning methods. After training, it outputs the inference path and path evaluation score.

[0130] This embodiment also provides an electronic device applicable to the knowledge reasoning path selection and evaluation method based on reinforcement learning, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the knowledge reasoning path selection and evaluation method based on reinforcement learning as proposed in the above embodiment.

[0131] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the knowledge reasoning path selection and evaluation method based on reinforcement learning as proposed in the above embodiments.

[0132] The storage medium proposed in this embodiment and the method for selecting and evaluating knowledge reasoning paths based on reinforcement learning proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0133] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0134] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A knowledge reasoning path selection and evaluation method based on reinforcement learning, characterized by: include, The knowledge graph is represented as a graph structure, and a deep reinforcement learning framework is used to model the reasoning process as a Markov decision process in which the agent selects a path along the edge in the graph structure. Based on the aforementioned Markov decision process, node embedding vectors are generated using graph contrastive learning and adaptive reinforcement mechanisms. Based on the Markov decision process and the node embedding vector, a generative network and a discriminative network are constructed, and candidate inference paths are obtained by optimizing the path generation strategy through adversarial training. Based on the evaluation results of the candidate inference paths and the discrimination network, a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards is constructed for reinforcement learning. Based on the candidate inference path, the dynamic propagation weight of the node in the inference process is calculated using the information propagation model, and the dynamic propagation weight is integrated into the node feature representation. Based on the multi-objective reward function and the node feature representation, a deep reinforcement learning method is used to train the agent end-to-end. After training, the inference path and path evaluation score are output.

2. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 1, characterized in that: The method represents the knowledge graph as a graph structure and employs a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which an agent selects paths along the edges of the graph structure. include, Map the entity relationships in the knowledge graph to nodes and edges in the graph structure to construct the graph structure corresponding to the knowledge graph. The state and actions of the agent during the reasoning process are determined based on the graph structure, and the actions correspond to the candidate relation edges of the current node; Based on the states and actions, state transition relationships and reward relationships are established to construct the Markov decision process. The control agent, based on the Markov decision process, selects paths along the edges of the graph structure.

3. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 2, characterized in that: The node embedding vectors generated based on the Markov decision process, using graph contrastive learning and adaptive reinforcement mechanisms, include: An adaptive enhancement strategy is determined based on the node attributes and structural features of the knowledge graph. The knowledge graph is perturbed based on the adaptive enhancement strategy to generate an enhanced view. Graph contrastive learning encoding is performed on the enhanced view to obtain node embedding vectors; Based on high-order relation modeling and attention mechanisms, the representation of complex relations and long-distance dependencies in the knowledge graph is enhanced.

4. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 3, characterized in that: Based on the Markov decision process and the node embedding vectors, a generative network and a discriminative network are constructed. Candidate inference paths are obtained by optimizing the path generation strategy through adversarial training. include, The current state information is constructed based on the Markov decision process and the node embedding vector. Based on the current state information, the generator network is used to generate a candidate action distribution, and the inference path is expanded based on the candidate action distribution. The extended inference path is evaluated using the aforementioned discrimination network to obtain the evaluation result; Based on the evaluation results, adversarial training is performed on the generator network and the discriminator network to obtain the candidate inference path.

5. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 4, characterized in that: Based on the evaluation results of the candidate inference paths and the discrimination network, a multi-objective reward function that integrates topological connectivity rewards and semantic consistency rewards is constructed for reinforcement learning. include, Based on the graph structure relationship between the endpoint of the candidate reasoning path and the target entity, a topological connectivity reward is constructed. Based on the semantic association between the path endpoint and the target entity, a semantic consistency reward is constructed; Based on the matching relationship between the relation sequence in the candidate reasoning path and the predefined logical rules, a logical consistency reward is constructed; Based on the topological connectivity reward, the semantic consistency reward, the logical consistency reward, and the evaluation results of the discrimination network, the multi-objective reward function is constructed. Based on the multi-objective reward function, reinforcement learning is performed on the path generation strategy of the candidate inference path.

6. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 5, characterized in that: Based on the candidate inference path, the dynamic propagation weight of the node during the inference process is calculated using an information propagation model, and the dynamic propagation weight is integrated into the node feature representation. include, Based on the candidate reasoning paths, determine the propagation relationships of nodes in the current reasoning process; Based on the propagation relationship and the information propagation model, calculate the dynamic propagation weight corresponding to each node; Based on the dynamic propagation weights, the node feature representations are weighted, or the dynamic propagation weights are concatenated with the node feature representations. The node feature representation, after incorporating the dynamic propagation weights, characterizes the dynamic association state of the node in the current inference process.

7. The knowledge reasoning path selection and evaluation method based on reinforcement learning as described in claim 6, characterized in that: Based on the multi-objective reward function and the node feature representation, a deep reinforcement learning method is used to train the agent end-to-end. After training, the inference path and path evaluation score are output. include, Based on the interaction process between the agent and the knowledge graph, obtain the interaction data corresponding to the state, action, reward and new state; Based on the interaction data and experience replay mechanism, the interaction data is stored and sampled; Based on the multi-objective reward function and the sampled interaction data, the network parameters corresponding to the agent are updated; Based on the target network mechanism, the update process of the network parameters is stably controlled; After completing end-to-end training, the inference path and path evaluation score are output based on the initial entity and the target entity.

8. A knowledge reasoning path selection and evaluation system based on reinforcement learning, employing the knowledge reasoning path selection and evaluation method based on reinforcement learning as described in any one of claims 1 to 7, characterized in that, include: The module includes a graph representation and modeling module, a robust feature encoding module, a path generation and evaluation module, a multi-objective reward design module, a dynamic propagation weight fusion module, and an end-to-end training and inference module. The graph representation and modeling module is used to represent the knowledge graph as a graph structure and adopts a deep reinforcement learning framework to model the reasoning process as a Markov decision process in which the agent selects a path along the edge in the graph structure. The robust feature encoding module is used to generate node embedding vectors based on the Markov decision process, utilizing graph contrastive learning and adaptive enhancement mechanisms. The path generation and evaluation module is used to construct a generator network and a discriminator network based on the Markov decision process and the node embedding vector, and to obtain candidate inference paths by optimizing the path generation strategy through adversarial training. The multi-objective reward design module is used to construct a multi-objective reward function that integrates topological connectivity reward and semantic consistency reward based on the evaluation results of the candidate inference path and the discrimination network, and to perform reinforcement learning. The dynamic propagation weight fusion module is used to calculate the dynamic propagation weight of the node in the reasoning process based on the candidate reasoning path using the information propagation model, and to integrate the dynamic propagation weight into the node feature representation. The end-to-end training and inference module is used to train the agent end-to-end based on the multi-objective reward function and the node feature representation using a deep reinforcement learning method. After training, it outputs the inference path and path evaluation score.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the knowledge reasoning path selection and evaluation method based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the knowledge reasoning path selection and evaluation method based on reinforcement learning as described in any one of claims 1 to 7.

Images