Knowledge graph reasoning method, system and equipment, program product and storage medium

Abstract

The invention discloses a knowledge graph reasoning method, system and device, a program product and a storage medium. According to the method, the concept of entropy is introduced, the random degree of the intelligent agent during path reasoning and selection can be adaptively evaluated, the target of the maximum entropy can enable the reasoning process to obtain higher exploration capability, and compared with the limitation of local optimum of a Monte Carlo method, the intelligent agent strategy is encouraged to explore more paths, so that the path selection efficiency is improved. Therefore, omission of reasoning information is reduced, reasoning accuracy is improved, and more useful potential information of the knowledge graph is mined.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a reasoning method, system, device, program product and storage medium for a knowledge graph. Background Art

[0002] In the field of modern finance and commerce, knowledge graphs, as a powerful data structure, are widely used in risk assessment, recommendation systems, and other aspects. Especially in commercial banks, by building a knowledge graph of customer risk information, potential risk transmission paths can be effectively identified and predicted, improving the accuracy and efficiency of risk management.

[0003] Existing knowledge graph reasoning methods based on reinforcement learning still have the problem of insufficient relational path mining capabilities. The relational paths in the knowledge graph are bridges connecting different entities and concepts, and are important clues in the reasoning process. However, existing methods often find it difficult to effectively mine and utilize these relational paths, resulting in the lack of sufficient contextual information and reasoning basis for the intelligent agent during reasoning. This not only limits the reasoning accuracy of the intelligent agent, but also makes it easy for it to fall into the local optimal solution, making it impossible to find the global optimal reasoning path. Summary of the invention

[0004] Based on the above problems, the present application provides a method.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] The first aspect of the embodiment of the present application provides a method for reasoning about a knowledge graph, including:

[0007] Get the target information to be processed;

[0008] Extracting entities from the target information, and constructing a knowledge graph based on the extracted entities and the relationships between the entities;

[0009] Modeling the constructed knowledge graph as a Markov decision environment, which is used to provide a decision-making and reasoning framework for the intelligent agent;

[0010] The agent-based target reasoning strategy is used to reason in a Markov decision environment, so that the agent can reach the end entity through action selection from the starting entity according to the target strategy;

[0011] The endpoint entity obtained in each round of iteration is saved in a collection. The endpoint obtained in each round of iteration is a customer entity discovered through reasoning that has a risk transmission relationship with the starting entity.

[0012] In a possible implementation, modeling the constructed knowledge graph as a Markov decision environment includes:

[0013] Define a state space, where each state represents an entity or a set of entities in the knowledge graph; define an action space, where each action represents a possible transfer path from the current entity to another entity; set a reward function to evaluate the reward obtained by the agent when it transfers from the current state to the next state, and the size of the reward is set according to the importance of the risk transmission relationship between the entities; set a transition probability, which is used to reflect the possibility of the agent transferring from the current state to the next state under different actions.

[0014] In a possible implementation, the method further includes:

[0015] Use the TransE embedding model to preprocess the knowledge graph;

[0016] Model the processed knowledge graph as a Markov decision environment;

[0017] Based on the training method of bidirectional BFS supervised learning, the agent is pre-trained;

[0018] Based on the maximum entropy strategy network, value network and experience replay pool, the agent's knowledge graph reasoning is retrained to obtain the agent's target reasoning strategy.

[0019] In a possible implementation, the experience replay pool is used to store four-tuple data obtained from each interaction in the environment, and when training the value network, a number of data are randomly sampled from the experience replay pool for training.

[0020] In one possible implementation, the reward function consists of a global reward, a path efficiency reward, and a path diversity reward, and is used to give a positive reward when the agent successfully reaches the target entity; rewards are given based on the length of the journey and the number of paths.

[0021] A second aspect of the embodiment of the present application provides a knowledge graph reasoning system, including:

[0022] An acquisition unit, used for acquiring target information to be processed;

[0023] A construction unit, used to extract entities from the target information, and construct a knowledge graph based on the extracted entities and the relationships between the entities;

[0024] A modeling unit, used to model the constructed knowledge graph into a Markov decision environment, wherein the Markov decision environment is used to provide a decision-making and reasoning framework for the intelligent agent;

[0025] The reasoning unit is used to perform reasoning in a Markov decision environment based on the target reasoning strategy of the intelligent agent, so that the intelligent agent can reach the terminal entity through action selection from the starting entity according to the target strategy;

[0026] The storage unit is used to store the endpoint entity obtained in each round of iteration in a collection. The endpoint obtained in each round of iteration is a customer entity discovered through reasoning that has a risk transmission relationship with the starting entity.

[0027] In a possible implementation, the modeling unit is specifically used to:

[0028] Define a state space, where each state represents an entity or a set of entities in the knowledge graph; define an action space, where each action represents a possible transfer path from the current entity to another entity; set a reward function to evaluate the reward obtained by the agent when it transfers from the current state to the next state, and the size of the reward is set according to the importance of the risk transmission relationship between the entities; set a transition probability, which is used to reflect the possibility of the agent transferring from the current state to the next state under different actions.

[0029] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the reasoning method of the knowledge graph as described in the first aspect above is implemented.

[0030] A fourth aspect of an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the reasoning method of the knowledge graph as described in the first aspect above.

[0031] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the reasoning method of the knowledge graph as described in the first aspect above.

[0032] Compared with the prior art, this application has the following beneficial effects:

[0033] Entities are extracted from the target information to be processed, and a knowledge graph is constructed based on these entities and their relationships. The constructed knowledge graph is modeled as a Markov decision environment. In this environment, the agent can select an action (i.e., move to the next entity) based on the current state (i.e., the current entity and its relationship), and obtain corresponding rewards or penalties based on the selected action. This modeling method not only provides a clear decision-making and reasoning framework for the agent, but also enables the agent to optimize its reasoning path by using the Markov property (i.e., the future state depends only on the current state). In the Markov decision environment, the agent uses a target reasoning strategy based on policy gradients for reasoning. This strategy enables the agent to explore and find the optimal reasoning path through action selection starting from the starting entity according to the goal (i.e., reaching the end entity). In this process, the agent not only considers the immediate reward of the current action, but also considers the long-term rewards that may be obtained in the future, thereby avoiding falling into a local optimal solution. After each round of iteration, the end entity reached by the agent is saved in a set. These end entities are customer entities that have a risk transmission relationship with the starting entity discovered through reasoning. Through iterative updates and set preservation, the agent can continuously accumulate effective reasoning paths, thereby gradually improving its reasoning strategy.

[0034] Because the Markov decision environment provides a more stable decision-making framework, this application not only solves the high variance problem caused by the Monte Carlo method, but also because the agent can obtain clearer feedback to guide its policy update in each iteration, optimizing the policy gradient update process. At the same time, by constructing a knowledge graph and modeling it as a Markov decision environment, the agent can more effectively mine and utilize relationship paths, thereby improving the accuracy and efficiency of reasoning. In addition, the strategy of iterative update and set preservation also ensures that the agent can continuously learn and improve its reasoning strategy to adapt to different knowledge graphs and reasoning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A flowchart of a knowledge graph reasoning method provided in an embodiment of the present application;

[0037] Figure 2 A flow chart of another method for reasoning about a knowledge graph provided in an embodiment of the present application;

[0038] Figure 3A structural diagram of the reasoning system of a knowledge graph provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the professional terms involved in the embodiments of the present application will be explained below.

[0041] Knowledge Graph: It is a graph-based data structure used to represent entities, concepts, attributes and their relationships. It aims to build a large, searchable knowledge network to achieve efficient management, search, reasoning and application of knowledge. The knowledge graph uses nodes (entities or concepts) and edges (relationships) to display things in the real world and their connections, thus forming a large, structured knowledge base.

[0042] Knowledge graph reasoning: Knowledge graph reasoning refers to the ability to obtain new knowledge based on existing knowledge in the knowledge graph. Specifically, it mainly revolves around the reasoning of relationships, that is, inferring unknown facts or relationships based on existing facts or relationships in the graph. This reasoning process generally focuses on the characteristic information of entities, relationships, and graph structures. Through knowledge graph reasoning, new facts, new relationships, new axioms, and new rules can be inferred.

[0043] Reinforcement Learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. This method is mainly used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment. The core of reinforcement learning is that the agent finds the optimal strategy to complete a given task through continuous trial and learning.

[0044] The SAC (Soft Actor-Critic) algorithm is a model-based deep reinforcement learning algorithm that combines the advantages of the Actor-Critic algorithm and the model-based policy optimization algorithm. The goal of the SAC algorithm is to achieve a balance between exploration and exploitation while maintaining high entropy (i.e. randomness) of the policy. This algorithm is particularly suitable for solving reinforcement learning problems in continuous action spaces with high-dimensional state space and action space, such as autonomous driving and robot control.

[0045] To facilitate understanding of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application will be described below.

[0046] As mentioned above, the existing technology mainly uses Monte Carlo policy gradient to maximize the cumulative reward in the knowledge graph reasoning method based on reinforcement learning, so as to train the policy network. However, this Monte Carlo-based sampling estimation method will produce high variance, resulting in instability in the policy training process. In addition, inappropriate setting of the update step size can easily lead to poor policy gradient updates, causing the agent to fall into the local optimum when performing reasoning, reducing the accuracy of reasoning.

[0047] Existing knowledge graph reasoning methods based on reinforcement learning also have shortcomings in relational path mining. These methods usually use simple policy gradient methods and lack effective mechanisms to enhance the agent's exploration ability, which makes it easy for the agent to fall into local optimal solutions when faced with complex relational paths and fail to discover more potential paths. In addition, due to the lack of effective control of policy entropy, the agent tends to choose known and relatively certain paths during training, while neglecting to explore new and potentially better paths.

[0048] In order to solve the above problems, the embodiment of the present application introduces the SAC (Soft Actor-Critic) algorithm. The present application not only pursues the maximization of the cumulative reward, but also maximizes the entropy of the strategy. The purpose of this is to increase the randomness of the strategy and improve the ability of the agent to explore new paths in the knowledge graph, thereby avoiding falling into the local optimal solution. This method helps to discover more potential relationship paths and improve the overall accuracy and effectiveness of reasoning.

[0049] The experience replay pool mechanism is introduced to store the data obtained from each interaction in the environment, and randomly sample data from it for training when training the value network. This not only improves the sample utilization rate, but also breaks the correlation between samples to meet the independence assumption, which helps to improve training efficiency and model performance.

[0050] A pre-training method based on supervised learning is adopted to achieve the reduction of action space. The paths between given entity pairs are searched through breadth-first search, and these paths are saved for retraining of the agent to reduce the dimensionality explosion problem and improve training efficiency.

[0051] During the training process, in addition to maximizing the cumulative reward, an entropy regularization term controlled by the parameter α is added to maintain the randomness of the strategy. This method can help the agent explore more possibilities in the early stages and reduce the risk of falling into suboptimal solutions.

[0052] The TransE model converts each triple (head entity, relationship, tail entity) in the knowledge graph into a vector representation for further processing while maintaining the consistency of semantic information and computational efficiency.

[0053] In summary, this application solves the problems of poor training stability and easy falling into local optimality in traditional knowledge graph reasoning methods based on Monte Carlo policy gradient by introducing technical means such as SAC algorithm, experience replay pool, supervised learning pre-training, etc., and significantly improves the accuracy and efficiency of reasoning.

[0054] It should be noted that the reasoning method, system, device and medium of the knowledge graph provided in this application can be applied to the field of computer technology. The above is only an example and does not limit the application field of the reasoning method, system, device and medium of the knowledge graph provided in this application. In addition, the embodiment of the present application may not limit the execution subject of the reasoning of the knowledge graph. For example, the reasoning method of the knowledge graph of the embodiment of the present application can be applied to data processing devices such as terminal devices or servers. Among them, the terminal device can be an electronic device such as a computer, a personal digital assistant (PDA). The server can be a stand-alone server, a cloud server, or a cluster server composed of multiple servers.

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] The following is an example of an example to illustrate a knowledge graph reasoning method provided by this application. Figure 1 ,Should Figure 1 A flowchart of a method for reasoning about a knowledge graph provided in an embodiment of the present application, the method comprising:

[0057] Step 1: Use the TransE embedding model to preprocess the knowledge graph.

[0058] The TransE embedding model based on representation learning is used to process each triple (h, r, t) in the knowledge graph.

[0059] The TransE (Translational Embedding) model is a representation learning method for knowledge graph embedding. Its core idea is to transform each triple (h, r, t) in the knowledge graph into a relational translation problem in the vector space, that is, the sum of the head entity vector and the relation vector is as close as possible to the tail entity vector.

[0060] In the TransE model, each entity (including head and tail entities) and relationship in the knowledge graph is represented as a low-dimensional vector. These vectors are embedded in a continuous vector space so that similar entities or relationships are close in the vector space.

[0061] For each triple (h, r, t), the TransE model hopes to satisfy the following relationship: h+r≈t, where h, r, and t represent the vector representations of the head entity, relation, and tail entity, respectively. In order to measure the proximity between entities in the vector space, the TransE model uses the L1 norm (also known as the Manhattan distance) or the L2 norm (also known as the Euclidean distance) to calculate the distance d between entities. The calculation formula is: d=||h+rt||.

[0062] Ideally, for the correct triple (h, r, t), the distance d(h+r, t) should be as small as possible, indicating that the semantic information of the head entity vector plus the relationship vector is very close to that of the tail entity vector. For the wrong triple, the distance should be as large as possible, indicating that the result of adding the head entity and relationship vectors is less correlated with the tail entity. To achieve this goal, the TransE model uses a distance-based loss function, such as a squared loss function based on the L1 or L2 norm. During the training process, the model continuously adjusts the vector representation of entities and relationships by optimizing the loss function, so that the distance of correct triplets is smaller and the distance of incorrect triplets is larger.

[0063] The low-dimensional vector representation makes subsequent calculations more efficient and reduces computational complexity. Through the TransE model, the semantic relationship between entities and relationships is maintained, which helps to make reasoning more accurate. Through the above steps, the entities and relationships in the knowledge graph can be effectively mapped to the low-dimensional vector space, which provides a basis for subsequent knowledge graph reasoning. Next, the method will further build an MDP environment and use the SAC reinforcement learning algorithm to train the agent to improve the accuracy and efficiency of knowledge graph reasoning.

[0064] Step 2: Model the processed knowledge graph as an MDP environment.

[0065] After obtaining the preprocessed knowledge graph, we need to establish an MDP environment for the agent to perform reinforcement learning training.

[0066] The MDP environment is implemented through the four-tuple<S,A,P,R> It represents the continuous state space set of all entities in the knowledge graph. Since each entity node in the knowledge graph is represented discretely, it is necessary to use the translation embedding method of the Trans series mentioned above to map the entity node to a low-dimensional vector space for processing. Mapping entities in this way can reduce computational efficiency costs while retaining semantic information.

[0067] A represents the action space, that is, the signal transmitted to the knowledge graph environment when the agent interacts. In the embodiment of the present application, A is modeled as a set of relation edges between all entities. Given the head and tail entities, the agent can explore the reasoning path between entity pairs by selecting a series of relation actions from A; P is the state transition probability matrix, which is interpreted as the agent's transition from the current state s in the environment. t Transition to next state s t+1 Finally, similar to the concept of state transfer matrix P, this paper defines R as the probability of the agent changing from state s after taking action a. t Transfer to t+1 The value of the reward that can be obtained.

[0068] Step 3: Pre-train the agent based on the training method of bidirectional BFS supervised learning.

[0069] Step 4: Retrain the agent’s knowledge graph reasoning based on the maximum entropy strategy network, value network, and experience replay pool to obtain the agent’s target reasoning strategy.

[0070] In this application, the agent's reasoning task is defined as: given an entity pair (e source ,r,e target ), train the agent from the beginning node e sourceTake a series of actions t Arriving at the end node e target process.

[0071] In the process of performing relational path reasoning, the state of the agent is represented as the difference between the head and tail entity vectors, that is, the current state is intuitively represented as the "distance" from the target node, because in the process of the agent performing the reasoning task, the entity vector obtained by the embedding model is insensitive to the choice of path, that is, the distance between the two head and tail nodes in the vector space remains unchanged. When the agent takes an action a in the knowledge graph to interact with the environment, it will not only obtain the corresponding state change information, but also receive a reward R from the knowledge graph environment. In this way, the agent can "judge" whether the action a it chooses is reasonable through this incentive, and make corresponding adjustments when facing the same state next time. Since the agent may take more wrong action sequences than correct action sequences during the execution of a reasoning task, a detailed reward function is needed to control its behavior of selecting relational actions.

[0072] In a possible implementation of the present application, the reward function can be designed to consist of three parts: global reward, path efficiency reward, and path diversity reward. Global reward: a positive reward is given when the agent successfully reaches the target entity e_target. Path efficiency reward: a reward is given based on the length of the path, and shorter paths usually receive higher rewards. Path diversity reward: encourages the agent to explore different paths, increases the randomness of the strategy, and prevents premature convergence to the local optimal solution.

[0073] In the prior art methods mentioned above, the agent directly samples an action from the environment each time it performs reasoning, and does not save the conversion information obtained after getting feedback from the environment, but directly discards it. This greatly reduces sample utilization and low training efficiency.

[0074] In a possible implementation of the present application, an experience replay pool is introduced. Specifically, a replay buffer is maintained, and the four-tuple data (state s, action a, reward R, next state s') obtained from each interaction in the environment is stored in the experience replay pool. When training the value network, some data is randomly sampled from the experience replay pool for training. This can not only improve the sample utilization rate, but also break the correlation between samples so that they meet the independence assumption.

[0075] Furthermore, in order to reduce the action space during agent training and avoid dimensionality explosion, the embodiment of the present application adopts a pre-training method based on supervised learning to implement it. It performs a breadth-first search between given entity pairs and saves the obtained paths in a set for the agent to be retrained. Next, the embodiment of the present application designs a policy network for knowledge graph reasoning based on the maximum entropy idea of ​​the SAC algorithm. Specifically, in the process of training the agent, in addition to maximizing the cumulative rewards obtained by the agent, the strategy must also be made more random. Entropy represents a measure of the degree of randomness of a random variable. Therefore, combined with the reinforcement learning strategy πθ, the degree of randomness of the strategy in a certain state is expressed as H(π(*|s t )) According to the objective function of the classic reinforcement learning problem mentioned in the previous article, an entropy regularization term controlled by the parameter α is added. The larger the parameter, the stronger the exploratory nature, which helps to accelerate subsequent strategy learning and reduce the possibility of the strategy falling into a poor local optimum.

[0076] In a possible implementation of the present application, by combining the policy network and the value network, efficient reasoning training can be performed on the knowledge graph for the objective function of maximizing the policy entropy.

[0077] For the policy network, state input layer: set the dimension of the input layer according to the shape and size of the input state vector. This layer is responsible for receiving the current state information from the knowledge graph.

[0078] Hidden layer: Designed as a linear neural network layer of size (512, 1024). This layer is used to extract and transform the features of the input state so that the subsequent output layer can generate the action probability distribution.

[0079] Output layer: Set the dimension of the output layer according to the number of relationship sets in the knowledge graph. Use the Softmax function to normalize the output action probabilities to ensure that the sum of all action probabilities is 1.

[0080] Activation function: The ReLU nonlinear activation function is used between the input layer and the hidden layer to enhance the approximation ability of the network and prevent the gradient from disappearing.

[0081] For the value network, the input layer: receives a batch of inference states sampled from the experience replay pool. This layer is responsible for passing state information to subsequent hidden layers.

[0082] Hidden layer: Designed as a two-layer fully connected neural network, the first layer uses the ReLU activation function to prevent the gradient from disappearing. This layer is used to extract and transform the features of state information.

[0083] Activation function: The ReLU activation function is also used between the input layer and the hidden layer.

[0084] In the reasoning process of the agent, a maximum number of attempts, max_step, is set. When the maximum number of attempts is reached or the reasoning has reached the target entity, the current reasoning process ends. N state transition pairs are sampled from the experience replay pool to update the policy network parameters, value network parameters, and entropy regularization coefficients.

[0085] Use the value network to evaluate the sampled states. Combine the policy evaluation and policy entropy output by the value network to update the policy network parameters. By maximizing the policy entropy objective function, the agent is encouraged to explore more action options and avoid falling into the local optimal solution. Invalid reasoning states are penalized to guide the agent to avoid choosing these states. The reasoning path to the target entity is rewarded to encourage the agent to find the correct reasoning path. The completed reasoning path is saved in the output set for subsequent analysis and evaluation.

[0086] In multiple iterations, the parameters of the policy network and value network, as well as the entropy regularization coefficient, are continuously updated. The training effect is evaluated by observing the reasoning performance of the agent on the knowledge graph. When the reasoning performance of the agent reaches stability or no longer improves significantly, the training is considered to have converged.

[0087] Compared with the previous method of training only with policy networks, this application introduces a value network, which optimizes the policy update process by combining policy evaluation and value functions, thereby improving training efficiency and reasoning accuracy. By maximizing the objective function of policy entropy, the agent is encouraged to explore more action options during the reasoning process, thereby enhancing the generalization ability and robustness of the model. The use of technical means such as the experience replay pool and the ReLU activation function further improves the stability and efficiency of the training process.

[0088] Step 5: Use the optimal reasoning strategy obtained through training to complete the knowledge graph of customer risk information to be processed.

[0089] This process can be seen in Figure 2 , Figure 2 Another flow chart of the reasoning method of the knowledge graph provided in the embodiment of the present application. It includes:

[0090] S101. Obtain target information to be processed.

[0091] S102: extract entities from the target information, and construct a knowledge graph based on the extracted entities and the relationships between the entities.

[0092] S103. Model the constructed knowledge graph as a Markov decision environment, which is used to provide a decision-making and reasoning framework for the intelligent agent.

[0093] S104. Reasoning is performed in a Markov decision environment based on the target reasoning strategy of the intelligent agent, so that the intelligent agent can reach the end entity through action selection from the starting entity according to the target strategy.

[0094] S105. The endpoint entity obtained in each round of iteration is saved in a collection. The endpoint obtained in each round of iteration is a customer entity discovered through reasoning that has a risk transmission relationship with the starting entity.

[0095] That is, after obtaining the best knowledge graph reasoning strategy, the entity relationship of the customer risk information to be processed (i.e. the target information to be processed) is extracted to form a knowledge graph, and the TransE model is used again to obtain the vector representation of entities and relationships, and then the knowledge graph is modeled as an MDP environment. The optimal strategy of the intelligent agent is used for reasoning, and the terminal entity obtained in each round of iteration is saved in a collection.

[0096] In actual application scenarios, based on the obtained set, we can mine the risk transmission information between customers, improve the corresponding risk warning level for entity pairs with abnormal transactions between customer entities, and increase the risk attention of other customer entities with potential social relationships that are inferred for entities that already have risks. This will complete the customer risk information knowledge graph and improve the bank's risk management capabilities.

[0097] In summary, compared with the traditional Monte Carlo policy gradient method, the SAC knowledge graph reasoning method has the following significant advantages:

[0098] In the SAC method, entropy is used as an important indicator to measure the randomness of the agent's strategy. By calculating the entropy value of the strategy distribution, the diversity of the agent in choosing the reasoning path can be quantified. The introduction of entropy enables the agent to choose a path more flexibly when faced with multiple optional paths, rather than being limited to the local optimal solution.

[0099] In the process of policy gradient optimization, the SAC method considers both the maximization of cumulative rewards and the maximization of policy entropy. This is achieved by constructing a composite loss function that includes these two objectives. By optimizing this composite loss function, the agent can not only find a high-reward path, but also maintain a high level of exploration ability during reasoning.

[0100] The SAC method adopts the strategy of collaborative training of policy network and value network. The policy network is responsible for generating action probability distribution, while the value network is used to evaluate the value of state or action-state pair. Both are trained by sharing samples in the experience replay pool, jointly optimizing the policy and value estimation, thereby improving the stability and efficiency of training.

[0101] During the reasoning process, the SAC method encourages the agent to explore more paths to reduce the omission of reasoning information. At the same time, by utilizing the information of the explored paths, the agent can gradually improve its reasoning strategy. This balance between exploration and utilization enables the agent to continuously improve the accuracy of reasoning while maintaining a high level of exploration ability.

[0102] The above are some specific implementations of the reasoning method of the knowledge graph provided in the embodiment of the present application. Based on this, the present application also provides a corresponding reasoning system of the knowledge graph. The system provided in the embodiment of the present application will be introduced from the perspective of functional modularization. Figure 3 A structural diagram of the reasoning system of a knowledge graph provided in an embodiment of the present application.

[0103] The system comprises:

[0104] An acquisition unit 110 is used to acquire target information to be processed;

[0105] A construction unit 111 is used to extract entities from the target information and construct a knowledge graph based on the extracted entities and the relationships between the entities;

[0106] A modeling unit 112, used to model the constructed knowledge graph as a Markov decision environment, wherein the Markov decision environment is used to provide a decision-making and reasoning framework for the intelligent agent;

[0107] The reasoning unit 113 is used to perform reasoning in a Markov decision environment based on the target reasoning strategy of the agent, so that the agent starts from the starting entity and reaches the end entity through action selection according to the target strategy;

[0108] The storage unit 114 is used to store the end point entity obtained in each round of iteration in a set. The end point obtained in each round of iteration is a customer entity that is found through reasoning to have a risk transmission relationship with the starting entity.

[0109] In a possible implementation manner, the modeling unit is specifically used to:

[0110] Define a state space, where each state represents an entity or a set of entities in the knowledge graph; define an action space, where each action represents a possible transfer path from the current entity to another entity; set a reward function to evaluate the reward obtained by the agent when it transfers from the current state to the next state, and the size of the reward is set according to the importance of the risk transmission relationship between the entities; set a transition probability, which is used to reflect the possibility of the agent transferring from the current state to the next state under different actions.

[0111] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the test method solutions provided in the embodiments of the present application.

[0112] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes the test method described in any embodiment of the present application.

[0113] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the test method described in any embodiment of the present application.

[0114] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0115] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0116] It should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0117] It should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0118] It should also be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0119] The steps of the method or algorithm described in conjunction with the embodiments disclosed in this application can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0120] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.

Claims

1. A knowledge graph reasoning method, characterized in that: include: Get the target information to be processed; Extracting entities from the target information, and constructing a knowledge graph based on the extracted entities and the relationships between the entities; Modeling the constructed knowledge graph as a Markov decision environment, which is used to provide a decision-making and reasoning framework for the intelligent agent; The agent-based target reasoning strategy is used to reason in a Markov decision environment, so that the agent can reach the end entity through action selection from the starting entity according to the target strategy; The endpoint entity obtained in each round of iteration is saved in a collection. The endpoint obtained in each round of iteration is a customer entity discovered through reasoning that has a risk transmission relationship with the starting entity.

2. The method according to claim 1, characterized in that The constructed knowledge graph is modeled as a Markov decision environment, including: Define a state space, where each state represents an entity or a set of entities in the knowledge graph; define an action space, where each action represents a possible transfer path from the current entity to another entity; set a reward function to evaluate the reward obtained by the agent when it transfers from the current state to the next state, and the size of the reward is set according to the importance of the risk transmission relationship between the entities; set a transition probability, which is used to reflect the possibility of the agent transferring from the current state to the next state under different actions.

3. The method according to claim 1, characterized in that The method further comprises: Use the TransE embedding model to preprocess the knowledge graph; Model the processed knowledge graph as a Markov decision environment; Based on the training method of bidirectional BFS supervised learning, the agent is pre-trained; Based on the maximum entropy strategy network, value network and experience replay pool, the agent's knowledge graph reasoning is retrained to obtain the agent's target reasoning strategy.

4. The method according to claim 3, characterized in that The experience replay pool is used to store the four-tuple data obtained from each interaction in the environment, and when training the value network, a number of data are randomly sampled from the experience replay pool for training.

5. The method according to claim 2, characterized in that: The reward function consists of a global reward, a path efficiency reward, and a path diversity reward, and is used to give a positive reward when the agent successfully reaches the target entity; rewards are given based on the length of the journey and the number of paths.

6. A knowledge graph reasoning system, characterized in that: include: An acquisition unit, used for acquiring target information to be processed; A construction unit, used to extract entities from the target information, and construct a knowledge graph based on the extracted entities and the relationships between the entities; A modeling unit, used to model the constructed knowledge graph into a Markov decision environment, wherein the Markov decision environment is used to provide a decision-making and reasoning framework for the intelligent agent; The reasoning unit is used to perform reasoning in a Markov decision environment based on the target reasoning strategy of the intelligent agent, so that the intelligent agent can reach the terminal entity through action selection from the starting entity according to the target strategy; The storage unit is used to store the endpoint entity obtained in each round of iteration in a collection. The endpoint obtained in each round of iteration is a customer entity discovered through reasoning that has a risk transmission relationship with the starting entity.

7. The system according to claim 6, characterized in that The modeling unit is specifically used for: Define a state space, where each state represents an entity or a set of entities in the knowledge graph; define an action space, where each action represents a possible transfer path from the current entity to another entity; set a reward function to evaluate the reward obtained by the agent when it transfers from the current state to the next state, and the size of the reward is set according to the importance of the risk transmission relationship between the entities; set a transition probability, which is used to reflect the possibility of the agent transferring from the current state to the next state under different actions.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, characterized in that When the computer program product runs on a computer, the computer executes the method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 5.

Images