Knowledge reasoning method based on adversarial reinforcement learning

Abstract

The invention relates to the field of knowledge engineering, and discloses a knowledge inference method based on adversarial reinforcement learning, comprising a generator and a discriminator, using an AttTuck Comp GCN embedding model in a KGE framework to obtain vector representation of entities and relationships, incorporating potential target entity vectors into an RL framework, and using a scoring function as a soft reward for calculating candidate entities; the generator maintains a coverage vector and then calculates the probability of each step of candidate action according to state information, and the discriminator captures context path representation and evaluates the accuracy of answers and the reasonability of reasoning paths; the method has the advantages that the problem that the long-distance dependency relationship and the high-dimensional complex relationship are insufficient in representation capability can be effectively solved, and the problems that convergence is slow and the performance is poor due to non-destination search are avoided. And the problem that false paths exist in the reinforcement learning framework is relieved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge engineering, and in particular to a knowledge reasoning method based on adversarial reinforcement learning. Background Art

[0002] Cloud manufacturing uses advanced information technologies such as cloud computing, the Internet of Things, and big data to virtualize and service decentralized manufacturing resources and capabilities. In a cloud manufacturing environment, product design knowledge presents the following characteristics: huge data volume, including rich design documents, historical data, market feedback, etc.; complex structure, with interdisciplinary and multi-level knowledge; multiple sources, including internal enterprise resources, external public resources, partner knowledge, and user-generated content. These characteristics pose huge challenges to knowledge management and application, and also provide broad space for the application of knowledge graphs (KGs) and question answering systems (QA).

[0003] KGs constructs semantic data models to structure various types of knowledge so that it can be understood and processed by computers, achieving efficient organization and intelligent retrieval. At the same time, KGs gives QA excellent reasoning ability and necessary explainability to answer factual questions. In the field of cloud manufacturing, a question-answering system based on knowledge graphs can quickly and accurately answer factual questions about product design, improving design efficiency and quality.

[0004] Inspired by generative adversarial learning, its generator and discriminator are jointly trained through an adversarial process. The present invention expects that the discriminator can effectively distinguish the correctness of the answer entity and the reasoning chain. In the reward function of the RL framework, the scoring function is used to calculate the probability score of the candidate entity as a soft reward for the target entity instead of the binary reward, alleviating the delay and sparsity problems of the reward. At the same time, the path reward is introduced to alleviate the false path problem. When the agent follows the correct reasoning path, the environment provides a higher positive reward signal; on the contrary, if it walks along a false path, a lower reward is provided to hinder the reasoning process. Combining the two rewards, the optimization of the agent is guided by the policy gradient. To this end, a multi-hop reasoning model based on generative adversarial reinforcement learning is proposed, which can effectively solve the problems of insufficient representation capabilities of long-distance dependencies and high-dimensional complex relationships, avoid aimless search leading to slow convergence, and alleviate the false path problem in the reinforcement learning framework. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a knowledge reasoning method based on adversarial reinforcement learning, which has the advantages of being able to effectively solve problems such as insufficient representation capabilities of long-distance dependencies and high-dimensional complex relationships, avoiding aimless searches that lead to slow convergence, and alleviating the problem of false paths in the reinforcement learning framework, thereby solving the problems in the above-mentioned background technology.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned purpose of effectively solving the problems of insufficient representation ability of long-distance dependencies and high-dimensional complex relationships, avoiding aimless searches that lead to slow convergence, and alleviating the problem of false paths in the reinforcement learning framework.

[0009] In order to solve the above problems, the present invention provides the following technical solutions:

[0010] A knowledge reasoning method based on adversarial reinforcement learning, including a generator G θ and the discriminator D φ , the AttTuck Comp GCN embedding model is used in the KGE framework to obtain vector representations of entities and relations, the potential target entity vector is incorporated into the RL framework, and the scoring function is used as the soft reward for calculating candidate entities; the generator maintains a coverage vector and then calculates the probability of each candidate action based on the state information, the discriminator captures the context path representation and evaluates the accuracy of the answer and the rationality of the reasoning path; the two feedbacks of the discriminator are incorporated into the reward function, and the generator learns to maximize the expected reward and execute the action;

[0011] Step S1: knowledge graph embedding;

[0012] Step S2: reinforcement learning;

[0013] Step S3: policy network;

[0014] Step S4: Discriminator;

[0015] Step S5: Model optimization.

[0016] According to the above technical solution, step S1 further includes:

[0017] Knowledge graph embedding converts entities and relations into low-dimensional vectors, a triple embedding <e h ,r,e t >∈G is denoted as <e h ,r,e t >And randomly initialize, define a scoring function f s (e h ,r,e t ) to measure the correctness of the triple;

[0018] In the AttTuck Comp GCN embedding model, the knowledge graph is represented as a complex relationship graph, denoted as G = (V, R, E, X, Z);

[0019] Where V represents the entity set, R represents the relationship set, and E represents the edge set. and Represent the initial input features of each entity and relation respectively. The initial vector z of the rth relation can be expressed as a set of basis vectors {v0,v1,...,v B}, the formula is as follows:

[0020]

[0021] Among them, B represents the number of basis vectors, α br Represents the weight coefficient of the rth relationship in the bth basis vector, which can be continuously learned and updated as a network parameter. AttTuck Comp GCN aggregates and updates neighbor entities and relationships through a message passing mechanism. The graph convolution operation of each layer can be expressed as:

[0022]

[0023] in, represents the node v obtained after k layers, represents the relationship r obtained after k layers, It is a matrix related to the relationship category, including three categories W O , W I , W S , corresponding to the positive relationship, the inverse relationship and the self-loop respectively; All relationships are projected into the same embedding space as nodes. When aggregating neighbor information, an attention mechanism is introduced in the propagation layer. The propagation matrix learned is It reflects the weight of the neighbor entity to the central node during the feature aggregation process. The formula is as follows:

[0024]

[0025] Among them, k is the number of propagation layers, β k is the adaptive coefficient, Calculate the cosine distance between entities u and v in the hidden state of the kth layer to capture the similarity between the two. The higher the similarity between the two, the greater the weight of the central node.

[0026] The semantic features between many entities and relations in the knowledge graph are the same, so a core tensor can be used to replace the shared semantic features. The formula is as follows:

[0027]

[0028] Among them, G represents the core tensor, which captures the shared semantic features. and Represents the semantic vector unique to each entity;

[0029] This method achieves efficient combined embedding of entities and relations, can capture the complex interactive information between entities and relations, and effectively reduce the number of parameters. It not only improves the model's ability to represent complex relations, but also enhances its performance in multi-hop reasoning tasks.

[0030] KGE has three ideal properties. First, cloud manufacturing product design knowledge entities and relations can be embedded into vector space for efficient representation and reasoning; second, the vectors of potential target entities are regarded as expected information of the RL framework to reduce aimless exploration; finally, the scoring function can calculate soft rewards, alleviating the problems of reward delay and sparsity.

[0031] According to the above technical solution, step S2 further includes:

[0032] The formalization of RL is based on the Markov decision process, which consists of a tuple<S,A,P,R> Composition, where S is the state space, A is the action space, P is the state transition, and R is the reward function;

[0033] At each time step t, the agent first observes the state of the environment, then selects an action according to the policy network, and provides a scalar reward signal to the agent after the environment changes to a new state. The agent's goal is to perform an optimal action in each state to maximize the expected cumulative reward.

[0034] Step S21, state space:

[0035] At time step t, the original state is defined as S t =(q,e0,e t ,h t ), where e0 represents the subject entity of a given question q, e t represents the entity visited in the current step during reasoning, h t is the decision history of the agent;

[0036] The vector of the potential target entity is embedded into the state of the RL framework as an expectation. First, the probability distribution of the candidate entities is calculated, and then the entity with the highest probability is selected as the predicted answer. The vector of this entity is the predicted embedding h ant Finally, the expected embedding is added to the state of the RL framework, and the state representation is changed to S t =(q,e0,e t ,h t ,h ant );

[0037] Step S22, action space:

[0038] Given state S t , action space A(S t ) by the current entity e t All outgoing edges in KGs are composed of A(S t )={(r,e)|(e t ,r,e)∈G}, since we cannot know the number of hops required to answer the question in advance, we t ) adds a self-loop edge (r loop ,e t ) is used to indicate the termination condition of the reasoning process. Once the agent selects a self-loop edge, it will stay at the current entity point and terminate the path search process.

[0039] Step S23: state transfer:

[0040] If the current state is S t , the agent performs action A t =(r * ,e * ), then the current state S t will be converted to S t+1 =(q,e0,e * ,h t+1 ,h ant ), where h t+1 =h t ∪{A t}, q, e0 and h ant is global information and does not change during the conversion;

[0041] Step S24, reward function:

[0042] The scoring function in KGE is used to calculate the soft reward of the candidate entity, and it is expected to add additional path rewards to the reward function. Formally, the reward function is defined as follows:

[0043]

[0044] Among them, S t =(q,e0,e T ,h T ,h ant ) represents the final state, e T Represents the predicted entity, e0 and h q denote the subject entity embedding and question embedding respectively, and the soft reward is determined by the KGE scoring function f s (·) provides a continuous feedback reward value, indicating the similarity between the predicted entity and the correct answer. The discriminator D φWith question q and relationship path As input, the correctness of the path is evaluated and λ is a hyperparameter that balances the two rewards.

[0045] According to the above technical solution, step S3 further includes:

[0046] Generator G θ It can be regarded as a policy network of RL, which calculates the probability of candidate actions, samples the most promising actions at each step, and selects an action for the next step of reasoning;

[0047] Given a problem q=(q1,q2,...,q n ), each token q i Send it to the embedding layer to get the corresponding word embedding. Then, send the word embedding sequence to the Transformer encoder to encode the question and get a series of hidden states Through the LSTM network, the decision history h t =(A1,A2,...,A t-1 ) is encoded, where A i =(r i ,e i ) represents the action selected in step i, A i The embedding representation of where r i and e i Initialize it through the pre-trained KGE module to obtain the corresponding vector representation, which is represented by h t =LSTM(h t-1 ,A t-1 ) Calculate the hidden state h of the decision history t ;

[0048] Borrowing the idea of ​​the coverage mechanism, a coverage vector c is maintained t ∈R n*1 , which is the sum of the attention distributions about the question token in all previous reasoning steps and is formulated as follows:

[0049]

[0050] Given a state S t =(q,e0,e t ,h t ,h ant ) and the corresponding action space A(S t )={(r,e)|(e t ,r,e)∈G}, compute the contextual problem representation based on the attention mechanism to learn each action A * =(r * ,e *)∈A(S t )’s entity relations and the word-level alignment between the question tokens, covering the vector c t is used as an additional input to the attention mechanism, and the formula is as follows:

[0051]

[0052] a * =softmax(e * ) (8)

[0053]

[0054] Among them, q i is the token q i The embedding of i and r i Represents entity e i and the relationship i The vector, v, W q , W e , W r , w c is a learnable parameter. When the agent selects action A at time step t * When * In formula (6), it is assigned to a t ;

[0055] Define a computational action space A(S t ) Probability distribution of policy network π θ (A * |S t ) to select the action, the formula is as follows:

[0056]

[0057] Among them, A * It is action A * Embedding, [h t ;q * ;h ant ] represents the concatenation of the encoded decision history, context-aware question vector, and expected goal embedding, and is the parameter matrix, which is the generator G θ (·) is considered as the policy network π θ (·).

[0058] According to the above technical solution, step S4 further includes:

[0059] Discriminator D φThe goal is to evaluate the accuracy of the answer and the rationality of the reasoning path, and to push the generator to infer the answer along the correct path. Similar to the generator, the question token It is encoded through a Transformer encoder and then a maximum pooling layer is applied to obtain the question context vector h q ∈R →d*1 , given a relationship path The attention mechanism is also used to calculate the context-aware question vector. The calculation formula of the context-aware question vector is:

[0060]

[0061] Among them, q i is the token q i The embedding of i and r i Represents entity e i and the relationship i The vector, v d , W q' , W e' , W r' is a learnable parameter;

[0062] By connecting t and r t Get a context vector o t =[q t ; r t ], so the relationship path The embedding of (o1,o2,...,o T ) means that the question context vector h q Add to Relationship Path (W v *h q ,o1,o2,...,o T ), where W v is the parameter matrix used to transform the dimension. Then, the relationship path is encoded through the LSTM network, and the last hidden state h' T+1 It is considered as the context representation of the relation path. Since the question context vector is added to the head of the original relation path, the path length increases from T to T+1. Finally, the score of the relation path is modeled as follows:

[0063]

[0064] Among them, σ(·) is the sigmoid function, w s is a learnable parameter;

[0065] The scoring function of the KGE framework is used to measure the credibility of each candidate question. The formula is as follows:

[0066] s k =f s (e0,h q ,e T ) (14)

[0067] According to the above technical solution, step S5 further includes:

[0068] Given a question q, and Represent the correct relationship path and the wrong relationship path respectively. The discriminator hopes Maximize the score, Score minimization, adversarial learning can be formulated as a minimum-maximum optimization problem, with the following formula:

[0069]

[0070] Among them, P R (q,p r ) and P R (q) represent the joint probability distribution and probability distribution from real data, respectively, P G (p r |q) is the conditional probability distribution of generated data;

[0071] In order to avoid the unsolvable gradient caused by the discrete nature of text data, the policy gradient is used to directly train the generative model. Drawing on the idea of ​​Ql earning, the goal of the generator is to obtain the answer entity and maximize the expected final reward. The formula is as follows:

[0072]

[0073] Among them, R(S T ) represents the reward of the terminal state, S0=(q,e0,e0,h0) is the initial state, Q(S t ,A t ) is in state S t Next take action A t The state-action value at the time, G θ (A t |S t ) Calculate the action probability;

[0074] Since the discriminator is trained to assign reward signals to fully generated reasoning paths, it cannot provide scores for intermediate steps. To improve training efficiency, the state-action value Q(S t ,A t ) as the final delayed reward R(S T ) discount, the formula is as follows:

[0075] Q(S t ,A t )=γ T-1 R(S T ) (17)

[0076] Among them, γ<1 is the discount factor;

[0077] Defining a value network To reduce the variance of the estimate, a single-layer perceptron is used to The modeling formula is as follows:

[0078]

[0079] Among them, [h t ;e t ;h ant ] represents the concatenation of the encoded decision history, current entity embedding, and expected target embedding, W v is a learnable parameter, Optimize by minimizing the expected mean square error, the formula is as follows:

[0080]

[0081] For the value network, the generator G θ The policy gradient derivation is as follows:

[0082]

[0083] The true relationship path is regarded as a positive sample, and the generator G θ The generated false relationship paths are regarded as negative samples, and the loss function of the discriminator is expressed as:

[0084]

[0085] (III) Beneficial effects

[0086] Compared with the prior art, the present invention provides a knowledge reasoning method based on adversarial reinforcement learning, which has the following beneficial effects:

[0087] This paper focuses on improving the KGQA system's ability to represent complex relationships and multi-hop reasoning to support product design in a cloud manufacturing environment.

[0088] This paper proposes the AttTuck Comp GCN knowledge embedding model to solve the problems of insufficient representation of long-distance dependencies and high-dimensional complex relationships, and designs a KGE framework to measure the credibility of <topic entity, question, answer entity> in the embedding space and assign soft rewards to the RL agent;

[0089] This paper proposes a new knowledge reasoning method based on reinforcement learning, introduces expected information for purposeful exploration, and uses an adversarial reinforcement learning framework to alleviate the false path problem in multi-hop KGQA. The generator generates answers and expands the reasoning path, and the discriminator evaluates the accuracy of the answer and the rationality of the reasoning path.

[0090] The reasoning model of the present invention not only improves the performance of the knowledge graph question-answering system, but also effectively manages and applies cloud manufacturing product design knowledge, promotes collaborative innovation in cloud manufacturing product design, improves design efficiency, and provides personalized product design solutions for smart manufacturing companies, thereby enhancing the overall competitiveness of my country's manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is an example diagram of a false path of the present invention;

[0092] Figure 2 It is the overall framework diagram of the model of the present invention;

[0093] Figure 3 This is an example diagram of the reward of the present invention;

[0094] Figure 4 It is the performance diagram of different G steps and D steps based on PQL of the present invention;

[0095] Figure 5 is the PQL-based discriminator score graph of the present invention;

[0096] Figure 6 It is a graph of different reward coefficients λ based on the PQL verification set of the present invention;

[0097] Figure 7 This is a case diagram of the present invention. DETAILED DESCRIPTION

[0098] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0099] Embodiment 1:

[0100] 1. Dataset

[0101] (1) PathQuestion (PQ) was constructed by Zhou et al. and uses a subset of Freebase to construct questions and reasoning paths, aiming to enhance the multi-relational question answering capabilities in knowledge graphs; PQ-2H and PQ-3H represent 2-hop and 3-hop questions, respectively; PQ-M represents the combination of all questions.

[0102] (2) PathQuestion-Large (PQL) is more challenging because PQL provides fewer training instances but uses larger KGs. Similarly, PQL-2H and PQL-3H represent 2-hop and 3-hop problems respectively; PQL-M is a mixed dataset.

[0103] Table 1 shows the detailed statistics of the dataset. The first block shows the diversity of training / validation / test / question modes; the question mode is obtained by replacing the subject entity in the question with “ <e>" mark; the second block details the scale of the KGs these datasets rely on.

[0104] Table 1. Statistics of benchmark datasets

[0105]

[0106] 2. Experimental details

[0107] Due to the complex relations and the inverse relations involved in the reasoning process, the direction of each triple in the KGs that the PQL depends on is also reversed. For example, given a triple "(3D Modeling-Software Tools-Autodesk Fusion 360)", the inverse triple "(Autodesk Fusion 360-Software Tools-3D Modeling)" is then added to the KGs. In addition, the optimization of the discriminator requires the generator to generate negative samples. In order to sample negative samples, beam search is first applied to maintain multiple relationship paths. If one of the paths is different from the true path, it is considered a negative sample. It is possible that all generated paths are false negative samples (i.e., the same as the golden path), so negative samples are empirically created by randomly replacing one of the relations in the path.

[0108] In the experiment, the CPU model used is Xeon(R) Platinum 8474C / 15 cores, the GPU model is NVIDIARTX 4090D / 24GB, the CUDA version is 12.1, the PyTorch version is 2.1.0, the Python version is 3.10, and the 300-dimensional pre-trained BERT model word embedding and 200-dimensional KGs embedding are applied. The KGs embedding is pre-trained by AttTuck Comp GCN.

[0109] For the Transformer encoder, the present invention sets 2 layers and 4 heads; the unidirectional LSTM has two layers and the hidden dimension is 200; the reward coefficient λ in formula (5) is 0.5; the path reasonable threshold τ is 0.9; the discount factor γ in equation (17) is 0.95; in each cycle G θ and D φ The number of training times is 5. ADAM optimizer is used for parameter optimization, and the initial learning rate is 0.0001. In the test phase, beam search is applied to generate multiple reasoning paths on KGs, and the beam size is 2.

[0110] The benchmark models cover a comprehensive range of relevant models. In general, the benchmark models are subdivided into the following three refined categories according to the technical routes: neural network-based methods, namely MemNN, KVMemNN, IRN and DRN; graph neural network (GNN)-based methods, namely GraftNet, HyperTransformer and RAID; path-based models, namely MINERVA, SRN and ARN.

[0111] 3. Experimental results and analysis

[0112] The Hits@1 score is used to evaluate the performance of the model, and the experimental results compared with 10 baseline models are shown in Table 2. From the results, the proposed method achieves satisfactory performance on all datasets. It can be observed that for neural network-based methods (such as MemNN, KVMemNN, and IRN), the performance on the PQL dataset is significantly reduced compared to the PQ dataset. This may be because PQL contains larger-scale KGs, (KV)MemNN must process more triples in the memory unit, and the multi-task training mode designed by IRN requires learning KG embedding every few epochs, which will mislead the goal of KGQA. DRN is trained in a strongly supervised manner, that is, it hopes to know the golden relationship in the middle in advance. In contrast, the model considers a weakly supervised training process.

[0113] Table 2 Comparative test results

[0114]

[0115] The GNN-based method performs better than PQ on the PQL dataset. GNN can better handle complex graph structures through message passing mechanism and enhance multi-hop reasoning ability. It is worth noting that Hyper Transformer improves performance by jointly optimizing hypergraph representation, optimizing problem and knowledge alignment. However, GNN focuses on answer acquisition and lacks interpretability. The model relies on the interpretable RL framework, and the strategy guides the agent to execute action sequences, forming a clear reasoning path to the answer entity, enhancing traceability and interpretability.

[0116] Compared to path-based models (i.e., MI NERVA, SRN, and ARN), our model performs better on the mixed dataset, demonstrating that GARL is capable of variable-length reasoning. With the help of the discriminator, the generator receives feedback on the correctness of the relation paths, thereby alleviating the false path problem faced by reinforcement learning-based methods.

[0117] The above analysis shows that the proposed model achieves consistent excellent results under different data scales, successfully answers multi-hop questions on KGs and improves interpretability and correctness.

[0118] Embodiment 2:

[0119] The present invention provides a technical solution: a knowledge reasoning method based on adversarial reinforcement learning, including an ablation experiment to verify the influence of different elements in the model, respectively changing the expected embedding, the context-aware problem representation in the discriminator and the reward function, as shown in Table 3.

[0120] Table 3 Ablation experiment results

[0121]

[0122] First, the expected target embedding h is cancelled in equations (10) and (17). ant Therefore, the state at time step t is defined in its original form S t =(q,e0,e t ,h t ). Policy network π θ (A * |S t ) is the concatenation of the encoding decision history and context-aware question vector. In addition, the model value network Only the encoded decision history is retained. The results show that the performance of the ablation model consistently decreases, which indicates that incorporating knowledge graph embeddings as expected information into RL states can reduce aimless exploration in KGs.

[0123] Secondly, ignore the discriminator D calculated by formulas (11)-(12) φ In the context-aware question representation in

[15] , only relation embeddings are retained in relation paths. Ablation results show the advantage of learning word-level alignments between entity, relation, and question tokens to obtain contextual information.

[0124] Finally, the reward function in formula (5) is modified. The reward function is divided into two parts: soft reward and path reward. By removing each reward, this paper conducts two ablation experiments respectively. The above observations show that both rewards are beneficial to multi-hop KGQA. The soft reward is used to guide the agent to predict the correct answer entity, and the path reward is used to evaluate the rationality of the reasoning path.

[0125] To further prove the effectiveness of the model, the influence of G-steps and D-steps, the effectiveness of the discriminator, and the influence of the hyperparameter λ are also considered.

[0126] (1) Impact of G-steps and D-steps

[0127] The stability of GARL depends on the training strategy. Specifically, the ratio of G-steps to D-steps has a significant impact on the convergence and performance of the model. Taking the PQL dataset as an example, Figure 4 The solid line in the figure represents the average performance, showing the impact of different proportions and the changes in the evaluation indicators as the number of epochs increases.

[0128] from Figure 4 It can be seen that when G-steps:D-steps=1:1, both components are underfitting. When G-steps is increased to 5 steps only, the generator is able to predict the correct answer, while the reasoning path may be wrong due to the inability to distinguish false paths. If D-steps is modified to 5, but G-steps is still equal to 1, the generator cannot generate high-quality negative samples for the discriminator, resulting in poor performance of answer prediction and path reasoning. Only when G-steps:D-steps=5:5, the adversarial learning process can be stable and effective.

[0129] (2) Effectiveness of the Discriminator

[0130] The discriminator hopes to maximize the score of positive samples and minimize the score of negative samples. Figure 5 The discriminator score on the PQL dataset is shown as the number of training times increases. The results show that the discriminator can gradually assign high rewards to correct answers and reasoning paths, and low rewards to incorrect answers and reasoning paths.

[0131] (3) Impact of hyperparameter λ

[0132] The hyperparameter λ in formula (5) is used to balance the soft reward and path reward. λ is tuned between {0.3, 0.5, 0.7} based on the validation set of the PQL dataset. Figure 6 The impact of different settings is shown. It can be observed that in the PQL dataset, the best value of answer Hits@1 and path accuracy should be 0.5, which indicates that soft rewards and path rewards play equally important roles in inferring the answer. Specific embodiment 1:

[0134] First, a case study of the sequential decision process is conducted. During the test phase, beam search is used to maintain multiple reasoning paths on KGs. An example is designed. Figure 7 The action probability of each jump is visualized in to illustrate the reasoning steps. The topic is "Which software tool is mainly used when designing smart bracelet products?", and the subject entity is "smart bracelet product design". The action space consists of all outgoing edges of the subject entity, namely (self, smart bracelet product design), (use_techno l ogy_i nv, 3D mode ling), (use_techno l ogy_i nv, embeddedsystems), and (is_part_i nv, sensor). Note that "i nv" is an abbreviation for "i nverse". The generator executes actions (use_techno l ogy_i nv, 3D mode ling) and (use_techno l ogy_i nv, embedded systems) in the first step. After the environment changes to the new state, the action space of the second step consists of the outgoing edges of all entities reached in the previous step. Similarly, the top-N actions are selected based on the action probabilities derived by the generator. Finally, the tail entity with the highest score, Autodesk Fusion 360, is taken as the predicted answer.

[0135] Table 4 Positive and negative score sample table

[0136]

[0137] Secondly, the running examples of positive and negative sample evaluation are instantiated in Table 4. The question is "Which software tool is mainly used for the technology-dependent design of smart bracelet products?" The discriminator can effectively assign high scores to the correct relationship paths and output low scores to negative samples to judge false paths.

[0138] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0139] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.< / e>

Claims

1. A knowledge reasoning method based on adversarial reinforcement learning, characterized in that: Including the generator Gθ and the discriminator In the KGE framework, the AttTuckCompGCN embedding model is used to obtain vector representations of entities and relations, the potential target entity vector is incorporated into the RL framework, and the scoring function is used as the soft reward for calculating candidate entities; the generator maintains a coverage vector and then calculates the probability of each candidate action based on the state information. The discriminator captures the context path representation and evaluates the accuracy of the answer and the rationality of the reasoning path; the feedback of the discriminator is incorporated into the reward function, and the generator learns to maximize the expected reward and executes the action; S1: Knowledge graph embedding; S2: Reinforcement learning; S3: policy network; S4: discriminator; S5: Model optimization.

2. A knowledge reasoning method based on adversarial reinforcement learning according to claim 1, characterized in that: The S1 further comprises: Knowledge graph embedding converts entities and relations into low-dimensional vectors, a triple embedding <e h ,r,e t >∈G is denoted as <e h ,r,e t >And randomly initialize, define a scoring function f s (e h ,r,e t ) to measure the correctness of the triple; In the AttTuckCompGCN model, the knowledge graph is represented as a complex relationship graph, denoted as G = (V, R, E, X, Z); Where V represents the entity set, R represents the relationship set, and E represents the edge set. and Represent the initialization input features of each entity and relation respectively. The initial vector z of the rth relation can be expressed as a set of basis vectors {v0, v1, ..., v B }, the formula is as follows: Among them, B represents the number of basis vectors, α br Represents the weight coefficient of the rth relationship in the bth basis vector, which can be continuously learned and updated as a network parameter. AttTuckCompGCN aggregates and updates neighbor entities and relationships through a message passing mechanism. The graph convolution operation of each layer can be expressed as: in, represents the node v obtained after k layers, represents the relationship r obtained after k layers, It is a matrix related to the relationship category, including three categories W O , W I , W S , corresponding to the positive relationship, the inverse relationship and the self-loop respectively; All relationships are projected into the same embedding space as nodes. When aggregating neighbor information, an attention mechanism is introduced in the propagation layer. The propagation matrix learned is It reflects the weight of the neighbor entity to the central node during the feature aggregation process. The formula is as follows: Among them, k is the number of propagation layers, β k is the adaptive coefficient, Calculate the cosine distance between entities u and v in the hidden state of the kth layer; The semantic features between many entities and relations in the knowledge graph are the same, so a core tensor can be used to replace the shared semantic features. The formula is as follows: Among them, G represents the core tensor, which captures the shared semantic features. and Represents the semantic vector unique to each entity.

3. The knowledge reasoning method based on adversarial reinforcement learning according to claim 1, characterized in that: The S2 further comprises: The formalization of RL is based on the Markov decision process, which consists of a tuple<S,A,P,R> Composition, where S is the state space, A is the action space, P is the state transition, and R is the reward function; At each time step t, the agent first observes the state of the environment, then selects an action based on the policy network, and provides a scalar reward signal to the agent after the environment changes to a new state. The agent's goal is to perform an optimal action in each state to maximize the expected cumulative reward; S21, state space: At time step t, the original state is defined as S t =(q, e0, e t ,h t ), where e0 represents the subject entity of a given question q, e t represents the entity visited in the current step during reasoning, h t is the decision history of the agent; The vector of the potential target entity is embedded into the state of the RL framework as an expectation. First, the probability distribution of the candidate entities is calculated, and then the entity with the highest probability is selected as the predicted answer. The vector of this entity is the predicted embedding h ant Finally, the expected embedding is added to the state of the RL framework, and the state representation is changed to S t =(q, e0, e t ,h t ,h ant ); S22, Action Space: Given a state S t , action space A(S t ) by the current entity e t All outgoing edges in KGs are composed of A(S t )={(r,e)|(e t , r, e)∈G}, since we cannot know the number of hops required to answer the question in advance, we t ) adds a self-loop edge (r loop , e t ) is used to indicate the termination condition of the reasoning process. Once the agent selects a self-loop edge, it will stay at the current entity point and terminate the path search process. S23, state transfer: If the current state is S t , the agent performs action A t =(r * , e * ), then the current state S t will be converted to S t+1 =(q, e0, e * ,h t+1 ,h ant ), where h t+1 =h t ∪{A t }, q, e0 and h ant is global information and does not change during the conversion; S24. Reward function: The scoring function in KGE is used to calculate the soft reward of the candidate entity, and it is expected to add additional path rewards to the reward function. Formally, the reward function is defined as follows: Among them, S t =(q, e0, e T ,h T ,h ant ) represents the final state, e T represents the predicted entity, e0 and hq represent the subject entity embedding and question embedding respectively, and the soft reward is calculated by the KGE scoring function f s (·) Provides a continuous feedback reward value, indicating the similarity between the predicted entity and the correct answer, the discriminator With question q and relationship path As input, the correctness of the path is evaluated and λ is a hyperparameter that balances the two rewards.

4. The knowledge reasoning method based on adversarial reinforcement learning according to claim 1, characterized in that: The S3 further comprises: Generator G θ It can be regarded as a policy network of RL, which calculates the probability of candidate actions, samples the most promising actions at each step, and selects an action for the next step of reasoning; Given a problem q=(q1, q2, ..., q n ), each token q i Send it to the embedding layer to get the corresponding word embedding. Then, send the word embedding sequence to the Transformer encoder to encode the question and get a series of hidden states Through the LSTM network, the decision history h t =(A1, A2, ..., A t-1 ) is encoded, where A i =(r i , e i ) represents the action selected in step i, A i The embedding of i =[r i ;e i ]∈R 2d*1 , where r i and e i Initialize it through the pre-trained KGE module to obtain the corresponding vector representation, which is represented by h t =LSTM(h t-1 , A t-1 ) Calculate the hidden state h of the decision history t ; Borrowing the idea of ​​the coverage mechanism, a coverage vector c is maintained t ∈R n*1 , which is the sum of the attention distributions about the question token in all previous reasoning steps and is formulated as follows: Given a state S t =(q, e0, e t ,h t ,h ant ) and the corresponding action space A(S t )={(r,e)|(e t ,r,e)∈G}, compute the contextual question representation based on the attention mechanism to learn each action A * =(r * , e * )∈A(S t )’s entity relations and the word-level alignment between the question tokens, covering the vector c t is used as an additional input to the attention mechanism, and the formula is as follows: a*=softmax(e*)(8) Among them, q i is the token q i The embedding of i and r i Represents entity e i and the relationship i The vector, v, W q , W e , W r , w c is a learnable parameter. When the agent selects action A at time step t * When * In formula (6), it is assigned to a t ; Define a computational action space A(S t ) Probability distribution of policy network π θ (A t |S t ) to select the action, the formula is as follows: Among them, A * It is action A * Embedding, [h t ;q * ;h ant ] represents the concatenation of the encoded decision history, context-aware question vector, and expected goal embedding, and is the parameter matrix, the generator G θ (·) is considered as the policy network π θ (·).

5. The knowledge reasoning method based on adversarial reinforcement learning according to claim 1, characterized in that: The S4 further comprises: Discriminator The goal is to evaluate the accuracy of the answer and the rationality of the reasoning path, and to push the generator to infer the answer along the correct path. Similar to the generator, the question token It is encoded through a Transformer encoder and then a maximum pooling layer is applied to obtain the question context vector h q ∈R →d*1 , given a relationship path The attention mechanism is also used to calculate the context-aware question vector. The calculation formula of the context-aware question vector is: Among them, q i is the token q i The embedding of i and r i Represents entity e i and the relationship i The vector, v d , W q′ , W e′ , W r′ is a learnable parameter; By connecting t and r t Get a context vector o t =[q t ; r t ], so the relationship path The embedding of (o1, o2, ..., o T ) means that the question context vector h q Add to Relationship Path (W v *h q , o1, o2, ..., o T ), where W v is the parameter matrix used to transform the dimension. Then, the relationship path is encoded through the LSTM network, and the last hidden state h′ T+1 It is considered as the context representation of the relation path. Since the question context vector is added to the head of the original relation path, the path length increases from T to T+1. Finally, the score of the relation path is modeled as follows: Among them, σ(·) is the sigmoid function, w s is a learnable parameter; The scoring function of the KGE framework is used to measure the credibility of each candidate question. The formula is as follows: s k =f s (e0,h q ,e T ) (14) 6. The knowledge reasoning method based on adversarial reinforcement learning according to claim 1, characterized in that: The S5 further comprises: Given a question q, and Represent the correct relationship path and the wrong relationship path respectively. The discriminator hopes Maximize the score, Score minimization, adversarial learning can be formulated as a minimum-maximum optimization problem, with the following formula: Among them, P R (q, p r ) and P R (q) represent the joint probability distribution and probability distribution from real data, respectively, P G (pr|q) is the conditional probability distribution of generated data; The goal of the generator is to obtain the answer entity and maximize the expected final reward, as follows: Among them, R(S T ) represents the reward of the terminal state, S0 = (q, e0, e0, h0) is the initial state, Q(S t , A t ) is in state S t Next take action A t The state-action value at the time, G θ (A t |S t ) Calculate the action probability; Calculate the state-action value Q(S t , A t ) as the final delayed reward R(S T ) is discounted by the following formula: Q(S t ,A t )=γ T-1 R(S T ) (17) Among them, Y < 1 is the discount factor; Defining a value network To reduce the variance of the estimate, a single-layer perceptron is used to The modeling formula is as follows: Among them, [h t ;e t ;h ant ] represents the concatenation of the encoded decision history, current entity embedding, and expected target embedding, W v is a learnable parameter, Optimize by minimizing the expected mean square error, the formula is as follows: For the value network, the generator G θ The policy gradient derivation is as follows: The true relationship path is regarded as a positive sample, and the generator G θ The generated false relationship paths are regarded as negative samples, and the loss function of the discriminator is expressed as:

Images