A fact detection method based on deep interactive reasoning model

Abstract

The present invention discloses a fact detection method based on a deep interactive reasoning model, including: using a BERT model to encode evidence and statements respectively; extracting the semantic information of evidence through a multi-head mapping mechanism and saving it in a memory pool; calculating the semantic relationship between the statement and each piece of evidence; obtaining the overall representation of the evidence set based on an attention mechanism, and calculating the reasoning relationship between the statement and the overall representation of the evidence; upgrading the reasoning structure to a multi-layer structure, iteratively updating the reasoning relationship; and using the updated evidence representation to judge the authenticity of the statement. The multi-head mapping mechanism in the method of the present invention can extract the semantic information of evidence from different angles; the memory pool structure gives the model the ability to handle "multi-hop" problems; and the multi-layer reasoning structure can deeply explore the reasoning relationship between evidence and statements. Compared with other methods, the method of the present invention has better prediction effects on the UKP-Snopes dataset.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a fact detection method based on a deep interactive reasoning model. Background Art

[0002] Fact detection based on natural language processing technology aims to automatically judge the authenticity of information by using natural language processing related technologies. Given a statement whose authenticity is to be verified, the detection method can be divided into three steps: (1) Article retrieval: In order to narrow the search space, a collection of articles related to the statement is retrieved from the knowledge base based on the statement as the basic knowledge for verifying the statement; (2) Evidence extraction: Articles often contain a lot of noise information, so it is necessary to further screen out the evidence collection related to the statement and refine the evidence granularity to the sentence level as the direct knowledge for verifying the statement; (3) True or false reasoning: Based on the evidence collection, the authenticity of the statement is judged through natural language processing methods and deep learning models. In the task of fact detection, article retrieval methods and evidence extraction methods are relatively mature and have high recall rate indicators. At present, most scholars focus their research on the true or false reasoning stage.

[0003] True or false reasoning methods can be divided into two categories: (1) The model first obtains the overall representation of the evidence set, and then combines the statement and the overall representation to predict the result. In this type of method, the overall representation of the evidence set is the key part. If the representation is inaccurate due to noise or irrelevant information, the prediction result will be greatly affected; (2) The model first determines the reasoning relationship between each piece of evidence and the statement, and then aggregates all relationships to judge the statement. There are some statements that need to be judged by combining multiple pieces of evidence at the same time, and this type of method cannot solve this "multi-hop" problem. Both types of methods have certain defects, and how to avoid these two defects is the difficulty of model design.

[0004] In the task of true or false reasoning, the limitations of existing methods are mainly the following three points: (1) The current method is relatively simple in constructing the reasoning relationship between evidence and statements, resulting in weak reasoning ability of the model; (2) The current method first determines the relationship between each piece of evidence and the statement, and infers the truth or falsity of the statement through relationship aggregation. Such methods cannot handle the "multi-hop" problem; (3) In current work, there is insufficient or excessive interaction between evidence and statements, which has a negative impact on the reasoning effect. Summary of the invention

[0005] To solve the above problems, the purpose of the invention is: In order to overcome the shortcomings of existing methods in true and false reasoning tasks, the present invention provides a fact detection method based on a deep interactive reasoning model, which can capture the information provided by evidence from different angles, has the ability to handle "multi-hop" problems, and can deeply explore the reasoning relationship between evidence and statements, and judge the authenticity of statements through reasoning relationships.

[0006] The present invention discloses a fact detection method based on a deep interactive reasoning model, comprising the following steps:

[0007] (1) Use the BERT model to encode statements and evidence separately;

[0008] (2) Extracting the semantic information of evidence through a multi-head mapping mechanism and saving it into the memory pool;

[0009] (3) Compute the semantic relationship between the statement and each piece of evidence; obtain the overall representation of the evidence set based on the attention mechanism, and calculate the inference relationship between the statement and the overall representation of the evidence;

[0010] (4) Upgrading the reasoning structure to a multi-layer structure and iteratively updating the reasoning relationship;

[0011] (5) The truthfulness of the statement is judged through a single-layer feedforward neural network and softmax operation.

[0012] Preferably, in step (1), the BERT model is used to encode the statement and the evidence respectively to obtain the statement encoding Emb c and evidence encoding {Emb1,Emb2,...,Emb N}. BERT uses Transformer as the encoder. The Transformer model uses the self-attention mechanism to capture the relationship between texts at different distances. The bidirectional Transformer model ensures that BERT can take into account both the previous and the following information.

[0013] For the statement c = [c1,c2,…,c lc ], where c i represents the i-th word in the statement, lc represents the length of the statement, and the corresponding evidence set is E = {e1, e2, ..., e M},e i =[e i,1 ,e i,2 ,…,e i,le ](1≤i≤M), where e i,j represents the jth word in the ith piece of evidence, le represents the length of the ith piece of evidence, and M represents the number of corresponding evidences.

[0014] For the statement and each piece of evidence, they are input into the BERT model to obtain the statement encoding vector and the evidence encoding vector {Emb1,Emb2,...,Emb M}, as shown in formulas (1) and (2), where Emblen represents the encoding dimension.

[0015] Emb c =BERT(c1,c2,…c lc ) (1)

[0016] Emb i =BERT(e i,1 ,e i,2 ,…e i,le ) (2)

[0017] Preferably, in step (2), extracting semantic information of evidence through a multi-head mapping mechanism and storing it in a memory pool specifically includes:

[0018] Construct N different mapping matrices To extract information from different angles in the evidence, Sublen = Emblen / N represents the dimension of the subspace. First, the evidence encoding vector Emb is transformed into i (1≤i≤M) are mapped to different subspaces,

[0019] As shown in formula (3)

[0020]

[0021] Get feature representation based on a specific angle Then the features extracted from different subspaces are concatenated as the updated evidence representation NewEmb i , and finally save the updated evidence representation into the memory pool H;

[0022] As shown in formula (4), the concat() function represents the concatenation operation.

[0023]

[0024] The updated evidence representation is saved in the memory pool, as shown in formula (5), where

[0025] H={NewEmb1,NewEmb2,...,NewEmb M} (5)

[0026] Preferably, in step (3), the semantic relationship between the statement and each piece of evidence is calculated; the overall representation of the evidence set is obtained based on the attention mechanism, and the reasoning relationship between the statement and the overall representation of the evidence is calculated, specifically including:

[0027] Construct two different embedding matrices and Map the feature representation H of the evidence in the memory pool to different feature spaces to obtain the feature representation M of all evidence a =[a1,a2,...,a M ] and M r =[r1,r2,...,r M ],

[0028] As shown in formulas (6) and (7),

[0029] a i =NewEmb i ·A (6)

[0030] r i =NewEmb i ·S (7)

[0031] Constructing the embedding matrix The statement encoding vector Emb c Mapped into the feature space, we get the statement feature representation b, as shown in formula (8).

[0032] b=Emb c ·B (8)

[0033] The statement feature representation b and the evidence feature representation a i (1≤i≤M) splicing to get l i , and then use the relationship matrix It is mapped to a space with a dimension size of Memlen, and the hyperbolic tangent function is further used to calculate the semantic relationship D = [d1, d2, ..., d M ].

[0034] As shown in formulas (9) and (10),

[0035] l i =concat(b;a i ) (9)

[0036] d i =tanh(l i Wd ) (10)

[0037] Introducing weight vector v∈R based on soft attention mechanism Memlen×1 , calculate the semantic relationship D between the statement and the evidence, and get the attention vector As shown in formula (11), where p i represents the attention weight of the statement on the i-th piece of evidence.

[0038] p i =softmax(d i v T ) (11)

[0039] In the formula,

[0040] Further use the attention vector P and the feature representation M of the evidence r Calculate the overall representation of evidence o.

[0041]

[0042] The reasoning relationship h between the two is obtained by summing the statement representation and the overall representation of evidence.

[0043] h=b+o (13)

[0044] Preferably, in step (4), the reasoning structure is upgraded to a multi-layer structure, and the reasoning relationship is iteratively updated, specifically including:

[0045] For multi-layer reasoning structures, construct multiple different embedding matrix combinations (A 1 ,S 1 ),(A 2 ,S 2 ),…,(A K ,S K ),in In the kth layer, through A k and S k Map the representation in the memory pool to different feature space combinations and execute the method in step (3). In addition, there are three points to note: 1) Different layers calculate semantic relationships and attention based on different angles. Therefore, the relationship matrix W of different layers is d Different from the weight vector v.

[0046] 2) The multi-level reasoning process is performed iteratively. In order to integrate the reasoning information in the historical iterations, the feature representation of the statement is updated to the reasoning relationship calculated in the next layer in each iteration.

[0047] b k+1 =bk +o k (14)

[0048] 3) The number of parameters of a multi-layer structure will increase exponentially with the number of layers. In order to reduce the number of parameters, this paper adopts the same embedding matrix for adjacent layers for training, and the embedding matrices A and S of adjacent layers are the same.

[0049] A k+1 =S k (15)

[0050] Preferably, the authenticity of the statement is judged in step (5) by a single-layer feedforward neural network and a softmax operation, specifically comprising:

[0051] First, a single-layer feedforward neural network is used to transform the feature representation of the combined evidence and the statement into a probability vector for the predicted label, as shown in formula (16), where and is the training parameter of a single-layer feedforward neural network.

[0052] β=W2(o K +b K ) T +u (16)

[0053] Then the predicted probability β corresponding to each label is calculated through the softmax operation. i Normalize so that the predicted probabilities of all labels are equal to 1. The obtained label probability vector The label corresponding to the maximum value is the prediction result; as shown in formula (17)

[0054]

[0055] Beneficial effects of the present invention:

[0056] 1. The fact detection method based on the deep interactive reasoning model provided by the present invention uses the BERT model to encode statements and evidence respectively, which has good semantic characteristics; the evidence encoding is mapped to different subspaces through a multi-head mapping mechanism, and the semantic information of the evidence is extracted from multiple angles; the evidence information is saved by introducing a memory pool structure, giving the model the ability to handle "multi-hop" problems;

[0057] 2. Based on semantic relations and attention mechanisms, the reasoning relationship between evidence sets and statements is calculated, and the reasoning structure is upgraded to a multi-layer structure to achieve the purpose of deep mining. Relevant experiments were conducted on the UKP-Snopes dataset, and the results showed that compared with other technologies, the present invention has higher prediction accuracy, recall rate and F1 value indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 , is a method flow chart of an embodiment of the present invention.

[0059] Figure 2 , is a flow chart of a fact detection method based on a deep interactive reasoning model. DETAILED DESCRIPTION

[0060] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0061] A fact detection method based on a deep interactive reasoning model in this embodiment, such as Figure 1 As shown in Figure 1, the method includes the following steps: (1) Use the BERT model to encode the statement and evidence respectively; (2) Extract the semantic information of the evidence through the multi-head mapping mechanism and save it in the memory pool; (3) Calculate the semantic relationship between the statement and each piece of evidence; Obtain the overall representation of the evidence set based on the attention mechanism, and calculate the reasoning relationship between the statement and the overall representation of the evidence; (4) Upgrade the reasoning structure to a multi-layer structure and iteratively update the reasoning relationship; (5) Judge the truth of the statement through a single-layer feedforward neural network and softmax operation. Each step is described in detail below.

[0062] The problem can be described as follows: Let c = [c1, c2, ..., c lc ] indicates a statement, where c i represents the i-th word in the statement, lc represents the length of the statement, and the number of evidences is set to M = 5, and the corresponding evidence set is E = {e1, e2, e3, e4, e5}, e i =[e i,1 ,e i,2 ,…,e i,le ](1≤i≤5), where e i,j represents the jth word in the i-th piece of evidence, and le represents the length of the i-th piece of evidence. The fact detection task is to predict the truthfulness of statement c based on the evidence set E.

[0063] (1) Use the BERT model to encode statements and evidence.

[0064] For the statement and each piece of evidence, they are input into the BERT model to obtain the statement encoding vector and evidence encoding vectors {Emb1,Emb2,...,Emb5}.

[0065] (2) The semantic information of the evidence is extracted through a multi-head mapping mechanism and saved in the memory pool.

[0066] Construct N different mapping matrices Extract information from different angles in the evidence, where Sublen = Emblen / N represents the dimension of the subspace. First, the evidence encoding vector Emb is transformed into i (1≤i≤5) is mapped to different subspaces to obtain feature representation based on specific angles Then the features extracted from different subspaces are concatenated as the updated evidence representation NewEmb i , and finally save the updated evidence representation into the memory pool H.

[0067] (3) Calculate the semantic relationship between the statement and each piece of evidence; obtain the overall representation of the evidence set based on the attention mechanism, and calculate the inference relationship between the statement and the overall representation of the evidence.

[0068] Construct two different embedding matrices and Memlen represents the feature space dimension. The feature representation H of the evidence in the memory pool is mapped to different feature spaces to obtain the feature representation M of all evidence. a =[a1,a2,...,a5] and M r =[r1,r2,...,r5],

[0069] Constructing the embedding matrix The statement encoding vector Emb c Mapped into the feature space, we get the statement feature representation b.

[0070] The statement b and the evidence a i (1≤i≤5) splicing to get l i , and then use the relationship matrix W d It is mapped into a space with dimension size Memlen, and the hyperbolic tangent function is further used to calculate the semantic relationship D = [d1, d2, ..., d5] between the statement and each piece of evidence.

[0071] d i =tanh(l i W d )

[0072] Introducing weight vector v∈R based on soft attention mechanism Memlen×1 , calculate the semantic relationship D between the statement and the evidence, and get the attention vector where p i represents the attention weight of the statement on the i-th piece of evidence.

[0073] p i =softmax(d i v T )

[0074] In the formula,

[0075] Further use the attention vector P and the feature representation M of the evidence r Calculate the overall representation of evidence o.

[0076]

[0077] The reasoning relationship h between the two is obtained by summing the statement representation and the overall representation of evidence.

[0078] h=b+o

[0079] (4) Upgrade the reasoning structure to a multi-layer structure and iteratively update the reasoning relationship.

[0080] For multi-layer reasoning structures, construct multiple different embedding matrix combinations (A 1 ,S 1 ),(A 2 ,S 2 ),…,(A K ,S K ),in In the kth layer, through A k and S k Map the representation in the memory pool to different feature space combinations and execute the method in step (3). In addition, there are three points to note:

[0081] 1) Different layers calculate semantic relations and attention based on different angles. Therefore, the relationship matrix W of different layers is d Different from the weight vector v.

[0082] 2) The multi-level reasoning process is iterative. In order to integrate the reasoning information in the historical iterations, the feature representation of the statement is updated to the reasoning relationship calculated in the next layer in each iteration.

[0083] b k+1 =b k +o k

[0084] 3) The number of parameters of a multi-layer structure will increase exponentially with the number of layers. In order to reduce the number of parameters, this paper adopts the same embedding matrix for adjacent layers for training, and the embedding matrices A and S of adjacent layers are the same.

[0085] A k+1 =Sk

[0086] (5) The truthfulness of the statement is judged through a single-layer feedforward neural network and softmax operation.

[0087] First, the inference relationship is converted into a probability vector for the predicted label through a single-layer feedforward neural network, and then the prediction probability β corresponding to each label is converted into a probability vector through a softmax operation. i Normalize so that the predicted probabilities of all labels are equal to 1. The obtained label probability vector The label corresponding to the maximum value is the prediction result.

[0088] like Figure 2 :A flow chart of a fact detection method based on a deep interactive reasoning model: The fact detection method based on a deep interactive reasoning model proposed in the present invention uses the BERT model to encode evidence and statements; extracts the semantic relationship of evidence through a multi-head mapping mechanism and saves it in a memory pool; calculates the semantic relationship between the statement and each piece of evidence, obtains the overall representation of the evidence set based on the attention mechanism, and calculates the reasoning relationship between the evidence as a whole and the statement; upgrades the reasoning structure to a multi-layer structure, and iteratively updates the reasoning relationship; finally, the authenticity of the statement is judged through a single-layer feedforward neural network and a softmax operation. The fact detection method based on a deep interactive reasoning model proposed in the present invention has an accuracy of 52.2%, a recall rate of 51.8%, and an F1 value of 51.9% on the UKP-Snopes dataset. All indicators are higher than the comparative experiments, among which the highest accuracy in the comparative experiments is 49.3%, the highest recall rate is 47.7%, and the highest F1 value is 48.5%.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fact detection method based on a deep interactive reasoning model, characterized in that: The steps include: (1) Using the BERT model to encode the statement and the evidence respectively; In step (1), using the BERT model to encode the statement and the evidence respectively specifically includes: For the statement c = [c1,c2,…,c lc ], where c i represents the i-th word in the statement, lc represents the length of the statement, and the corresponding evidence set is E = {e1, e2, ..., e M }, e i =[e i，1 , e i，2 ,…,e i,le ], 1≤i≤M where e i,j represents the jth word in the i-th piece of evidence, le represents the length of the i-th piece of evidence, and M represents the number of corresponding evidences; For the statement and each piece of evidence, they are input into the BERT model to obtain the statement encoding vector and the evidence encoding vector {Emb1,Emb2,...,Emb M }, as shown in formulas (1) and (2), where Emblen represents the encoding dimension; Emb c =BERT(c1,c2,…c lc ) (1) Emb i =BERT(e i,1 ,e i,2 ,…e i,le ) (2) (2) Extracting semantic information of evidence through a multi-head mapping mechanism and saving it to a memory pool; The step (2) extracting semantic information of evidence through a multi-head mapping mechanism and saving it to a memory pool specifically includes: Construct N different mapping matrices To extract information from different angles in the evidence, where Sublen = Emblen / N represents the dimension of the subspace; the evidence encoding vector Emb is transformed into i 1≤i≤M is mapped to different subspaces to obtain feature representation based on specific angles. As shown in formula (3) The feature representations extracted from different subspaces are concatenated as the updated evidence representation, as shown in formula (4), where the concat() function represents the concatenation operation. The updated evidence representation is saved in the memory pool, as shown in formula (5), where H={NewEmb1,NewEmb2,...,NewEmb M } (5); (3) calculating the semantic relationship between the statement and each piece of evidence; obtaining the overall representation of the evidence set based on the attention mechanism, and calculating the inference relationship between the statement and the overall representation of the evidence; the step (3) calculating the semantic relationship between the statement and each piece of evidence; obtaining the overall representation of the evidence set based on the attention mechanism, and calculating the inference relationship between the statement and the overall representation of the evidence, specifically includes: Construct two different embedding matrices and Memlen represents the dimension of feature space. The feature representation H of evidence in the memory pool is mapped to different feature spaces to obtain the feature representation M of all evidence. a =[a1,a2,...,a M ] and M r =[r1,r2,...,r M ], as shown in formulas (6) and (7), where a i =NewEmb i ·A (6) r i =NewEmb i ·S (7) Constructing the embedding matrix n will represent the encoding vector Emb c Mapped into the feature space, we get the statement feature representation b, as shown in formula (8); b=Emb c ·B (8) Using the feature representation b of the statement and the feature representation M of the evidence a Compute the semantic relationship between the statement and each piece of evidence; associate the statement representation b with the evidence representation a i 1≤i≤M splicing to get l i , and then use the relationship matrix W d It is mapped to a space with a dimension size of Memlen, and the hyperbolic tangent function is further used to calculate the semantic relationship D = [d1, d2, ..., d M ], as shown in formulas (9) and (10), where l i =concat(b;a i ) (9) d i =tanh(l i W d ) (10) Introducing weight vector v∈R based on soft attention mechanism Memlen×1 , calculate the semantic relationship D between the statement and the evidence, and get the attention vector As shown in formula (11), where p i represents the attention weight of the statement on the i-th piece of evidence; p i =softmax(d i v T ) (11) In the formula, Further use the attention vector P and the feature representation M of the evidence r Calculate the overall representation of evidence o, as shown in formula (12); The reasoning relationship h between the statement representation and the overall evidence representation is obtained by summing them, as shown in formula (13); h=b+o (13) (4) Upgrading the reasoning structure to a multi-layer structure and iteratively updating the reasoning relationship; (5) judging the truth of the statement through a single-layer feedforward neural network and a softmax operation; in step (4), the reasoning structure is upgraded to a multi-layer structure, and the reasoning relationship is iteratively updated, specifically including: For multi-layer reasoning structures, construct multiple different embedding matrix combinations (A 1 ,S 1 ),(A 2 ,S 2 ),…,(A K ,S K ),in In the kth layer, through A k and S k Map the representation in the memory pool to different feature space combinations and execute the method in step (3); in addition, there are three points to note: 1) Different layers calculate semantic relations and attention based on different angles. Therefore, the relationship matrix W of different layers is d Different from the weight vector v; 2) The multi-level reasoning process is iterative. In order to integrate the reasoning information in the historical iterations, the feature representation of the statement is updated to the reasoning relationship calculated in the next layer in each iteration, as shown in formula (14); b k+1 =b k +o (14) 3) The number of parameters of a multi-layer structure will increase exponentially with the number of layers. In order to reduce the number of parameters, this paper adopts the same embedding matrix for adjacent layers for training. As shown in formula (15), the embedding matrices A and S of adjacent layers are the same; A k+1 =S k (15); In step (4), the authenticity of the statement is judged by a single-layer feedforward neural network and a softmax operation, specifically including: A single-layer feedforward neural network is used to transform the feature representation of the combined evidence and statement into a probability vector for the predicted label, as shown in formula (16), where and is the training parameter of a single-layer feedforward neural network; β=W2(o K +b K ) T +u (16) The softmax operation is further used to transform the predicted probability β corresponding to each label i Normalize so that the predicted probabilities of all labels are equal to 1, as shown in formula (17); the label probability vector obtained at this time is The label corresponding to the maximum value is the prediction result;

Images