A refutation and argumentation sentence retrieval method and device based on the BERT model

Abstract

The present invention discloses a retrieval method and device for refutation argument sentences based on a BERT model, the method comprising: constructing a data set in the form of #imgabs0#, wherein #imgabs1# is a context sentence corresponding to the argument sentence #imgabs2#, y i To mark whether the second argumentative statement is a refutation of the first; based on the pre-trained model, a bipolar-encoder model is proposed, which uses joint training to measure whether the contextual statements are related and whether the two argumentative statements refute each other. If they are refutations of each other, they should have the same topic but different positions. This invention simultaneously considers topic similarity and position dissimilarity without prior topic information, has low asymptotic time complexity during retrieval, and can maintain good results and performance in large data environments.

Description

Technical Field

[0001] The present invention relates to the field of argument mining, and in particular to a retrieval method and device for providing the best rebuttal argument statement when a given argument statement is given. Background Art

[0002] The core of the argumentation process is to retrieve the best rebuttal statement for a given input argument statement. This is a very challenging task for both humans and computers, partly because it is difficult to define what constitutes a good argument (or counter-argument). To date, the quality of arguments has been extensively studied, both in argumentation theory and computer science. Although it is generally believed that qualities such as persuasiveness, logic, good contribution, and local acceptability should play an important role in evaluating counter-arguments, these qualities are difficult to formalize specifically for machine learning. One solution is to avoid formalizing the task by adopting a data-driven approach, but this requires a lot of labeling work and is not very practical.

[0003] Based on the definition that the best counterargument emphasizes the same aspects as the input argument but takes opposite positions, Wachmuth et al. proposed a simple scoring method (SimpleSD) in reference [1] (Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the Best Counterargument without Prior Topic Knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241-251, Melbourne, Australia. Association for Computational Linguistics). This method combines similarity metrics such as Manhattan and EarthMover's distance. Although simple, it demonstrates the potential of simultaneously modeling similarity and dissimilarity for counterargument retrieval. Orbach et al. in Reference [2] (Matan Orbach, Yonatan Bilu, Assaf Toledo, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim. 2020. Out of the Echo Chamber: Detecting Countering Debate Speeches. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7073-7086, Online. Association for Computational Linguistics) studied a related but slightly different task, where they attempted to “identify responses to supporting speeches from a set of opposing speeches, all discussing the same topic” to combat fake news and disinformation on social media. The main difference between the two settings is that the former uses arguments without prior information about topics or positions, while the latter focuses on speeches that each contain several arguments with known topics and positions. The authors report several benchmarks, including BERT-based classification and SimpleSD. Surprisingly, even with fine-tuning, BERT-based methods do not show any advantage over SimpleSD.This may be because the input sentences are usually very long and need to be truncated for BERT, where the input is limited to 512 characters. Therefore, the present invention focuses on the setting in Wachmuth, that is, searching for refutation argument sentences without topic and stance information.

[0004] Only by supporting big data retrieval can comprehensive and reliable rebuttal retrieval be achieved without prior topics or pre-set scenarios. Ensuring efficiency in this context is crucial, especially for real-time debate tasks, which place even higher demands on time complexity. Summary of the Invention

[0005] Purpose of the invention: The present invention proposes a refutation and argument sentence retrieval method and device based on the BERT model, which at least partially solves the problems of the prior art.

[0006] Technical solution: In order to achieve the above purpose, the present invention adopts the following technical solution:

[0007] First, a BERT-based refutation and argumentation sentence retrieval method includes the following steps:

[0008] Crawling from debate forums or extracting data from available text information to construct a training dataset. Each piece of data in the dataset has a quadruple form. in Argumentative Sentences The corresponding context sentence, It is an argumentative statement refutation argumentative statement;

[0009] Use two BERT models to encode BERT1 and BERT2 for the two pairs of context sentences and argument sentence pairs in the quadruple respectively. and Encoding;

[0010] Construct a context relevance layer to obtain the context similarity of the two debate sentences. The context relevance layer is used to represent the vectors corresponding to the context sentences in the output of BERT1 and BERT2. and Perform standardization, RELU activation and linear transformation to obtain and Calculate the distance between the two;

[0011] Construct a refutation classification layer to measure whether two argumentative statements refute each other. The refutation classification layer is used to classify the vectors corresponding to the argumentative statement representations in the outputs of BERT1 and BERT2. and Take the absolute value of the difference and then add After splicing, normalization, RELU activation, linear transformation, and classification processing are performed;

[0012] Perform negative sampling and construct sextuples Perform joint training, where q i is a negative sample debate sentence, Negative sample context sentences. Negative sampling means sampling a sentence with a different context as a negative sample for a pair of correct mutually refuting argument sentences. The loss function of the context-related task and the refutation classification task are added together as the model loss function. After training, a refutation argument sentence retrieval model is obtained.

[0013] Use the trained refutation argument sentence retrieval model to retrieve a new argument sentence binary Process and give rebuttal arguments The candidate set is then classified using the refutation classification layer to obtain the refutation degree probability score, and the binary groups in the refutation candidate set are sorted again to obtain the desired result.

[0014] Furthermore, the encoding process of the BERT model includes: adding a delimiter identifier to each pair of context sentence and argument sentence (C, p), constructing a string of the form "[STR1]C[STR2]p", and inputting this string into the corresponding BERT model:

[0015]

[0016] The first string STR1 is used to mark the beginning of the context sentence, and the second string STR2 is used to mark the beginning of the argument sentence. The aggregation of character vectors corresponding to the C part in the output of the BERT model is used as the context sentence representation, and the aggregation of character vectors corresponding to the p part is used as the argument sentence representation.

[0017] Furthermore, the average pooling aggregation method is used for the aggregation of character vectors.

[0018] Furthermore, the processing of the context relevance layer includes:

[0019]

[0020] Among them, Layernorm is the normalization processing, relu is the nonlinear transformation processing, and linear1 is the linear transformation processing; the contextual similarity of two argumentative statements is measured using Euclidean distance.

[0021] Furthermore, the processing of the refutation classification layer includes:

[0022]

[0023] abs means taking the absolute value, layernorm means normalization, relu means nonlinear transformation, and linear2 means linear transformation.

[0024] The classification process uses the softmax function to determine the probability that the argument statement pairs are mutually contradictory.

[0025] Furthermore, the context-dependent task is trained using the contrastive learning triplet loss function, which is calculated as: in Context vector representing the negative sample.

[0026] Furthermore, the refutation classification task is trained using the cross-entropy loss function.

[0027] Furthermore, when using the trained refutation argumentation retrieval model to process new argumentation sentences and corresponding context sentences, BallTree is used to construct The index of the collection, when given a new argument statement, obtains its corresponding context vector Search the BallTree for the most similar statements as candidate results.

[0028] In a second aspect, a computer device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the refutation and argumentation sentence retrieval method based on the BERT model as described in the first aspect of the present invention are implemented.

[0029] In a third aspect, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the BERT model-based rebuttal argument sentence retrieval method as described in the first aspect of the present invention.

[0030] Beneficial effects: In order to improve the accuracy of refutation argument retrieval, the present invention proposes a BERT-based framework, which simultaneously considers topic similarity and position dissimilarity without prior information about the topic. In turn, this will be helpful for other argument mining applications. Compared with the Wachmuth sentence pair model classification scheme, the present invention improves efficiency, and the asymptotic time complexity during retrieval is reduced from O(n) to O(logn) compared to the original sentence pair calculation and sorting scheme, and can continue to maintain good results and performance in the case of big data. For the original sentence pair sorting scheme, the algorithm with linear time complexity of real-time tasks is almost impossible to apply. At the same time, the present invention also has better performance in terms of retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the BERT model-based rebuttal argument sentence retrieval method of the present invention;

[0032] Figure 2 It is a schematic diagram of the refutation argument sentence retrieval model (Bipolar-encoder, referred to as Bipolar model for short) of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0034] Reference Figure 1 The present invention proposes a refutation argument sentence retrieval method based on the BERT model, comprising the following steps:

[0035] Step S1: construct a training data set.

[0036] According to the embodiment of the present invention, it is necessary to prepare sample argument data and its corresponding rebuttal argument data. The dataset can be crawled from debate forums, or extracted and constructed from text information such as news articles, legal documents, and debate discussions. Documents (news articles, legal documents, debate discussion texts, etc.) can be divided into some fragments, each of which can be regarded as a context sentence. Some websites such as Debatepedia have context sentences themselves, which can be crawled using web tools and directly constructed into the form of a quadruple, or some forum data only has debate sentences and rebuttal debate sentences. At this time, each of them is regarded as a context sentence, and a quadruple can also be obtained, that is, the specific context is contained in the debate sentence itself. This method is used to construct a dataset for training. Each data in the dataset has a quadruple form. in Argumentative Sentences The corresponding context sentence, It is an argumentative statement In a degenerate version, C i Can also be used with p i same.

[0037] It should be noted that the model of the present invention has no a priori theme. No a priori theme means that there is no need to clarify the scope of the specific discussion topic of the debate statement. The data of all topics can use the same model, and there is no need to clearly limit the specific debate scope in advance. When the collected data set is large enough, the model will be sufficient to discuss the topics of various subjects without limiting the topic to a narrow range.

[0038] In step S2, two BERT models are used to encode the two argument sentence pairs in the quadruple.

[0039] BERT is a pre-trained model that can be obtained directly from the internet or pre-trained using collected debate text data. This produces a pre-trained BERT model that can be used as a model component. Once these BERT encoders are trained, the debate text and context can be fed into the BERT model to generate dense vectors representing pairs of debate sentences with corresponding contextual sentences, thereby measuring whether they refute each other.

[0040] Use the BERT model to encode each pair of context sentences and argument sentences (C (1) ,p (1) ) and (C (2) ,p (2) ), the quadruplets in the training set are encoded by two different BERTs, denoted as BERT1 and BERT2, which respectively process argument sentence pairs and rebuttal sentence pairs. Each BERT model encodes a argument sentence and its corresponding context sentence.

[0041] In one embodiment, STR1 uses CLS and STR2 uses SEP. For each quadruple in the training set, a string in the form of "[CLS]C[SEP]p" is constructed for each pair (C, p), and then the string is input into the BERT model, where C represents the context sentence and p represents the corresponding argument sentence; the corresponding [CLS] string in the output of BERT is used as the context representation, and the output of the characters corresponding to the p part uses certain aggregation methods to serve as the argument representation. Here, [CLS] corresponds to the start of the context sentence, and [SEP] separates the two sentences, which can also be regarded as the start of the second sentence. Therefore, the context sentence representation can correspond to the [CLS] string, or it can be the aggregation of the vector corresponding to C. Similarly, the argument sentence can correspond to [SEP] or the aggregation of the character vectors corresponding to p. Exemplarily, the aggregation method here can be average pooling. It should be noted that the use of [CLS] and [SEP] strings embedded in the sentence is only an example, not a limitation. In other words, in the specific implementation process, other strings or symbols can be used to complete such separation and identification functions.

[0042] use Representing the context representation and argument representation quad Output after BERT;

[0043]

[0044] Step S3: constructing a context relevance layer for obtaining the context similarity of two argumentative statements and a refutation classification layer for measuring whether the two argumentative statements refute each other.

[0045] Reference Figure 2 , the context relevance layer (also called retrieval module) is built on top of the BERT encoder to measure whether two context sentences are similar. In this embodiment of the present invention, after obtaining the CLS embeddings from BERT1 and BERT2 and After that, a Layer-norm layer, a RELU activation function layer, and a linear transformation layer are used to form a context-dependent layer to obtain the embedding vector for retrieval, which is expressed as follows:

[0046]

[0047] The contextual similarity between two argumentative sentences is measured using the Euclidean distance. The Euclidean distance between two vectors is Where xi and yi are the values ​​of each dimension of the two vectors.

[0048] The vector obtained by BERT is input into the context relevance layer to obtain a vector representing the context. The smaller the Euclidean distance, the higher the context similarity. In the context relevance layer, Layer-norm is a normalization process. The mean μ and variance σ of the vector are calculated, and then each element in the vector is normalized. That is, for the input vector x = [x1, x2,, ..., xn], the variance σ and mean μ are first calculated, and then each component xi of x is normalized. Get new This is the result of the layer-norm operation, which adjusts all values ​​in the vector to a standard normal distribution. The standard normal distribution is more suitable for neural network fitting, ensuring a smoother deep learning training process, better model convergence, and preventing model training crashes. This standardization process is different from regularization, which is used to prevent overfitting. This is more for smooth model training. Without the normalization layer, deep learning training is prone to failure, resulting in vanishing and exploding gradients.

[0049] The RELU activation function is Linear transformations superimposed on linear transformations can only fit linear transformations, not arbitrary functions. The introduction of the RELU nonlinear transformation function allows for better function fitting and classification results. The reason linear transformations are used for function fitting is that, compared to the pre-trained BERT layer, the newly added linear transformation layer is more suitable for fitting learning between context vector pairs. This is because the original BERT vectors are affected by the original BERT pre-training task, thus affecting model performance.

[0050] Reference Figure 2 , the refutation classification layer (also called classification module) is built on top of the BERT encoder to measure whether two argumentative statements refute each other. and To represent the mean embedding from BERT1 and BERT2, the mean is obtained by pooling the output vectors of the corresponding part of the argument (i.e., the average of p). The classification module is used to obtain the probability that the argument sentence pair refutes each other:

[0051]

[0052] Where abs represents the absolute value. The output of linear2 is a real value representing the relative size of the probability, which is converted into a probability value after calculation in the softmax layer.

[0053] Softmax function, that is, for the input vector x = [x1, x2, x3, x4, ..., xn], calculate have So yi can be used to fit the probability, as long as e xi When (that is, xi) corresponds to a suitable value, the output of linear2 fits the probability, and the output real number size is transformed by the softmax function to obtain a real number with a size between [0,1], and the sum is 1, which is the probability.

[0054] The vectors obtained by the BERT1 model and the BERT2 model are input into the refutation classification layer together to obtain the probability score of the similarity between the two. The higher the score, the more likely it is that the sentences are mutually refuting.

[0055] Step S4: perform model training.

[0056] In this paper, the refutation and argumentation sentence retrieval model built based on BERT, the contextual relevance layer, and the refutation classification layer is called the Bipolar-encoder model. The relevance and refutation tasks are trained jointly, using the contrastive learning triplet loss function for the contextual relevance task and the cross-entropy loss function for the refutation classification task.

[0057] Specifically, negative sampling is performed and a sextuple is constructed in is a negative sample sentence pair, q i Negative sample debate sentences, Negative sample context sentences. Negative samples refer to sampling a sentence with a different context as a negative sample for any correct mutually refuting argument sentence pair. For example, for the prepared training data Four tuples, correct one-to-one correspondence, can be taken from the data set any one not equal to One of the two-tuples (C, p) is used as a negative sample, whether it is a debate sentence or a rebuttal sentence, and is put together with the original four-tuple to form a sextuple. For each negative sample, the following three-tuple loss function is used for training: in Context vector representing the negative sample.

[0058] The cross entropy loss is calculated to measure the refutation relevance score. The calculation expression is: CrossEntropy(p,label)=-label*log(p)-(1-label)*log(1-p), where log is the logarithmic operation and the base here is e, i.e. ln. Because it is optimized by deep learning, the base-changing formula logab=lnb / lna is added. Therefore, no matter how many are selected, it does not affect the result. p is the probability that the predicted model is true, and label is the actual label, marking whether it is a correct statement pair.

[0059] The final loss function is obtained by adding the retrieval loss function and the classification loss function. In this loss function, true represents a correct statement pair and false represents an incorrect statement pair.

[0060]

[0061] Six-tuple joint training means that the argument sentence and the corresponding context sentence are input into BERT1, the rebuttal argument sentence and the corresponding context sentence are input into BERT2, and the negative sample argument sentence and the corresponding context sentence are input into BERT2. The three get the context sentence vector Corresponding to the loss part of the triplet. true The corresponding p part vectors of the argument sentence and the rebuttal sentence are concatenated together and input into the rebuttal classification layer to obtain the rebuttal probability. Similarly, prob false The vectors corresponding to the argument and the negative sample are concatenated and input into the refutation classification layer to obtain the refutation probability. These two probabilities are input into the above loss formula, corresponding to the two crossentropy loss parts.

[0062] The model is trained and saved, and then used to index a large dataset of argumentative texts for online retrieval of rebuttal argumentative statements.

[0063] Step S5: Use the trained model to perform inference retrieval.

[0064] First, use the Bipolar-encoder model to transform all the argument sentences in the collected dataset into embedding vectors. These embedding vectors contain two parts, one of which is used for retrieval, as described in step S3. and The other part is used for further sentence pair classification inference, such as u in step S3 p and v p . Use BallTree to build The index of the collection. Given a new argument statement, get its context vector Used to search for the most similar statements in BallTree as candidate results. For the re-ranked candidate set, when a debate statement and a candidate result are given, the cached embedding vector u is used p and v p to proceed; using the classification module of the Bipolar model, they are used for further classification and get scores for re-ranking.

[0065] Specifically, for all query refutation argument (C, p) statement pairs, each corresponding context vector is obtained and these vectors are indexed using BallTree. When a new argument statement is given, a new (C', p') pair is also given, and a query vector is obtained. The BallTree is queried to obtain a candidate set of vectors corresponding to the pair (C', p'). The refutation classification layer is then used to obtain a refutation probability score, and the pairs in the refutation candidate set are re-ranked to obtain the desired result. That is, for a argument statement and its corresponding context statement, a retrieval model query is performed to obtain a candidate refutation argument statement. The refutation argument statement has already been pre-calculated with the argument statement representation required by the refutation layer. At this time, the three vectors—the argument statement representation vector, the refutation argument statement representation vector, and the absolute value of the difference between the two—are concatenated together and passed through the classification layer to obtain the refutation probability. The probabilities are then ranked to obtain the most appropriate refutation argument statement.

[0066] The present invention also provides a computer device comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the BERT model-based rebuttal argument sentence retrieval method are implemented as described above.

[0067] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for retrieving refutation and argument sentences based on the BERT model as described above.

[0068] It will be understood by those skilled in the art that all or part of the steps to implement the above embodiments may be accomplished by hardware or by a program to instruct the relevant hardware to accomplish the steps, and the program may be stored in a computer-readable storage medium. In the context of the present invention, the computer-readable medium may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital tapes or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs), etc.

[0069] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0070] In addition, although adopting specific order to describe each operation, this should be understood as requiring such operation to be carried out in the specific order shown or in sequential order, or requiring all illustrated operations to be carried out to obtain desired result.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be construed as limiting the scope of the present invention.Some features described in the context of independent embodiment can also be realized in single realization in combination.On the contrary, the various features described in the context of independent realization also can be realized in multiple realizations individually or in the mode of any suitable subcombination.

[0071] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A refutation and argumentation sentence retrieval method based on the BERT model, characterized by: The following steps are involved: Crawling from debate forums or extracting data from available text information to construct a training dataset. Each piece of data in the dataset has a quadruple form. in Argumentative Sentences The corresponding context sentence, It is an argumentative statement refutation argumentative statement; Use two BERT models to encode BERT1 and BERT2 for the two pairs of context sentences and argument sentence pairs in the quadruple respectively. and Encoding; Construct a context relevance layer to obtain the context similarity of the two debate sentences. The context relevance layer is used to represent the vectors corresponding to the context sentences in the output of BERT1 and BERT2. and After normalization, RELU activation and linear transformation, we get and Calculate the distance between the two; Construct a refutation classification layer to measure whether two argumentative statements refute each other. The refutation classification layer is used to classify the vectors corresponding to the argumentative statement representations in the outputs of BERT1 and BERT2. and Take the absolute value of the difference and then add After splicing, normalization, RELU activation, linear transformation, and classification processing are performed; Perform negative sampling and construct sextuples Perform joint training, where q i is a negative sample debate sentence, Negative sample context sentences. Negative sampling means sampling a sentence with a different context as a negative sample for a pair of correct mutually refuting argument sentences. The loss function of the context-related task and the refutation classification task are added together as the model loss function. After training, a refutation argument sentence retrieval model is obtained. Use the trained refutation argument sentence retrieval model to retrieve a new argument sentence binary Process and give rebuttal arguments The candidate set is then classified using the refutation classification layer to obtain the refutation degree probability score, and the binary groups in the refutation candidate set are re-sorted to obtain the desired result.

2. The method according to claim 1, characterized in that The encoding process of the BERT model includes: adding a delimiter identifier to each pair of context sentence and argument sentence (C, p), constructing a string of the form "[STR1]C[STR2]p", and inputting this string into the corresponding BERT model: The first string STR1 is used to mark the beginning of the context sentence, and the second string STR2 is used to mark the beginning of the argument sentence. The aggregation of character vectors corresponding to the C part in the output of the BERT model is used as the context sentence representation, and the aggregation of character vectors corresponding to the p part is used as the argument sentence representation.

3. The method according to claim 2, characterized in that Aggregate character vectors using average pooling.

4. The method according to claim 1, wherein The context-relevance layer includes: Among them, layernorm is the normalization processing, relu is the nonlinear transformation processing, and linear1 is the linear transformation processing; the contextual similarity of two argumentative statements is measured using Euclidean distance.

5. The method according to claim 1, wherein The processing of the refutation classification layer includes: abs means taking the absolute value, layernorm means normalization, relu means nonlinear transformation, and linear2 means linear transformation. The classification process uses the softmax function to determine the probability that the argument statement pairs are mutually contradictory.

6. The method according to claim 1, characterized in that The context-dependent task is trained using the contrastive learning triplet loss function, which is calculated as: in Context vector representing the negative sample.

7. The method according to claim 1, characterized in that The refutation classification task is trained using the cross-entropy loss function.

8. The method according to claim 6, characterized in that When using the trained refutation argumentation retrieval model to process new argumentation sentences and corresponding context sentences, BallTree is used to construct The index of the collection, when given a new argument statement, obtains its corresponding context vector Search the BallTree for the most similar statements as candidate results.

9. A computer device, characterized in that: include: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the refutation argument sentence retrieval method based on the BERT model as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the BERT model-based rebuttal argument sentence retrieval method as described in any one of claims 1 to 8 are implemented.

Images