Method and system for enhancing performance of question and answer adversarial samples based on aggregated representation

Abstract

A method and system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation, the method comprising: S1: preprocessing the text of the question and answer to obtain the corresponding input text sequence X. q and X a S2: X q and X a Input a twin-tower model and obtain X. q and X a High-level representation vector H q and H a And syntactic representation vectors and S3: H q and H a The probability distribution mapped to the vocabulary is used to construct a representation vector v containing local lexical information through max pooling. q and v a S4: For v q and v a Pruning is performed to obtain a low-dimensional lexical representation vector. S5: The reduced-dimensional syntactic representation vector and the low-dimensional lexical representation vector are concatenated and aggregated to obtain the aggregated representation vector e. q and e a S6: According to e q and e a Calculate the matching score between question-answer pairs to obtain the matching loss function L. match S7: Based on L match Calculate the loss function L with added perturbation. adv The method of this invention can obtain more effective adversarial examples under adversarial training, thereby improving retrieval and question-answering performance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and specifically to a method and system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation. Background Technology

[0002] In recent years, with the continuous development of the information age, information on the internet has expanded rapidly, and the ways to obtain information have become increasingly diverse, making information sharing and acquisition easier. However, this has also made fast and efficient information retrieval a challenge. Traditional search engine retrieval technology uses keyword retrieval, returning a large number of documents for users to choose from. While this has achieved some results, the quality of documents retrieved by search engines varies greatly, and there are also misleading advertisements, causing annoyance to users. Users also need to repeatedly try keywords to get the desired results. In recent years, with the development of machine learning and deep learning technologies, neural network models have begun to achieve remarkable results in various natural language processing tasks, and the application and research of related technologies are increasing. Natural language question answering, as one of the important applications of natural language processing, is an ideal human-computer interaction mode. Users can directly describe questions in natural language, and the question answering system can directly provide accurate and concise answers. This relies on the powerful text understanding, representation, and even generation capabilities of pre-trained models such as BERT. Based on the model selection, question answering systems can be divided into retrieval-based question answering systems, generative question answering systems, and extraction-based question answering systems. Generative question-answering systems rely on generative models to generate answers character by character from scratch based on the question. While flexible, this approach is slow and may produce answers that violate laws or ethics. Extractive question-answering systems retrieve answers through a two-stage process of recall and ranking, but suffer from error propagation and inefficiency. Retrieval-based question-answering systems, on the other hand, do not generate false answers out of thin air, making them more reliable than generative systems. They are also more efficient than extractive systems. In recent years, retrieval-based question answering has received increasing attention and research.

[0003] In retrieval-based question answering systems, negative examples are often constructed for model training. To enhance model robustness, adversarial training methods can be used to add perturbations to increase the difficulty of negative examples. However, previous research has pointed out that in resource-rich scenarios, due to the lack of information in the representation vectors, it is impossible to construct high-quality adversarial examples, which means that adversarial training may not bring performance improvements. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation.

[0005] The technical solution of this invention is: a method for enhancing the performance of adversarial examples in retrieval and question answering based on aggregated representation, comprising:

[0006] Step S1: Preprocess the question and answer texts, transforming them into a sequence of question input text X composed of words from the vocabulary. q and the answer input text sequence X a ;

[0007] Step S2: Place X q and X a The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain X. q and X a The high-level representation vector H of each symbol in the middle q and H a and syntactic representation vectors and

[0008] Step S3: Utilize the knowledge information from the masked language model header of the pre-trained model to... q and H a The probability distribution mapped to the vocabulary is used to construct a representation vector v containing local lexical information through max pooling. q and v a ;

[0009] Step S4: For v q and v a Pruning is performed to obtain the pruned low-dimensional lexical representation vector. and

[0010] Step S5: Concatenate and aggregate the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector e. q and e a ;

[0011] Step S6: According to e q and e a Calculate the matching score between question-answer pairs to obtain the matching loss function.

[0012] Step S7: Based on Calculate the loss function with added perturbation

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] 1. This invention discloses a method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation. The question-answering representation vector is directly related to the retrieval matching result and is crucial for retrieval question answering tasks. Previous ReQA works often directly use the vector representation of [CLS] or use simple average pooling and attention weighting strategies to obtain the representation vector. A single strategy will lead to missing information. For example, [CLS] only contains global syntactic information and lacks fine-grained bag-of-words features, while average pooling dilutes syntactic information. This invention uses aggregated representation that simultaneously contains syntactic and lexical information. The organic combination of the two enhances the retrieval performance of ReQA.

[0015] 2. In traditional adversarial training, representation vectors are used to calculate the loss, which in turn affects the direction of adversarial perturbations and the quality of adversarial examples. A single representation strategy limits the diversity of adversarial examples and is detrimental to model performance. In this invention, by aggregating representations, the quality of adversarial examples and the model's retrieval performance are enhanced. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation, as described in an embodiment of the present invention.

[0017] Figure 2 This is a schematic flowchart of the method of the present invention;

[0018] Figure 3 This is a structural block diagram of a system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation, as described in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a method for enhancing the performance of adversarial examples in retrieval and question answering based on aggregated representations. By using special symbols and knowledge information from pre-trained masked prediction heads, an aggregated representation that combines global syntactic information and local lexical information is obtained. Under adversarial training, more effective adversarial examples are obtained, thereby improving retrieval and question answering performance.

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0021] Example 1

[0022] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation, comprising the following steps:

[0023] Step S1: Preprocess the question and answer texts, transforming them into a sequence of question input text X composed of words from the vocabulary. qand the answer input text sequence X a ;

[0024] Step S2: Place X q and X a The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain X. q and X a The high-level representation vector H of each symbol in the middle q and H a and syntactic representation vectors and

[0025] Step S3: Utilize the knowledge information from the masked language model header of the pre-trained model to... q and H a The probability distribution mapped to the vocabulary is used to construct a representation vector v containing local lexical information through max pooling. q and v a ;

[0026] Step S4: For v q and v a Pruning is performed to obtain the pruned low-dimensional lexical representation vector. and

[0027] Step S5: Concatenate and aggregate the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector e. q and e a ;

[0028] Step S6: According to e q and e a Calculate the matching score between question-answer pairs to obtain the matching loss function.

[0029] Step S7: Based on Calculate the loss function with added perturbation

[0030] In one embodiment, step S1 above involves preprocessing the text of the question and answer to transform it into a sequence of question input text X composed of words from a vocabulary. q and the answer input text sequence X a Specifically, it includes:

[0031] Step S11: Convert the question and candidate answers to Unicode encoding, then perform word segmentation on the question and candidate answers to obtain the word segmentation results for the question and the answer respectively.

[0032]

[0033]

[0034] Where q and a represent a single question and answer, and They represent the i-th word in the question and the i-th word in the answer, respectively. q and l a These are the text lengths of the question and the answer, respectively.

[0035] Given a user-submitted question and candidate answers, question answering aims to retrieve the correct answer from the candidate answers end-to-end. First, the question and candidate answers are converted to Unicode encoding (e.g., the commonly used UTF-8 encoding). Then, the WordPiece word segmentation model is used to segment the question and candidate answers, yielding the segmented results for the question and answer.

[0036] Step S12: Concatenate the question segmentation result and the answer segmentation result with the special characters [CLS] and [SEP] in BERT to obtain the question input text sequence X. q and the answer input text sequence X a :

[0037]

[0038]

[0039] In one embodiment, step S2 above: X q and X a The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain X. q and X a The high-level representation vector H of each symbol in the middle q and H a and syntactic representation vectors and Specifically, it includes:

[0040] X q and X a Convert to a sequence of token_ids for each word, and assign X based on the length of the sequence. q and X a Set the position code position_id to the appropriate length; input the token_id sequence and position_id into the BERT encoder to extract the high-level representation vectors of each symbol in the answer to the question:

[0041]

[0042]

[0043] in, and These represent [CLS] and [SEP] with their corresponding high-level representation vectors, respectively. Let represent the high-level representation vector corresponding to the i-th word in the problem. Let d represent the high-level representation vector corresponding to the i-th word in the answer, where d is the dimension of the high-level representation vector. These are high-level representation vector matrices consisting of all the words in the question and the answer, respectively.

[0044] This invention employs a dual-tower model constructed based on a shared-parameter BERT model to encode questions and answers, obtaining high-level representation vectors H for each symbol in the questions and answers. q and H a And take the vector of the top-level [CLS] symbol. and These serve as syntactic representation vectors for the question and the answer, respectively.

[0045] In one embodiment, step S3 above: using the knowledge information from the masked language model header of the pre-trained model to... q and H a The probability distribution mapped to the vocabulary is used to construct a representation vector v containing local lexical information through max pooling. q and v a Specifically, it includes:

[0046] Step S31: Use the masked language model header to embed the H q and H a The probability distribution mapped to the vocabulary is represented as follows:

[0047]

[0048]

[0049] in, A vector representation of the i-th symbol in the question and answer; These are the linear layer parameters of the masked language model; and It is a |V|-dimensional context vector, with each dimension conforming to a probability distribution, where |V| is the size of the BERT model vocabulary;

[0050] Step S32: Through max pooling, respectively l q and l a A series of |V|-dimensional vectors are aggregated into a single |V|-dimensional vector v. q and v a The formula is as follows:

[0051]

[0052]

[0053] in, The weights are positive scalars. These are the trainable linear layer parameters, where v∈{1,2,…,|V|} are the dimension indices of the vector; q and v a It is a |V|-dimensional vector containing fine-grained lexical information from bag-of-words features.

[0054] To facilitate matching and retrieval, this embodiment of the invention requires processing vector v. q and v a Pruning is performed to obtain a low-dimensional vector. The specific steps are as follows.

[0055] In one embodiment, S4 above: for v q and v a Pruning is performed to obtain the pruned low-dimensional lexical representation vector. and Specifically, it includes:

[0056] Step S41: Transform the |V| dimension of v q and v a The vector is split into d blocks, respectively. Each block has a dimension of v is obtained through intra-block max pooling q and v a The formula for compressing into a d-dimensional vector is as follows:

[0057]

[0058]

[0059]

[0060]

[0061] Where n∈{1,2,…,d} are vector indices. It is a d-dimensional vector obtained from aggregation, containing the most important item within each block, id q , It is an index vector of the maximum value within the block;

[0062] Step S42: Use For v q ,v a The dot product can be approximated by the following formula:

[0063]

[0064] The idea is that the sum of the dot products of the entire array and the corresponding blocks is equal. The dot products within a block can be approximated as the product of the maximum values ​​within that block. It is important to note that when the index vector id... q ,id a When misaligned, the approximate estimate in the above formula contains an error, namely:

[0065] id q [n]≠id a [n]

[0066] this means, This does not represent the same fraction; the original dot product term does not include this product term, leading to an error. To resolve this mismatch, each S... n The set n∈{1,2,…d} is split into two disjoint subsets S. n+ ,S n- Adjust agg based on the value of id[n]. + The pruned low-dimensional lexical representation vector is obtained. and The formula is as follows:

[0067]

[0068]

[0069] in,

[0070] To obtain a dense vector of d=768 dimensions for retrieval that contains both syntactic and lexical information, the two vector parts are further compressed and concatenated.

[0071] In one embodiment, step S5 above involves concatenating and aggregating the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector e. q and e a Specifically, it includes:

[0072] Step S51: Compress the [CLS] vector using a linear layer, as shown in the following formula:

[0073]

[0074]

[0075] in, These are linear layer parameters. d1 is the [CLS] compressed syntactic representation vector; d1 is the dimension of the compressed syntactic representation vector.

[0076] Step S52: Based on step S4, prune the lexical vectors to... And for A vector that satisfies the following equation:

[0077] d1+d2=d

[0078] Where d2 is the dimension of the pruned lexical representation vector;

[0079] In this embodiment of the invention, d1 = 128 and d2 = 640 are set;

[0080] Step S53: Concatenate the two to obtain the final d-dimensional aggregated representation vector, as shown in the following formula;

[0081]

[0082]

[0083] Among them, e q , That is, an aggregated representation vector that contains both syntactic and lexical information.

[0084] In one embodiment, step S6 above: according to e q and e a Calculate the matching score between question-answer pairs to obtain the matching loss function. Specifically, it includes:

[0085] Step S61: Calculate the matching degree of a pair of vectors using the dot product of the vectors, as shown in the following formula:

[0086] sim(e q ,e a ) = e q ·e a

[0087] Among them, sim(e q ,e a () represents the unnormalized matching score between question-answer pairs;

[0088] Step S62: For sim(e) q ,e a Perform a Softmax operation to obtain the normalized probability distribution, as shown in the following formula:

[0089]

[0090] in, This represents the set of all answers that do not match question q; these answers are called negative samples or negative examples.

[0091] Because the number of these negative examples is too large, the training efficiency is low. Therefore, this embodiment of the invention uses... As The estimation of the problem is as follows. This embodiment of the invention uses ANCE-style dynamic hard negative examples. Before the start of each training round, the top-k negative examples corresponding to each problem are calculated, and samples are obtained from them.

[0092] Step S63: Use the negative log-likelihood as the aggregation representation loss function, as shown in the following formula:

[0093]

[0094] Where B is the size of the training batch, that is, the number of questions contained in a training batch;

[0095] Using the same method, the negative log-likelihood loss obtained by matching [CLS] vectors and lexical vectors is denoted as follows: The matching loss function is obtained as follows:

[0096]

[0097] Wherein, λ1 and λ2 are the weights of the loss function; in this embodiment of the invention, λ1 = 0.5 and λ2 = 0.5 are set.

[0098] To further improve the robustness of the model, adversarial perturbations are added to the question and answer vector embeddings, while the difficulty of negative examples within a batch is further increased.

[0099] In one embodiment, step S7 above: based on Calculate the loss function with added perturbation Specifically, it includes:

[0100] Step S71: Define the objective of adversarial training as:

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] Where D represents the training dataset, θ represents the model parameters, and y is a label indicating whether a question-answer pair matches. These are the aggregated representation vectors of the question and answer, obtained by calculating and aggregating the perturbed embedding vectors using the BERT model. They are based on the perturbed high-level symbolic representation H′. q ,H′ a Calculated according to the above formula; These represent the symbolic embedding vectors of question q and answer a, respectively, obtained by multiplying token_id with the symbolic embedding layer of the BERT model; It is the perturbation vector within space Ω;

[0107] Step S72: In order to maximize Using the fast gradient sign method, adversarial perturbations are added through gradient ascent, as defined below:

[0108]

[0109]

[0110] Where ∈ represents the maximum norm of the perturbation. Expressing the request For V q The differential, Expressing the request For V a The differential;

[0111] Step S73: To avoid δ q δ a If it is too large, normalize it using the following formula:

[0112]

[0113]

[0114] Where sign is the sign function;

[0115] The final loss function is calculated using the following formula:

[0116]

[0117] Figure 2 A schematic diagram illustrating the method flow for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representations is presented.

[0118] The method in this invention is named Adversarial Aggregate Representation Retrieval (AARR), and uses BERT as the pre-trained model encoder version, ultimately named AARR-BERT.

[0119] The method of this invention was tested on the retrieval question-answering version of the Stanford Question Answering Dataset, ReQA (SQuAD), and compared with the top methods on the corresponding dataset, as shown in Table 1. Three experiments were conducted and the mean and standard deviation are reported.

[0120] Table 1 shows the model's performance on ReQA and SQuAD.

[0121]

[0122] As shown in Table 1, this embodiment of the invention uses three popular retrieval metrics: MRR, P@K, and R@K to evaluate the results, following the conventions of previous ReQA work. MRR (Mean Reciprocal Rank) measures the overall performance of the ranking results, and the formula is as follows:

[0123]

[0124] Where |Q| represents the number of questions in the question set, and rank i This represents the ranking of the first correct answer among the prediction results for the i-th question.

[0125] P@K (PrecisionAtTop K) measures the model's ability to predict the correct answer. It represents the proportion of correct answers among the top-k ranked results, and the formula is as follows:

[0126]

[0127] in, These represent the predicted answer ranking and the set of correct answers for the i-th question, respectively. Considering that the main purpose of ReQA is to find the correct answer for each question, this embodiment of the invention obtains the result of P@1.

[0128] R@K (RecallAt Top K) measures the model's ability to recall correct answers. It represents the proportion of correct answers recalled from the top-k ranked results out of the total number of correct answers, as shown in the formula below:

[0129]

[0130] Consistent with previous ReQA work, the results of R@5 are obtained in this embodiment of the invention.

[0131] As can be seen, AARR-BERT has significantly improved in the three metrics of MRR, P@1 and R@5 compared to the basic model Dual-BERT, and has significantly surpassed the previous best method A-ILAG-BERT, achieving the best results. It has also surpassed previous methods based on dual-tower interactive reinforcement and transfer learning, such as Cross-VAE, ENDX and RBAR, as well as the A-ILAG framework based on difficult negative examples.

[0132] To further verify the improvement of adversarial examples by aggregated representations, the performance improvement brought by adversarial training under the classic [CLS] representation and aggregated representations is compared, as shown in Table 2. Here, w / adv represents adversarial training, and [CLS] and AGG represent [CLS] representation and aggregated representation, respectively. It can be seen that under the [CLS] representation, adversarial training only brings about a 1 percentage point improvement in P@1, while under the aggregated representation, adversarial training can bring about a 2.4 percentage point improvement in P@1. This confirms that aggregated representations are beneficial for enhancing the quality of adversarial examples, thereby improving model performance.

[0133] Table 2 shows the improvement in adversarial training under [CLS] and aggregated representation.

[0134]

[0135] This invention discloses a method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation. The question-answering representation vector directly relates to the retrieval matching result and is crucial for retrieval question answering tasks. Previous ReQA works often directly used the vector representation of [CLS] or simple average pooling and attention-weighted strategies to obtain the representation vector. A single strategy can lead to information gaps; for example, [CLS] only contains global syntactic information and lacks fine-grained bag-of-words features, while average pooling dilutes syntactic information. This invention uses aggregated representation that simultaneously includes syntactic and lexical information; the organic combination of the two enhances the retrieval performance of ReQA. Furthermore, in traditional adversarial training, the representation vector is used for loss calculation, thus affecting the direction of adversarial perturbations and the quality of adversarial examples. A single representation strategy limits the diversity of adversarial examples and is detrimental to model performance. In this invention, the aggregated representation method enhances the quality of adversarial examples and the retrieval performance of the model.

[0136] Example 2

[0137] like Figure 3 As shown, this embodiment of the invention provides a system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representations, comprising the following modules:

[0138] The question and answer input text sequence generation module 81 is used to preprocess the question and answer texts, converting them into a question input text sequence X composed of words from the vocabulary. q and the answer input text sequence X a ;

[0139] Syntactic representation vector generation module 82, used to generate X q and X a The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain X. q and X a The high-level representation vector H of each symbol in the middle q and H a and syntactic representation vectors and

[0140] The lexical representation vector generation module 83 is used to utilize the knowledge information of the masked language model head of the pre-trained model to generate H... q and H a The probability distribution mapped to the vocabulary is used to construct a representation vector v containing local lexical information through max pooling. q and v a ;

[0141] Lexical representation vector pruning module 84, used for v q and v a Pruning is performed to obtain the pruned low-dimensional lexical representation vector. and

[0142] The aggregation representation vector module 85 is used to concatenate and aggregate the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector e. q and e a ;

[0143] Construct a matching loss function 86, used to determine e q and e a Calculate the matching score between question-answer pairs to obtain the matching loss function.

[0144] Construct a loss function with added perturbations 87, for use based on Calculate the loss function with added perturbation

[0145] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A method for enhancing the performance of adversarial examples in retrieval and question answering based on aggregated representation, characterized in that, include: Step S1: Preprocess the question and answer texts, transforming them into a sequence of question input texts composed of words from the vocabulary. and the sequence of input text for the answer ; Step S2: and The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain... and High-level representation vectors of each symbol and and syntactic representation vectors and ; Step S3: Utilize the knowledge information from the masked language model head of the pre-trained model to... and The probability distribution mapped to the vocabulary is used to construct a representation vector containing local lexical information through max pooling. and ; Step S4: For and Pruning is performed to obtain the pruned low-dimensional lexical representation vector. ; Step S5: Concatenate and aggregate the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector. Specifically, it includes: Step S51: Compress the [CLS] vector using a linear layer, as shown in the following formula: in, These are linear layer parameters. It is the [CLS] compressed syntactic representation vector; It is the dimension of the compressed syntactic representation vector; Step S52: Based on step S4, prune the lexical vectors to... And for A vector that satisfies the following equation: in, It is the dimension of the pruned lexical representation vector; Step S53: Concatenate the two to obtain the final d-dimensional aggregated representation vector, as shown in the following formula; in, That is, an aggregated representation vector that simultaneously contains syntactic and lexical information; Step S6: According to Calculate the matching score between question-answer pairs to obtain the matching loss function. ; Step S7: Based on Calculate the loss function with added perturbation .

2. The method for enhancing the performance of adversarial examples in retrieval and question answering based on aggregated representation according to claim 1, characterized in that, Step S1: Preprocess the text of the question and answer, transforming it into a sequence of question input text composed of words from the vocabulary. and the sequence of input text for the answer Specifically, it includes: Step S11: Convert the question and candidate answers to Unicode encoding, then perform word segmentation on the question and candidate answers to obtain the word segmentation results for the question and the answer respectively. Where q and a represent a single question and answer, and These represent the i-th word in the question and the i-th word in the answer, respectively. and These are the text lengths of the question and the answer, respectively. Step S12: Concatenate the question segmentation result and the answer segmentation result with the special characters [CLS] and [SEP] in BERT to obtain the question input text sequence. and the sequence of input text for the answer :

3. The method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation according to claim 2, characterized in that, Step S2: ... and The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain... and High-level representation vectors of each symbol and and syntactic representation vectors and Specifically, it includes: Will and Convert to a sequence of token_ids for words, and based on the length of the sequence, assign... and Set the position code position_id to the appropriate length; input the token_id sequence and position_id into the BERT encoder to extract the high-level representation vectors of each symbol in the answer to the question: in, and They represent and With the corresponding high-level representation vector, In the problem, the first The high-level representation vector corresponding to each word Indicates the first in the answer The high-level representation vector corresponding to each word The dimension of the high-level representation vector. These are high-level representation vector matrices consisting of all the words in the question and the answer, respectively.

4. The method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation according to claim 3, characterized in that, Step S3: Utilize the knowledge information from the masked language model head of the pre-trained model to... and The probability distribution mapped to the vocabulary is used to construct a representation vector containing local lexical information through max pooling. and Specifically, it includes: Step S31: Use the masked language model head to... The probability distribution mapped to the vocabulary is represented as follows: in, A vector representation of the i-th symbol in the question and answer; , These are the linear layer parameters of the masked language model; and It is A context vector of dimensions, where each dimension conforms to a probability distribution. It is the size of the BERT model vocabulary; Step S32: Through max pooling, respectively... and indivual dimensional vectors are aggregated into one dimensional vector and The formula is as follows: in, The weights are positive scalars. These are trainable linear layer parameters. It is the dimension index of the vector; and yes A dimensional vector containing fine-grained lexical information from bag-of-words features.

5. The method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation according to claim 4, characterized in that, Step S4: [The sentence is incomplete and requires more context.] and Pruning is performed to obtain the pruned low-dimensional lexical representation vector. Specifically, it includes: Step S41: ... Vie and The vectors are split into Blocks, respectively Each of these blocks has a dimension of Max pooling within the block and Compress to A dimensional vector, the formula is as follows: in, It is a vector index. It is a d-dimensional vector obtained from aggregation, containing the most important terms within each block. It is an index vector of the maximum value within the block; Step S42: Use right The dot product can be approximated by the following formula: When index vector When misaligned, the approximate estimate in the above formula contains an error, namely: Therefore, each Split into two disjoint subsets ,according to Value adjustment The pruned low-dimensional lexical representation vector is obtained. The formula is as follows: in, .

6. The method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation according to claim 5, characterized in that, Step S6: According to Calculate the matching score between question-answer pairs to obtain the matching loss function. Specifically, it includes: Step S61: Calculate the matching degree of a pair of vectors using the dot product of the vectors, as shown in the following formula: in, This represents the unnormalized matching score between question-answer pairs; Step S62: For Performing a Softmax operation yields a normalized probability distribution, as shown in the following formula: in, Representing all and issues The set of mismatched answers; these answers are called negative samples or negative examples. Step S63: Use the negative log-likelihood as the aggregation representation loss function, as shown in the following formula: in, It is the size of the training batch, that is, the number of questions contained in a training batch; Using the same method, the negative log-likelihood loss obtained by matching [CLS] vectors and lexical vectors is denoted as follows: The matching loss function is obtained as follows: in, These are the weights of the loss function.

7. The method for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation according to claim 6, characterized in that, Step S7: Based on Calculate the loss function with added perturbation Specifically, it includes: Step S71: Define the objective of adversarial training as: in, Represents the training dataset. Represents model parameters, It is a tag that indicates whether a question-and-answer pair matches. These are the aggregated representation vectors of the question and answer, obtained by calculating and aggregating the perturbed embedding vectors using the BERT model. They are based on the perturbed high-level symbolic representation. Calculated according to the above formula; Each represents a problem and answer The symbolic embedding vector is obtained by multiplying token_id with the symbolic embedding layer of the BERT model; It is space The perturbation vector within; Step S72: In order to maximize Using the fast gradient sign method, adversarial perturbations are added through gradient ascent, as defined below: in, The maximum norm represents the perturbation. Expressing the request right The differential, Expressing the request right The differential; Step S73: To avoid If it is too large, normalize it using the following formula: in, It is a symbolic function; The final loss function is calculated using the following formula: 。 8. A system for enhancing the performance of adversarial examples in retrieval question answering based on aggregated representation, characterized in that, Includes the following modules: The question and answer input text sequence generation module is used to preprocess the question and answer texts, transforming them into a question input text sequence composed of words from a vocabulary list. and the sequence of input text for the answer ; The syntactic representation vector generation module is used to generate syntactic representation vectors. and The input is a dual-tower model constructed based on a BERT model with shared parameters, which is then encoded to obtain... and High-level representation vectors of each symbol and and syntactic representation vectors and ; The lexical representation vector generation module is used to utilize the knowledge information from the masked language model head of the pre-trained model to... and The probability distribution mapped to the vocabulary is used to construct a representation vector containing local lexical information through max pooling. and ; The lexical representation vector pruning module is used to... and Pruning is performed to obtain the pruned low-dimensional lexical representation vector. ; The aggregation representation vector module is used to concatenate and aggregate the dimensionality-reduced syntactic representation vector and the low-dimensional lexical representation vector to obtain the aggregated representation vector. Specifically, it includes: Step S51: Compress the [CLS] vector using a linear layer, as shown in the following formula: in, These are linear layer parameters. It is the [CLS] compressed syntactic representation vector; It is the dimension of the compressed syntactic representation vector; Step S52: Based on step S4, prune the lexical vectors to... And for A vector that satisfies the following equation: in, It is the dimension of the pruned lexical representation vector; Step S53: Concatenate the two to obtain the final d-dimensional aggregated representation vector, as shown in the following formula; in, That is, an aggregated representation vector that simultaneously contains syntactic and lexical information; Construct a matching loss function to be used based on Calculate the matching score between question-answer pairs to obtain the matching loss function. ; Construct a loss function with added perturbations, for use based on Calculate the loss function with added perturbation .

Images