A common sense question answering method and system based on a pre-trained language model

Abstract

The application discloses a common sense question and answer method and system based on a pre-training language model, belongs to the technical field of common sense question and answer, and can solve the problems that the performance of an existing model is poor in a common sense question and answer task and the calculation cost is high. The method comprises the following steps: S1, acquiring a query answer pair; the query answer pair comprises a query sentence and one candidate answer in a plurality of candidate answers corresponding to the query sentence; S2, extracting external related knowledge of the query answer pair from an external knowledge base; S3, inputting the query answer pair into a pre-training language model, and integrating the external related knowledge and query output features output by the pre-training language model by using an adapter to obtain integrated features; and S4, calculating a credibility score of the query answer pair according to the integrated features, and taking the candidate answer in the query answer pair with the highest credibility score as the correct answer of the query sentence. The application is used for common sense question and answer.

Description

Technical Field

[0001] This invention relates to a common sense question answering method and system based on a pre-trained language model, belonging to the field of common sense question answering technology. Background Technology

[0002] Common sense question answering is a process that combines individual observation with knowledge related to the world and human behavior to reason about common situations in daily life, and it is an important task in natural language understanding. Common sense question answering plays a crucial role in people's daily decision-making. In recent years, pre-trained language models (PLMs) based on full fine-tuning have revolutionized many subfields of Natural Language Processing (NLP) with their outstanding performance. After fine-tuning, they have been applied to common sense question answering tasks, such as... Figure 1 Method a is shown in the table. However, as the scale of model parameters increases, pre-trained language models also have some drawbacks, including increasingly unacceptable computational costs and shortcomings in transparency and interpretability.

[0003] To address the aforementioned shortcomings, researchers combined pre-trained language models with external knowledge to improve the interpretability of the reasoning process. Their methods include... Figure 1 As shown in method b, the first step is to extract the entity-related knowledge subgraph from the query statement, and then design a Graph Neural Network (GNN) module for reasoning. Although this method provides some empirical support for the reasoning process, designing an effective GNN module requires a great deal of expertise. In addition, these methods usually only consider the structured triples related to entities in the query statement, while ignoring the textual knowledge about the query itself, resulting in poor model performance on commonsense question answering tasks and high computational cost. Summary of the Invention

[0004] This invention provides a commonsense question answering method and system based on a pre-trained language model, which can solve the problems of poor performance and high computational cost of existing models in commonsense question answering tasks.

[0005] On the one hand, the present invention provides a commonsense question-answering method based on a pre-trained language model, the method comprising:

[0006] S1. Obtain a query answer pair; the query answer pair includes a query statement and one of the multiple candidate answers corresponding to the query statement;

[0007] S2. Extract relevant external knowledge about the query answer pair from an external knowledge base;

[0008] S3. Input the query answer pair into the pre-trained language model, and use an adapter to integrate the external relevant knowledge with the query output features output by the pre-trained language model to obtain the integrated features;

[0009] S4. Calculate the credibility score of the query answer pair based on the integrated features, and take the candidate answer in the query answer pair with the highest credibility score as the correct answer to the query statement.

[0010] Optionally, S2 specifically includes:

[0011] S21. Identify all entities in the query answer pair, and search for entity-related knowledge in an external knowledge base based on the entities;

[0012] S22. Based on the query answer, search the external knowledge base to obtain the query-related knowledge from the external related knowledge.

[0013] Optionally, S21 specifically includes:

[0014] S211. Identify all entities in the query answer pair and associate all entities with triples in the knowledge graph to obtain a set of triples;

[0015] S212. Convert the triples in the triple set into statements, and use the statements as keywords to search in an external knowledge base to obtain multiple first-related knowledge fragments.

[0016] S213. Calculate the knowledge relevance score between each first relevant knowledge fragment and the query statement, and take the top K first relevant knowledge fragments with the highest knowledge relevance scores as entity-related knowledge.

[0017] Optionally, S22 specifically includes:

[0018] S221. Use the query answer pair as keywords to search in an external knowledge base to obtain multiple second related knowledge fragments;

[0019] S222. Calculate the knowledge relevance score between each second relevant knowledge fragment and the query statement, and take the top K second relevant knowledge fragments with the highest knowledge relevance scores as the query relevant knowledge.

[0020] Optionally, S3 specifically includes:

[0021] S31. Input the query answer pair into the pre-trained language model to obtain the query output features;

[0022] S32. Integrate one piece of knowledge from the external related knowledge and the query output feature into the adapter for L-layer integration to obtain the integrated feature and knowledge output feature; where L is the adapter layer number; the input of the current adapter layer is the query output feature output by the corresponding pre-trained language model layer and the knowledge output feature of the previous adapter layer.

[0023] Optionally, before step S32, the method further includes:

[0024] S33. Filter the query output features using a gating function.

[0025] Optionally, after S32, the method further includes:

[0026] S34. Update the parameters of the adapter using the span masking method.

[0027] Optionally, after S32, the method further includes:

[0028] S35. Update the parameters of the adapter using a relational discrimination method.

[0029] Optionally, after S211, the method further includes:

[0030] S214. Score each triple in the triple set and delete triples with scores below a preset threshold from the triple set.

[0031] On the other hand, the present invention provides a commonsense question-answering system based on a pre-trained language model, the system comprising:

[0032] The acquisition module is used to acquire query answer pairs; the query answer pair includes a query statement and one of a plurality of candidate answers corresponding to the query statement;

[0033] The knowledge extraction module is used to extract relevant external knowledge from an external knowledge base for the query answer pair;

[0034] The feature integration module is used to input the query answer pair into the pre-trained language model and use an adapter to integrate the external relevant knowledge with the query output features output by the pre-trained language model to obtain integrated features;

[0035] The scoring module calculates the credibility score of the query answer pair based on the integrated features, and selects the candidate answer from the query answer pair with the highest credibility score as the correct answer to the query statement.

[0036] The beneficial effects that this invention can produce include:

[0037] (1) The commonsense question answering method based on pre-trained language models provided by this invention can significantly reduce the cost of incorporating external relevant knowledge (i.e., entity-related knowledge and query-related knowledge) by designing a trainable parameter-sharing adapter and connecting it to a parameter-frozen PLM. This invention can effectively incorporate external relevant knowledge and significantly improve the performance of commonsense question answering.

[0038] (2) The commonsense question answering method based on pre-trained language models provided in this invention employs gating and attention mechanisms to filter and fuse query information (i.e., query output features) from PLMs, respectively, in order to enable the adapter to focus on relevant knowledge. Furthermore, this invention utilizes knowledge-related auxiliary tasks such as span masking and relation discrimination to optimize the adapter's parameters, thereby promoting the adapter's memorization of knowledge and helping it understand the inherent relationships within the text. Additionally, this invention constructs a simple yet efficient Knowledgeable Parameter Efficient Tuning Network (KPE) based on this method, combining PLMs with external knowledge to solve commonsense question answering tasks. Experiments on two benchmark datasets demonstrate that the proposed Knowledgeable Parameter Efficient Tuning Network (KPE) can effectively incorporate external knowledge, significantly improving the model's performance in commonsense question answering. Attached Figure Description

[0039] Figure 1 This diagram illustrates the comparison between existing common sense question-and-answer methods and the method proposed in this paper.

[0040] Figure 2 This is a schematic diagram of the KPE model structure provided in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the knowledge adapter structure provided in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram illustrating the impact of the dimensionality reduction dimension r on the performance of the adapter provided in this embodiment of the invention on two datasets;

[0043] Figure 5 A schematic diagram illustrating the KPE model results case analysis provided in this embodiment of the invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.

[0045] With the significant success of pre-trained language models (PLMs) on various tasks, some researchers have proposed using fully fine-tuned PLMs for commonsense question answering tasks. While the use of PLMs is becoming increasingly widespread, fine-tuning all parameters of a model incurs extremely high computational costs, a problem that becomes more pronounced as the scale of PLM parameters grows. Furthermore, pre-trained language models lack transparency and interpretability due to the absence of modules that explicitly model knowledge. In light of this, some existing methods improve PLMs by using carefully designed graph neural network modules to integrate relevant knowledge from knowledge graphs. However, these methods suffer from several drawbacks: firstly, they require expertise to design effective graph neural network modules to encode knowledge subgraphs; secondly, they only consider structured knowledge related to entities, neglecting query-related knowledge that may exist in textual form.

[0046] Because fine-tuning all parameters of a pre-trained language model is too costly, researchers have proposed fine-tuning a small subset of the model's parameters while freezing a portion of them. Adapter fine-tuning is a common and efficient parameter-tuning method that inserts trainable adapter modules between frozen PLM layers to guide the PLMs to achieve the desired input and output. Existing techniques utilize adapters to incorporate knowledge into pre-trained language models. Inspired by prompting methods, some researchers have also proposed prefix fine-tuning and prompting fine-tuning. They use a set of trainable prompt lexical units as input or intermediate layers and update only these lexical units during training. Furthermore, existing techniques have explored low-rank adaptation methods for efficient parameter fine-tuning, which inject and optimize low-rank matrices of attention weights into frozen PLMs.

[0047] Based on the aforementioned research, this invention presents a common-sense knowledge question-answering task in the form of multiple-choice questions. Given a natural language query (question) q and a set of candidate answers (or options). For each answer, a credibility score ρ(q,a) is measured, and the most credible answer a is selected. * To improve general knowledge quiz skills, based on q and This invention utilizes an external knowledge base to extract knowledge fragments related to entities and queries.

[0048] like Figure 2 As shown, the Knowledge Fusion Module (KPE) proposed in this invention combines PLMs with external knowledge by connecting a parameter-sharing knowledge adapter to frozen pre-trained language models (PLMs). The PLMs take (q,a) as input and output a credibility score ρ(q,a). The purpose of the knowledge adapter is to integrate knowledge fragments k.

[0049] Specifically, embodiments of the present invention provide a commonsense question-answering method based on a pre-trained language model, the method comprising:

[0050] S1. Obtain the query answer pair; the query answer pair includes a query statement and one of the multiple candidate answers corresponding to the query statement.

[0051] S2. Extract relevant external knowledge about the query answer pair from an external knowledge base.

[0052] S2 specifically includes:

[0053] S21. Identify all entities in the query answer pair, and retrieve entity-related knowledge from the external knowledge base based on the entities.

[0054] Specifically, it includes:

[0055] S211. Identify all entities in the query answer pair and associate all entities with triples in the knowledge graph to obtain a set of triples.

[0056] Preferably, after S211, the method further includes:

[0057] S214. Score each triple in the triple set and delete triples with scores below a preset threshold from the triple set.

[0058] S212. Convert the triples in the triple set into statements, and use the statements as keywords to search in an external knowledge base to obtain multiple first-related knowledge fragments.

[0059] S213. Calculate the knowledge relevance score between each first relevant knowledge fragment and the query statement, and take the top K first relevant knowledge fragments with the highest knowledge relevance scores as entity-related knowledge.

[0060] S22. Based on the query answer, search the external knowledge base to obtain the query-related knowledge from the external related knowledge.

[0061] Specifically, it includes:

[0062] S221. Use the query answer pair as keywords to search in an external knowledge base to obtain multiple second-related knowledge fragments;

[0063] S222. Calculate the knowledge relevance score between each second relevant knowledge fragment and the query statement, and take the top K second relevant knowledge fragments with the highest knowledge relevance scores as the query relevant knowledge.

[0064] One traditional source of common sense knowledge is the triple-based knowledge graph; however, triples can only encode a limited number of types of knowledge. This invention uses a general sentence corpus of common sense facts, GenericsKB, as an external knowledge base (or knowledge source). The text in this external knowledge base can represent more complex common sense knowledge, involving facts with more than three concepts.

[0065] For entity-related knowledge, this invention first identifies all entities in the query statement and candidate answers, and associates them with triples in the knowledge graph (ConceptNet). Then, these triples are serialized into statements and used as keys to retrieve knowledge fragments from GenericsKB.

[0066] Given a query q and candidate answers This invention first extracts entity 'e'. Then, it associates all triples in ConceptNet that contain 'e' to obtain a set of triples. To compress and filter the extracted triples, this invention scores each triple:

[0067]

[0068] Where p i Represents the i-th triple (h) i ,r i ,t i The score of w i These are the triple weights provided by ConceptNet, where N is... Size, yes There is a relationship r in i The number of triples. If p i The score is higher than the predefined score threshold p * The triple will then be added to the selected set of triples. middle.

[0069] Next, the present invention will These triples are converted into a series of statements, retrieving knowledge from the unstructured external knowledge base GenericsKB. Specifically, for each triple (h... i ,r i ,t i First, a set of predefined relational templates is used to generate statement s. i For example, the triple (sweltering, RelatedTo, hot) can be serialized into the statement "sweltering is related to hot". Then, s... iRetrieve relevant knowledge fragments in sentence form from GenericsKB using keywords. Excludes entity pairs (h i ,t i Knowledge fragments will be directly ignored.

[0070] Then, the knowledge fragments most relevant to the query are selected to enhance the commonsense knowledge question answering. Specifically, a pre-trained SimCSE model is used to obtain the statement embedding, and based on this, k is calculated for each retrieved knowledge fragment. i The cosine similarity between the query q and the query q is used as the knowledge relevance score. Finally, after processing all triples, the top K (in the example, K=5) retrieved knowledge fragments are selected as entity-related knowledge (i.e., entity-related knowledge) based on the calculated knowledge relevance scores.

[0071]

[0072] Considering the rich semantic information contained in query q, it is necessary to leverage query-related knowledge to improve common-sense question-answering tasks. Similar to entity-related knowledge retrieval, this invention retrieves query-related knowledge fragments from GenericsKB by concatenating the query with all candidate answers as retrieval keywords. Then, a knowledge relevance score is calculated, and the K knowledge fragments with the highest scores are selected as query-related knowledge (i.e., query-relevant knowledge).

[0073] S3. Input the query answer pair into the pre-trained language model, and use an adapter to integrate entity-related knowledge or query-related knowledge with the query output features of the pre-trained language model to obtain integrated features.

[0074] Specifically, it includes:

[0075] S31. Input the query answer pair into the pre-trained language model to obtain the query output features;

[0076] S32. Input one piece of knowledge from the relevant external knowledge and the query output feature into the adapter for integration at layer L to obtain the integrated feature and knowledge output feature; where L is the adapter layer number; the input of the current adapter layer is the query output feature output by the corresponding pre-trained language model layer and the knowledge output feature of the previous adapter layer.

[0077] Furthermore, prior to S32, the method further includes:

[0078] S33. Use gating functions to filter the query output features.

[0079] In practical applications, after S32, the method may further include:

[0080] S34. Update the adapter parameters using the span mask method.

[0081] Following S32, the method may further include:

[0082] S35. Update the parameters of the adapter using the relation discrimination method.

[0083] In this embodiment of the invention, each pre-trained language model layer (PLMs layer) is connected to a parameter-sharing adapter layer, such as... Figure 3 As shown.

[0084] For the l-th adapter layer (l∈[1,L]), the input The knowledge output features of the vertically connected (l-1)th adapter layer and the query output features of the l-th PLMs layer The structure consists of m and n, representing the lengths of the PLMs input sequence and knowledge fragment, respectively, and d, the hidden size. Note that this invention uses learnable gating functions (also called gating units) to filter the query output features of the PLMs. To obtain the key information needed for the query. Specifically,

[0085]

[0086] Where G∈R m×d It is a trainable matrix that learns during training, and ⊙ represents element-wise multiplication.

[0087] Now, given the input The adapter layer first projects the data to r dimensions using a linear projection layer. Then, a self-attention layer is applied to better fuse the knowledge and query information from the PLMs. Afterward, another linear projection layer projects the data back to the original dimension d. Finally, the output features of the upper projection layer are... Divided into two parts: Residual connection layer for PLMs For use in the next adapter layer.

[0088] To enhance the adapter's knowledge modeling capabilities, this invention also designs the following two knowledge-related tasks, which are based on the adapter's final output. As input.

[0089] Span masking (also known as fragment masking) tasks can help improve the adapter's memorization of knowledge. Specifically, for tasks involving triples (h... i ,r i ,t i Corresponding entity-related knowledge This invention masks the corresponding markers of the tail entity and replaces them with the same number of [MASK] to generate a masked sequence. The masked sequence is then input into an adapter for forward inference. Based on the final adapter output... Predict the masked tags and calculate their cross-entropy loss. For knowledge fragments related to the query This invention masks a total of 15% of the tags at the fragment level and uses the same method as SpanBERT for prediction.

[0090] The relation discrimination task helps the adapter understand the inherent relationships in text and improves the robustness of the learned representation through contrastive learning. This task is only applicable to entity-related knowledge fragments containing entity pairs. The specific task is as follows:

[0091] Given a fragment of entity-related knowledge and its corresponding triple (h) i ,r i ,t i ), through the embedding of entity references (from adapter output) Perform average pooling to obtain the entity representation. and Then, and Connect them as a relation representation To improve the understanding of relational facts, this invention will use relation r I Positive samples are considered positive samples, while other relationships are considered negative samples. Finally, the InfoNCE loss function is used to reduce the distance between positive samples and increase the distance between negative samples.

[0092]

[0093] Where τ is the temperature hyperparameter, and |ε| is the relation r in ConceptNet. i The quantity, f(r) (i) ) represents a relationship based on PLMs. i The lookup operation for the tag ID. If r i There are multiple tags that can be used to average the pooling of applications.

[0094] S4. Calculate the credibility score of the query answer pair based on the integrated features, and take the candidate answer in the query answer pair with the highest credibility score as the correct answer to the query statement.

[0095] Given a query q and candidate answers Utilizing the output of the final PLMs layer Calculate credibility score And maximize the correct answer a using cross-entropy loss.* Credibility score:

[0096]

[0097] In summary, the overall training objective function of the Knowledge Awareness Parameter Efficient Fine-Tuning Network (KPE) provided by this invention is as follows:

[0098]

[0099] During the training process, this invention will extract entity-related and query-related knowledge. and A knowledge item is randomly selected from the query. Please note that for query-related knowledge fragments that are not suitable for relation discrimination tasks, this invention will ignore the corresponding loss function.

[0100] Another embodiment of the present invention provides a commonsense question-answering system based on a pre-trained language model, which is the aforementioned Knowledge Awareness Parameter Efficient Fine-Tuning Network (KPE). The system includes:

[0101] The retrieval module is used to retrieve query answer pairs; a query answer pair includes a query statement and one of the multiple candidate answers corresponding to the query statement;

[0102] The knowledge extraction module is used to extract relevant external knowledge from external knowledge bases for query answer pairs;

[0103] The feature integration module is used to input query answer pairs into the pre-trained language model and use an adapter to integrate external relevant knowledge with the query output features output by the pre-trained language model to obtain integrated features.

[0104] The scoring module calculates the credibility score of the query answer pair based on the integrated features, and selects the candidate answer from the query answer pair with the highest credibility score as the correct answer to the query statement.

[0105] For a detailed description of each module in the above system, please refer to the description of each step in the method. It will not be repeated here. The above system can achieve the same functionality as the method.

[0106] This invention evaluates the Knowledge-Aware Parameter Efficient Fine-Tuning Network (KPE) on two benchmark datasets: OpenbookQA and CommonsenseQA 2.0. OpenbookQA is a question-and-answer dataset for elementary science, with four different options for each question. This dataset contains a total of 5,957 questions, with the official dataset divided into training, validation, and test sets. CommonsenseQA 2.0 (CSQA2) is a binary classification dataset containing 14,343 questions. Note that the CSQA2 test set is not publicly available; this invention requires submitting the model's predictions to the official leaderboard for evaluation.

[0107] For knowledge retrieval, this invention first stores the GenericsKB using Elasticsearch and then performs retrieval using Elasticsearch retrieval functions based on BM25. This invention selects parameter values ​​that yield the best results on the development set. In the experiment, the temperature hyperparameter τ in the relation discrimination task is set to 0.1, and the score threshold p for triplet matching is set... * The dimensionality reduction dimension size r in the adapter is set to 3.5. Following previous work, the hidden layer dimension d = 1024 and the number of layers L = 24. The AdamW optimizer is used in the experiments. For model training, the batch size can be set to 32 and the learning rate to 2 × 10⁻⁶. -5 Based on AdapterHub, a baseline model for efficient parameter fine-tuning of commonsense question answering was implemented.

[0108] This invention compares KPE with parameter-based efficient fine-tuning methods and existing high-performance common sense question answering models.

[0109] This invention compares KPE with the following parameter-based efficient fine-tuning methods.

[0110] Bottleneck Adapter: The first method to use an adapter for fine-tuning in natural language processing.

[0111] Prefix Tuning: Inserting a series of learnable prompts into the model input or intermediate layers to reduce training costs.

[0112] LoRA: Pre-sets a trainable rank decomposition matrix in each layer of the PLM to reduce the number of trainable parameters.

[0113] MAM Adapter: Constructs an efficient adapter module that combines the advantages of adapters, prefix tuning, and low-rank methods.

[0114] Compactor: Based on the ideas of adapter, low-rank optimization and parameterized hypercomplex multiplication layers, it can better balance task performance and the number of trainable parameters.

[0115] To ensure fair comparisons, this invention improves upon these baseline methods by concatenating the extracted knowledge with the original inputs of these benchmark models.

[0116] This invention also compares KPE with existing powerful commonsense question answering methods. For the OpenbookQA dataset, the model is compared with the following baseline models that incorporate knowledge into PLMs via GNN modules:

[0117] (1)RN; (2)RGCN; (3)GconAttn; (4)MHGRN; (5)QA-GNN; (6)GSC; (7)JointLK.

[0118] For fair comparison, this invention uses the same pre-trained language model (i.e., RoBERTa-large) on all the aforementioned baseline models and the KPE provided by this invention.

[0119] For the CSQA2 dataset, this invention uses the Unicorn-11B initial model as the pre-trained language model for KPE, and compares KPE with the following fully fine-tuned models from the official leaderboard:

[0120] (1)T5-large; (2)Unicorn-large; (3)T5-11B; (4)Unicorn-11B; (5)GKP+Unicorn-11B-ft.

[0121] Among these baseline models, GKP+Unicorn11B-ft performed best. It guides GPT3 to generate knowledge through manually designed demonstrations and integrates this knowledge into commonsense question answers through prompts.

[0122] Table 1. Performance comparison of the parameter-efficient fine-tuning method on the two datasets.

[0123]

[0124] Because the CSQA2 dataset is more difficult, this invention uses RoBERTa-large with 355.36 million parameters as PLMs on the OpenbookQA dataset, and Unicorn-11B with 11.31 billion parameters on the CSQA2 dataset. Due to the use of different underlying PLMs, the trainable parameters may differ between the two datasets.

[0125] Table 1 shows the results of the proposed KPE and common parameter-based efficient fine-tuning methods on the OpenbookQA and CSQA2 datasets. Note that for fair comparison, the common-sense knowledge extracted in this invention is concatenated with the original inputs of these baseline methods. Since the annotations for the CSQA2 test set are not publicly available, this invention only reports comparison results on the validation set. From Table 1, it can be observed that:

[0126] (1) KPE consistently outperforms all baseline methods on both datasets. Compared to the best baseline method, KPE achieves an improvement of approximately 12.5% ​​on the CSQA2 development set and approximately 1.3% on the OpenbookQA test set. This invention argues that KPE benefits from a well-designed knowledge adapter, which is parametrically efficient and effectively integrates common-sense knowledge. Furthermore, KPE's improvement on the CSQA2 dataset is significantly greater than its improvement on the OpenbookQA dataset (+12.5% ​​vs. +1.3%). This is likely because the problems on the CSQA2 dataset are more difficult and require greater utilization of external knowledge. Therefore, KPE achieves a greater improvement on the CSQA2 dataset by effectively integrating external knowledge.

[0127] (2) Compared to the Bottleneck Adapter, Prefix Tuning, and MAM Adapter, KPE introduces fewer parameters while achieving significant improvements on both datasets. This is because the knowledge adapter employs an efficient parameter-sharing strategy and better integrates knowledge through two knowledge-related auxiliary tasks. Gating and attention mechanisms also help the adapter focus on useful knowledge for improving commonsense question answers.

[0128] (3) Although the baseline methods Compacter and LoRA introduce fewer parameters, their performance is far inferior to KPE. This shows that the method of the present invention achieves a better balance between the number of trainable parameters and task performance.

[0129] Table 2 Comparison of test accuracy on OpenbookQA

[0130]

[0131] Table This indicates the results reported in GSC.

[0132] Table 3 compares the performance of the full fine-tuning method on CSQA2.

[0133]

[0134] Table This indicates the results reported in the paper. This refers to the results reported in the official rankings.

[0135] Tables 2 and 3 show the comparison of the results of the model of the present invention with existing high-performance common sense question answering methods on the OpenbookQA dataset and the CSQA2 dataset.

[0136] As shown in Table 2, KPE outperforms all GNN-based methods, achieving the best performance. This demonstrates that KPE integrates knowledge more effectively than existing methods through a knowledge adapter, improving the ability of commonsense question answering. KPE can further benefit from the advancements in Large Language Models (LLMs) and has significant value for research on efficient parameter fine-tuning.

[0137] To verify the importance of each module in the KPE model, this invention compares it with the following variants:

[0138] (1) KPE-w / oE: A variant of KPE that removes knowledge associated with entities.

[0139] (2) KPE-w / oQ: KPE variant that removes knowledge associated with the query.

[0140] (3) KPE-w / oE&Q: A KPE variant that removes entity and query-related knowledge.

[0141] Accordingly, the three span masking and relation identification tasks mentioned above were removed.

[0142] (4) KPE-w / oS: KPE variant for removing span mask task.

[0143] (5) KPE-w / oR: KPE variant for removing relation identification task.

[0144] (6) KPE-w / oS&R: A KPE variant that removes span masking and relation identification tasks.

[0145] (7) KPE-w / oA: Replace the self-attention mechanism in the knowledge adapter with a KPE variant of a regular nonlinear function.

[0146] (8) KPE-w / oG: Replace the learnable gating function in the knowledge adapter with a directly connected KPE variant.

[0147] Table 4 shows the ablation experiment results on the OpenbookQA and CSQA2 datasets.

[0148]

[0149] Table 4 shows the results of the ablation experiment. The following conclusions can be drawn:

[0150] (1) On both datasets, removing query-related knowledge resulted in a greater performance degradation than removing entity-related knowledge. This indicates the importance of query-related knowledge for answering common-sense questions. When both entity and query-related knowledge were removed simultaneously, performance degraded significantly (by -2.8% and -5.05% on OpenbookQA and CSQA2, respectively).

[0151] (2) Disabling any knowledge-related auxiliary tasks will result in a performance degradation, which suggests that these two tasks enable the adapter to better capture and simulate knowledge, thereby improving common sense question answering capabilities.

[0152] (3) KPE consistently outperforms KPE-w / oG and KPE-w / oA on both datasets, which confirms that gating and self-attention mechanisms help with knowledge integration.

[0153] To explore the impact of downsampling r in the adapter on model performance, this invention takes values ​​for r from 16 to 1024 and observes the results on two datasets, such as... Figure 4 As shown.

[0154] It can be observed that on both datasets, accuracy generally increases and reaches its maximum as r increases from 16 to 256, while it begins to decline when r is greater than 256. Overall, KPE's best performance on the OpenbookQA and CSQA2 datasets occurs at r=256.

[0155] To gain a more intuitive understanding of how external knowledge in KPE helps improve answers to common sense questions, this invention compares KPE with KPE-w / oE&Q and uses two examples from the OpenbookQA and CSQA2 datasets to visualize the distribution of question options.

[0156] like Figure 5 As shown in (a), for the query "Desert environments are generally_", KPE selected the correct option "sweltering", while KPE-w / o E&Q assigned a higher score to the incorrect option "arctic like". Extracted knowledge (e.g., "some plants grow in the hot, dry desert", "sweltering is related to hot") helps in answering common-sense questions. Furthermore, from Figure 5As can be observed in (b), although both KPE and KPE-w / oE&Q correctly predicted the answer, KPE had a higher confidence in the result due to the knowledge it extracted.

[0157] This invention proposes for the first time an efficient fine-tuning network for knowledge perception parameters in commonsense question answering, which uses a parameter-sharing adapter to incorporate external knowledge into the model.

[0158] The adapter designed in this invention integrates entity-related and query-related knowledge through two auxiliary tasks. Furthermore, this invention employs gating and attention mechanisms to filter and fuse query information from a pre-trained language model, respectively, enabling the adapter to focus on relevant knowledge in commonsense question answering.

[0159] This invention conducts extensive experiments on two benchmark datasets. The experimental results show that the KPE model can effectively incorporate external knowledge, improve the performance of commonsense question answering, and has a very low computational cost.

[0160] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A common sense question answering method based on a pre-trained language model, characterized in that, The method comprises: S1, acquiring a query answer pair; the query answer pair comprises a query sentence and a candidate answer in a plurality of candidate answers corresponding to the query sentence; S2, extracting external related knowledge of the query answer pair from an external knowledge base; S3, inputting the query answer pair into a pre-trained language model, and integrating the external related knowledge and query output features output by the pre-trained language model by using an adapter to obtain integrated features; S4, calculating a credibility score of the query answer pair according to the integrated features, and taking the candidate answer in the query answer pair with the highest credibility score as the correct answer of the query sentence; The S3 specifically comprises: S31, inputting the query answer pair into a pre-trained language model to obtain query output features; S32, input one piece of knowledge in the external related knowledge and the query output feature into an adapter for L-layer integration, to obtain an integrated feature and a knowledge output feature; wherein, L is the number of adapter layers; connect each pre-training language model layer with an adapter layer sharing a parameter; the input expression of the first adapter layer is as follows: ​​ ； wherein, is the query output feature of the th pre-trained language model layer, , is a trainable matrix that is learned during training, denotes element-wise multiplication; is the knowledge output feature of the th adapter layer; and denote the length of the pre-trained language model input sequence and the knowledge snippet, respectively, is the hidden size; Before the S32, the method further comprises: S33, filtering the query output features by using a gating function; After the S32, the method further comprises: S35, updating parameters of the adapter by using a relationship discrimination method; Given a fragment of entity-related knowledge and its corresponding triples Entity representations are obtained by performing average pooling on the token embeddings of entity mentions. and ;Will and Connect them as a relation representation ; Relationship Treating each relation as a positive sample and other relations as negative samples; the InfoNCE loss function is used for adapter parameter updates, and the expression of the InfoNCE loss function is as follows: ； wherein, is a temperature hyperparameter, is the number of relations in ConceptNet, denotes a lookup operation based on the label ID of the relation based on the pre-trained language model.​ 2. The method of claim 1, wherein, The S2 specifically comprises: S21, identifying all entities of the query answer pair, and retrieving entity related knowledge in the external related knowledge according to the entities in the external knowledge base; S22, retrieving query related knowledge in the external related knowledge according to the query answer pair in the external knowledge base.

3. The method of claim 2, wherein, The S21 specifically comprises: S211, identifying all entities of the query answer pair, and associating all the entities with triples in a knowledge graph to obtain a triple set; S212, converting triples in the triple set into sentences, and retrieving the sentences as keywords in the external knowledge base to obtain a plurality of first related knowledge segments; S213, calculating a knowledge correlation score of each first related knowledge segment with the query sentence, and taking the first related knowledge segments with top K knowledge correlation scores as entity related knowledge.

4. The method of claim 2, wherein, The S22 specifically comprises: S221, retrieving the query answer pair as keywords in the external knowledge base to obtain a plurality of second related knowledge segments; S222, calculating a knowledge correlation score of each second related knowledge segment with the query sentence, and taking the second related knowledge segments with top K knowledge correlation scores as query related knowledge.

5. The method of claim 1, wherein, After the S32, the method further comprises: S34, updating parameters of the adapter by using a span mask method.

6. The method of claim 3, wherein, After the S211, the method further comprises: S214, scoring each triple in the triple set, and deleting triples with scores lower than a preset threshold from the triple set.

7. A common sense question answering system based on a pre-trained language model, characterized in that, The system comprises: An acquisition module configured to acquire a query answer pair; the query answer pair comprises a query sentence and a candidate answer in a plurality of candidate answers corresponding to the query sentence; A knowledge extraction module configured to extract external related knowledge of the query answer pair from an external knowledge base; The feature integration module is configured to input the query answer pair into a pre-trained language model, and integrate the external related knowledge and query output features output by the pre-trained language model by using an adapter to obtain integrated features. The scoring module is configured to calculate a credibility score of the query answer pair according to the integrated features, and take the candidate answer in the query answer pair with the highest credibility score as the correct answer of the query sentence. The feature integration module is specifically configured to: input the query answer pair into a pre-trained language model to obtain query output features; and input one piece of knowledge in the external related knowledge and the query output features into an adapter for L-layer integration to obtain integrated features and knowledge output features; wherein L is the number of adapter layers; each pre-trained language model layer is connected with one parameter-shared adapter layer; the input expression of the first adapter layer is as follows: ​​ ； wherein, is the query output feature of the th pre-trained language model layer, , is a trainable matrix that is learned during training, denotes element-wise multiplication; is the knowledge output feature of the th adapter layer; and denote the length of the pre-trained language model input sequence and the knowledge snippet, respectively, is the hidden size; The feature integration module is further configured to filter the query output features by using a gating function, and update parameters of the adapter by using a relationship discrimination method. Given entity related knowledge snippets and their corresponding triples , the entity representation is obtained by performing average pooling operation on the token embedding of the entity mention and ; the and are connected as the relation representation ; the relation is regarded as its positive sample, and other relations are regarded as negative samples; the InfoNCE loss function is used for adapter parameter updating, and the expression of the InfoNCE loss function is as follows: ； wherein, is a temperature hyperparameter, is the number of relations in ConceptNet, denotes a lookup operation based on the relation ID of the pre-trained language model.​

Images