Scientific choice question automatic answering method based on large language model, medium and equipment

Abstract

The invention discloses a scientific choice question automatic answering method based on a large language model, a medium and equipment, and the method comprises the steps: carrying out the retrieval enhancement based on an external database, and obtaining a document set D related to a question; secondly, abstracting and preprocessing each document in the document set D to generate a preprocessed document set D '; and splicing the questions, the answer options and the preprocessed document information to obtain an input sequence S. And extracting answer information based on a Flash Attention mechanism, and voting based on a multi-thinking chain to obtain b answer options as answers of scientific choice questions. The method has higher answering accuracy and reliability, the experimental result on a Kaggle scientific subject selection question data set shows that the score of the method reaches 0.909, the method is remarkably superior to other baseline models, and the answering accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to a method, medium and device for automatically answering scientific multiple-choice questions based on a large language model. Background Art

[0002] In the field of natural language processing, machine reading comprehension is an important research task. Its goal is to understand and automatically answer questions based on given relevant materials and questions. At present, the field of machine reading comprehension mainly involves four tasks: multiple-choice selection, free answering, cloze, and fragment extraction. With the popularization and development of artificial intelligence in the fields of education and employee training, the understanding and application of science-themed multiple-choice questions have received widespread attention. This type of multiple-choice question aims to select the correct answer from several options for cutting-edge knowledge such as high-precision technology. It is widely used in the training of high-end technical talents in engineering, communications, biology, medicine, and economics.

[0003] Existing science-themed multiple-choice question answering methods are generally based on deep learning technology, and mainly fall into the following categories:

[0004] 1. Direct matching method: by directly matching words or paragraphs between the question stem and the options, find the answer with the highest similarity. Although this method is simple and efficient, it performs poorly for questions that require complex reasoning or rely on background knowledge, especially when dealing with semantically diverse options, its accuracy is not high.

[0005] 2. Methods based on pre-trained language models: Use pre-trained language models (such as BERT, GPT-3) for question understanding and answer generation. These models are trained with a large amount of corpus and have strong language understanding capabilities. For example, in the MCTest dataset, the answer accuracy using the DCMN model reached 86.5%; in the SQuAD dataset, the F1 value using the BERT model was as high as 93.2%. Although these models have strong semantic processing capabilities, they are still insufficient when dealing with complex reasoning scenarios and are difficult to deal with problems that require multi-level reasoning.

[0006] 3. Enhanced retrieval methods: Some of the latest research has begun to focus on enhanced retrieval technology, which improves the accuracy of answer selection by obtaining additional information related to the question stem from external databases. However, although this type of method has improved information acquisition, it still has shortcomings, especially the lack of effective combination of semantic analysis and reasoning ability. Existing enhanced retrieval methods have not been widely used in complex reasoning tasks, and most of them rely on the accuracy of external data, while ignoring the deep semantic integration with the question itself.

[0007] Although the existing technology has made some progress in answering scientific questions, it still has the following major defects:

[0008] ① Insufficient semantic analysis ability: Most methods rely too much on simple word matching or paragraph matching, and lack an understanding of the deep semantic relationship between the question stem and the options. For questions that require complex reasoning and background knowledge, existing methods are difficult to accurately answer.

[0009] ② Insufficient use of external information: Although enhanced retrieval technology can improve the accuracy of answer selection, the existing methods still do not make full use of external information and lack effective integration of the relationship between external data and the question stem and options. The retrieved external information often fails to fully optimize the combination with the logical reasoning ability of the model, resulting in limited results in solving complex problems.

[0010] ③ Lack of logical reasoning ability: Although the existing enhanced retrieval and multi-chain reasoning technologies have improved the reasoning ability to a certain extent (such as multi-chain meta-reasoning MCR), these technologies lack an effective mechanism for integrating reasoning results between multiple thinking chains, resulting in low consistency of results between reasoning chains. In addition, the existing technologies lack a dynamic weighting mechanism to adjust the weights of different reasoning chains during the reasoning process, which is prone to bias.

[0011] ④Long text processing and information loss problems: Existing technologies (multi-hop question answering methods) improve processing efficiency by summarizing and preprocessing long documents to generate compact representations. However, excessive information compression may lead to the loss of key semantic information, especially when facing multi-level reasoning problems. Overly simplified preprocessing often ignores important contextual relationships, limiting the model's ability to answer complex problems.

[0012] ⑤ Dataset quality issues: Some data sets are of low quality, contain a lot of duplicate content, and cover a single field, which limits the generalization ability of the model. In a wider range of application scenarios, the model cannot effectively handle a variety of problem types, showing the limitation of poor adaptability. Summary of the invention

[0013] The purpose of the present invention is to provide a method, medium and device for automatically answering scientific multiple-choice questions based on a large language model, aiming to solve the above-mentioned problems.

[0014] The present invention is mainly achieved through the following technical solutions:

[0015] The method for automatically answering scientific multiple-choice questions based on a large language model includes the following steps:

[0016] Step S1: Analyze the question text Q, perform retrieval enhancement based on the external database, perform retrieval matching from the external database, and obtain a document set D = {d1, d2, ..., d n};

[0017] Step S2: Then, each document in the document set D is summarized and preprocessed to generate a preprocessed document set D'; the question, answer options and preprocessed document information are concatenated to obtain an input sequence S = concat (Q, A, D'), where A is the answer option set, and A = {a1, a2, ..., a m};

[0018] Step S3: Convert the concatenated input sequence S into an embedding vector Embed(S), and then extract the answer information based on the FlashAttention mechanism:

[0019] E(S)=FlashAttention(Embed(S))

[0020] Step S4: Based on multi-thinking chain voting, each thinking chain uses the decision-making layer FFN of the Platypus2 model decision Score each answer option F(A):

[0021] F(A)=FFN decision (FlashAttention(Embed(S)))

[0022] Each thinking chain ranks the score F(A) of each answer option, and multiple thinking chains finally vote to obtain b answer options as the answer to the scientific multiple-choice question.

[0023] In order to better implement the present invention, further, in the step S1, the question text Q is first analyzed, and the question stem is accurately segmented and preprocessed using the BERT word segmenter, and then the segmentation result is encoded by using the pre-trained gte-base model to generate an embedding vector E(Q); then based on the nearest neighbor search technology, a search is performed in an external database to locate the context of the question and obtain a document set D.

[0024] In order to better implement the present invention, further, in step S1, the embedding vector E(Q) is combined into an embedding matrix and L2 norm normalization is performed:

[0025]

[0026] Where: Norm(E(Q)) is the normalized embedding vector;

[0027] n is the number of elements in the vector,

[0028] e i is the i-th element in the embedding vector E(Q);

[0029] Then, based on Norm(E(Q)), the nearest neighbor search technique is used to perform a search in the external database to locate the context of the question and obtain the document set D.

[0030] In order to better implement the present invention, further, the nearest neighbor search technology is to find the embedding vector closest to the embedding vector to be queried in the external database by minimizing the Euclidean distance between the embedding vector to be queried and the embedding vector in the database, thereby obtaining the context of the problem.

[0031] In order to better implement the present invention, further, step S4 includes the following steps:

[0032] Step S41: Each thought chain randomly selects a document subset D of the preprocessed document set D' i , then score the answer options F i (A):

[0033] F i (A) = FFN decision (FlashAttention(Embed(concat(Q,A,D i ))));

[0034] Step S42: sorting all answer options according to the scores obtained from each thinking chain, and selecting the first a answer options with the highest scores as the answer options recommended by the thinking chain;

[0035] Step S43: For each answer option, calculate the number of times the selected thinking chain is recommended in all thinking chains; then, sort all the answer options according to the number of times to obtain the final recommended b answer options as the answers to the scientific multiple-choice questions.

[0036] In order to better implement the present invention, further, a=b=3.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for automatically answering scientific multiple-choice questions based on a large language model.

[0038] A computer device includes one or more processors and a storage device, wherein the storage device is used to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for automatically answering scientific multiple-choice questions based on a large language model.

[0039] The beneficial effects of the present invention are as follows:

[0040] (1) When processing long texts and complex document information, the present invention uses the Flash Attention mechanism to reduce data dimensionality, extracts key information in the sequence through the feedforward layer, combines questions, answer options and retrieved document information, and constructs a comprehensive input sequence to improve the model's ability to process long texts and the accuracy of information extraction. The method of the present invention adopts a multi-thinking chain voting mechanism, which reduces the bias that may be introduced by relying on a single data source or thinking chain by integrating the results of different reasoning paths, thereby achieving higher accuracy and reliability when comprehensively evaluating answer options. Experimental results on the Kaggle science-themed multiple-choice question dataset show that the present invention is significantly better than other baseline models and has good practicality.

[0041] (2) The present invention optimizes resource management: When hardware resources are limited, the present invention can still run efficiently. It runs smoothly and achieves good results on a single-core RTX 4090 with 24G video memory, which proves the advantages of this method in resource management and model optimization.

[0042] (3) The present invention enhances the model’s comprehension ability: The present invention introduces the Retrieval Augmented Generation (RAG) technology, uses an external database (such as Wikipedia) to retrieve relevant information, embeds question prompts through the SentenceTransformer model, and uses the FAISS index to perform similarity search to obtain contextual information related to the question content, thereby enriching the semantic relationship between the question stem and options, enriching the model’s understanding of the question stem and options, and improving the accuracy and reliability of question answering.

[0043] (4) The present invention has strong adaptability: the present invention is based on the existing large-scale pre-trained language model (such as Platypus2-70B), has zero-shot capabilities, and can be directly applied to new multiple-choice question-answering tasks without involving a parameter training process, showing strong adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0045] Embodiment 1:

[0046] The automatic answering method for scientific multiple-choice questions based on a large language model mainly includes three parts: retrieval enhancement based on an external database, large model data dimensionality reduction based on Flash Attention, and voting based on multiple thinking chains to obtain more accurate results. Since the present invention uses the existing Platypus2 large model, the model has zero-shot capabilities. The following is a mathematical formula description of the process:

[0047] R final =Vote(C1(text(Q),FA(X1)),C2(text(Q),FA(X2)),…,C N (text(Q), FA(X N )))

[0048] Among them, Q is the input question content,

[0049] text(Q) represents the corresponding search information retrieved from the external database;

[0050] X1, X2, ...X N Indicates different options;

[0051] FA(X) represents the data dimension reduction operation based on the Flash Attention mechanism;

[0052] Vote(C1,C2,…,C N ) represents the multi-thinking chain voting process;

[0053] R final It is the final result obtained based on multi-thinking chain voting.

[0054] like Figure 1 As shown, it mainly includes indexing, retrieval and generation stages, which specifically include the following steps:

[0055] First, in the question stem analysis stage, the question stem content is analyzed, and search and matching are performed from the external database to obtain the corresponding search information.

[0056] Then, the search information data is spliced. Since the search text is long, the present invention appropriately introduces the Flash Attention mechanism in the feedforward layer to extract important information in the sequence. Then a series of data analysis and processing are performed to obtain the correctness of the data. The results of the external database search are randomized to a certain extent, and the questions are asked and answered again from different thinking angles to view the corresponding results.

[0057] Preferably, the search enhancement based on the external database is as follows:

[0058] For the multiple-choice question-answering system on scientific topics, introducing an external database for enhanced retrieval can significantly improve the accuracy of the model's answering questions. In this process, the question text is first analyzed in depth. The BERT tokenizer is used to accurately segment and preprocess the question stem, and then the segmentation results are encoded using the pre-trained 'gte-base' model to generate high-quality text embedding. Specifically, assuming that the question text sequence is represented as Q, after processing by the BERT tokenizer, we convert the text sequence Q into a series of tokens Q = [q1, q2, ..., q n ], and then map each token to the corresponding word embedding E = [e1, e2, ..., e n ]. This process can be expressed as:

[0059] E(Q)=Embed(Q)

[0060] exist Figure 1 In the model, the embedding layer refers to the layer that converts the token after word segmentation into an embedding vector.

[0061] Next, we use these generated embedding vectors E(Q) to perform similarity searches in external databases to extract information related to the question as context to enhance the model's ability to answer multiple-choice questions. After processing all the data, we combine these embedding vectors into a large embedding matrix and adjust the possible imbalance of the vectors in different dimensions by implementing L2 norm normalization:

[0062]

[0063] Where Norm(E(Q)) is the normalized embedding vector, n is the number of elements in the vector, and e i is the i-th element in the vector E(Q).

[0064] Furthermore, we use the nearest neighbor search technique to accurately locate the context of the question. Let E(Q) be the embedding vector to be queried, and D represents the set of embedding vectors in the database D = {d1, d2, ..., d n}. The nearest neighbor search aims to find the embedding vector d that is closest to E(Q) in the database. nn , which is usually achieved by minimizing the Euclidean distance between the two:

[0065] dnn=argmin d∈D ∥E(Q)-d∥2

[0066] The method of enhanced retrieval based on external databases can not only improve the hit rate of answers to multiple-choice questions, but also further enrich and improve the model's understanding of the questions by introducing contextual information closely related to the questions, significantly improving the accuracy and reliability of the answers.

[0067] Preferably, the dimensionality reduction of large model data based on Flash Attention is as follows:

[0068] When building a multiple-choice question answering system based on retrieval enhancement, one of the challenges is to deal with the large amount of information in the retrieved documents due to long text search. In order to effectively integrate this information and process it together with the questions and answers of the multiple-choice questions, the present invention uses the Flash Attention mechanism of the feedforward layer to extract the key information in the sequence. Specifically, suppose we have a question text sequence Q, and the corresponding answer options A = {a1, a2, ..., a m By searching the external database, we obtain a series of documents D = {d1, d2, ..., d n Since directly processing these documents may be infeasible due to the large amount of information, the present invention first summarizes and preprocesses each document to generate a more compact representation. By concatenating the question, answer options, and preprocessed document information, we construct a comprehensive input sequence:

[0069] S = concat(Q, A, D')

[0070] Where D' represents the preprocessed document set.

[0071] In order to extract the most critical information from this comprehensive sequence, we applied the Flash Attention mechanism:

[0072] E(S)=FlashAttention(Embed(S))

[0073] Among them, the embedding layer Embed(S) converts the concatenated sequence S into an embedding vector, and Flash Attention is responsible for performing efficient self-attention operations on these embedding vectors to capture the important connections between different parts of the sequence.

[0074] Then, we use the decision layer of the Platypus2 model to score each answer option. Specifically, we set the scoring function F(A) to calculate the score of each answer option:

[0075] F(A)=FFN decision (FlashAttention(Embed(S)))

[0076] This score reflects the likelihood of each option being the correct answer. Next, the model uses these scores for sorting, selecting the top three answer options with the highest scores as the model's answer to the multiple-choice question. The sorting algorithm is as follows:

[0077] Top3(A)=sort(F(A)) top3

[0078] This approach not only improves the ability to process long text and complex document information, but also enhances the model's performance in multiple-choice question answering through precise scoring and ranking mechanisms. The Platypus2 model, by combining the FlashAttention mechanism with a refined decision layer, can accurately identify and extract the most critical information for answering questions from a large amount of information, thereby making more accurate predictions.

[0079] Preferably, more accurate results are obtained based on multi-thinking chain voting as follows:

[0080] In terms of improving the accuracy and robustness of the multiple-choice question answering system, the introduction of a multi-thinking chain voting mechanism can significantly improve the model performance. This mechanism achieves more accurate and reliable decisions by combining the results of different thinking chains (or reasoning paths). Specifically, each thinking chain can be regarded as an independent reasoning process, which is based on random sampling of data indexed from an external database and independently evaluates the question.

[0081] Set N thought chains, each thought chain C i Each will independently perform the following steps:

[0082] 1. Based on the indexed document set D = {d1, d2, ..., d n} or the preprocessed document set D', each thought chain is randomly sampled, and the subset D is selected i For analysis.

[0083] 2. Using the Flash Attention and Platypus2 model decision layer described earlier, each thought chain is for answer option A = {a1, a2, ..., a m} to rate:

[0084] F i (A) = FFN decision (FlashAttention(Embed(concat(Q,A,D i ))));

[0085] 3. Sort the answer options according to the scores obtained for each thinking chain, and select the top three answers with the highest scores as recommendations for that chain:

[0086] Top3 i (A)=sort(F i (A) top3

[0087] After all N chains of thought have been scored and ranked, a multi-chain voting process is performed. For each answer option, the number of times it was selected as the top three in all chains of thought is calculated:

[0088]

[0089] Where I is the indicator function. If the answer is option a j In Top3 i (A), then I(a j ∈Top3 i (A))=1, otherwise 0. Then, according to vote(a j ) For all answer options a j Sort and get the final recommended answer:

[0090] Top3 final (A) = sort(vote(A)) top3

[0091] Through this multi-thinking chain voting mechanism, the model of the present invention integrates viewpoints from different reasoning paths, reduces the bias that may be introduced by relying on a single data source or thinking chain, and thus achieves higher accuracy and reliability when comprehensively evaluating answer options. This method enhances the model's ability to handle complex problems and improves its performance in multiple-choice question answering scenarios. As shown in Table 1, compared with a variety of advanced models in the prior art, such as FLAN-T5-base, BERT fine-tuning, mistral+Wikipedia RAG, GPT-3, etc., the accuracy of the present invention in scientific multiple-choice question answering tasks is significantly improved. The experimental results on the Kaggle science-themed multiple-choice question dataset show that the score of this method reached 0.909, which is significantly better than other baseline models.

[0092] Table 1

[0093]

[0094]

[0095] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for automatically answering scientific multiple-choice questions based on a large language model, characterized in that: The following steps are involved: Step S1: Analyze the question text Q, perform retrieval enhancement based on the external database, perform retrieval matching from the external database, and obtain a document set D = {d1, d2, ..., d n }; Step S2: Then, each document in the document set D is summarized and preprocessed to generate a preprocessed document set D'; the question, answer options and preprocessed document information are concatenated to obtain an input sequence S = concat (Q, A, D'), where A is the answer option set, and A = {a1, a2, ..., a m }; Step S3: Convert the concatenated input sequence S into an embedding vector Embed(S), and then extract the answer information based on the Flash Attention mechanism: E(S)=FlashAttention(Embed(S)) Step S4: Based on multi-thinking chain voting, each thinking chain uses the decision-making layer FFN of the Platypus2 model decision Score each answer option F(A): F(A)=FFN decision (FlashAttention(Embed(S))) Each thinking chain ranks the score F(A) of each answer option, and multiple thinking chains finally vote to obtain b answer options as the answer to the scientific multiple-choice question.

2. The method for automatically answering scientific multiple-choice questions based on a large language model according to claim 1, characterized in that: In the step S1, the question text Q is first analyzed, and the question stem is accurately segmented and preprocessed using the BERT word segmenter. The segmentation result is then encoded using a pre-trained gte-base model to generate an embedding vector E(Q). Then, a search is performed in an external database based on the nearest neighbor search technology to locate the context of the question and obtain a document set D.

3. The method for automatically answering scientific multiple-choice questions based on a large language model according to claim 2, characterized in that: The step S1 also includes combining the embedding vectors E(Q) into an embedding matrix and performing L2 norm normalization processing: Where: Norm(E(Q)) is the normalized embedding vector; n is the number of elements in the vector, e i is the i-th element in the embedding vector E(Q); Then, based on Norm(E(Q)), the nearest neighbor search technique is used to perform a search in the external database to locate the context of the question and obtain the document set D.

4. The method for automatically answering scientific multiple-choice questions based on a large language model according to claim 2 or 3, characterized in that: The nearest neighbor search technology is to find the embedding vector closest to the embedding vector to be queried in the external database by minimizing the Euclidean distance between the embedding vector to be queried and the embedding vector in the database, thereby obtaining the context of the problem.

5. The method for automatically answering scientific multiple-choice questions based on a large language model according to claim 1, characterized in that: The step S4 comprises the following steps: Step S41: Each thought chain randomly selects a document subset D of the preprocessed document set D' i , then score the answer options F i (A): F i (A)=FFN decision (FlashAttention(Embed(concat(Q,A,D i )))); Step S42: sorting all answer options according to the scores obtained from each thinking chain, and selecting the first a answer options with the highest scores as the answer options recommended by the thinking chain; Step S43: For each answer option, calculate the number of times the selected thinking chain is recommended in all thinking chains; then, sort all the answer options according to the number of times to obtain the final recommended b answer options as the answers to the scientific multiple-choice questions.

6. The method for automatically answering scientific multiple-choice questions based on a large language model according to claim 5, characterized in that: a=b=3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for automatically answering scientific multiple-choice questions based on a large language model as described in any one of claims 1 to 6 is implemented.

8. A computer device, characterized in that: It includes one or more processors and a storage device, wherein the storage device is used to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method for automatically answering scientific multiple-choice questions based on a large language model as described in any one of claims 1 to 6.

Images