Intelligent course question-answering method and system based on retrieval enhancement generation and large model

Abstract

The invention discloses a course intelligent question-answering method and system based on retrieval enhancement generation and a large model, and the method comprises the steps: processing a to-be-answered artificial intelligence course knowledge field question, obtaining a question sequence, and obtaining a text representation vector of the question; obtaining a similarity score based on the text representation vector and the paragraph representation vector, and calculating a BM25 score between the question and the paragraph; based on the similarity score and the BM25 score, calculating to obtain a mixed score corresponding to each paragraph, and sorting the paragraphs from high to low according to the mixed scores to obtain a topN document; paragraphs in the topN document are reordered, and a topK document is obtained; and taking the topK document as the context of the question, taking the topK document and the question as input by the large language model, and outputting the answer of the question. According to the method, intelligent question answering of artificial intelligence course knowledge in the education field is realized, the learning efficiency of students is improved, and the teaching burden of teaching assistance of teachers is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of retrieval, large models and intelligent question answering, and specifically relates to a course intelligent question answering method and system based on retrieval enhancement generation and large models. Background Art

[0002] With the widespread application of large models, various industries and fields have paid attention to the application and learning of artificial intelligence technology. In particular, in the field of education and teaching, colleges and universities have promoted the popularization of artificial intelligence knowledge and talent training. Knowledge popularization has become an urgent task in the field of education and teaching in colleges and universities. An important way to popularize artificial intelligence knowledge is to promote the construction of artificial intelligence courses.

[0003] However, with the opening of artificial intelligence courses, the number of students' questions and answers will inevitably explode during the course learning process. In order to deal with student questions and answers, the traditional method based on teachers or teaching assistants answering students' questions can no longer meet the question and answer requests of many students, and it will also take up a lot of time for teachers or teaching assistants. In addition, due to the shortage of teachers or teaching assistants, the need to deal with a large number of students' questions and answers will also cause students' doubts to not be answered in time, which will further affect students' interest in learning artificial intelligence and learning efficiency. The intelligent question and answer system built based on artificial intelligence natural language processing technology can effectively alleviate this problem. The existing typical intelligent question and answer method based on BERT uses an attention mechanism to implement extractive question and answer. From a technical point of view, the BERT method uses a cross-attention mechanism to interact with the representation vectors of questions and paragraphs, and then obtains the start and end tags of the answer in the paragraph through a classification layer. However, this method cannot accurately answer all questions, especially some questions in specific professional fields cannot directly extract answers from a given text paragraph, and must be given certain reasoning and summary requirements.

[0004] With the advent of generative large language models, relying on the large language model's powerful understanding of human language, it can have good performance in common sense, reasoning, semantics, knowledge and other aspects, and can further align the large model with new data sets through large model fine-tuning technologies such as Lora, which can improve accuracy. However, there are many inaccurate answers to proprietary knowledge questions and answers by large language models, and the answers given may mislead students' new knowledge learning.

[0005] In order to address the shortcomings of large language model question answering, retrieval enhancement generation technology will introduce additional professional knowledge bases into large models, which can improve the performance of large models in specific question answering tasks, and the performance of large model question answering is greatly dependent on retrieval performance. Existing retrieval technologies include sparse retrieval and dense retrieval. Sparse retrieval includes BM25, which performs retrieval based on word frequency statistics and term matching, and cannot effectively capture the semantic features of words. Dense retrieval uses deep learning methods to obtain vector representations of texts, and uses vector representations to calculate the similarity between texts for retrieval. However, this method can only capture limited semantic features in professional data sets, resulting in limited question answering effects. In order to make up for the problem of limited semantic feature capture of dense retrieval on professional data sets, it is necessary to propose a hybrid retrieval algorithm that combines sparse retrieval and dense retrieval, and add re-ranking technology to improve retrieval performance.

[0006] Therefore, there is an urgent need for an intelligent question-answering system for courses that integrates retrieval enhancement generation and large models to help teachers or teaching assistants answer students' questions, thereby improving students' learning efficiency and helping teachers reduce their teaching burden. Summary of the invention

[0007] In order to solve the problems in the prior art, the present invention proposes a course intelligent question answering method and system based on retrieval enhanced generation and large model.

[0008] The technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention discloses a course intelligent question answering method based on retrieval enhancement generation and large model, comprising the following steps:

[0010] Obtain and train a twin BERT network for obtaining text representation vectors, build and train a cross encoder for reordering, and obtain and train a large language model for obtaining answers to questions;

[0011] Perform word segmentation, filling and id mapping on the questions in the knowledge domain of the artificial intelligence course to be answered to obtain a question sequence, and then use the trained twin Bert network to encode the question sequence to obtain a text representation vector of the question; then calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database to obtain a similarity score, and the vector database is obtained based on the trained twin Bert network;

[0012] Calculate the correlation score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; the paragraph database is obtained based on the trained twin Bert network;

[0013] The mixed score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the mixed score to obtain the topN documents. The paragraphs in the topN documents are reordered using the trained cross encoder to obtain the final topK documents.

[0014] The obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and questions as input, and the trained large language model outputs the answer to the question.

[0015] In a second aspect, the present invention discloses a course intelligent question-answering system based on retrieval enhancement generation and large models, comprising:

[0016] The question preprocessing module is used to obtain and train the twin Bert network for obtaining text representation vectors, perform word segmentation, filling and ID mapping on the questions in the knowledge field of the artificial intelligence course to be answered, obtain the question sequence, and then use the trained twin Bert network to encode the question sequence to obtain the text representation vector of the question;

[0017] A document retrieval module is used to build and train a cross encoder for re-ranking, calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database, and obtain a similarity score. The vector database is obtained based on the trained twin Bert network.

[0018] Calculate the correlation score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; the paragraph database is obtained based on the trained twin Bert network;

[0019] The mixed score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the mixed score to obtain the topN documents. The paragraphs in the topN documents are reordered using the trained cross encoder to obtain the final topK documents.

[0020] The large model inference module is used to obtain and train a large language model for obtaining answers to questions; the obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and the question as input, and the trained large language model outputs the answer to the question.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention first constructs a question-and-answer dataset for artificial intelligence course domain knowledge from different data sources, realizing the creation of a Chinese artificial intelligence dataset from scratch. Secondly, the present invention constructs a hybrid retrieval algorithm and uses a cross-coding reordering method to improve retrieval performance. Finally, the present invention uses a retrieval-enhanced generated question-and-answer method, and uses the retrieved results as the context of the question, which are input into the LLM fine-tuned by lora, and finally outputs the answer to the question, thereby improving the question-and-answer performance of the large language model and solving the problem of large language model hallucination. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the architecture of the course intelligent question-answering system according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of the workflow of the intelligent course question-answering system according to an embodiment of the present invention;

[0025] Figure 3 This is a workflow diagram of the document retrieval module;

[0026] Figure 4 Schematic diagram of the reordering algorithm;

[0027] Figure 5 Schematic diagram of fine-tuning the large model for LoRa. DETAILED DESCRIPTION

[0028] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0029] In view of the shortcomings of intelligent question answering with large language models, the present invention provides an intelligent question answering method and system for answering questions in artificial intelligence course teaching based on retrieval enhanced generation and large models. By utilizing retrieval enhanced generation and large language model technology, intelligent course question answering in the field of artificial intelligence is realized, which is conducive to improving students' learning efficiency and reducing teachers' teaching burden.

[0030] like Figure 1 As shown, the course intelligent question answering system of an embodiment of the present invention includes a question preprocessing module, a document retrieval module and a large model reasoning module.

[0031] The question preprocessing module is used to obtain and train a twin Bert network for obtaining a text representation vector, perform word segmentation, filling and id mapping on the artificial intelligence course knowledge field questions to be answered, obtain a question sequence, and then use the trained twin Bert network to encode the question sequence to obtain a text representation vector of the question;

[0032] The document retrieval module is used to construct and train a cross encoder for re-ranking, calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database, and obtain a similarity score, wherein the vector database is obtained based on the trained twin Bert network;

[0033] Calculate the correlation score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; the paragraph database is obtained based on the trained twin Bert network;

[0034] The hybrid score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the hybrid score to obtain the topN documents. The paragraphs in the topN documents are re-sorted using the trained cross encoder to obtain the final topK documents. The workflow of the document retrieval module is as follows: Figure 3 shown.

[0035] The large model inference module is used to obtain a large language model for obtaining answers to questions and train the large language model; the obtained topK documents are used as the context of the question, the trained large language model takes the topK documents and the question as input, and the trained large language model outputs the answer to the question.

[0036] The workflow diagram of the course intelligent question-answering system is as follows: Figure 2 As shown, the details are as follows: obtain and train a twin Bert network for obtaining a text representation vector, build and train a cross encoder for reordering, obtain a large language model for obtaining the answer to the question and train the large language model;

[0037] Perform word segmentation, filling and id mapping on the questions in the knowledge domain of the artificial intelligence course to be answered to obtain a question sequence, and then use the trained twin Bert network to encode the question sequence to obtain a text representation vector of the question; then calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database to obtain a similarity score, and the vector database is obtained based on the trained twin Bert network;

[0038] Calculate the correlation score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; the paragraph database is obtained based on the trained twin Bert network;

[0039] The mixed score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the mixed score to obtain the topN documents. The paragraphs in the topN documents are reordered using the trained cross encoder to obtain the final topK documents.

[0040] The obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and questions as input, and the trained large language model outputs the answer to the question. This process is implemented step by step by the question preprocessing module, document retrieval module and large model inference module.

[0041] The course intelligent question-answering method of the present invention uses rich network resources and books to construct a high-quality question-answering data set in the field of artificial intelligence; and retrieves the best matching paragraph according to the DPR-based intensive retrieval algorithm and re-ranking optimization; finally, the retrieved best paragraph is used as the context of the question and the question together with the question as the input of the large language model. After fine-tuning the large language model, the answer to the question can be obtained, thereby realizing effective intelligent course question-answering.

[0042] The training of the twin BERT network, cross encoder, and large language model mainly includes the following steps:

[0043] (1) First, we extract paragraphs of relevant knowledge points from forums, web pages and books such as Wikipedia and popular science books on artificial intelligence ("TensorFlow Deep Learning", "Advanced Deep Learning: Natural Language Processing", and "Introduction to Deep Learning: Theory and Practice Based on Python"), and summarize at least one question-answer pair for each paragraph, thereby constructing a question-answer dataset for the knowledge field of artificial intelligence courses. The question-answer dataset includes multiple question-answer pairs, each consisting of a paragraph, a question and an answer. In the end, there are a total of 1,833 question-answer pairs (paragraph-question-answer pairs), and some examples of paragraph-question-answer pairs are shown in Table 1. The training set and the test set are divided according to a ratio of 8:2. The final training set size is 1,466 question-answer pairs, and the test set size is 367 question-answer pairs.

[0044] Table 1 Examples of AI question answering datasets

[0045]

[0046]

[0047] (2) A dense retrieval algorithm based on DPR (Dense Passage Retrieval) is used to retrieve paragraphs related to the question as context. Specifically, the twin Bert network is first used to obtain the paragraph text representation vector of the paragraph and the text representation vector of the question, and the dot product similarity between the paragraph text representation vector and the text representation vector of the question is calculated to obtain the first similarity score; then the BM25 (Best Matching 25) score is calculated based on the question and the paragraph to obtain the first BM25 score; the BM25 algorithm is then used as a difficult negative sample sampling strategy to obtain the corresponding difficult negative sample; finally, the first similarity score and the first BM25 score are combined to calculate the first mixed score, and the input paragraphs are sorted from high to low according to the first mixed score to obtain the first topN documents. The topN paragraphs retrieved in the initial retrieval are then reordered using the cross encoder reordering algorithm to obtain the first topK documents. Finally, the InfoNCE loss function is used to optimize the twin Bert network to obtain a trained twin Bert network; the cross encoder is optimized based on the cross entropy loss function to obtain a trained cross encoder. The process is as follows:

[0048] (2-1) Construct DPR dense retrieval algorithm.

[0049] In a specific embodiment of the present invention, the question-answering dataset is used to train the twin Bert network; acquiring and training the twin Bert network for obtaining a text representation vector comprises the following steps:

[0050] The twin Bert network refers to the use of a shared Bert network to encode paragraphs and questions. The Bert contains an embedding layer and an encoding layer. The embedding layer includes a word vector layer, a paragraph vector layer, and a position vector layer; and the encoding layer is composed of 12 layers of bidirectional transformers. To input paragraphs and questions into Bert, it is necessary to first use the Bert word segmenter to segment, fill, and id map the paragraph and question texts, and finally obtain a question sequence and a paragraph sequence, an input sequence composed of each word id.

[0051] The question-answer pairs in the question-answer dataset are divided into k batches, and the questions and paragraphs of the question-answer pairs in a batch are segmented, filled and id mapped to obtain the corresponding second question sequence and second paragraph sequence. The twin Bert network takes the second question sequence and the second paragraph sequence as input, and the twin Bert network outputs the paragraph text representation vector and the text representation vector of the question; the InfoNCE loss is calculated and the parameters of the twin Bert network are updated based on the InfoNCE loss; when updating the parameters of the twin Bert network, the question-answer pairs in the first batch are first used to update the parameters of the twin Bert network to obtain the first twin Bert network, and then the question-answer pairs in the second batch are used to update the parameters of the first twin Bert network to obtain the second twin Bert network; then the question-answer pairs in the i-th batch are used to update the parameters of the i-1-th twin Bert network to obtain the i-th twin Bert network; finally, the question-answer pairs in the k-th batch are used to update the parameters of the k-1-th twin Bert network to obtain a trained twin Bert network.

[0052] After the second question sequence and the second paragraph sequence are input into the twin Bert network (the second question sequence and the second paragraph sequence are combined as the input sequence), the hidden state of the text sequence is obtained. The hidden state of the text sequence refers to the vector obtained after the input sequence passes through the Bert embedding layer and the encoding layer. Then the hidden state vector at [CLS] is used as the representation vector of the entire input, and finally a vector of fixed length is obtained as the paragraph text representation vector of the paragraph and the text representation vector of the question.

[0053] Based on the trained twin BERT network, the paragraph representation vector of each paragraph in the question and answer dataset is obtained. The paragraph representation vectors of all batches constitute a vector database; all paragraphs in the question and answer dataset constitute a paragraph database.

[0054] Paragraph D in the question-answering dataset b Indicated as D b =(d1,d2,…,d n ), matching question Q b Represented as Q b =(q1,q2,…,q m ). Use the twin network Bert to extract the corresponding representations for paragraphs and questions. The expression is as follows:

[0055]

[0056] Among them, d represents the dimension of the representation vector, For paragraph D b The paragraph text representation vector, For question Q b The text representation vector of .

[0057] When training the twin BERT network, the BM25 algorithm is used to calculate the relevance scores of the questions and paragraphs in the question-answering dataset to obtain the first BM25 score. The first BM25 score is

[0058]

[0059] Among them, score(D b ,Q c ) refers to paragraph D b and question Q c The first BM25 score, D b is the bth paragraph among all the paragraphs in the question-answering dataset, 1≤b≤M, M is the total number of question-answer pairs in the question-answering dataset, Q c is the cth question among all questions in the question-answering dataset, 1≤b≤M;

[0060] IDF(q d ) is the question Q c The word q in d The inverse document frequency, q d For question Q c The dth word in

[0061]

[0062] Where B is the total number of all paragraphs in the question-answering dataset, B = M; n(q d ) are all paragraphs containing word q d The number of paragraphs; n Qc For question Q c Total number of words;

[0063] TF(q d ,D b ) is the question Q c The word q in d In paragraph D b The frequency of occurrence in

[0064]

[0065] Among them, f(q d ,D b ) is the word q d In paragraph D b The number of times it appears in |D b | for paragraph D b avgdl′ is the average number of words in all paragraphs; b′ is a preset parameter; k1′ is a preset parameter;

[0066] TF(q d,Q c ) is the question Q c The word q in d In question Q c The frequency of occurrence in

[0067]

[0068] Among them, f(q d ,Q c ) is the word q d In question Q c The number of times it appears in |Q c |For question Q c The total number of words in the paragraph; avgdl′ is the average number of words in all paragraphs; b″ is a preset parameter; k1″ is a preset parameter;

[0069] The paragraph corresponding to the first BM25 score with the largest value and not the direct source of the answer to the question is taken as the question Q c In short, difficult negative sample sampling means selecting one paragraph that is most similar to any question in the question-answering dataset but is not the direct source of the answer to the question from all the paragraphs in the question-answering dataset, and using this paragraph as the difficult negative sample for the corresponding question.

[0070] Since a batch of paragraphs contains a difficult negative sample obtained by BM25 sampling in addition to the paragraphs corresponding to each question, the sampled difficult negative sample is the paragraph text with the greatest similarity to the question but not corresponding to the question, expressed as Finally, the InfoNCE loss function is used to optimize the twin Bert network parameters, and its expression is as follows:

[0071]

[0072] Among them, InfoNCE i is the InfoNCE loss of the i-th batch of question-answer pairs; I is the paragraph text representation vector of all paragraphs in the i-th batch and the paragraph text representation vector of the difficult negative samples; d q is the qth paragraph text representation vector in I; τ is the temperature coefficient, which is 1.0 here; sim(x,y)=x·y; is the text representation vector of the lth question in the i-th batch of question-answer pairs, 1≤l≤M i , M i is the total number of question-answer pairs in the i-th batch; is the paragraph text representation vector of the paragraph from which the answer to the lth question in the i-th batch of question-answer pairs directly comes; g is the total number of question-answer pairs in the i-th batch.

[0073] The optimization process uses the stochastic gradient descent algorithm with a learning rate of 1e-4, a momentum parameter of 0.9, a weight decay of 5e-5, 10 training rounds, and a batch size of 16.

[0074] (2-2) Mixed score retrieval of topN paragraphs.

[0075] Based on the trained twin BERT network, we obtain the paragraph representation vector of each paragraph in the question-answering dataset and the text representation vector of each question, and calculate the first similarity score between each question and all paragraphs. The first similarity score is:

[0076]

[0077] in, for The first similarity score of is the paragraph representation vector of the mth paragraph in the question-answering dataset, 1≤m≤M; is the question representation vector of the lth question in the question-answering dataset, 1≤l≤M

[0078] The first mixed score of each question and paragraph is obtained according to the first BM25 score and the first similarity score. The first mixed score is: Among them, hybrid (D m ,Q l ) is the question Q l With paragraph D m The first mixed score of; x1 is a hyperparameter, here x is taken as 20. According to the first mixed score, the paragraphs in the ith batch are sorted from high to low, and the first topN documents of the initial retrieval are obtained. Here, the first topN documents include 20 paragraphs.

[0079] All paragraphs are sorted from high to low according to all the first mixed scores of each question to obtain M first topN documents, that is, one question has one first topN document, and M questions have M first topN documents.

[0080] (2-3) Use the cross encoder to re-rank the initial retrieval results, such as Figure 4The cross encoder includes a Bert network and a linear layer, performs word segmentation, padding and id mapping on the questions corresponding to each first topN document to obtain a first question sequence, and then performs word segmentation, padding and id mapping on each paragraph in the first topN document to obtain a first paragraph sequence of each paragraph, and the first question sequence and each first paragraph sequence are combined into a first input sequence, and the first input sequence is

[0081] x=[CLS]q1,q2,…,q n [SEP]p1,p2,…,p m

[0082] Where x is the first input sequence; q i is the i-th word in the first question sequence; p i is the i-th word in the first paragraph sequence;

[0083] The training cross encoder uses the paragraphs in the first topN documents obtained by the initial retrieval. Specifically, each question in the training set contains a best matching document and the topN-1 documents obtained by the initial retrieval of DPR (excluding the best document), and the category of the best matching document is set to 1, and the category of the remaining documents is set to 0 (the cross encoder is trained based on M first topN documents). After the input sequence is input into the cross encoder, it is reduced to 2 dimensions through a linear layer, and finally the task is converted into a binary classification task.

[0084] Specifically: the Bert network takes the first input sequence as input and outputs the first vector to the linear layer. The linear layer performs dimensionality reduction processing on the first vector to obtain a two-dimensional vector. The two-dimensional vector is classified into two categories using the softmax function to obtain the classification result and the probability that the paragraph is the direct source of the answer to the question. The paragraphs are reordered based on the probability to obtain the first topK documents, where the first topK documents include 1 paragraph. The training process uses the cross entropy loss function and is optimized using the stochastic gradient descent algorithm. The learning rate is 1e-4, the momentum parameter is 0.9, the weight decay is 5e-5, and the number of training rounds is 20.

[0085] (3) Use the chatglm-6b large language model to answer the questions and return the answers. This process is completed by the large model inference module, which includes:

[0086] (3-1) Take the first topK document as the context of the corresponding question, construct a question-context-answer pair, and use the question-context-answer pair as the input of the large language model. The large language model outputs the answer to the question generated by the large language model. Use the lora model to supervise and fine-tune the large language model, and finally obtain the trained large language model.

[0087] Chatglm-6b is based on the glm architecture, which is an autoregressive and improved transformer structure. The improved transformer includes changing the order of layer normalization and residual connection, using a single linear layer to predict output tokens, and using GeLUs to replace the ReLU activation function. At the same time, the architecture uses autoregressive fill-in-the-blank pre-training to improve the model's ability to understand natural language; and multi-task pre-training is performed, pre-training at the sentence level and document level to meet the needs of generating long texts.

[0088] The large language model fine-tuning adopts lora fine-tuning, such as Figure 5 As shown. The lora fine-tuning refers to externalizing a module in the large language model parameters. During fine-tuning, the pre-trained parameters of the large language model are fixed, and only the external lora module is fine-tuned, thereby saving time and memory overhead for fine-tuning. The idea of ​​lora fine-tuning is that the large language model has a lower intrinsic dimension. Even if the weights of the large language model are projected into a smaller subspace, the model can still learn effectively. Lora fine-tuning uses low-rank decomposition. For the pre-trained weights W0∈R d×k , which can be decomposed into the product of two matrices with smaller dimensions, and its expression is as follows:

[0089] W0+δW=W0+BA

[0090] Where B is the decomposition matrix, B∈R d×r ; A is the decomposition matrix, A∈R r×o , r is the hidden state dimension, d is the input dimension, o is the output dimension, W is the pre-training parameter, W∈R d×o ; δ is a small coefficient; where r<<min(d,o). The forward propagation expression after Lora fine-tuning is as follows:

[0091] h=W0u+δWu

[0092] Among them, h is the output feature and u is the input feature.

[0093] The question-paragraph pair is segmented to obtain the corresponding input sequence y, and a special character [M] is added after the input sequence to indicate the answer to be predicted. The expression is as follows:

[0094] y=(q1,q2,…,q n ; d1, d2, …, d m ; [M])

[0095] The large language model predicts the next word based on the input sequence and the words that have been generated. Its expression is as follows:

[0096]

[0097] Among them, E z is the expected probability; p θ For the zth i Predicted word probabilities; For the zth i word units, Z is the answer label, Represented as the generated question tag. Ultimately, the predicted answer probability expectation needs to be maximized. The large language model fine-tuning uses Lora fine-tuning, with a fine-tuning batch size of 4, a training round number of 5, a learning rate of 5e-5, and a Lora rank of 8.

[0098] In a specific embodiment of the present invention, the process of obtaining answers to questions in the field of artificial intelligence course knowledge to be answered includes the following steps: performing word segmentation, filling and ID mapping processing on the questions in the field of artificial intelligence course knowledge to be answered to obtain a question sequence, and then using the trained twin Bert network to encode the question sequence to obtain a text representation vector of the question; then calculating the dot product similarity between the text representation vector and each paragraph representation vector in the vector database to obtain a similarity score, wherein the vector database is constructed when training the twin Bert network;

[0099] Calculate the relevance score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; the paragraph database is constructed when training the twin Bert network;

[0100] The mixed score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the mixed score to obtain the topN documents. The paragraphs in the topN documents are reordered using the trained cross encoder to obtain the final topK documents.

[0101] The obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and questions as input, and the trained large language model outputs the answer to the question.

[0102] Furthermore, the BM25 score is

[0103]

[0104] Among them, score(D a ,Q) refers to paragraph D a and the BM25 score of the AI ​​course knowledge domain question Q to be answered, D a The paragraph database D total In the ath paragraph, Dtotal =(D1,D2,…,D a ,…,D N ); IDF(q i ) is the word q in question Q i The inverse document frequency, q i is the i-th word in the question,

[0105]

[0106] Where M is the paragraph database D total The total number of paragraphs; n(q i ) contains the word q i The number of paragraphs; n is the total number of words in question Q;

[0107] TF(q i ,D a ) is the word q in question Q i In paragraph D a The frequency of occurrence in

[0108]

[0109] Among them, f(q i ,D a ) is the word q i In paragraph D a The number of times it appears in |D a | for paragraph D a The total number of words in the paragraph database D total The average number of words in a paragraph; b is a preset parameter; k1 is a preset parameter;

[0110] TF(q i ,Q) is the word q in question Q i The frequency of occurrence in question Q,

[0111]

[0112] Among them, f(q i ,Q) is word q i The number of times it appears in question Q; |Q| is the total number of questions; b1 is a preset parameter; k 11 are the preset parameters;

[0113] The similarity score is:

[0114]

[0115] in, V Q With V Da Similarity score of VQ is the text representation vector of question Q; V Da For paragraph D a The paragraph representation vector of

[0116] The mixed score is:

[0117]

[0118] Among them, hybrid (D a ) is paragraph D a The mixed score of ; x is a hyperparameter.

[0119] Furthermore, the method of using the trained cross encoder to reorder the paragraphs in the topN documents to obtain the final topK documents includes:

[0120] Each paragraph in the topN documents is processed with word segmentation, filling and id mapping in turn to obtain a paragraph sequence of each paragraph, and the questions in the knowledge field of the artificial intelligence course to be answered are processed with word segmentation, filling and id mapping to obtain a question sequence, and each paragraph sequence and the question sequence are combined into an input sequence in turn. The trained cross encoder takes the input sequence as input, and the trained cross encoder sorts the paragraphs in the topN documents from high to low according to the probability that the paragraphs in the input sequence are the direct source of the answer to the question, and obtains the final topK documents; among them, the topN documents include 20 paragraphs and the topK documents include 1 paragraph.

[0121] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A course intelligent question answering method based on retrieval enhancement generation and large model, characterized in that: The following steps are involved: Obtain and train a twin BERT network for obtaining text representation vectors, build and train a cross encoder for reordering, and obtain and train a large language model for obtaining answers to questions; Perform word segmentation, filling and id mapping on the questions in the knowledge domain of the artificial intelligence course to be answered to obtain a question sequence, and then use the trained twin Bert network to encode the question sequence to obtain a text representation vector of the question; then calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database to obtain a similarity score, and the vector database is obtained based on the trained twin Bert network; Calculate the relevance score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; The paragraph database is obtained based on the trained twin Bert network; Use the similarity score and BM25 score to calculate the mixed score corresponding to each paragraph, sort the paragraphs from high to low according to the mixed score, and get the topN documents; Use the trained cross encoder to reorder the paragraphs in the topN documents to obtain the final topK documents; The obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and questions as input, and the trained large language model outputs the answer to the question.

2. The method according to claim 1, characterized in that The training is used to obtain a twin BERT network of a text representation vector, including: Construct a question-answering dataset for the knowledge field of artificial intelligence courses, wherein the question-answering dataset includes multiple question-answering pairs, each of which consists of a paragraph, a question, and an answer; divide the question-answering pairs in the question-answering dataset into k batches, perform word segmentation, padding, and id mapping on the questions and paragraphs of the question-answering pairs in each batch, and obtain a corresponding second question sequence and a second paragraph sequence; the twin Bert network takes the second question sequence and the second paragraph sequence as input, and the twin Bert network outputs a paragraph text representation vector and a text representation vector of the question; calculate the InfoNCE loss and update the parameters of the twin Bert network based on the InfoNCE loss; Based on the trained twin BERT network, the paragraph representation vector of each paragraph in the question and answer dataset is obtained, and all paragraph representation vectors constitute a vector database; all paragraphs in the question and answer dataset constitute a paragraph database.

3. The method according to claim 2, characterized in that The construction of a question-answering dataset for the knowledge domain of artificial intelligence courses includes: Using Wikipedia, AI popular science books, AI popular science forums, and AI popular science web pages as data sources, relevant paragraphs of AI knowledge are extracted from the data sources, and for each paragraph, at least one AI-related question that students will ask and the corresponding reference answer to the question are summarized to form a paragraph-question-answer question and answer dataset.

4. The method according to claim 2, characterized in that: When training the twin BERT network, the BM25 algorithm is used to calculate the relevance scores of the questions and paragraphs in the question-answering dataset to obtain the first BM25 score. The first BM25 score is Among them, score(D D , Q c ) refers to paragraph D b and question Q c The first BM25 score, D b is the bth paragraph among all the paragraphs in the question-answering dataset, 1≤b≤M, M is the total number of question-answer pairs in the question-answering dataset, Q c is the cth question among all questions in the question-answering dataset, 1≤b≤M; IDF(q d ) is the question Q c The word q in d The inverse document frequency, q d For question Q c The dth word in Where B is the total number of all paragraphs in the question-answering dataset, B = M; n(q d ) are all paragraphs containing word q d The number of paragraphs; For question Q c Total number of words; TF(q d , D b ) is the question Q c The word q in d In paragraph D b The frequency of occurrence in Among them, f(q d , D b ) is the word q a In paragraph D b The number of times it appears in |D b | for paragraph D b avgdl′ is the average number of words in all paragraphs; b′ is a preset parameter; k1′ is a preset parameter; TF(q d , Q c ) is the question Q c The word q in d In question Q c The frequency of occurrence in Among them, F(q d , Q c ) is the word q d In question Q c The number of times it appears in |Q c |For question Q c The total number of words in the paragraph; avgdl′ is the average number of words in all paragraphs; b″ is a preset parameter; k1″ is a preset parameter; The paragraph corresponding to the first BM25 score with the largest value is taken as a difficult negative sample, where the paragraph is not the direct source of the answer to the question; The InfoNCE loss is: Among them, InfoNCE i is the InfoNCE loss of the i-th batch of question-answer pairs; I is the paragraph text representation vector of all paragraphs in the i-th batch and the paragraph text representation vector of the difficult negative samples; d q is the qth paragraph text representation vector in I; τ is the temperature coefficient; sim(x, y) = x·y; is the text representation vector of the lth question in the i-th batch of question-answer pairs, 1≤l≤M i , M i is the total number of question-answer pairs in the i-th batch; is the paragraph text representation vector of the paragraph from which the answer to the lth question in the i-th batch of question-answer pairs directly comes; g is the total number of question-answer pairs in the i-th batch.

5. The method according to claim 4, characterized in that Based on the trained twin BERT network, we obtain the paragraph representation vector of each paragraph in the question-answering dataset and the text representation vector of each question, and calculate the first similarity score between each question and all paragraphs. The first similarity score is: in, for and The first similarity score of is the paragraph representation vector of the mth paragraph in the question-answering dataset, 1≤m≤M; is the question representation vector of the lth question in the question-answering dataset, 1≤l≤M The first mixed score of each question and paragraph is obtained according to the first BM25 score and the first similarity score. The first mixed score is: Among them, hybrid (D m , Q l ) is the question Q l With paragraph D m The first mixed score of; x1 is a hyperparameter; All paragraphs are sorted from high to low according to all the first mixed scores of each question to obtain M first topN documents.

6. The method according to claim 5, characterized in that The construction and training of the cross encoder includes: The cross encoder includes a Bert network and a linear layer. The question corresponding to each first topN document is segmented, filled and id mapped to obtain a first question sequence. Then, each paragraph in the first topN document is segmented, filled and id mapped to obtain a first paragraph sequence of each paragraph. The first question sequence and each first paragraph sequence are combined into a first input sequence. The first input sequence is x=[CLS]q1,q2,...,q n [SEP]p1,p2,...,p m Where x is the first input sequence; q i is the i-th word in the first question sequence; p i is the i-th word in the first paragraph sequence; The Bert network takes the first input sequence as input and outputs the first vector to the linear layer. The linear layer performs dimensionality reduction processing on the first vector to obtain a two-dimensional vector. The two-dimensional vector is classified into two categories using the softmax function to obtain the classification result and the probability that the paragraph is the direct source of the answer to the question. The paragraphs are reordered based on the probability to obtain the first topK documents. The cross-encoder is optimized and trained based on the cross-entropy loss function and the stochastic gradient descent algorithm to obtain a trained cross encoder.

7. The method according to claim 6, characterized in that Training the large language model includes: The first topK document is used as the context of the corresponding question, and a question-context-answer pair is constructed. The question-context-answer pair is used as the input of the large language model, and the large language model outputs the generated answer. The lora model is used to supervise and fine-tune the large language model, and finally the trained large language model is obtained; wherein the large language model is a chatglm-6b model.

8. The method according to claim 1, characterized in that The BM25 score is Among them, score(D a , Q) refers to paragraph D a and the BM25 score of the AI ​​course knowledge domain question Q to be answered, D a The paragraph database D total In the ath paragraph, D total =(D1, D2, ..., D a , ..., D N ); IDF(q i ) is the word q in question Q i The inverse document frequency, q i is the i-th word in the question, Where N is the paragraph database D total The total number of paragraphs; n(q i ) contains the word q i The number of paragraphs; n is the total number of words in question Q; TF(q i , D a ) is the word q in question Q i In paragraph D a The frequency of occurrence in Among them, f(q i , D a ) is the word q i In paragraph D a The number of times it appears in |D a | for paragraph D a The total number of words in the paragraph database D total The average number of words in a paragraph; b is a preset parameter; k1 is a preset parameter; TF(q i , Q) is the word q in question Q i The frequency of occurrence in question Q, Among them, f(q i , Q) is word q i The number of times it appears in question Q; |Q| is the total number of questions; bb1 is a preset parameter; k 11 are the preset parameters; The similarity score is: in, V Q and Similarity score of V Q is the text representation vector of question Q; For paragraph D a The paragraph representation vector of The mixed score is: Among them, hybrid (D a ) is paragraph D a The mixed score of ; x is a hyperparameter.

9. The method according to claim 1, characterized in that: The method uses the trained cross encoder to reorder the paragraphs in the topN documents to obtain the final topK documents; including: Each paragraph in the topN documents is processed with word segmentation, filling and id mapping in turn to obtain a paragraph sequence of each paragraph, and the questions in the knowledge field of the artificial intelligence course to be answered are processed with word segmentation, filling and id mapping to obtain a question sequence, and each paragraph sequence and the question sequence are combined into an input sequence in turn. The trained cross encoder takes the input sequence as input, and the trained cross encoder sorts the paragraphs in the topN documents from high to low according to the probability that the paragraphs in the input sequence are the direct source of the answer to the question, and obtains the final topK documents; among them, the topN documents include 20 paragraphs and the topK documents include 1 paragraph.

10. A system for implementing the course intelligent question-answering method according to claim 1, characterized in that: include: The question preprocessing module is used to obtain and train the twin Bert network for obtaining text representation vectors, perform word segmentation, filling and ID mapping on the questions in the knowledge field of the artificial intelligence course to be answered, obtain the question sequence, and then use the trained twin Bert network to encode the question sequence to obtain the text representation vector of the question; A document retrieval module is used to build and train a cross encoder for re-ranking, calculate the dot product similarity between the text representation vector and each paragraph representation vector in the vector database, and obtain a similarity score. The vector database is obtained based on the trained twin Bert network. Calculate the relevance score between the question and each paragraph in the paragraph database using the BM25 algorithm to obtain a BM25 score; The paragraph database is obtained based on the trained twin Bert network; The mixed score corresponding to each paragraph is calculated using the similarity score and the BM25 score. The paragraphs are sorted from high to low according to the mixed score to obtain the topN documents. The paragraphs in the topN documents are reordered using the trained cross encoder to obtain the final topK documents. The large model inference module is used to obtain and train a large language model for obtaining answers to questions; the obtained topK documents are used as the context of the question, and the trained large language model takes the topK documents and the question as input, and the trained large language model outputs the answer to the question.

Images