Course intelligent question answering method and system based on retrieval enhancement generation and large model

Abstract

The application discloses a kind of based on retrieval enhancement generation and big model's course intelligent question and answer method and system, method includes: the artificial intelligence course knowledge field question of being processed to the problem to be answered, obtain question sequence, again obtain the text representation vector of question;Similarity score is obtained based on text representation vector and paragraph representation vector, the BM25 score between question and paragraph is calculated;Based on similarity score and BM25 score, the mixed score corresponding to each paragraph is calculated, paragraph is sorted according to mixed score from high to low, and topN document is obtained;Paragraph in topN document is re-sequenced, and topK document is obtained;TopK document is regarded as the context of question, and large language model is inputted with topK document and question, and the answer of question is outputted.The method realizes the intelligent question and answer of artificial intelligence course knowledge in education field, and is favorable for improving the learning efficiency of student and reducing the teaching burden of teacher assistant.

Description

Technical Field

[0001] This invention belongs to the field of retrieval, large model and intelligent question answering technology, specifically involving a course intelligent question answering method and system based on retrieval enhancement generation and large model. Background Technology

[0002] With the widespread application of large-scale models, various industries are paying increasing attention to the application and learning of artificial intelligence technology. In particular, in the field of education, universities are actively promoting the popularization of AI knowledge and the cultivation of talent. Knowledge dissemination has become an urgent task in higher education. A crucial way to popularize AI knowledge is to promote the development of AI courses.

[0003] However, with the introduction of artificial intelligence courses, an explosive increase in student questions and clarifications is inevitable during the learning process. Traditional methods relying on teachers or teaching assistants to answer student questions are no longer sufficient to meet the numerous requests, and also consume a significant amount of teachers' or teaching assistants' time. Furthermore, the shortage of teachers or teaching assistants to handle the large number of student questions can lead to unanswered questions, further impacting students' interest and learning efficiency in artificial intelligence. Intelligent question-answering systems built on artificial intelligence natural language processing technology can effectively alleviate this problem. Existing typical intelligent question-answering methods based on BERT employ an attention-based mechanism for extractive question answering. Technically, the BERT method uses a cross-attention mechanism to interact with the representation vectors of the question and the paragraph, and then uses a classification layer to obtain the start and end labels of the answer within the paragraph. However, this method cannot accurately answer all questions, especially those in specific professional fields where answers cannot be directly extracted from a given text paragraph and require a certain degree of reasoning and summarization.

[0004] With the advent of generative large language models, leveraging their powerful understanding of human language, they can demonstrate excellent performance in various aspects such as common sense, reasoning, semantics, and knowledge. Furthermore, they can be further fine-tuned using techniques like LoRa to align with new datasets, thus improving accuracy. However, large language models suffer from inaccurate responses to specialized knowledge-based questions, potentially misleading students' learning of new knowledge.

[0005] To address the shortcomings of large language model question answering, retrieval augmentation techniques introduce additional professional knowledge bases into large models, improving their performance on specific question answering tasks. The performance of large model question answering, in turn, heavily relies on retrieval performance. Existing retrieval techniques include sparse retrieval and dense retrieval. Sparse retrieval, such as BM25, relies on word frequency statistics and term matching, failing to effectively capture semantic features. Dense retrieval uses deep learning to obtain vector representations of text and calculates similarity between texts for retrieval. However, this method captures limited semantic features on specialized datasets, resulting in limited question answering effectiveness. To compensate for the limited semantic feature capture of dense retrieval on specialized datasets, it is necessary to propose a hybrid retrieval algorithm combining sparse and dense retrieval, along with re-ranking techniques, to improve retrieval performance.

[0006] Therefore, there is an urgent need for an intelligent question-and-answer system for courses that integrates retrieval enhancement generation and large models to help teachers or teaching assistants answer students' questions, thereby improving students' learning efficiency and helping teachers reduce their teaching burden. Summary of the Invention

[0007] To address the problems in the existing technology, this invention proposes a course intelligent question answering method and system based on retrieval-enhanced generation and large models.

[0008] The technical solution adopted in this invention is as follows:

[0009] In a first aspect, this invention discloses a course-based intelligent question-answering method based on retrieval-enhanced generation and a large model, comprising the following steps:

[0010] Acquire and train a Siamese BERT network to obtain text representation vectors, construct and train a cross encoder for reordering, and acquire and train a large language model to obtain the answer to the question.

[0011] The questions in the knowledge domain of the artificial intelligence course to be answered are processed by word segmentation, filling and ID mapping to obtain a question sequence. Then, the question sequence is encoded using a trained Siamese BERT network to obtain the text representation vector of the question. The dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database is calculated to obtain a similarity score. The vector database is obtained based on the trained Siamese BERT network.

[0012] The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, resulting in the BM25 score; the paragraph database is obtained based on a trained Siamese BERT network.

[0013] The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are sorted from high to low according to the mixed score to obtain the top N documents. The trained cross-encoder is used to re-sort the paragraphs in the top N documents to obtain the final top K documents.

[0014] The obtained topK documents are used as the context of the question. The trained large language model takes the topK documents and the question as input and outputs the answer to the question.

[0015] Secondly, this invention discloses a course-based intelligent question-answering system based on retrieval-enhanced generation and a large model, comprising:

[0016] The question preprocessing module is used to acquire and train a Siamese BERT network to obtain text representation vectors. It performs word segmentation, padding and ID mapping on the questions in the knowledge domain of the artificial intelligence course to be answered, and obtains a question sequence. Then, the trained Siamese BERT network is used to encode the question sequence to obtain the text representation vector of the question.

[0017] The document retrieval module is used to construct and train a cross encoder for reordering, calculate the dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database, and obtain a similarity score. The vector database is obtained based on a trained Siamese BERT network.

[0018] The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, resulting in the BM25 score; the paragraph database is obtained based on a trained Siamese BERT network.

[0019] The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are sorted from high to low according to the mixed score to obtain the top N documents. The trained cross-encoder is used to re-sort the paragraphs in the top N documents to obtain the final top K documents.

[0020] The large model inference module is used to acquire and train a large language model to obtain the answer to the question. The obtained top K documents are used as the context of the question. The trained large language model takes the top K documents and the question as input and outputs the answer to the question.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] This invention first constructs a question-answering dataset for the domain of artificial intelligence courses from different data sources, realizing the creation of a Chinese artificial intelligence dataset from scratch. Secondly, this invention constructs a hybrid retrieval algorithm and uses a cross-coding reordering method to improve retrieval performance. Finally, this invention uses a retrieval-enhanced question-answering method, incorporating the retrieved results as context for the question and inputting them into an LLM fine-tuned by LoRa, ultimately outputting the answer to the question. This improves the question-answering performance of the large language model and solves the problem of large language model illusion. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the intelligent question-and-answer system for courses according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the workflow of the intelligent course question-and-answer system according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart illustrating the workflow of the document retrieval module.

[0026] Figure 4 This is a schematic diagram of the reordering algorithm;

[0027] Figure 5 This is a schematic diagram of LoRa fine-tuning a large model. Detailed Implementation

[0028] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0029] To address the shortcomings of large language model-based intelligent question answering, this invention provides an intelligent question answering method and system for AI course teaching and Q&A based on retrieval-enhanced generation and large language model. By utilizing retrieval-enhanced generation and large language model technology, intelligent course question answering in the field of artificial intelligence is realized, which helps improve students' learning efficiency and reduce teachers' teaching burden.

[0030] like Figure 1 As shown, the course intelligent question-answering system of this invention includes a question preprocessing module, a document retrieval module, and a large model reasoning module.

[0031] The question preprocessing module is used to acquire and train a Siamese BERT network for obtaining text representation vectors. It performs word segmentation, padding, and ID mapping on the AI ​​course knowledge domain questions to be answered to obtain a question sequence. Then, the trained Siamese BERT network is used to encode the question sequence to obtain the text representation vector of the question.

[0032] The document retrieval module is used to construct and train a cross encoder for reordering, calculate the dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database, and obtain a similarity score. The vector database is obtained based on a trained Siamese BERT network.

[0033] The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, resulting in the BM25 score; the paragraph database is obtained based on a trained Siamese BERT network.

[0034] The mixed score for each paragraph is calculated using similarity scores and BM25 scores. The paragraphs are then sorted from highest to lowest mixed score to obtain the top N documents. A trained cross-encoder is used to re-sort the paragraphs within the top N documents to obtain the final top K documents. The workflow of the document retrieval module is as follows: Figure 3 As shown.

[0035] The large-scale model inference module is used to obtain a large language model for obtaining the answer to the question and to train the large language model; the obtained top K documents are used as the context of the question, the trained large language model takes the top K documents and the question as input, and outputs the answer to the question.

[0036] The workflow diagram of the course intelligent question-and-answer system is as follows: Figure 2 As shown, the specific steps are as follows: Obtain and train a Siamese BERT network to obtain text representation vectors, construct and train a cross encoder for reordering, obtain a large language model to obtain the answer to the question, and train the large language model.

[0037] The questions in the knowledge domain of the artificial intelligence course to be answered are processed by word segmentation, filling and ID mapping to obtain a question sequence. Then, the question sequence is encoded using a trained Siamese BERT network to obtain the text representation vector of the question. The dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database is calculated to obtain a similarity score. The vector database is obtained based on the trained Siamese BERT network.

[0038] The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, resulting in the BM25 score; the paragraph database is obtained based on a trained Siamese BERT network.

[0039] The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are sorted from high to low according to the mixed score to obtain the top N documents. The trained cross-encoder is used to re-sort the paragraphs in the top N documents to obtain the final top K documents.

[0040] The obtained top-K documents are used as the context of the question. The trained large language model takes the top-K documents and the question as input and outputs the answer to the question. This process is implemented step by step by a question preprocessing module, a document retrieval module, and a large model inference module.

[0041] The intelligent question-answering method for courses in this invention utilizes abundant online resources and books to construct a high-quality question-answering dataset in the field of artificial intelligence. It then retrieves the best-matching paragraphs based on a DPR-based dense retrieval algorithm and after reordering optimization. Finally, the retrieved best paragraphs are used as the context of the question, and together with the question, they serve as input to a large language model. After fine-tuning by the large language model, the answer to the question can be obtained, thereby achieving effective intelligent question-answering for courses.

[0042] The training of the Siamese BERT network, cross-encoder, and large language model mainly includes the following steps:

[0043] (1) First, relevant knowledge points were extracted from Wikipedia, popular science books on artificial intelligence, forums, web pages, and books (such as "TensorFlow Deep Learning," "Advanced Deep Learning: Natural Language Processing," and "Introduction to Deep Learning: Theory and Practice Based on Python"), and at least one question-answer pair was summarized for each paragraph, thus constructing a question-answer dataset for the knowledge domain of artificial intelligence courses. The question-answer dataset includes multiple question-answer pairs, which consist of paragraphs, questions, and answers. In the end, there are a total of 1833 question-answer pairs (paragraph-question-answer pairs), and some examples of paragraph-question-answer pairs are shown in Table 1. The training set and the test set were divided according to an 8:2 ratio. The final training set size was 1466 question-answer pairs, and the test set size was 367 question-answer pairs.

[0044] Table 1 Examples of AI Question Answering Datasets

[0045]

[0046]

[0047] (2) A dense retrieval algorithm based on DPR (Dense Passage Retrieval) is used to retrieve paragraphs related to the question as context. Specifically, firstly, the Siamese BERT network is used to obtain the paragraph text representation vector and the question text representation vector, and the dot product similarity between the paragraph text representation vector and the question text representation vector is calculated to obtain the first similarity score; then, the BM25 (Best Matching 25) score is calculated based on the question and paragraph to obtain the first BM25 score; then, the BM25 algorithm is used as a hard negative sample sampling strategy to obtain the corresponding hard negative samples; finally, the first similarity score and the first BM25 score are combined to calculate the first mixed score, and the input paragraphs are sorted from high to low according to the first mixed score to obtain the first top N documents. Then, the re-ranking algorithm of the cross encoder is used to re-rank the top N paragraphs retrieved in the first retrieval to obtain the first top K documents. Finally, the Siamese BERT network is optimized using the InfoNCE loss function to obtain the trained Siamese BERT network; the cross encoder is optimized based on the cross-entropy loss function to obtain the trained cross encoder. The process is as follows:

[0048] (2-1) Construct the DPR dense retrieval algorithm.

[0049] In a specific embodiment of the present invention, the question-answering dataset is used to train the Siamese BERT network; obtaining and training the Siamese BERT network for obtaining text representation vectors includes the following steps:

[0050] The aforementioned Siamese BERT network refers to the use of a shared BERT network for encoding paragraphs and questions. The BERT network comprises an embedding layer and an encoding layer. The embedding layer includes word vector layers, paragraph vector layers, and position vector layers; while the encoding layer consists of 12 layers of bidirectional transformers. Inputting the paragraphs and questions into BERT requires first using a BERT tokenizer to perform word segmentation, padding, and ID mapping on the paragraph and question texts, ultimately resulting in a question sequence and a paragraph sequence, which are input sequences composed of each word ID.

[0051] The question-answer pairs in the question-answer dataset are divided into k batches. Within each batch, the questions and paragraphs of each pair are processed through word segmentation, padding, and ID mapping to obtain the corresponding second question sequence and second paragraph sequence. The Siamese BERT network takes these second question sequences and second paragraph sequences as input and outputs paragraph text representation vectors and question text representation vectors. The InfoNCE loss is calculated, and the parameters of the Siamese BERT network are updated based on this loss. When updating the Siamese BERT network parameters, the parameters are first updated using the question-answer pairs from the first batch to obtain the first Siamese BERT network. Then, the parameters of the first Siamese BERT network are updated using the question-answer pairs from the second batch to obtain the second Siamese BERT network. Next, the parameters of the (i-1)th Siamese BERT network are updated using the question-answer pairs from the i-th batch to obtain the i-th Siamese BERT network. Finally, the parameters of the (k-1)th Siamese BERT network are updated using the question-answer pairs from the k-th batch to obtain the trained Siamese BERT network.

[0052] After the second question sequence and the second paragraph sequence are input into the Siamese BERT network (the second question sequence and the second paragraph sequence are combined as the input sequence), the hidden state of the text sequence is obtained. The hidden state of the text sequence refers to the vector obtained after the input sequence passes through the BERT embedding layer and the encoding layer. Then, the hidden state vector at [CLS] is used as the representation vector of the entire input, and finally a fixed-length vector is obtained, which serves as the paragraph text representation vector and the question text representation vector.

[0053] The paragraph representation vector of each paragraph in the question-answering dataset is obtained based on the trained Siamese BERT network, and the paragraph representation vectors of all batches constitute a vector database; all paragraphs in the question-answering dataset constitute a paragraph database.

[0054] Paragraph D in the question-and-answer dataset b Represented as D b =(d1,d2,…,d n The matching problem Q b Represented as Q b =(q1,q2,…,q m The Siamese network BERT is used to extract corresponding representations for paragraphs and questions, and its expression is shown below.

[0055]

[0056] Where d represents the dimension of the representation vector. For paragraph D b The paragraph text representation vector, For question Q b The text representation vector.

[0057] When training the Siamese BERT network, the BM25 algorithm is used to calculate the relevance scores between questions and paragraphs in the question-answering dataset, resulting in a first BM25 score. This first BM25 score is...

[0058]

[0059] Among them, score(D b Q c ) refers to paragraph D b And Question Q c The first BM25 score, D b Let Q be the b-th paragraph in the question-and-answer dataset, where 1 ≤ b ≤ M, and M is the total number of question-and-answer pairs in the dataset. c Let b be the c-th question in the question-answering dataset, where 1 ≤ b ≤ M;

[0060] IDF(q d Question Q c The word q d inverse document frequency, q d For question Q c The dth word in

[0061]

[0062] Where B is the total number of paragraphs in the question-and-answer dataset, B = M; n(q d ) contains the word q in all paragraphs d The number of paragraphs; n Qc For question Q c Total word count;

[0063] TF(q d D b Question Q c The word q d In paragraph D b Frequency of occurrence in

[0064]

[0065] Where, f(q) d D b ) is the word q d In paragraph D b The number of times it appears in |D b |For paragraph D b The total word count; avgdl′ is the average word count of all paragraphs; b′ is a preset parameter; k1′ is a preset parameter;

[0066] TF(q dQ c Question Q c The word q d In question Q c Frequency of occurrence in

[0067]

[0068] Where, f(q) d Q c ) is the word q d In question Q c The number of times it appears in |Q c |For question Q c The total word count; avgdl′ is the average word count of all paragraphs; b″ is a preset parameter; k1″ is a preset parameter;

[0069] The paragraph corresponding to the highest BM25 score, but not the paragraph from which the answer to the question is directly derived, will be used as question Q. c The difficult negative sample (i.e., the paragraph corresponding to the highest BM25 score is used as the difficult negative sample, where the paragraph is not the direct source of the answer to the question). In summary, difficult negative sample sampling refers to selecting one paragraph from all paragraphs in the question-answering dataset that is most similar to any question in the question-answering dataset, but is not the direct source of the answer to the question, and this paragraph is used as the difficult negative sample for the corresponding question.

[0070] Since a batch of paragraphs contains not only the paragraph corresponding to each question, but also a difficult negative sample obtained by BM25 sampling, the sampled difficult negative sample is the paragraph text with the highest similarity to the question but not the question itself, denoted as... Finally, the InfoNCE loss function is used to optimize the parameters of the Siamese BERT network, and its expression is as follows:

[0071]

[0072] Among them, InfoNCE i Let be the InfoNCE loss for the i-th batch of question-answer pairs; I is the paragraph text representation vector of all paragraphs in the i-th batch and the paragraph text representation vector of the hard negative samples; d q Let be the text representation vector of the q-th paragraph in I; τ is the temperature coefficient, which is set to 1.0 here; sim(x,y) = x·y; Let M be the text representation vector of the l-th question in the i-th batch of question-answer pairs, 1≤l≤M i M i Let be the total number of question-answer pairs in the i-th batch; is the paragraph text representation vector of the paragraph from which the answer to the l-th question in the i-th batch of question-answer pairs directly originates; g is the total number of question-answer pairs in the i-th batch.

[0073] The optimization process uses the stochastic gradient descent algorithm with a learning rate of 1e-4, a momentum parameter of 0.9, a weight decay of 5e-5, 10 training epochs, and a batch size of 16.

[0074] (2-2) Retrieve the top N paragraphs using mixed scores.

[0075] Based on the trained Siamese BERT network, the paragraph representation vector of each paragraph and the text representation vector of each question in the question-answering dataset are obtained. The first similarity score between each question and all paragraphs is calculated. The first similarity score is:

[0076]

[0077] in, for The first similarity score; Let m be the paragraph representation vector of the m-th paragraph in the question-and-answer dataset, where 1 ≤ m ≤ M; Let M be the question representation vector of the l-th question in the question-answering dataset, where 1 ≤ l ≤ M.

[0078] A first mixed score for each question and paragraph is obtained based on the first BM25 score and the first similarity score. The first mixed score is: Among them, hybrid(D m Q l Question Q l With paragraph D m The first mixed score is used; x1 is a hyperparameter, which is set to 20 here. The paragraphs in the i-th batch are sorted from high to low according to the first mixed score to obtain the first top N documents of the initial retrieval. The first top N documents here include 20 paragraphs.

[0079] Sort all paragraphs from highest to lowest based on the first mixed score of each question, and obtain M first top N documents. That is, each question has one first top N document, and M questions have M first top N documents.

[0080] (2-3) Use a cross-encoder to reorder the initial search results, such as... Figure 4As shown. The cross-encoder includes a BERT network and a linear layer. For each question corresponding to the first top N documents, it performs word segmentation, padding, and ID mapping to obtain a first question sequence. Then, it performs word segmentation, padding, and ID mapping on each paragraph in the first top N documents to obtain a first paragraph sequence. The first question sequence and each first paragraph sequence are combined to form a first input sequence. The first input sequence is...

[0081] x = [CLS]q1,q2,…,q n [SEP]p1,p2,…,p m

[0082] Where x is the first input sequence; q i p is the i-th word in the first question sequence; i It is the i-th word element in the first paragraph sequence;

[0083] The cross-encoder is trained using segments from the first top N documents obtained from the initial retrieval. Specifically, each question in the training set includes a best-matching document and the top N-1 documents (excluding the best-matching document) obtained from the initial DPR retrieval. The best-matching document is assigned to category 1, and the remaining documents are assigned to category 0 (the cross-encoder is trained based on the M first top N documents). After the input sequence is fed into the cross-encoder, it is then passed through a linear layer to reduce its dimensionality to 2 dimensions, ultimately transforming the task into a binary classification task.

[0084] Specifically: The BERT network takes the first input sequence as input, outputs the first vector to the linear layer, the linear layer performs dimensionality reduction on the first vector to obtain a two-dimensional vector, and uses the softmax function to perform binary classification on the two-dimensional vector to obtain the classification result and the probability that the paragraph is the direct source of the answer to the question. Based on the probability, the paragraphs are re-ranked to obtain the first top K documents, where the first top K documents include 1 paragraph; the training process uses the cross-entropy loss function and is optimized by the stochastic gradient descent algorithm, with a learning rate of 1e-4, a momentum parameter of 0.9, a weight decay of 5e-5, and 20 training epochs.

[0085] (3) The chatglm-6b large language model is used to answer the questions and return the answers. This process is completed by the large model inference module, specifically including:

[0086] (3-1) The first topK document is used as the context of the corresponding question to construct question-context-answer pairs. The question-context-answer pairs are used as the input of the large language model. The large language model outputs the answer to the question generated by the large language model. The LoRa model is used to supervise and fine-tune the large language model to finally obtain the trained large language model.

[0087] Chatglm-6b is based on the GLM architecture, which is a modified autoregressive transformer structure. The modified transformer includes changes to the order of layer normalization and residual connections, the use of a single linear layer to predict output terms, and the replacement of the ReLU activation function with GeLUs. Furthermore, this architecture employs autoregressive fill-in-the-blank pre-training to improve the model's ability to understand natural language; and it undergoes multi-task pre-training, performing pre-training at both the sentence and document levels to meet the requirements for generating long texts.

[0088] The large language model fine-tuning uses LoRa fine-tuning, such as... Figure 5 As shown. Lora fine-tuning refers to adding an external module to the parameters of a large language model. During fine-tuning, the pre-trained parameters of the large language model are fixed, while only the external Lora module is fine-tuned, thus saving fine-tuning time and memory overhead. The idea behind Lora fine-tuning is that the large language model has a low intrinsic dimensionality; even if the weights of the large language model are projected onto a smaller subspace, the model can still learn effectively. Lora fine-tuning uses low-rank decomposition, where for pre-trained weights W0∈R... d×k It can be decomposed into the product of two smaller matrices, as shown in the following expression:

[0089] W0+δW=W0+BA

[0090] Where B is the decomposition matrix, B∈R d×r A is a decomposition matrix, A∈R r×o r is the hidden state dimension, d is the input dimension, o is the output dimension, and W is the pre-trained parameter, W∈R. d×o δ is a small coefficient; where r << min(d,o). The forward propagation expression after LoRa fine-tuning is shown below:

[0091] h=W0u+δWu

[0092] Where h represents the output feature and u represents the input feature.

[0093] The question-paragraph pair is tokenized to obtain the corresponding input sequence y. A special character [M] is added to the end of the input sequence to indicate the answer to be predicted. The expression is as follows:

[0094] y = (q1, q2, ..., q n ;d1,d2,…,d m [M])

[0095] Large language models predict the next word based on the input sequence and the words already generated. The expression is as follows:

[0096]

[0097] Among them, E z p represents the expected probability; θ For the zth i Each predicted word probability; For the zth i There are 1 word elements, and Z is the answer label. This represents the generated question labels. The ultimate goal is to maximize the expected probability of the predicted answer. The large language model fine-tuning uses LoRa tuning with a batch size of 4, 5 training epochs, a learning rate of 5e-5, and a LoRa rank of 8.

[0098] In one specific embodiment of the present invention, the process of obtaining the answer to an AI course knowledge domain question includes the following steps: performing word segmentation, padding, and ID mapping on the AI ​​course knowledge domain question to obtain a question sequence; then encoding the question sequence using a trained Siamese BERT network to obtain a text representation vector of the question; and then calculating the dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database to obtain a similarity score, wherein the vector database is constructed when training the Siamese BERT network.

[0099] The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, resulting in the BM25 score; the paragraph database was constructed during the training of the Siamese BERT network.

[0100] The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are sorted from high to low according to the mixed score to obtain the top N documents. The trained cross-encoder is used to re-sort the paragraphs in the top N documents to obtain the final top K documents.

[0101] The obtained topK documents are used as the context of the question. The trained large language model takes the topK documents and the question as input and outputs the answer to the question.

[0102] Furthermore, the BM25 score is

[0103]

[0104] Among them, score(D a Q) refers to paragraph D a And the BM25 score of the unanswered AI course knowledge domain question Q, D a For paragraph database D total The a-th paragraph, Dtotal =(D1,D2,…,D a ,…,D N ); IDF(q i ) represents the word q in question Q. i inverse document frequency, q i For the i-th word in the question,

[0105]

[0106] Where M is the paragraph database D total Total number of paragraphs; n(q) i ) is the word q i The number of paragraphs; n is the total number of words in question Q;

[0107] TF(q i D a ) represents the word q in question Q. i In paragraph D a Frequency of occurrence in

[0108]

[0109] Where, f(q) i D a ) is the word q i In paragraph D a The number of times it appears in |D a |For paragraph D a Total word count; avgdl is the paragraph database D total The average word count of a paragraph; b is a preset parameter; k1 is a preset parameter;

[0110] TF(q i ,Q) represents the word q in question Q. i The frequency of occurrence in question Q

[0111]

[0112] Where, f(q) i ,Q) is the word q i The number of times it appears in question Q; |Q| is the total number of times the question appears; b1 is a preset parameter; k 11 These are preset parameters;

[0113] The similarity score is:

[0114]

[0115] in, For V Q With V Da Similarity score; VQ V is the text representation vector of question Q; Da For paragraph D a The paragraph representation vector;

[0116] The combined score is:

[0117]

[0118] Among them, hybrid(D a (This is paragraph D) a The mixed score; x is a hyperparameter.

[0119] Furthermore, the step of reordering the paragraphs within the top N documents using a trained cross-encoder to obtain the final top K documents includes:

[0120] Each paragraph in the top N documents is sequentially segmented, padded, and ID-mapped to obtain a paragraph sequence. The AI ​​course knowledge domain questions to be answered are then segmented, padded, and ID-mapped to obtain a question sequence. Each paragraph sequence and the question sequence are combined to form an input sequence. The trained cross-encoder takes the input sequence as input and sorts the paragraphs in the top N documents from highest to lowest probability that the paragraphs in the input sequence are the direct source of the answer to the question, resulting in the final top K documents. The top N documents contain 20 paragraphs, and the top K documents contain 1 paragraph.

[0121] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A course-based intelligent question-answering method based on retrieval-enhanced generation and large models, characterized in that, Includes the following steps: Acquire and train a Siamese BERT network to obtain text representation vectors, construct and train a cross encoder for reordering, and acquire and train a large language model to obtain the answer to the question. The questions in the knowledge domain of the artificial intelligence course to be answered are processed by word segmentation, filling and ID mapping to obtain a question sequence. Then, the question sequence is encoded using a trained Siamese BERT network to obtain the text representation vector of the question. The dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database is calculated to obtain a similarity score. The vector database is obtained based on the trained Siamese BERT network. The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, thus obtaining the BM25 score. The paragraph database is obtained based on a trained Siamese BERT network; The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are then sorted from high to low according to the mixed score to obtain the top N documents. The paragraphs in the top N documents are reordered using a trained cross-encoder to obtain the final top K documents; The obtained topK documents are used as the context of the question. The trained large language model takes the topK documents and the question as input and outputs the answer to the question. The training of the Siamese BERT network used to obtain text representation vectors includes: Construct a question-and-answer dataset for the knowledge domain of artificial intelligence courses. The question-and-answer dataset includes multiple question-and-answer pairs, each consisting of a paragraph, a question, and an answer.

2. The method according to claim 1, characterized in that, The training of the Siamese BERT network used to obtain text representation vectors also includes: The question-answer pairs in the question-answer dataset are divided into In each batch, the questions and paragraphs of the question-answer pairs in each batch are segmented, filled, and ID-mapped to obtain the corresponding second question sequence and second paragraph sequence. The Siamese BERT network takes the second question sequence and second paragraph sequence as input and outputs the paragraph text representation vector and the question text representation vector. The InfoNCE loss is calculated and the parameters of the Siamese BERT network are updated based on the InfoNCE loss. The paragraph representation vector of each paragraph in the question-answering dataset is obtained based on the trained Siamese BERT network, and all paragraph representation vectors constitute a vector database; all paragraphs in the question-answering dataset constitute a paragraph database.

3. The method according to claim 2, characterized in that, The construction of the question-answer dataset for the knowledge domain of artificial intelligence courses includes: Using Wikipedia, popular science books on artificial intelligence, popular science forums on artificial intelligence, and popular science websites on artificial intelligence as data sources, relevant paragraphs of artificial intelligence knowledge are extracted from the data sources. For each paragraph, at least one question related to artificial intelligence that students might ask is summarized, along with the corresponding reference answer, forming a paragraph-question-answer question and answer dataset.

4. The method according to claim 2, characterized in that, When training the Siamese BERT network, the BM25 algorithm is used to calculate the relevance scores between questions and paragraphs in the question-answering dataset, resulting in a first BM25 score. This first BM25 score is... ； in, It refers to paragraphs and the problem The first BM25 score, For the b-th paragraph in the question-and-answer dataset, , The total number of question-answer pairs in the question-answering dataset. The first question in the question-answering dataset. One question, ; For the question words in Inverse document frequency, For the question The first in One word, ； Where B is the total number of paragraphs in the question-and-answer dataset, B=M; For words containing all paragraphs The number of paragraphs; For the question Total word count; For the question words in In paragraph Frequency of occurrence in ； in, For words In paragraph The number of times it appears in; For paragraph Total word count; The average number of words in all paragraphs; These are preset parameters; These are preset parameters; For the question words in In the question Frequency of occurrence in ； in, For words In the question The number of times it appears in; For the question Total word count; The average number of words in all paragraphs; These are preset parameters; These are preset parameters; The paragraph corresponding to the first BM25 score with the largest value is taken as a difficult negative sample, where the paragraph is not the direct source of the answer to the question; The InfoNCE loss is: ； in, For the first InfoNCE loss per batch of question-and-answer pairs; For the first Paragraph text representation vectors of all paragraphs in each batch and paragraph text representation vectors of hard negative samples; for The text representation vector of the q-th paragraph; Temperature coefficient; ; Let be the text representation vector of the l-th question in the i-th batch of question-answer pairs. , Let be the total number of question-answer pairs in the i-th batch; Let be the paragraph text representation vector of the paragraph from which the answer to the l-th question in the i-th batch of question-answer pairs directly originates; For the first The total number of correct answers in each batch of questions.

5. The method according to claim 4, characterized in that, Based on the trained Siamese BERT network, the paragraph representation vector of each paragraph and the text representation vector of each question in the question-answering dataset are obtained. The first similarity score between each question and all paragraphs is calculated. The first similarity score is: ； in, for and The first similarity score; Let m be the paragraph representation vector of the m-th paragraph in the question-and-answer dataset. ; Let l be the question representation vector of the l-th question in the question-answering dataset. ; A first mixed score for each question and paragraph is obtained based on the first BM25 score and the first similarity score. The first mixed score is: in, For the question With paragraph The first mixed score; For hyperparameters; Based on the first mixed score of each question, all paragraphs are sorted from highest to lowest to obtain the M first top N documents.

6. The method according to claim 5, characterized in that, The construction and training of the cross-encoder includes: The cross-encoder includes a BERT network and linear layers. It performs word segmentation, padding, and ID mapping on the questions corresponding to each of the first top N documents to obtain a first question sequence. Then, it performs word segmentation, padding, and ID mapping on each paragraph in the first top N documents to obtain a first paragraph sequence. The first question sequence and each first paragraph sequence are combined to form a first input sequence. The first input sequence is... ； in, This is the first input sequence; It is the i-th word element in the first question sequence; It is the i-th word element in the first paragraph sequence; The BERT network takes the first input sequence as input, outputs the first vector to a linear layer, performs dimensionality reduction on the first vector to obtain a two-dimensional vector, uses the softmax function to perform binary classification on the two-dimensional vector, obtains the classification result and the probability that the paragraph is the direct source of the answer to the question, re-ranks the paragraphs based on the probability to obtain the first top K documents; and optimizes the cross-encoder based on the cross-entropy loss function and uses the stochastic gradient descent algorithm to obtain the trained cross-encoder.

7. The method according to claim 6, characterized in that, Training the large language model includes: The first top-K document is used as the context of the corresponding question to construct question-context-answer pairs. These pairs are then used as input to the large language model, which outputs the generated answer. The large language model is then fine-tuned under supervision using a LoRa model to obtain the trained large language model. The large language model is the chatglm-6b model.

8. The method according to claim 1, characterized in that, The BM25 score is ； in, It refers to paragraphs And the BM25 score of the unanswered AI course knowledge domain question Q. For paragraph database The first in One paragraph, ; For the words in question Q Inverse document frequency, For the question's first One word, ； Where N is the paragraph database. Total number of paragraphs; For containing words The number of paragraphs; n is the total number of words in question Q; For the words in question Q In paragraph Frequency of occurrence in ； in, For words In paragraph The number of times it appears in; For paragraph Total word count; avgdl is a paragraph database. Average word count of a middle paragraph; These are preset parameters; These are preset parameters; For the words in question Q The frequency of occurrence in question Q ； in, For words The number of times it appears in question Q; The total number of questions; These are preset parameters; These are preset parameters; The similarity score is: ； in, for and Similarity score; Let be the text representation vector of question Q; For paragraph The paragraph representation vector; The combined score is: ； in, For paragraph The mixed score; This is a hyperparameter.

9. The method according to claim 1, characterized in that, The step of reordering the paragraphs within the top N documents using a trained cross-encoder to obtain the final top K documents includes: Each paragraph in the top N documents is sequentially segmented, padded, and ID-mapped to obtain a paragraph sequence. The AI ​​course knowledge domain questions to be answered are then segmented, padded, and ID-mapped to obtain a question sequence. Each paragraph sequence and the question sequence are combined to form an input sequence. The trained cross-encoder takes the input sequence as input and sorts the paragraphs in the top N documents from highest to lowest probability that the paragraphs in the input sequence are the direct source of the answer to the question, resulting in the final top K documents. The top N documents contain 20 paragraphs, and the top K documents contain 1 paragraph.

10. A system for implementing the intelligent question-and-answer method for courses according to claim 1, characterized in that, include: The question preprocessing module is used to acquire and train a Siamese BERT network to obtain text representation vectors. It performs word segmentation, padding and ID mapping on the questions in the knowledge domain of the artificial intelligence course to be answered, and obtains a question sequence. Then, the trained Siamese BERT network is used to encode the question sequence to obtain the text representation vector of the question. The document retrieval module is used to construct and train a cross encoder for reordering, calculate the dot product similarity between the text representation vector and the representation vector of each paragraph in the vector database, and obtain a similarity score. The vector database is obtained based on a trained Siamese BERT network. The BM25 algorithm is used to calculate the relevance score between the question and each paragraph in the paragraph database, thus obtaining the BM25 score. The paragraph database is obtained based on a trained Siamese BERT network; The similarity score and BM25 score are used to calculate the mixed score for each paragraph. The paragraphs are sorted from high to low according to the mixed score to obtain the top N documents. The trained cross-encoder is used to re-sort the paragraphs in the top N documents to obtain the final top K documents. The large model inference module is used to acquire and train a large language model to obtain the answer to the question. The obtained top K documents are used as the context of the question. The trained large language model takes the top K documents and the question as input and outputs the answer to the question.

Images