A method for training an answer prediction model and a method for checking answers

Abstract

The present invention discloses a method for training an answer prediction model and a method for checking answers, which are applied to the field of question checking, and include: sorting each large language model to obtain all sorting vectors; calculating the similarity between the sorting vectors, and training a sorting embedding model according to the similarity to obtain all sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors; training a prediction model according to the mutual evaluation among the question, the best sorting embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain a trained prediction model; the trained prediction model and the constructed binary tree together constitute an answer prediction model. The present invention can broaden the knowledge of question checking based on each large language model, and make full use of the similarity to ensure that as many large language models as possible are the same in a certain area, so that there is a higher error tolerance when searching for the optimal large language model, and the answer output based on the answer prediction model is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of question checking, and particularly to a method for training an answer prediction model and a method for checking answers. Background Art

[0002] Existing large-scale question banks are generally directly generated by algorithm models. Therefore, it is necessary to check the authenticity of the answers to the questions in the question bank. Since it is difficult to conduct manual checks due to the large scale of the question bank, the reliability of the sampling inspection method is also difficult to guarantee. Therefore, an automated algorithm can be used for answer checking. For example, existing knowledge base matching algorithms and question-answering algorithms based on large language models. However, they all have certain defects. For example, the knowledge base has limitations. When the question bank exceeds the scope of the knowledge base, the similarity obtained by matching is poor and not referenceable; large language models are good at different fields. Therefore, it is not reliable to determine the accuracy of the answers in the question bank based on the answers obtained from large language models.

[0003] In summary, the existing answer checking methods at the present stage have problems of poor accuracy and unreliability. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a method and device for training an answer prediction model, a method and device for checking answers, an electronic device, and a storage medium, which solve the problems of poor accuracy and unreliability in answer checking in the prior art.

[0005] To solve the above technical problems, the present invention provides a method for training an answer prediction model, including:

[0006] Sorting each large language model to obtain all sorting vectors;

[0007] Calculating the similarity between the sorting vectors, and training a sorting embedding model according to the similarity to obtain all sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors;

[0008] Training a prediction model according to the mutual evaluation of the question, the best sorting embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain a trained prediction model;

[0009] The trained prediction model and the binary tree together constitute an answer prediction model.

[0010] Optionally, calculating the similarity between the sorting vectors, and training a sorting embedding model according to the similarity to obtain all sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors, includes:

[0011] Using an embedding technique to embed all the sorting vectors into a Euclidean space, and calculating the similarity between each sorting vector;

[0012] Train the ranking embedding model according to the similarity to obtain a trained ranking embedding model;

[0013] Obtain all the ranking embedding vectors based on the trained ranking embedding model, and construct the binary tree based on the ranking embedding vectors;

[0014] Among them, the calculation formula of the similarity is: ;

[0015] represents a preset positive hyperparameter, ; represents the i-th element in the ranking vector a; represents the i-th element in the ranking vector b; M represents a set composed of m elements; represents the number of large language models.

[0016] Optionally, the loss function in the training process of the ranking embedding model is:

[0017] ;

[0018] Among them, represents the i-th ranking vector; represents the j-th ranking vector; is an embedding function to be learned; represents the similarity between ranking vectors; represents a parameter; L represents the loss value of the ranking embedding model.

[0019] Optionally, train a prediction model according to the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language models based on the question to obtain a trained prediction model, including:

[0020] Encode the question using the encoder of the prediction model to obtain a question encoding vector;

[0021] Input the question encoding vector into each of the large language models to obtain the output answers of each large language model;

[0022] Compare the output answers of each large language model with the output answers of other large language models to obtain an evaluation tensor;

[0023] Use the question encoding vector and the evaluation tensor as the input of the prediction model, and the best ranking embedding vector corresponding to the question as the output of the prediction model to train the prediction model to obtain the trained prediction model.

[0024] Optionally, the loss function in the training process of the prediction model is:

[0025] ;

[0026] ;

[0027] Wherein, represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; represents the difference between the predicted value and the actual value; represents the loss value of the prediction model.

[0028] The present invention also provides an answer checking method, including:

[0029] Using the answer prediction model training method described above to obtain an answer prediction model; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0030] Inputting the question to be checked into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options;

[0031] Based on the predicted sorted embedding vector, searching in the binary tree to determine the target sorted embedding vector and the target sorted vector corresponding to the target sorted embedding vector;

[0032] Determining the predicted answer according to the preset conditions and the target sorted vector;

[0033] Comparing the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0034] The present invention also provides an answer prediction model training device, including:

[0035] A large language model sorting module, configured to sort each large language model to obtain all sorted vectors;

[0036] A binary tree construction module, configured to calculate the similarity between the sorted vectors, and train a sorted embedding model according to the similarity to obtain all sorted embedding vectors, and construct a binary tree based on the sorted embedding vectors;

[0037] A prediction model training module, configured to train a prediction model according to the mutual evaluation of the question, the best sorted embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain a trained prediction model;

[0038] An answer prediction model construction module, which is used to jointly form an answer prediction model with the trained prediction model and the binary tree.

[0039] The present invention also provides an answer checking device, including:

[0040] An answer prediction model acquisition module, which is used to obtain the answer prediction model obtained by the above answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0041] A prediction module, which is used to input the question to be checked into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options;

[0042] A search module, which is used to search in the binary tree based on the predicted sorted embedding vector to determine the target sorted embedding vector and the target sorted vector corresponding to the target sorted embedding vector;

[0043] A predicted answer determination module, which is used to determine the predicted answer according to a preset condition and the target sorted vector;

[0044] An inspection module, which is used to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0045] The present invention also provides an electronic device, including:

[0046] A memory, which is used to store a computer program;

[0047] A processor, which is used to implement the steps of the above answer prediction model training method and / or answer checking method when executing the computer program.

[0048] The present invention also provides a readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are loaded and executed by a processor, the steps of the above answer prediction model training method and / or answer checking method are implemented.

[0049] It can be seen that in the present invention, by sorting each large language model, all sorting vectors are obtained; the similarity between the sorting vectors is calculated, and the sorting embedding model is trained based on the similarity to obtain all sorting embedding vectors, and a binary tree is constructed based on the sorting embedding vectors; the prediction model is trained according to the mutual evaluation among the question, the best sorting embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain the trained prediction model; the trained prediction model and the constructed binary tree together constitute the answer prediction model. Based on each large language model, the present invention can broaden the knowledge scope of question checking, map the similar sorting sequences of the large language models into similar vectors, and fully ensure that as many large language models as possible in a certain area are the same. In this way, no matter which sorting vector of the large language model is finally searched, the same optimal-level large language model can be obtained, with a higher error tolerance rate, making the answer output based on the answer prediction model more accurate. Moreover, when it is found that the optimal large model is incorrect, the sub-optimal large model can also be selected as a second choice based on the sorting vector.

[0050] In addition, the present invention also provides an answer prediction model training device, an answer checking method and device, an electronic device, and a readable storage medium, which also have the above beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0052] Figure 1 It is a flowchart of an answer prediction model training method provided by an embodiment of the present invention;

[0053] Figure 2 It is a flow example diagram of a binary tree construction method provided by an embodiment of the present invention;

[0054] Figure 3 It is a flow example diagram of a prediction model training method proposed by an embodiment of the present invention;

[0055] Figure 4 It is a flowchart of an answer checking method provided by an embodiment of the present invention;

[0056] Figure 5 It is a flow example diagram of an answer checking method provided by an embodiment of the present invention;

[0057] Figure 6 It is a structural schematic diagram of an answer prediction model training device provided by an embodiment of the present invention;

[0058] Figure 7 A structural schematic diagram of an answer checking device provided by an embodiment of the present invention;

[0059] Figure 8 A structural schematic diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0061] Existing large-scale question banks are generated by algorithm models, but the authenticity of the answers to the questions in the question banks needs to be checked. Since it is difficult to manually verify due to the large scale of the question banks, a set of automated algorithm quality inspection solutions is decided to be adopted. The current automated quality inspection methods are as follows: (1) By semantic analysis or deep learning models, the most suitable answer is matched in the knowledge base and compared with the answers to the questions in the question bank. The main idea belongs to the category of classic KBQA (knowledge-based question answering, a question answering system that uses a knowledge base to answer natural language questions). For example, first match the questions in the question bank, and then use the most similar question and the corresponding answer obtained by the match as additional inputs to the model to output a new answer. The solution with KBQA as the main design idea highly depends on the scale and coverage of the existing question bank. Once the coverage area of the question bank to be tested is much larger than the existing knowledge base, either the optimal similar questions cannot be matched, or the similarity of the matched optimal similar questions is very low and not referenceable. (2) Question answering based on large language models, inducing large language models to give answers through prompts in a set format (a prompt refers to an initial input or suggestion provided to an AI system when using the AI system to guide the AI system to generate a specific output). The quality inspection process is mainly to interact with the large language model to obtain answers and then compare them with the existing answers. The main disadvantages of using large language models to answer questions and conduct quality inspections are that each large language model has questions it is not good at, and large language models do not have commonalities in the field. It is impossible to train large language models through the field. Therefore, this quality inspection method is unreliable and difficult to be actually applied. Moreover, answering questions by combining multiple different large language models usually focuses on the logic flow and is difficult to learn and combine automatically.

[0062] Based on the above problems, the present invention provides a method for training an answer prediction model. The answer prediction model trained based on this method can effectively verify the questions in the question bank. Specifically, please refer to Figure 1 , Figure 1The flowchart of a method for training an answer prediction model provided by an embodiment of the present invention. The method may include:

[0063] S101: Sort each large language model to obtain all sorted vectors.

[0064] For example, there are 3 large language models a1, a2, and a3. Then there are the following 6 sequence results for the large language models: (1) a1 < a2 < a3; (2) a1 < a3 < a2; (3) a2 < a1 < a3; (4) a2 < a3 < a1; (5) a3 < a1 < a2; (6) a3 < a2 < a1. For all possible sorting results, they can be regarded as a vector, that is, all sorted vectors are obtained. It should be noted that < here only represents a sorting, rather than the size relationship of real numbers, similar to the meaning of →.

[0065] S102: Calculate the similarity between the sorted vectors, and train the sorted embedding model according to the similarity to obtain all sorted embedding vectors, and construct a binary tree based on the sorted embedding vectors.

[0066] In this embodiment, in order to find the relationship between the sorted vectors, the similarity (i.e., distance) can be used to represent the relationship between each sorted vector. Train the sorted embedding model based on the similarity between all sorted vectors. After the sorted embedding model is trained, all sorted embedding vectors corresponding to the sorted vectors can be obtained, and a binary tree kd-tree can be constructed based on the sorted embedding vectors.

[0067] Further, the above calculation of the similarity between the sorted vectors, training the sorted embedding model according to the similarity to obtain all sorted embedding vectors, and constructing a binary tree based on the sorted embedding vectors may specifically include the following steps:

[0068] Step 21: Use the embedding technology to embed all sorted vectors into the Euclidean space, and calculate the similarity between each sorted vector.

[0069] Specifically, the similarity between two sorting sequences is reflected in the similarity between vectors. Here, the similarity uses the distance criterion, that is, the smaller the similarity score, the more similar. Use the embedding technology to embed all sorted vectors into the Euclidean space. Among them, the similarity between sorted vectors can be calculated according to the following formula:

[0070] ;

[0071] represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set composed of m elements; Indicates the number of large language models. Represents the preset positive hyperparameter. In order to achieve the actual performance, this scheme sets , which ensures that, under other conditions being the same, the high The closer they are, the smaller the final score. The ultimate goal of this solution is to find the optimal large language model, so we should try to embed high-order similar sorting vectors into the Euclidean space with a close distance, so that the answer prediction model can better recognize these similar high-order sortings. No matter which sorting is predicted, the same optimal large language model can be located.

[0072] Step 22: Train the sorting embedding model according to the similarity to obtain a trained sorting embedding model.

[0073] Specifically, all sorting vectors are mapped to a high-dimensional plane according to their similarity. Since each sorting vector itself is a sequence, the sorting embedding model of this embodiment adopts LSTM (Long Short-Term Memory Network) + BERT (Transformer-based Bidirectional Encoder Representation Technology). First, the sorting vector is encoded into a OneHot (One-Hot Encoding, a method of converting categorical data into numerical representation) matrix as the input of the LSTM layer, and the encoded result is input to BERT. The vector output by BERT is divided by the last component (usually non-zero, this operation is called projection) to obtain the sorting embedding vector .

[0074] in, , it is clear that the vector y is in the plane , ignoring the last component 1 (because the Euclidean distance calculation on this plane is equivalent to projecting onto ), the sorting embedding model maps the time series matrix (i.e., the sorting vector) to a point on a high-dimensional hyperplane. It should be noted that the sorting embedding model should not use the output of the Softmax activation function to replace the projection operation in this solution. The reason is that the components after Softmax are non-negative and the sum of the components is 1, which limits the range of the distance. In actual operations, the difference in w will cause large distance problems, and setting a larger w is more helpful to increase the distance between vectors.

[0075] It should be noted that the loss function in the training process of the above ranking embedding model is:

[0076] ;

[0077] in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents a parameter; L represents the loss value of the ranking embedding model.

[0078] Step 23: Obtain all ranking embedding vectors based on the trained ranking embedding model, and construct a binary tree based on the ranking embedding vectors.

[0079] Specifically, during the training process of the ranking embedding model, the optimal ranking embedding model parameters are obtained for the purpose of optimizing the loss function, such as the embedding function and the ranking embedding vector corresponding to each ranking vector. Based on these ranking embedding vectors, a binary tree (kd-tree, K-Dimensional Search Tree, a data structure used to store data points in a K-dimensional space for fast retrieval) in the space is constructed to facilitate subsequent nearest vector search. At the same time, maintain the mapping relationship between each ranking vector and its corresponding ranking embedding vector. The specific construction process of the binary tree can refer to Figure 2 . Figure 2 It is a flowchart example of a binary tree construction method provided by an embodiment of the present invention. Obtain all ranking vectors and calculate the similarity between the ranking vectors, train the ranking embedding model based on the similarity between the ranking vectors. After training is completed, a binary tree can be obtained. This binary tree can reflect the correlation between each ranking vector, and the mapping relationship between each ranking vector and the ranking embedding vector can also be obtained.

[0080] S103: Train a prediction model based on the mutual evaluation of the question, the best ranking embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain a trained prediction model.

[0081] The questions in this embodiment are questions including a stem and options, that is, multiple-choice questions. The questions used for training in the question bank in this embodiment are input into each large language model, and the answers given by each large language model can be obtained. Assume there are m different large language models (numbered M = {1, 2,..., m}). A certain amount (such as 100,000) of questions are extracted from the current question bank and given to the m large models for answering respectively. The quality of the m answers to each question is sorted in ascending order manually, and the ranking embedding vector corresponding to the optimal ranking vector is used as the best ranking embedding vector corresponding to the question. Each large language model can evaluate the output answers of all other large language models except itself, so as to obtain mutual evaluation. Train a prediction model based on the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the large language models, to obtain a trained prediction model. Further, it may include the following steps:

[0082] Step 31: Encode the question using the encoder of the prediction model to obtain a question encoding vector.

[0083] Specifically, for example, using the BERT model to encode the question and options of the question together into a question encoding vector, the format before encoding is Q = { "Q": "Which is the highest peak in the world?", "O": ["Mount Everest", "Mount Tai"]}.

[0084] Step 32: Input the question encoding vector into each large language model to obtain the output answers of each large language model.

[0085] Specifically, the question encoding vector Q can be given to each large language model for answering to obtain the answers A of m large models. i 。

[0086] Step 33: Compare the output answers of each large language model with the output answers of other large language models to obtain an evaluation tensor.

[0087] Specifically, it should be noted that a high-dimensional array is usually called a tensor, and a vector or matrix is just a 1-dimensional tensor and a 2-dimensional tensor. In this embodiment, the output answer A of each large language model i is given to each large language model other than i for evaluation to obtain m 2 -m evaluations (self-evaluation is not required), and the BERT model is used to encode each evaluation into a 2D (two-dimensional) initial evaluation tensor, so a 4D (four-dimensional) evaluation tensor P (2D large language model matrix + 2D initial evaluation tensor) is obtained. P (i,j) represents the evaluation tensor of large language model i for large language model j. When i = j, the tensor is set to 0. It should be noted that a prompt (the starting point of the interaction between the user and the language model, which tells the model the user's intention and expects the model to respond in a meaningful and relevant way) template requirement can be given in advance to make the large language model output in a specified format, facilitating the acquisition of the target answer given by the large language model.

[0088] Step 34: Use the question encoding vector and the evaluation tensor as the input of the prediction model, and the best sorted embedding vector corresponding to the question as the output of the prediction model, and train the prediction model to obtain a trained prediction model.

[0089] Specifically, taking the question encoding vector Q and the evaluation tensor P as inputs, and the best sorted embedding vector corresponding to the question as the output, a prediction model is trained by fitting the inputs and outputs. The structure of the prediction model is Concat(BERT(Q), Sum(P, dim=1)->CNN->BERT)->LSTM->projection operation, where the Sum operation sums along the second dimension to obtain the total evaluation of a large language model. The CNN part is an operation to convert a 3D tensor into a 2D tensor, Concat is to concatenate two 2D tensors, and the outer layer is encoded again using LSTM. The projection operation is the same as the above method. Among them, the loss function for training uses weighted MSE (mean squared error, a measure to evaluate the difference between the predicted value and the actual value of the model). The specific formula is:

[0090] ;

[0091] ;

[0092] Among them, represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; represents the difference between the predicted value and the actual value; represents the loss value of the prediction model.

[0093] The specific method for training the prediction model can refer to Figure 3 , Figure 3 which is a flowchart example of a method for training a prediction model proposed in an embodiment of the present invention. Each large language model answers based on the question vector encoding of the question and options to obtain the answers of each large language model. Each large language model evaluates the answers of other large language models to obtain an evaluation tensor. The prediction model is trained based on the question vector encoding and the evaluation tensor, so that the purpose of the prediction model is to establish the relationship between the encoding (question vector encoding and evaluation tensor) and the sorted embedding vector.

[0094] S104: The trained prediction model and the constructed binary tree together constitute an answer prediction model.

[0095] The prediction model trained in step S103 and the binary tree obtained in step S102 together constitute an answer prediction model.

[0096] Applying the answer prediction model training method provided by the embodiments of the present invention, by sorting each large language model, all sorting vectors are obtained; calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all sorting embedding vectors, constructing a binary tree based on the sorting embedding vectors; training a prediction model according to the mutual evaluation among the question, the best sorting embedding vector corresponding to the question, and the answer output by the large language model based on the question, to obtain a trained prediction model; the trained prediction model and the constructed binary tree together constitute an answer prediction model. Based on each large language model, the present invention can broaden the knowledge scope of question checking, map the similar sorting sequences of large language models into similar vectors, and fully ensure that as many large language models as possible are the same within a certain area. In this way, no matter which sorting vector of the large language model is finally searched, the same optimal-level large language model can be obtained, with a higher error tolerance, making the answer output based on the answer prediction model more accurate. Moreover, when it is found that the optimal large model is incorrect, the sub-optimal large model can also be selected as a fallback based on the sorting vector.

[0097] The present invention also provides an answer checking method, specifically refer to Figure 4 , Figure 4 which is a flowchart of an answer checking method provided by the embodiments of the present invention. The method may include:

[0098] S201: Obtain the answer prediction model obtained by using the above answer prediction model training method; the answer prediction model includes the trained prediction model and the constructed binary tree.

[0099] It should be noted that the purpose of the answer prediction model in this embodiment is to find the best large language model, and use the answer of the best large language model as the output answer of the answer prediction model.

[0100] S202: Input the question to be checked into the trained prediction model to obtain a predicted sorting embedding vector; the question to be checked includes a question and options.

[0101] Specifically, in this embodiment, the question to be checked is input into the prediction model, and the question to be checked is encoded by the trained encoder in the prediction model to obtain a question encoding vector Q; based on the question encoding vector Q, each large language model is made to answer to obtain the answers of each large language model, and then the answers are mutually evaluated pairwise among the large language models to obtain an evaluation tensor P. The prediction model outputs a predicted prediction result based on the question encoding vector Q and the evaluation tensor P, that is, a predicted sorting embedding vector.

[0102] S203: Based on the predicted sorting embedding vector, search in the binary tree to determine the target sorting embedding vector and the target sorting vector corresponding to the target sorting embedding vector.

[0103] Specifically, in the sorted projection space (i.e., the sorted embedding space), the nearest sorted embedding vector is searched for based on the predicted sorted embedding vector and the already created binary tree to obtain the target sorted embedding vector. The target sorted vector corresponding to the target sorted embedding vector is determined based on the mapping table between the sorted vector and the sorted embedding vector.

[0104] S204: Determine the predicted answer according to the preset conditions and the target sorted vector.

[0105] Specifically, the target sorted vector is, for example, in the following form:

[0106] .

[0107] Select the answer in the sorted vector according to the preset rules. For example, the prediction rule is usually to take the answer output by the optimal large language model in the target sorted vector as the final predicted answer. Or, the prediction rule can also be to take the answer output by the sub-optimal large language model in the target sorted vector as the final predicted answer.

[0108] S205: Compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0109] Specifically, the answer output by the answer prediction model is compared with the standard answer in the question bank to check the correctness of the question bank.

[0110] Applying the answer checking method provided by the embodiments of the present invention, through the answer prediction model obtained by using the above answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree; the question to be checked is input into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options; based on the predicted sorted embedding vector, search in the binary tree to determine the target sorted embedding vector and the target sorted vector corresponding to the target sorted embedding vector; determine the predicted answer according to the preset conditions and the target sorted vector; compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct. Based on each large language model, the present invention can broaden the knowledge scope of question checking, map the similar sorting sequences of large language models to similar vectors, and fully make as many large language models as possible the same within a certain area, so that no matter which large language model's sorting vector is finally searched, the same optimal-level large language model can be obtained, with a higher error tolerance rate, making the answer output based on the answer prediction model more accurate. And when it is found that the optimal large model is incorrect, the sub-optimal large model can also be selected as a second choice based on the sorted vector. Based on the above answer prediction model, various questions in the question bank can be effectively verified, and the answer prediction model can output reliable answers.

[0111] To better understand the answer checking method provided in this embodiment, reference can be specifically made to Figure 5 , Figure 5 which is a flowchart example of an answer checking method provided in an embodiment of the present invention.

[0112] After the answer prediction model is trained and the relevant configurations are ready, for a given question and options, first encode them using the trained encoder, then let each large language model answer the encoded question and options, and then let the large language models evaluate each other's answers pairwise. The following constants can be obtained from the above process: the question encoding vector Q, and the pairwise evaluation tensor P between the answers of the large language models. Extract the answers T of the large language models according to the answer format.

[0113] Input Q and P into the prediction model M to obtain the output prediction sorted embedding vector x. Search for the nearest sorted embedding vector in the sorted projection space using the pre-created binary tree, and further obtain the sorted vector (also called the sorted sequence) corresponding to this sorted embedding vector according to the mapping table. Select the large language model with the highest level in this sorted vector and obtain the answer provided by this large language model from the answers T of all large language models. Finally, compare the answer provided by the large language model with the standard answer to check the correctness of the question bank.

[0114] Next, an answer prediction model training device provided in an embodiment of the present invention will be introduced. The answer prediction model training device described below can be correspondingly referred to the answer prediction model training method described above.

[0115] Specifically, reference can be made to Figure 6 , Figure 6 which is a structural schematic diagram of an answer prediction model training device provided in an embodiment of the present invention, and may include:

[0116] A large language model sorting module 100, configured to sort each large language model to obtain all sorted vectors;

[0117] A binary tree construction module 200, configured to calculate the similarity between the sorted vectors, train a sorted embedding model according to the similarity to obtain all sorted embedding vectors, and construct a binary tree based on the sorted embedding vectors;

[0118] A prediction model training module 300, configured to train a prediction model according to the question, the best sorted embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question, to obtain a trained prediction model;

[0119] An answer prediction model construction module 400, configured to jointly form an answer prediction model with the trained prediction model and the binary tree.

[0120] Based on the above embodiments, the binary tree construction module 200 may include:

[0121] A similarity calculation unit, configured to embed all the sorted vectors into a Euclidean space by using an embedding technique and calculate the similarity between each sorted vector;

[0122] A sorted embedding model training unit, configured to train the sorted embedding model according to the similarity to obtain a trained sorted embedding model;

[0123] A binary tree construction unit, configured to obtain all the sorted embedding vectors based on the trained sorted embedding model and construct the binary tree based on the sorted embedding vectors;

[0124] Wherein, the calculation formula of the similarity is: ;

[0125] represents a preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set composed of m elements; represents the number of large language models.

[0126] Based on the above embodiments, the loss function in the process of training the sorted embedding model is:

[0127] ;

[0128] Wherein, represents the i-th sorted vector; represents the j-th sorted vector; is an embedding function to be learned; represents the similarity between sorted vectors; represents a parameter; L represents the loss value of the sorted embedding model.

[0129] Based on the above embodiments, the prediction model training module 300 may include:

[0130] A question encoding unit, configured to encode the question by using the encoder of the prediction model to obtain a question encoding vector;

[0131] An answer unit, configured to input the question encoding vector into each of the large language models to obtain the output answers of each large language model;

[0132] An evaluation unit, configured to compare the output answers of each large language model with the output answers of other large language models to obtain an evaluation tensor;

[0133] A prediction model training unit, configured to use the question encoding vector and the evaluation tensor as inputs of the prediction model, and the optimal sorting embedding vector corresponding to the question as the output of the prediction model, and train the prediction model to obtain the trained prediction model.

[0134] Based on the above embodiments, the loss function in the prediction model training process is:

[0135] ;

[0136] ;

[0137] Wherein, represents the predicted embedding vector output by the i-th prediction model; represents the i-th optimal embedding vector; n represents the total number of vectors; represents the difference between the predicted value and the actual value; represents the loss value of the prediction model.

[0138] It should be noted that the modules and units in the above answer prediction model training device can be changed in order before and after without affecting the logic.

[0139] Applying the answer prediction model training device provided by the embodiments of the present invention, through the large language model sorting module 100, which is configured to sort each large language model to obtain all sorting vectors; the binary tree construction module 200, which is configured to calculate the similarity between the sorting vectors and train the sorting embedding model based on the similarity to obtain all sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors; the prediction model training module 300, which is configured to train the prediction model according to the mutual evaluation between the question, the optimal sorting embedding vector corresponding to the question, and the answers output by the large language model based on the question, to obtain the trained prediction model; the answer prediction model construction module 400, which is configured to jointly form the answer prediction model with the trained prediction model and the constructed binary tree. This device can broaden the knowledge of question checking based on each large language model, map the similar sorting sequences of the large language models to similar vectors, and fully ensure that as many large language models as possible in a certain area are the same. In this way, no matter which sorting vector of the large language model is finally searched, the same optimal-level large language model can be obtained, with a higher error tolerance, making the answers output based on the answer prediction model more accurate. And when it is found that the optimal large model is incorrect, the sub-optimal large model can also be selected as a second choice based on the sorting vector.

[0140] Next, the answer checking device provided by the embodiments of the present invention will be introduced. The answer checking device described below can be correspondingly referred to the answer checking method described above.

[0141] For details, please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an answer checking device provided by an embodiment of the present invention and may include:

[0142] An answer prediction model acquisition module 500, configured to obtain an answer prediction model obtained by using the above answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0143] A prediction module 600, configured to input the question to be checked into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options;

[0144] A search module 700, configured to search in the binary tree based on the predicted sorted embedding vector to determine a target sorted embedding vector and a target sorted vector corresponding to the target sorted embedding vector;

[0145] A predicted answer determination module 800, configured to determine a predicted answer according to a preset condition and the target sorted vector;

[0146] An inspection module 900, configured to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0147] It should be noted that the modules and units in the above answer checking device can be changed in order before and after without affecting the logic.

[0148] Applying the answer checking device provided by the embodiments of the present invention, it includes an answer prediction model acquisition module 500, which is used to obtain an answer prediction model obtained by using the above answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree; a prediction module 600, which is used to input the question to be checked into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options; a search module 700, which is used to search in the binary tree based on the predicted sorted embedding vector to determine a target sorted embedding vector and a target sorted vector corresponding to the target sorted embedding vector; a predicted answer determination module 800, which is used to determine a predicted answer according to preset conditions and the target sorted vector; an inspection module 900, which is used to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct. Based on various large language models, this device can broaden the knowledge scope of question checking, map similar sorting sequences of large language models into similar vectors, and make it possible that as many large language models as possible are the same in a certain area. In this way, no matter which large language model's sorting vector is finally searched, the same optimal large language model can be obtained, with a higher error tolerance, making the answer output based on the answer prediction model more accurate. Moreover, when it is found that the optimal large model is incorrect, the sub-optimal large model can also be selected as a second choice based on the sorting vector. Based on the above answer prediction model, various types of questions in the question bank can be effectively verified, and the answer prediction model can output reliable answers.

[0149] Next, the electronic device provided by the embodiments of the present invention will be introduced. The electronic device described below can be correspondingly referred to the answer prediction model training method and / or the answer checking method described above.

[0150] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, and may include:

[0151] A memory 10, which is used to store computer programs;

[0152] A processor 20, which is used to execute the computer program to implement the above answer prediction model training method and / or answer checking method.

[0153] The memory 10, the processor 20, and the communication interface 31 all complete mutual communication through the communication bus 32.

[0154] In the embodiments of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiments of the present invention, the memory 10 may store programs for implementing the following functions:

[0155] Sort each large language model to obtain all sorted vectors;

[0156] Calculate the similarity between the sorted vectors and train the sorted embedding model based on the similarity to obtain all sorted embedding vectors, and construct a binary tree based on the sorted embedding vectors;

[0157] Train a prediction model based on the mutual evaluation among the question, the best sorted embedding vector corresponding to the question, and the answer output by the large language model based on the question to obtain a trained prediction model;

[0158] The trained prediction model and the constructed binary tree together constitute an answer prediction model;

[0159] and / or;

[0160] An answer prediction model obtained by using the above answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0161] Input the question to be checked into the trained prediction model to obtain a predicted sorted embedding vector; the question to be checked includes a question and options;

[0162] Based on the predicted sorted embedding vector, search in the binary tree to determine the target sorted embedding vector and the target sorted vector corresponding to the target sorted embedding vector;

[0163] Determine the predicted answer according to the preset conditions and the target sorted vector;

[0164] Compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0165] In a possible implementation, the memory 10 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function, etc.; the data storage area may store data created during use.

[0166] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include NVRAM. The memory stores an operating system and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof. Among them, the operation instructions may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0167] The processor 20 may be a Central Processing Unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices. The processor 20 may be a microprocessor or any conventional processor, etc. The processor 20 may call the program stored in the memory 10.

[0168] The communication interface 31 may be an interface of a communication module for connecting to other devices or systems.

[0169] Of course, it should be noted that Figure 8 the structure shown does not constitute a limitation on the electronic device in the embodiments of the present invention. In practical applications, the electronic device may include more or fewer components than those shown, or combine certain components. Figure 8

[0170] Next, the readable storage medium provided by the embodiments of the present invention will be introduced. The readable storage medium described below may be correspondingly referred to the answer prediction model training method and / or the answer checking method described above.

[0171] The present invention also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned answer prediction model training method and / or answer checking method are implemented.

[0172] The computer-readable storage medium may include various media that can store program codes, such as a USB flash drive, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc.

[0173] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0174] ​Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in the form of hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0175] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0176] The above has introduced in detail a method and apparatus for training an answer prediction model, a method and apparatus for checking answers, an electronic device, and a readable storage medium provided by the present invention. Specific examples are used herein to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A method for training an answer prediction model, characterized in that: include: Sort each large language model to obtain all sorting vectors; Calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors; the similarity is the distance; Training a prediction model according to the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question to obtain a trained prediction model; The trained prediction model and the binary tree together constitute an answer prediction model; Calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors, including: Embed all the sorting vectors into the Euclidean space using an embedding technique, and calculate the similarity between the sorting vectors; Training the sorting embedding model according to the similarity to obtain a trained sorting embedding model; Obtaining all the sorting embedding vectors based on the trained sorting embedding model, and constructing the binary tree based on the sorting embedding vectors; The calculation formula of the similarity is: ; represents the preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models.

2. The answer prediction model training method according to claim 1, characterized in that: The loss function during the training of the ranking embedding model is: ； in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents the parameter; L represents the loss value of the sorting embedding model.

3. The answer prediction model training method according to claim 1, characterized in that: A prediction model is trained based on the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question, to obtain a trained prediction model, including: Encoding the topic using the encoder of the prediction model to obtain a topic encoding vector; Inputting the question encoding vector into each of the large language models to obtain an output answer from each of the large language models; Compare the output answers of each of the large language models with the output answers of other large language models to obtain an evaluation tensor; The title encoding vector and the evaluation tensor are used as inputs of the prediction model, the best ranking embedding vector corresponding to the title is used as output of the prediction model, and the prediction model is trained to obtain the trained prediction model.

4. The answer prediction model training method according to claim 3, characterized in that: The loss function in the prediction model training process is: ； ； in, Represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; Indicates the difference between the predicted value and the actual value; Represents the loss value of the prediction model.

5. A method for checking an answer, characterized in that: include: An answer prediction model obtained by using the answer prediction model training method described in any one of claims 1 to 4; the answer prediction model includes a trained prediction model and a constructed binary tree; Inputting the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options; Based on the predicted sort embedding vector, searching in the binary tree to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector; Determine a predicted answer according to preset conditions and the target ranking vector; The predicted answer is compared with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

6. An answer prediction model training device, characterized in that: include: A large language model sorting module is used to sort each large language model to obtain all sorting vectors; A binary tree construction module is used to calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors; the similarity is the distance; A prediction model training module is used to train a prediction model based on a question, an optimal ranking embedding vector corresponding to the question, and a mutual evaluation between answers output by a large language model based on the question, to obtain a trained prediction model; An answer prediction model construction module, used for the trained prediction model and the binary tree to jointly form an answer prediction model; The binary tree construction module comprises: A similarity calculation unit, used to embed all the sorting vectors into the Euclidean space by using an embedding technique, and calculate the similarity between the sorting vectors; A sorting embedding model training unit, used for training the sorting embedding model according to the similarity to obtain a trained sorting embedding model; A binary tree construction unit, used to obtain all the sorting embedding vectors based on the trained sorting embedding model, and construct the binary tree based on the sorting embedding vectors; The calculation formula of the similarity is: ; represents the preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models.

7. An answer checking device, characterized in that: include: An answer prediction model acquisition module, used to obtain an answer prediction model using the answer prediction model training method described in any one of claims 1 to 4; the answer prediction model includes a trained prediction model and a constructed binary tree; A prediction module, used to input the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options; A search module, configured to search in the binary tree based on the predicted sort embedding vector to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector; A prediction answer determination module, used to determine the prediction answer according to preset conditions and the target ranking vector; The checking module is used to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the answer prediction model training method and / or the answer checking method as described in any one of claims 1 to 5 when executing the computer program.

9. A readable storage medium, characterized in that: The readable storage medium stores computer executable instructions, which, when loaded and executed by a processor, implement the steps of the answer prediction model training method and / or the answer checking method as described in any one of claims 1 to 5.

Images