Answer prediction model training method and answer checking method

Abstract

The invention discloses an answer prediction model training method and an answer checking method, which are applied to the field of question checking, and comprise the following steps: sorting each large language model to obtain all sorting vectors; calculating the similarity between the sorting vectors, training the sorting embedding model according to the similarity to obtain all sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors; training a prediction model based on mutual evaluation between answers output by the questions according to the questions, the optimal sorting embedding vectors corresponding to the questions and the large language model, and obtaining a trained prediction model; the trained prediction model and the constructed binary tree jointly form an answer prediction model. According to the method, the scope of knowledge of question examination can be broadened based on each large language model, and the similarity is utilized to fully enable the same large language models to exist in a certain area as far as possible, so that a higher error-tolerant rate is achieved when the optimal large language model is searched, and the answers output based on the answer prediction model are more accurate.

Description

Technical Field

[0001] The present invention relates to the field of question checking, and in particular to an answer prediction model training method and an answer checking method. Background Art

[0002] The existing large-scale question banks are generally generated directly by algorithm models, so it is necessary to check the authenticity of the answers to the questions in the question bank. Due to the large scale of the question bank, it is difficult to conduct manual inspection, and the reliability of the sampling method is difficult to guarantee. Therefore, an automated algorithm can be used to check the answers. For example, the existing knowledge base matching algorithm and the question-answering algorithm based on the large language model. However, there are certain defects. For example, the knowledge base is limited. When the question bank exceeds the scope of the knowledge base, the similarity obtained by matching is poor and has no reference value; the large language model excels in different fields, so it is not reliable to judge the accuracy of the answers in the question bank based on the answers obtained by the large language model.

[0003] In summary, the existing answer checking methods at this stage have problems of poor accuracy and unreliability. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an answer prediction model training method and device, an answer checking method and device, an electronic device and a storage medium, which solve the problems of poor accuracy and unreliability of answer checking in the prior art.

[0005] In order to solve the above technical problems, the present invention provides an answer prediction model training method, comprising:

[0006] Sort each large language model to obtain all sorting vectors;

[0007] Calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors;

[0008] Training a prediction model according to the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question to obtain a trained prediction model;

[0009] The trained prediction model and the binary tree together constitute an answer prediction model.

[0010] Optionally, calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors, including:

[0011] Embed all the sorting vectors into the Euclidean space using an embedding technique, and calculate the similarity between the sorting vectors;

[0012] Training the sorting embedding model according to the similarity to obtain a trained sorting embedding model;

[0013] Obtaining all the sorting embedding vectors based on the trained sorting embedding model, and constructing the binary tree based on the sorting embedding vectors;

[0014] The calculation formula of the similarity is: ;

[0015] represents the preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models.

[0016] Optionally, the loss function in the training process of the sorting embedding model is:

[0017] ;

[0018] in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents the parameter; L represents the loss value of the sorting embedding model.

[0019] Optionally, a prediction model is trained according to the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question to obtain a trained prediction model, including:

[0020] Encoding the topic using the encoder of the prediction model to obtain a topic encoding vector;

[0021] Inputting the question encoding vector into each of the large language models to obtain an output answer from each of the large language models;

[0022] Compare the output answers of each of the large language models with the output answers of other large language models to obtain an evaluation tensor;

[0023] The title encoding vector and the evaluation tensor are used as inputs of the prediction model, the best ranking embedding vector corresponding to the title is used as output of the prediction model, and the prediction model is trained to obtain the trained prediction model.

[0024] Optionally, the loss function in the prediction model training process is:

[0025] ;

[0026] ;

[0027] in, Represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; Indicates the difference between the predicted value and the actual value; Represents the loss value of the prediction model.

[0028] The present invention also provides an answer checking method, comprising:

[0029] An answer prediction model obtained by using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0030] Inputting the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options;

[0031] Based on the predicted sort embedding vector, searching in the binary tree to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector;

[0032] Determine a predicted answer according to preset conditions and the target ranking vector;

[0033] The predicted answer is compared with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

[0034] The present invention also provides an answer prediction model training device, comprising:

[0035] A large language model sorting module is used to sort each large language model to obtain all sorting vectors;

[0036] A binary tree construction module, used to calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors;

[0037] A prediction model training module is used to train a prediction model based on a question, an optimal ranking embedding vector corresponding to the question, and a mutual evaluation between answers output by a large language model based on the question, to obtain a trained prediction model;

[0038] The answer prediction model construction module is used for the trained prediction model and the binary tree to jointly form an answer prediction model.

[0039] The present invention also provides an answer checking device, comprising:

[0040] An answer prediction model acquisition module is used to obtain an answer prediction model using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0041] A prediction module, used to input the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options;

[0042] A search module, configured to search in the binary tree based on the predicted sort embedding vector to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector;

[0043] A prediction answer determination module, used to determine the prediction answer according to preset conditions and the target ranking vector;

[0044] The checking module is used to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

[0045] The present invention also provides an electronic device, comprising:

[0046] Memory for storing computer programs;

[0047] A processor is used to implement the steps of the above-mentioned answer prediction model training method and / or answer checking method when executing the computer program.

[0048] The present invention also provides a readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, the steps of the answer prediction model training method and / or the answer checking method as described above are implemented.

[0049] It can be seen that the present invention obtains all sorting vectors by sorting each large language model; calculates the similarity between the sorting vectors, and trains the sorting embedding model according to the similarity to obtain all sorting embedding vectors, and constructs a binary tree based on the sorting embedding vector; trains the prediction model based on the mutual evaluation between the question, the best sorting embedding vector corresponding to the question, and the large language model based on the question output answer, to obtain a trained prediction model; the trained prediction model and the constructed binary tree together constitute the answer prediction model. The present invention can broaden the knowledge of question inspection based on each large language model, map the sorting sequence similar to the large language model to a similar vector, and make the large language model as identical as possible in a certain area, so that no matter which large language model's sorting vector is finally searched, the same optimal level of large language model can be obtained, with a higher fault tolerance rate, so that the answer output based on the answer prediction model is more accurate. In addition, when the optimal large model is found to be wrong, the suboptimal large model can be selected as a second choice based on the sorting vector.

[0050] In addition, the present invention also provides an answer prediction model training device, an answer checking method and device, an electronic device and a readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A flowchart of an answer prediction model training method provided by an embodiment of the present invention;

[0053] Figure 2 A flowchart of a binary tree construction method provided by an embodiment of the present invention;

[0054] Figure 3 A flowchart of a prediction model training method proposed in an embodiment of the present invention;

[0055] Figure 4 A flowchart of an answer checking method provided by an embodiment of the present invention;

[0056] Figure 5 An example flowchart of an answer checking method provided by an embodiment of the present invention;

[0057] Figure 6 A schematic diagram of the structure of an answer prediction model training device provided by an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of the structure of an answer checking device provided by an embodiment of the present invention;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The existing large-scale question bank is generated by an algorithm model, but the authenticity of the answers to the questions in the question bank needs to be checked. Since the question bank is large in scale and difficult to verify manually, it is decided to use an automated algorithm quality inspection solution. The current automated quality inspection methods include: (1) Matching the most appropriate answer in the knowledge base through semantic analysis or deep learning models and comparing it with the answers to the questions in the question bank. The main idea belongs to the classic KBQA (knowledge question answering, a question-answering system that uses a knowledge base to answer natural language questions). For example, first match the questions in the question bank, and then use the most similar questions and corresponding answers obtained as additional input to give the model a new answer. The solution with KBQA as the main design idea is extremely dependent on the scale and coverage of the existing question bank. Once the scope of the question bank to be tested is much larger than the existing knowledge base, either the best similar question cannot be matched, or the best similar question matched has a very low similarity and is not referenceable. (2) Question answering based on a large language model, through the format of a prompt (prompt refers to an initial input or suggestion provided to the AI ​​system when using it, in order to guide the AI ​​system to generate a specific output) to induce the large language model to give an answer, the quality inspection process is mainly to interact with the large language model to obtain the answer and then compare it with the existing answer. The main disadvantage of using a large language model to answer questions and quality inspection is that each large language model has its own weak points, and the large language model does not have commonality in the field, and the large language model cannot be trained by the field. Therefore, this quality inspection method is unreliable and difficult to apply in practice. In addition, the combination of multiple different large language models to answer questions is usually based on logical flow, which is difficult to self-learn and combine.

[0062] Based on the above problems, the present invention provides a method for training an answer prediction model. The answer prediction model trained by this method can effectively verify the questions in the question bank. Figure 1 , Figure 1A flowchart of a method for training an answer prediction model provided by an embodiment of the present invention. The method may include:

[0063] S101: Sort each large language model to obtain all sorting vectors.

[0064] For example, there are 3 large language models a 1 、a 2 、a 3 The sequence results of the large language model are as follows: (1) a 1 <a 2 <a 3 ; (2) a 1 <a 3 <a 2 ; (3) a 2 <a 1 <a 3 ; (4) a 2 <a 3 <a 1 ; (5) a 3 <a 1 <a 2 ; (6) a 3 <a 2 <a 1 All possible sorting results can be regarded as a vector, that is, all sorting vectors are obtained. It should be noted that here < only represents a sorting, not the size relationship of the real numbers, which is similar to the meaning of →.

[0065] S102: Calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors.

[0066] In order to find the relationship between the sorting vectors, the present embodiment can use similarity (i.e., distance) to represent the relationship between the sorting vectors. The sorting embedding model is trained based on the similarity between all the sorting vectors. After the sorting embedding model is trained, the sorting embedding vectors corresponding to all the sorting vectors can be obtained, and the binary tree kd-tree can be constructed based on the sorting embedding vectors.

[0067] Furthermore, the above calculation of the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors may specifically include the following steps:

[0068] Step 21: Use embedding technology to embed all sorting vectors into the Euclidean space and calculate the similarity between each sorting vector.

[0069] Specifically, the similarity between two sorted sequences is reflected in the similarity between vectors. Here, the similarity is based on the distance standard, that is, the smaller the similarity score, the more similar. The embedding technology is used to embed all sorted vectors into the Euclidean space. Among them, the similarity between sorted vectors can be calculated according to the following formula:

[0070] ;

[0071] represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models. Represents the preset positive hyperparameter. In order to achieve the actual performance, this scheme sets , which ensures that, under other conditions being the same, the high The closer they are, the smaller the final score. The ultimate goal of this solution is to find the optimal large language model, so we should try to embed high-order similar sorting vectors into the Euclidean space with a close distance, so that the answer prediction model can better recognize these similar high-order sortings. No matter which sorting is predicted, the same optimal large language model can be located.

[0072] Step 22: Train the sorting embedding model according to the similarity to obtain a trained sorting embedding model.

[0073] Specifically, all sorting vectors are mapped to a high-dimensional plane according to their similarity. Since each sorting vector itself is a sequence, the sorting embedding model of this embodiment adopts LSTM (Long Short-Term Memory Network) + BERT (Transformer-based Bidirectional Encoder Representation Technology). First, the sorting vector is encoded into a OneHot (One-Hot Encoding, a method of converting categorical data into numerical representation) matrix as the input of the LSTM layer, and the encoded result is input to BERT. The vector output by BERT is divided by the last component (usually non-zero, this operation is called projection) to obtain the sorting embedding vector .

[0074] in, , it is clear that the vector y is in the plane , ignoring the last component 1 (because the Euclidean distance calculation on this plane is equivalent to projecting onto ), the sorting embedding model maps the time series matrix (i.e., the sorting vector) to a point on a high-dimensional hyperplane. It should be noted that the sorting embedding model should not use the output of the Softmax activation function to replace the projection operation in this solution. The reason is that the components after Softmax are non-negative and the sum of the components is 1, which limits the range of the distance. In actual operations, the difference in w will cause large distance problems, and setting a larger w is more helpful to increase the distance between vectors.

[0075] It should be noted that the loss function in the training process of the above ranking embedding model is:

[0076] ;

[0077] in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents the parameter; L represents the loss value of the sorting embedding model.

[0078] Step 23: Obtain all sorting embedding vectors based on the trained sorting embedding model, and construct a binary tree based on the sorting embedding vectors.

[0079] Specifically, the training process of the sorting embedding model aims to optimize the loss function to obtain the optimal sorting embedding model parameters, such as the embedding function and the sorting embedding vector corresponding to each sorting vector. Based on these sorting embedding vectors, a binary tree (kd-tree, K-Dimensional Search Tree, is a data structure used to store data points in K-dimensional space for fast retrieval) is constructed in the space to facilitate the subsequent nearest vector search. At the same time, the mapping relationship between the sorting embedding vector corresponding to each sorting vector is maintained. The specific process of building a binary tree can be referred to Figure 2 . Figure 2 A flowchart of a binary tree construction method provided by an embodiment of the present invention. All sorting vectors are obtained and the similarity between the sorting vectors is calculated. The sorting embedding model is trained based on the similarity between the sorting vectors. After the training is completed, a binary tree can be obtained, which can reflect the correlation between the sorting vectors, and the mapping relationship between the sorting vectors and the sorting embedding vector can also be obtained.

[0080] S103: A prediction model is trained based on the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question, to obtain a trained prediction model.

[0081] The questions in this embodiment include a stem and options, that is, multiple-choice questions. In this embodiment, the questions used for training in the question bank are input into each large language model, and the answers given by each large language model can be obtained. Assuming that there are a total of m different large language models (numbered M={1,2,…,m}), a certain amount (for example, 100,000) of questions are extracted from the current question bank and given to m large models to answer respectively, and the m answers to each question are manually sorted in ascending order, and the sorting embedding vector corresponding to the best sorting vector is used as the best sorting embedding vector corresponding to the question. Each large language model can evaluate the output answers of all large language models except itself, so as to obtain mutual evaluation. The prediction model is trained based on the question, the best sorting embedding vector corresponding to the question, and the mutual evaluation between the large language model to obtain a trained prediction model. Further, the following steps may be included:

[0082] Step 31: Encode the question using the encoder of the prediction model to obtain a question encoding vector.

[0083] Specifically, if the BERT model is used to encode the question and options together into a question encoding vector, the format before encoding is Q={"Q":"Which is the highest peak in the world?","O":["Mount Everest","Mount Taishan"]}.

[0084] Step 32: Input the question encoding vector into each large language model to obtain the output answer of each large language model.

[0085] Specifically, the question encoding vector Q can be asked to each large language model to answer, and the answer A of m large models can be obtained. i .

[0086] Step 33: Compare the output answers of each large language model with the output answers of other large language models to obtain an evaluation tensor.

[0087] Specifically, it should be noted that high-dimensional arrays are usually called tensors, and vectors or matrices are just 1-dimensional tensors and 2-dimensional tensors. i Submit it to each large language model except i for evaluation, and get m 2 -m reviews (no need for self-review), each review is encoded into a 2D (two-dimensional) initial review tensor using the BERT model, thus obtaining a 4D (four-dimensional) review tensor P (2D large language model matrix + 2D initial review tensor). P (i,j)The evaluation tensor of large language model i on large language model j. When i=j, the tensor is set to 0. It should be noted that the prompt (the starting point of the user's interaction with the language model, which tells the model the user's intention and expects the model to respond in a meaningful and relevant way) template requirement can be given in advance so that the large language model can output in the specified format, making it easier to obtain the target answer given by the large language model.

[0088] Step 34: Use the question encoding vector and the evaluation tensor as the input of the prediction model, and the best ranking embedding vector corresponding to the question as the output of the prediction model. Train the prediction model to obtain a trained prediction model.

[0089] Specifically, the question encoding vector Q and the evaluation tensor P are used as input, and the best ranking embedding vector corresponding to the question is used as output. The input and output are fitted to train a prediction model. The structure of the prediction model is Concat(BERT(Q), Sum(P, dim=1)->CNN->BERT)->LSTM->projection operation, where the Sum operation is to sum according to the second dimension to obtain the sum of the evaluations of a large language model. The CNN part is the operation of converting a 3D tensor into a 2D tensor. Concat is to splice two 2D tensors, and the outer layer is encoded once again with LSTM. The projection operation is the same as the above method. Among them, the training loss function uses weighted MSE (mean square error, a measurement method to measure the difference between the model prediction value and the actual value). The specific formula is:

[0090] ;

[0091] ;

[0092] in, Represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; Indicates the difference between the predicted value and the actual value; Represents the loss value of the prediction model.

[0093] For more information on the prediction model training method, please refer to Figure 3 , Figure 3 This is a flowchart of a prediction model training method proposed in an embodiment of the present invention. Each large language model answers based on the question vector encoding of the question and the options to obtain the answer of each large language model, each large language model evaluates the answer of each other large language model to obtain an evaluation tensor, and the prediction model is trained based on the question vector encoding and the evaluation tensor, so that the purpose of the prediction model is to establish a relationship between the encoding (question vector encoding and evaluation tensor) and the sorting embedding vector.

[0094] S104: The trained prediction model and the constructed binary tree together constitute an answer prediction model.

[0095] The prediction model trained in step S103 and the binary tree obtained in step S102 together constitute the answer prediction model.

[0096] The answer prediction model training method provided by the embodiment of the present invention is applied to sort each large language model to obtain all sorting vectors; calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all sorting embedding vectors, and construct a binary tree based on the sorting embedding vector; train the prediction model according to the mutual evaluation between the question, the best sorting embedding vector corresponding to the question, and the large language model based on the question output answer, to obtain a trained prediction model; the trained prediction model and the constructed binary tree together constitute the answer prediction model. The present invention can broaden the knowledge of question inspection based on each large language model, map the sorting sequence similar to the large language model to a similar vector, and make the large language model as identical as possible in a certain area, so that no matter which large language model sorting vector is finally searched, the same optimal level of large language model can be obtained, with a higher fault tolerance rate, so that the answer output based on the answer prediction model is more accurate. In addition, when the optimal large model is found to be wrong, the suboptimal large model can be selected as a second choice based on the sorting vector.

[0097] The present invention also provides an answer checking method, please refer to Figure 4 , Figure 4 A flowchart of an answer checking method provided by an embodiment of the present invention. The method may include:

[0098] S201: An answer prediction model is obtained using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree.

[0099] It should be noted that the purpose of the answer prediction model in this embodiment is to find the best large language model and use the answer of the best large language model as the output answer of the answer prediction model.

[0100] S202: Input the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes questions and options.

[0101] Specifically, this embodiment inputs the question to be checked into the prediction model, and encodes the question to be checked through the encoder trained in the prediction model to obtain the question encoding vector Q; based on the question encoding vector Q, each large language model is asked to answer, and the answers of each large language model are obtained, and then the large language models evaluate each other's answers to obtain the evaluation tensor P. The prediction model outputs the predicted result based on the question encoding vector Q and the evaluation tensor P, that is, the predicted ranking embedding vector.

[0102] S203: Based on the predicted sort embedding vector, search in the binary tree to determine the target sort embedding vector and the target sort vector corresponding to the target sort embedding vector.

[0103] Specifically, in the sorting projection space (i.e., sorting embedding space), the nearest sorting embedding vector is searched based on the predicted sorting embedding vector and the already created binary tree to obtain the target sorting embedding vector. The target sorting vector corresponding to the target sorting embedding vector is determined based on the mapping table between the sorting vector and the sorting embedding vector.

[0104] S204: Determine a predicted answer according to preset conditions and the target sorting vector.

[0105] Specifically, the target sorting vector is, for example, in the following form:

[0106] .

[0107] The answer in the sorting vector is selected according to the preset rules. For example, the prediction rule is usually to take the answer output by the best large language model in the target sorting vector as the final predicted answer. Alternatively, the prediction rule can also be to take the answer output by the second best large language model in the target sorting vector as the final predicted answer.

[0108] S205: Compare the predicted answer with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

[0109] Specifically, the answer output by the answer prediction model is compared with the standard answer of the question in the question bank to check the correctness of the question bank.

[0110] The answer checking method provided by the embodiment of the present invention is applied, and the answer prediction model is obtained by using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree; the question to be checked is input into the trained prediction model to obtain a prediction sorting embedding vector; the question to be checked includes questions and options; based on the prediction sorting embedding vector, a search is performed in the binary tree to determine the target sorting embedding vector and the target sorting embedding vector corresponding to the target sorting embedding vector; the predicted answer is determined according to the preset conditions and the target sorting vector; the predicted answer is compared with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct. The present invention can broaden the knowledge of question checking based on each large language model, map the sorting sequence similar to the large language model to a similar vector, and fully make the large language model as identical as possible in a certain area, so that no matter which large language model's sorting vector is finally searched, the same optimal level of large language model can be obtained, with a higher fault tolerance rate, so that the answer output based on the answer prediction model is more accurate. In addition, when the optimal large model is found to be wrong, the suboptimal large model can be selected as a second choice based on the sorting vector. Based on the above answer prediction model, various types of questions in the question bank can be effectively verified, and the answer prediction model can output reliable answers.

[0111] To better understand the answer checking method provided in this embodiment, please refer to Figure 5 , Figure 5 This is a flowchart of an answer checking method provided in an embodiment of the present invention.

[0112] The answer prediction model has been trained and the relevant configurations are ready. For a given question and option, it is first encoded with the trained encoder, and then the encoded question and option are answered by each large language model. Then the large language models evaluate each other's answers. The above process can obtain the following constants: question encoding vector Q, large language model answer and large language model mutual evaluation tensor P. The answer T of the large language model is extracted according to the answer format.

[0113] Input Q and P into the prediction model M to obtain the output prediction sort embedding vector x. Use the created binary tree to search for the nearest sort embedding vector in the sort projection space, and further obtain the sort vector (also called sort sequence) corresponding to the sort embedding vector according to the mapping table. Take the largest language model in this sort vector and obtain the answer provided by the largest language model from all the answers T of the largest language models. Finally, compare the answer provided by the large language model with the standard answer to check the correctness of the question bank.

[0114] The answer prediction model training device provided in an embodiment of the present invention is introduced below. The answer prediction model training device described below and the answer prediction model training method described above can be referenced to each other.

[0115] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of an answer prediction model training device provided by an embodiment of the present invention may include:

[0116] A large language model sorting module 100 is used to sort each large language model to obtain all sorting vectors;

[0117] A binary tree construction module 200 is used to calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors;

[0118] A prediction model training module 300 is used to train a prediction model based on a question, an optimal ranking embedding vector corresponding to the question, and a mutual evaluation between answers output by a large language model based on the question, to obtain a trained prediction model;

[0119] The answer prediction model construction module 400 is used to form an answer prediction model together with the trained prediction model and the binary tree.

[0120] Based on the above embodiment, the binary tree construction module 200 may include:

[0121] A similarity calculation unit, used to embed all the sorting vectors into the Euclidean space by using an embedding technique, and calculate the similarity between the sorting vectors;

[0122] A sorting embedding model training unit, used for training the sorting embedding model according to the similarity to obtain a trained sorting embedding model;

[0123] A binary tree construction unit, used to obtain all the sorting embedding vectors based on the trained sorting embedding model, and construct the binary tree based on the sorting embedding vectors;

[0124] The calculation formula of the similarity is: ;

[0125] represents the preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models.

[0126] Based on the above embodiment, the loss function in the training process of the sorting embedding model is:

[0127] ;

[0128] in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents the parameter; L represents the loss value of the sorting embedding model.

[0129] Based on the above embodiment, the prediction model training module 300 may include:

[0130] A topic encoding unit, used to encode the topic using the encoder of the prediction model to obtain a topic encoding vector;

[0131] An answer unit, used for inputting the question encoding vector into each of the large language models to obtain an output answer of each of the large language models;

[0132] An evaluation unit, configured to compare the output answers of each of the large language models with the output answers of other large language models to obtain an evaluation tensor;

[0133] The prediction model training unit is used to take the question encoding vector and the evaluation tensor as the input of the prediction model, and the best sorting embedding vector corresponding to the question as the output of the prediction model, train the prediction model, and obtain the trained prediction model.

[0134] Based on the above embodiment, the loss function in the prediction model training process is:

[0135] ;

[0136] ;

[0137] in, Represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; Indicates the difference between the predicted value and the actual value; Represents the loss value of the prediction model.

[0138] It should be noted that the order of the modules and units in the above-mentioned answer prediction model training device can be changed without affecting the logic.

[0139] The answer prediction model training device provided by the embodiment of the present invention is used to sort each large language model through a large language model sorting module 100 to obtain all sorting vectors; a binary tree construction module 200 is used to calculate the similarity between sorting vectors, and train the sorting embedding model according to the similarity to obtain all sorting embedding vectors, and construct a binary tree based on the sorting embedding vector; a prediction model training module 300 is used to train the prediction model according to the mutual evaluation between the question, the best sorting embedding vector corresponding to the question and the large language model based on the question output answer, to obtain a trained prediction model; an answer prediction model construction module 400 is used to form an answer prediction model with the trained prediction model and the constructed binary tree. This device can broaden the knowledge of question inspection based on each large language model, map the sorting sequence similar to the large language model into a similar vector, and fully make the large language model as identical as possible in a certain area, so that no matter which large language model sorting vector is finally searched, the same optimal level of large language model can be obtained, with a higher fault tolerance rate, so that the answer output based on the answer prediction model is more accurate. Furthermore, when an error is found in the optimal large model, a suboptimal large model can be selected as a second choice based on the sorting vector.

[0140] The answer checking device provided in an embodiment of the present invention is introduced below. The answer checking device described below and the answer checking method described above can be referenced to each other.

[0141] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of an answer checking device provided by an embodiment of the present invention may include:

[0142] The answer prediction model acquisition module 500 is used to obtain the answer prediction model using the above-mentioned answer prediction model training method; the answer prediction model includes the trained prediction model and the constructed binary tree;

[0143] Prediction module 600, used to input the subject to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the subject to be checked includes a question and options;

[0144] A search module 700 is used to search in the binary tree based on the predicted sort embedding vector to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector;

[0145] A prediction answer determination module 800, configured to determine a prediction answer according to a preset condition and the target ranking vector;

[0146] The checking module 900 is used to compare the predicted answer with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

[0147] It should be noted that the order of the modules and units in the above-mentioned answer checking device can be changed without affecting the logic.

[0148] The answer checking device provided by the embodiment of the present invention is applied, through the answer prediction model acquisition module 500, which is used to obtain the answer prediction model using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree; a prediction module 600 is used to input the question to be checked into the trained prediction model to obtain a predicted ranking embedding vector; the question to be checked includes a question and options; a search module 700 is used to search in the binary tree based on the predicted ranking embedding vector to determine the target ranking embedding vector and the target ranking vector corresponding to the target ranking embedding vector; a predicted answer determination module 800 is used to determine the predicted answer according to preset conditions and the target ranking vector; a checking module 900 is used to compare the predicted answer with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct. This device can broaden the knowledge of the question inspection based on each large language model, map the similar sorting sequence of the large language model to a similar vector, and make sure that the large language model is as identical as possible in a certain area. In this way, no matter which large language model's sorting vector is finally searched, the same optimal large language model can be obtained, which has a higher fault tolerance rate, making the answer output based on the answer prediction model more accurate. Moreover, when the optimal large model is found to be wrong, the suboptimal large model can be selected as a second choice based on the sorting vector. Based on the above answer prediction model, various types of questions in the question bank can be effectively verified, and the answer prediction model can output reliable answers.

[0149] The electronic device provided by an embodiment of the present invention is introduced below. The electronic device described below and the answer prediction model training method and / or answer checking method described above can be referenced to each other.

[0150] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention may include:

[0151] A memory 10, used for storing computer programs;

[0152] The processor 20 is used to execute a computer program to implement the above-mentioned answer prediction model training method and / or answer checking method.

[0153] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0154] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0155] Sort each large language model to obtain all sorting vectors;

[0156] Calculate the similarity between the sorting vectors, and train the sorting embedding model based on the similarity to obtain all the sorting embedding vectors, and build a binary tree based on the sorting embedding vectors;

[0157] The prediction model is trained based on the questions, the best ranking embedding vectors corresponding to the questions, and the mutual evaluation between the answers output by the large language model based on the questions, to obtain a trained prediction model;

[0158] The trained prediction model and the constructed binary tree together constitute the answer prediction model;

[0159] and / or;

[0160] An answer prediction model is obtained by using the above-mentioned answer prediction model training method; the answer prediction model includes a trained prediction model and a constructed binary tree;

[0161] Input the questions to be checked into the trained prediction model to obtain the prediction ranking embedding vector; the questions to be checked include questions and options;

[0162] Based on the predicted sort embedding vector, search in the binary tree to determine the target sort embedding vector and the target sort vector corresponding to the target sort embedding vector;

[0163] Determine the predicted answer based on the preset conditions and the target ranking vector;

[0164] The predicted answer is compared with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

[0165] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0166] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0167] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0168] The communication interface 31 may be an interface of a communication module, and is used to connect to other devices or systems.

[0169] Of course, it should be noted that Figure 8 The structure shown does not constitute a limitation on the electronic device in the embodiment of the present invention. In actual applications, the electronic device may include Figure 8 More or fewer components than shown, or combinations of certain components.

[0170] The following is an introduction to the readable storage medium provided in an embodiment of the present invention. The readable storage medium described below and the answer prediction model training method and / or answer checking method described above can be referenced to each other.

[0171] The present invention also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned answer prediction model training method and / or answer checking method are implemented.

[0172] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0173] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0174] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0175] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0176] The above is a detailed introduction to an answer prediction model training method and device, an answer checking method and device, an electronic device and a readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for training an answer prediction model, characterized in that: include: Sort each large language model to obtain all sorting vectors; Calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors; Training a prediction model according to the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question to obtain a trained prediction model; The trained prediction model and the binary tree together constitute an answer prediction model.

2. The answer prediction model training method according to claim 1, characterized in that: Calculating the similarity between the sorting vectors, and training the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and constructing a binary tree based on the sorting embedding vectors, including: Embed all the sorting vectors into the Euclidean space using an embedding technique, and calculate the similarity between the sorting vectors; Training the sorting embedding model according to the similarity to obtain a trained sorting embedding model; Obtaining all the sorting embedding vectors based on the trained sorting embedding model, and constructing the binary tree based on the sorting embedding vectors; The calculation formula of the similarity is: ; represents the preset positive hyperparameter, ; represents the i-th element in the sorted vector a; represents the i-th element in the sorted vector b; M represents a set of m elements; Indicates the number of large language models.

3. The answer prediction model training method according to claim 2, characterized in that: The loss function during the training of the ranking embedding model is: ； in, represents the i-th sorting vector; represents the jth sorting vector; is the embedding function to be learned; Represents the similarity between sorted vectors; represents the parameter; L represents the loss value of the sorting embedding model.

4. The answer prediction model training method according to claim 1, characterized in that: A prediction model is trained based on the question, the best ranking embedding vector corresponding to the question, and the mutual evaluation between the answers output by the large language model based on the question, to obtain a trained prediction model, including: Encoding the topic using the encoder of the prediction model to obtain a topic encoding vector; Inputting the question encoding vector into each of the large language models to obtain an output answer from each of the large language models; Compare the output answers of each of the large language models with the output answers of other large language models to obtain an evaluation tensor; The title encoding vector and the evaluation tensor are used as inputs of the prediction model, the best ranking embedding vector corresponding to the title is used as output of the prediction model, and the prediction model is trained to obtain the trained prediction model.

5. The answer prediction model training method according to claim 4, characterized in that: The loss function in the prediction model training process is: ； ； in, Represents the predicted embedding vector output by the i-th prediction model; represents the i-th best embedding vector; n represents the total number of vectors; Indicates the difference between the predicted value and the actual value; Represents the loss value of the prediction model.

6. A method for checking an answer, characterized in that: include: An answer prediction model obtained by using the answer prediction model training method described in any one of claims 1 to 5; the answer prediction model includes a trained prediction model and a constructed binary tree; Inputting the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options; Based on the predicted sort embedding vector, searching in the binary tree to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector; Determine a predicted answer according to preset conditions and the target ranking vector; The predicted answer is compared with the answer to the question to be checked in the question bank to determine whether the answer to the question to be checked in the question bank is correct.

7. An answer prediction model training device, characterized in that: include: A large language model sorting module is used to sort each large language model to obtain all sorting vectors; A binary tree construction module, used to calculate the similarity between the sorting vectors, and train the sorting embedding model according to the similarity to obtain all the sorting embedding vectors, and construct a binary tree based on the sorting embedding vectors; A prediction model training module is used to train a prediction model based on a question, an optimal ranking embedding vector corresponding to the question, and a mutual evaluation between answers output by a large language model based on the question, to obtain a trained prediction model; The answer prediction model construction module is used for the trained prediction model and the binary tree to jointly form an answer prediction model.

8. An answer checking device, characterized in that: include: An answer prediction model acquisition module, used to obtain an answer prediction model using the answer prediction model training method described in any one of claims 1 to 5; the answer prediction model includes a trained prediction model and a constructed binary tree; A prediction module, used to input the question to be checked into the trained prediction model to obtain a prediction ranking embedding vector; the question to be checked includes a question and options; A search module, configured to search in the binary tree based on the predicted sort embedding vector to determine a target sort embedding vector and a target sort vector corresponding to the target sort embedding vector; A prediction answer determination module, used to determine the prediction answer according to preset conditions and the target ranking vector; The checking module is used to compare the predicted answer with the answer of the question to be checked in the question bank to determine whether the answer of the question to be checked in the question bank is correct.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the answer prediction model training method and / or the answer checking method as described in any one of claims 1 to 6 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores computer executable instructions, which, when loaded and executed by a processor, implement the steps of the answer prediction model training method and / or the answer checking method as described in any one of claims 1 to 6.

Images