Large language model evaluation method and device, equipment, storage medium and program product

Abstract

The invention discloses a big language model evaluation method and device, equipment, a storage medium and a program product, and aims at evaluating a big language model in the aspect of subjective questions and answers by using an evaluation mode capable of automatically calculating a quantitative evaluation index, and improving the evaluation efficiency during capability testing. At least one of consistency evaluation of model questions and model answers and consistency evaluation of model answers and reference answers can be realized, and targeted automatic evaluation of the model is realized according to the similarity between the model answers and the model questions and the similarity between the model answers and the reference answers. The manual workload and error interference caused by manual subjective evaluation are reduced, and the evaluation efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence interaction technology, and in particular to a large language model evaluation method, device, equipment, storage medium and program product. Background Art

[0002] With the continuous development and progress of artificial intelligence technology, large language models can summarize relevant semantic information from massive corpora for questions input by users and automatically generate responses. Existing large language model evaluation schemes usually use manual evaluation and automated evaluation. Manual evaluation requires a lot of human resources, so automated evaluation has gradually become the mainstream method for large language model evaluation. Automated evaluation usually involves asking large language models to answer standardized multiple-choice questions, then using regular expressions and automated scripts to extract the model's answers and calculate the accuracy of the model's answers. When evaluating the open-ended question-answering capabilities of large language models, due to the lack of direct quantitative evaluation indicators, automated evaluation methods may no longer be applicable, and manual annotators are required to conduct quality assessments on the texts generated by the models. Therefore, the ability assessment of large language models has not yet been fully automated, and some automation is mainly limited to the field of objective questions. When evaluating the quality of content generated subjectively by large language models, it still mainly relies on manual judgment, which affects the efficiency and accuracy of the evaluation. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a large language model evaluation method, device, equipment, storage medium and program product to reduce manual intervention in the model evaluation process and improve the efficiency and accuracy of model evaluation.

[0004] To achieve the above purpose, an embodiment of the present invention provides a large language model evaluation method, including:

[0005] Performing an ability assessment on the large language model to obtain corresponding assessment indicators; wherein the ability assessment includes a subjective question-answering ability assessment, and the subjective question-answering ability assessment includes at least one of a consistency assessment between model questions and model answers and a consistency assessment between model answers and reference answers;

[0006] An evaluation score of the large language model is calculated according to at least one evaluation indicator, and the evaluation score is used as an evaluation result of the large language model.

[0007] In an embodiment of the present invention, for the evaluation of a large language model in terms of subjective question and answer, an evaluation method using automatically calculated quantitative evaluation indicators is used, and when conducting an ability test, at least one of the consistency evaluation of model questions and model answers and the consistency evaluation of model answers and reference answers can be achieved. Targeted automatic evaluation of the model is achieved based on the similarity between model answers and model questions, and between model answers and reference answers, thereby reducing manual workload and error interference caused by manual subjective evaluation, and improving evaluation efficiency and accuracy.

[0008] As an improvement of the above solution, when the subjective answering ability evaluation includes the consistency evaluation of the model questions and the model answers, the ability evaluation of the large language model is performed to obtain the corresponding evaluation indicators, including:

[0009] The natural language processing model is used to extract the vectors of the model question and the model answer to obtain the question vector and the first answer vector;

[0010] A first similarity index between the model question and the model answer is calculated based on the question vector and the first answer vector.

[0011] In the embodiment of the present invention, since there is no reference answer in an open scenario, it is impossible to calculate the consistency between the model answer and the reference answer, so the consistency between the model question and the model answer becomes an important indicator for evaluating the performance of the model. The consistency between the model question and the model answer can evaluate the understanding of the input question by the large language model and the relevance of the generated answer. In addition, it can also evaluate the performance of the large language model in new problem scenarios.

[0012] As an improvement of the above solution, when the subjective answering ability evaluation includes consistency evaluation of the model answer and the reference answer, the ability evaluation of the large language model is performed to obtain corresponding evaluation indicators, including:

[0013] The natural language processing model is used to extract the vectors of the model answer and the reference answer to obtain the second answer vector and the answer vector;

[0014] A second similarity index between the model answer and the reference answer is calculated based on the answer vector and the second answer vector.

[0015] In an embodiment of the present invention, after obtaining the answer of the large language model to be tested, the semantic similarity between the model answer of the large language model and the reference answer is calculated, and the semantic similarity calculation is used as a quantitative evaluation indicator. After vector extraction by the natural language processing model, the different meanings of words in different contexts can be understood, thereby providing more accurate semantic similarity.

[0016] As an improvement of the above-mentioned scheme, the consistency evaluation of the model answer and the reference answer includes the consistency evaluation of the model answer and the reference answer under a specific retrieval framework and without a specific retrieval framework, and the corresponding second similarity index includes a second professional similarity index and a second general similarity index.

[0017] In an embodiment of the present invention, in order to solve the problem that the large language model lacks specific domain knowledge and the knowledge cannot be updated in real time after training is completed, a specific retrieval framework is often used to plug in a specific knowledge base to solve the problem. Therefore, the embodiment of the present invention also evaluates the answer effect of the large language model under a specific retrieval framework, which can more comprehensively evaluate the answer accuracy of the large language model and improve the reliability of the large language model evaluation.

[0018] As an improvement of the above solution, under a specific search framework, the data input into the large language model includes the original question and the corresponding simulated search content, and the simulated search content includes the following three search fragments:

[0019] Search snippets that are completely relevant to the original question;

[0020] A search fragment partially related to the original question;

[0021] Retrieve snippets that are not relevant to the original question.

[0022] In the embodiment of the present invention, by setting a variety of search segments, the ability of a large language model in a specific search framework can be better evaluated.

[0023] As an improvement of the above solution, the ability assessment also includes objective question-answering ability assessment; then, the ability assessment of the large language model is performed to obtain corresponding assessment indicators, including:

[0024] Inputting objective question cases and corresponding prompts in the data set into the large language model, so that the large language model outputs a reasoning process based on each objective question case, and determines a corresponding answer based on the reasoning process;

[0025] Recording the number of successful outputs of the large language model for correct answers;

[0026] The proportion of the number of successes to the total number of answers is used as the answer accuracy indicator of the large language model.

[0027] In an embodiment of the present invention, for the evaluation of a large language model in objective question-answering, the large language model needs to complete reasoning during the evaluation process so that the evaluation process does not damage the reasoning ability of the large language model, and uses objective questions for automated evaluation, thereby reducing manual intervention and improving evaluation efficiency and accuracy.

[0028] As an improvement of the above scheme, the objective question answering ability assessment includes objective question answering ability assessment in general fields and professional fields, and the corresponding answer accuracy index includes a general field accuracy index and a professional field accuracy index.

[0029] In the embodiment of the present invention, the objective answering ability in general fields and special fields is evaluated separately, which can more comprehensively evaluate the answer accuracy of the large language model and improve the reliability of the large language model evaluation.

[0030] As an improvement of the above solution, the data set corresponding to the general field is a general data set, and the data set corresponding to the professional field is a dedicated data set.

[0031] In the embodiment of the present invention, a dedicated data set is used to evaluate the answering ability of the large language model in the professional field, thereby improving the reliability of the evaluation of the answering ability of the large language model in the professional field.

[0032] As an improvement of the above solution, the method for constructing objective question cases in the dedicated data set includes:

[0033] Acquire at least one target keyword from at least one document in a professional field;

[0034] Extracting original text containing the target keyword from the document, and hiding the target keyword in the original text;

[0035] Using a natural language processing model to predict the original text in which the target keyword is hidden, to obtain at least one predicted answer;

[0036] An objective question case of a dedicated data set is constructed by taking the original text with the target keyword hidden as the question stem, and taking the target keyword and at least one predicted answer different from the target keyword as options.

[0037] In the embodiment of the present invention, a dedicated data set is automatically constructed from documents in professional fields to evaluate the answering ability of the large language model in the professional field, which makes up for the defect in the prior art that only original document data can be used for evaluation of the large language model in the professional field, so that the evaluation of the large language model is more in line with the actual needs of professional field applications.

[0038] As an improvement of the above solution, the step of obtaining at least one target keyword from at least one document in a professional field includes:

[0039] Acquire a number of candidate keywords from at least one document in a professional field;

[0040] For each candidate keyword, calculate the frequency of occurrence of the candidate keyword in the corresponding document, and calculate the proportion of the number of documents containing the candidate keyword in the document collection;

[0041] Calculate the importance value of each candidate keyword in the corresponding document according to the occurrence frequency and the document proportion;

[0042] The candidate keywords of each document in the document set are sorted according to the importance value, and the first K candidate keywords in each document are extracted as target keywords; wherein K≥1, and K is an integer.

[0043] In an embodiment of the present invention, by screening out important keywords in professional field documents as objective question cases, the generated objective question cases are made more consistent with the meaning expressed by the professional field documents, thereby making the evaluation of the large language model more in line with the actual needs of professional field applications.

[0044] As an improvement of the above solution, the capability assessment also includes tool call capability assessment; then, the capability assessment of the large language model is performed to obtain corresponding assessment indicators, including:

[0045] Inputting a plurality of questions containing reference keywords into the large language model; wherein each reference keyword corresponds to the call of at least one query tool;

[0046] Record the effective number of successful calls to the tool corresponding to the reference keyword;

[0047] The ratio of the effective number of times to the total number of calls is used as the tool call index of the large language model.

[0048] In an embodiment of the present invention, the tool calling capability of the large language model is also added to the capability evaluation of the large language model. The tool calling capability can directly reflect whether the model's answer is correct. Therefore, by evaluating the tool calling capability of the large language model, the capability evaluation process of the large language model is improved, and the answer accuracy of the large language model can be more comprehensively evaluated, thereby improving the reliability of the large language model evaluation.

[0049] To achieve the above purpose, an embodiment of the present invention further provides a large language model evaluation device, comprising:

[0050] A capability assessment module, used to conduct capability assessment on the large language model and obtain corresponding assessment indicators; wherein the capability assessment includes subjective question answering capability assessment, and the subjective question answering capability assessment includes at least one of consistency assessment between model questions and model answers and consistency assessment between model answers and reference answers;

[0051] An evaluation result generating module is used to calculate an evaluation score of the large language model according to at least one evaluation indicator, and use the evaluation score as the evaluation result of the large language model.

[0052] To achieve the above objectives, an embodiment of the present invention further provides a large language model evaluation device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the large language model evaluation method as described in any of the above embodiments is implemented.

[0053] To achieve the above objectives, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model evaluation method as described in any of the above embodiments.

[0054] To achieve the above objectives, an embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the large language model evaluation method as described in any of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a large language model evaluation method provided by an embodiment of the present invention;

[0056] Figure 2 is a flow chart of generating a first similarity index provided by an embodiment of the present invention;

[0057] Figure 3 A flowchart of generating a first similarity index by extracting a vector through a natural language processing model is provided in an embodiment of the present invention;

[0058] Figure 4 is a flow chart of generating a second similarity index provided by an embodiment of the present invention;

[0059] Figure 5 A flowchart of generating a second similarity index by extracting a vector through a natural language processing model is provided in an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of the training process of the natural language processing model provided by an embodiment of the present invention;

[0061] Figure 7 is a flow chart of generating a second professional similarity index provided by an embodiment of the present invention;

[0062] Figure 8 is a flow chart of generating an answer accuracy index provided by an embodiment of the present invention;

[0063] Fig. 9 is a flowchart of an example of constructing an objective question in a dedicated data set provided by an embodiment of the present invention;

[0064] Fig.10 is another flow chart of an example of constructing an objective question in a dedicated data set provided by an embodiment of the present invention;

[0065] Fig.11 is a schematic diagram of predicting a hidden part of a text through a natural language processing model provided by an embodiment of the present invention;

[0066] Fig.12 is a flow chart of a tool calling indicator provided by an embodiment of the present invention;

[0067] Fig.13 is a structural block diagram of a large language model evaluation device provided by an embodiment of the present invention;

[0068] Fig.14 It is a structural block diagram of a large language model evaluation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] As large language models become more popular in various industries, it is particularly important to evaluate their performance in professional fields. This is because different large language models may perform differently in different application scenarios. Only through systematic evaluation can we ensure that the performance of the model in a specific task meets the expected standards and avoid mistakes or deviations in practical applications. At the same time, the evaluation results can reveal the shortcomings of large language models in certain aspects and potential directions for improvement, and guide researchers to optimize the design and training process of the model, thereby improving the overall performance of the model.

[0071] The existing technology allows the large language model to answer the specified questions to complete the evaluation, and limits the model's answer range through LoRa (Low-Rank Adaptation) fine-tuning or prompts, so that the model's response is only one of the multiple options provided, and then uses regular expressions and automated scripts to extract the model's answer, and the accuracy of the model's answer is statistically calculated. Although the output of the large language model can be limited to only between the provided options through prompts or fine-tuning, it will damage the reasoning ability of the large language model. The principle of the large language model determines that the generation of each token of its output depends on the input information and the previously generated content. By putting summary and reasoning in front, the model needs to fully understand the text first, and the output reasoning chain is displayed. The model will consider the reasoning process when making a judgment, and directly requiring the model to give the final answer may lead to a decrease in reasoning accuracy. In addition, in terms of open question answering, due to the lack of direct quantitative evaluation indicators, the automated evaluation method may no longer be applicable, and manual annotators are required to conduct quality assessment of the text generated by the model. Therefore, the ability assessment of large language models has not yet been fully automated, and partial automation is mainly limited to the field of objective questions. When evaluating the quality of content subjectively generated by large language models, it still mainly relies on manual judgment, which affects the evaluation efficiency.

[0072] In order to solve the problem of low evaluation efficiency and low accuracy due to reliance on manual judgment when evaluating the quality of content subjectively generated by a large language model, an embodiment of the present invention provides a large language model evaluation method, which can reduce manual intervention in the model evaluation process and improve the model evaluation efficiency and accuracy.

[0073] See also Figure 1 , Figure 1 : is a flow chart of a large language model evaluation method provided by an embodiment of the present invention, the large language model evaluation method comprising:

[0074] S1. Evaluate the capability of the large language model and obtain corresponding evaluation indicators;

[0075] S2. Calculate an evaluation score of the large language model according to at least one evaluation indicator, and use the evaluation score as an evaluation result of the large language model.

[0076] Exemplarily, the ability assessment includes a subjective question-answering ability assessment, and the subjective question-answering ability assessment includes at least one of a consistency assessment between model questions and model answers and a consistency assessment between model answers and reference answers.

[0077] In an embodiment of the present invention, for the evaluation of a large language model in terms of subjective question and answer, an evaluation method using automatically calculated quantitative evaluation indicators is used, and when conducting an ability test, at least one of the consistency evaluation of model questions and model answers and the consistency evaluation of model answers and reference answers can be achieved. Targeted automatic evaluation of the model is achieved based on the similarity between model answers and model questions, and between model answers and reference answers, thereby reducing manual workload and error interference caused by manual subjective evaluation, and improving evaluation efficiency and accuracy.

[0078] Furthermore, the evaluation indicators include the six types shown in Table 1. Each indicator will generate a quantitative score. Users can select the appropriate large language model based on the indicators they need to use and the weights they set. The following text will explain the generation process of the six evaluation indicators one by one.

[0079] Table 1 Examples of evaluation indicators

[0080]

[0081] In the embodiment of the present invention, not only a quantitative evaluation index is provided for the selection of a large language model for a professional domain base, but also, since the large language model may have a decline in its ability in a general domain after using professional domain data for full parameter or lora efficient fine-tuning, when evaluating the large language model in a professional domain, it is still necessary to evaluate its general ability to observe the extent of the decline in its ability in the general domain. Therefore, the evaluation index provided in the embodiment of the present invention is also applicable to the ability evaluation of a general large language model, thereby providing more comprehensive and accurate guidance for the evaluation of large language models.

[0082] See also Figure 2 , Figure 2 The embodiment of the present invention provides a flowchart for generating a first similarity index. When the subjective answering ability evaluation includes a consistency evaluation of model questions and model answers, the ability evaluation of the large language model is performed to obtain a corresponding evaluation index, including:

[0083] S11, extracting vectors of model questions and model answers using a natural language processing model to obtain a question vector and a first answer vector;

[0084] S12. Calculate a first similarity index between the model question and the model answer based on the question vector and the first answer vector.

[0085] It should be noted that in open scenarios, since there are no reference answers, it is impossible to calculate the consistency between the model answer and the reference answer, so the consistency between the model question and the model answer becomes an important indicator for evaluating model performance. The consistency between the model question and the model answer can evaluate the large language model's understanding of the input question and the relevance of its generated answers. A higher consistency score can indicate that the large language model can accurately capture the intent of the question and generate appropriate answers. Conversely, it indicates that the large language model's understanding ability is insufficient. In addition, the first similarity indicator can also detect the performance of the large language model in new problem scenarios. A high score means that the model has good generalization ability and can handle unseen types of questions.

[0086] For example, see Figure 3 , Figure 3 The present invention provides a flowchart of extracting vectors through a natural language processing model to generate a first similarity index. Due to the large semantic gap between the model question and the model answer, the natural language processing model (such as BERT) cannot be used directly to calculate the similarity. Usually, the model question is relatively short, mainly in the form of questions or imperative sentences, while the model answer is often a longer analysis and statement. It is not very meaningful to directly use the BERT model for vectorization and similarity calculation. Therefore, it is necessary to vectorize the model question and model answer separately, and align the corresponding question and answer pairs as much as possible in the vector space, eliminate the semantic gap, and then calculate the similarity. This process requires the use of existing question and answer pairs as training data, using e.g. Figure 3 The structure shown in the figure is used for training. At this time, the vectors of the model question and the model answer are obtained by a BERT model respectively, and the question vector bert is obtained. (query) and the first answer vector bert (answer) , the problem vector bert (query) It is used to encode the user's original query into a vector, and the first answer vector bert (answer) It is used to encode the answer of the large language model to be tested into a vector. There is no parameter sharing between the two BERT-like models. At this time, the questions and answers of the same question and answer pair are used as positive samples, and the samples of different question and answer pairs are used as negative samples. The contrast loss function makes the positive sample pairs as close as possible in space and the negative sample pairs as far away as possible in space. Training with the contrast loss function can be used to align the features of the model answers and model questions in the latent space, eliminate the semantic gap, and make the similarity score calculation meaningful. The training process satisfies the following formula:

[0087]

[0088] Among them, l qacRepresents the loss function in the comparative learning training process of model questions and model answers, where the numerator represents the similarity score of a positive sample pair and the denominator represents the similarity scores of all negative sample pairs; Bert (query) (query) represents the model problem of the positive sample pair input into the large language model during training, Bert (answer) (ans + ) represents the model answer (can be multiple) to the positive sample pairs input into the large language model during training; Bert (query) (query) represents the model problem of negative sample pairs input into the large language model during training; BERT (answer) (ans) represents the model answer to the negative sample pairs input into the large language model during training; τ represents the temperature hyperparameter, indicating the control difficulty; the sim() function represents the cosine similarity function between two vectors; k represents the kth negative sample, k = 1, 2...N, N is the total number of negative samples.

[0089] BERT is used as a model for extracting vectors because the BERT model has excellent vector extraction capabilities. Therefore, while training the model for contrastive learning, the original capabilities of the BERT model should be maintained as much as possible to avoid space collapse caused by using only contrastive learning loss functions, that is, mapping all inputs to the same point in the vector space. Therefore, it is necessary to use the original loss function for training BERT, randomly mask the question and answer pairs, and use the MLM (Masked Language Model l) loss function as the second loss function to train together to obtain the total loss function L. Among them, the pre-training task of the BERT-type model is MLM, that is, in each input sentence, some words are randomly masked, and the model is required to predict these masked words. The MLM loss function and the total loss function satisfy the following formula:

[0090]

[0091] L = l mlm +l qac (3);

[0092] Among them, x i is the ground truth (true value) of the masked word, are words other than the masked word, It represents the probability of the model predicting the content of the masked word when the unmasked word is known. This is a cross entropy loss function. The goal of this loss function is to minimize the difference between the predicted word and the actual word, so that the model can be more accurate in predicting the masked word. BERT-like models are naturally good at cloze tasks because they have learned how to infer missing words from the context during training. After training, they obtain a question vector bert (query) and a first answer vector bert (answer) .

[0093] When evaluating the consistency between model questions and model answers, you can use any common questions to ask and get the corresponding answers. The calculation process of the first similarity index Score(query,ans) satisfies:

[0094]

[0095] Among them, query represents the model question; ans represents the reference answer; || bert (query) || represents the question vector bert (query) Model; || bert (answer) || represents the first answer vector bert (answer) Model.

[0096] In the embodiment of the present invention, since there is no reference answer in an open scenario, it is impossible to calculate the consistency between the model answer and the reference answer, so the consistency between the model question and the model answer becomes an important indicator for evaluating the performance of the model. The consistency between the model question and the model answer can evaluate the understanding of the input question by the large language model and the relevance of the generated answer. In addition, it can also evaluate the performance of the large language model in new problem scenarios.

[0097] See also Figure 4 , Figure 4 The embodiment of the present invention provides a flowchart for generating a second similarity index. When the subjective answering ability evaluation includes a consistency evaluation of the model answer and the reference answer, the ability evaluation of the large language model is performed to obtain the corresponding evaluation index, including:

[0098] S21, extracting vectors of the model answer and the reference answer using a natural language processing model to obtain a second answer vector and an answer vector;

[0099] S22. Calculate a second similarity index between the model answer and the reference answer based on the answer vector and the second answer vector.

[0100] It should be noted that since the large language model in a professional field is often obtained by fine-tuning the base model through full parameter fine-tuning or Lora fine-tuning, and fine-tuning requires a paired question-answer dataset, the questions of the question-answer pairs in the dataset can be used as input to the large language model to be tested, see Figure 5 , Figure 5 The embodiment of the present invention provides a flowchart for generating a second similarity index by extracting vectors through a natural language processing model. After obtaining the answer of the large language model to be tested, the model answer and the reference answer are processed by the vector of the natural language processing model to obtain an answer vector and a second answer vector. Then, semantic similarity is calculated for these two vectors to obtain a second similarity index, and the second similarity index is used as a quantitative evaluation index.

[0101] Furthermore, when calculating semantic similarity, a BERT-type model optimized and trained by a contrastive learning method is used to obtain the representation of the model answer sentence and the reference answer sentence in the vector space, and the cosine similarity is used to calculate the similarity between the two. The BERT model can capture the context and deep semantic relationship of the sentence through a bidirectional encoder architecture based on self-attention. It can understand the different meanings of words in different contexts, thereby providing more accurate semantic similarity. In addition, since evaluation indicators such as BLEU and ROUGE mainly rely on n-gram matching and word order matching, it is impossible to fully capture the deep semantics of the sentence, and it is insensitive to synonyms, restatements, and word order changes. Therefore, in the embodiment of the present invention, evaluation indicators such as BLEU and ROUGE are not used, but the BERT model is used to calculate the second similarity indicator Score (res, ans) between the model answer and the reference answer. The calculation process satisfies the following formula:

[0102]

[0103] Among them, res represents the model answer; ans represents the reference answer; the f() function represents the BERT-type model after optimization training; f(res) and f(ans) represent the question vector and the second answer vector generated by the BERT-type model after optimization training, and ||f(res)|| and ||f(ans)|| represent the modulus of the corresponding vectors respectively.

[0104] It should be noted that in order to avoid the model answers generated by the large language model being too similar to the reference answers, the question-answer pairs used for evaluation and the question-answer pairs used for training are two complementary subsets of the original question-answer pair dataset. The vectors generated by the native BERT model that has not been optimized and trained are usually concentrated in a certain direction of the vector space. This same directionality will cause the vectors of different sentences in the same professional field to be very close in the vector space. Even if the semantic differences between these sentences are very large, this concentration effect will make the cosine similarity between vectors unreliable, resulting in the inability to effectively distinguish different sentences. Therefore, in order to enhance the distinguishing ability of BERT-like models, contrastive learning methods can be used for optimization training. By allowing the model to learn to distinguish between similar and dissimilar sample pairs, the model's distinguishing ability can be significantly improved. Specifically, by constructing positive and negative sample pairs, the model will learn how to bring similar samples closer and dissimilar samples farther apart. This training method makes the vectors generated by the model more distinguishable in the vector space and avoids the same directionality problem.

[0105] In order to make similar samples closer in the vector space and dissimilar samples farther away in the vector space, the loss function l of contrastive learning is i,j The construction is as follows:

[0106]

[0107] Among them, (Z i ,Z j ) represents a pair of positive samples; (Z i ,Z k ) represents a pair of negative samples; τ represents the temperature hyperparameter, which indicates the control difficulty; the sim() function represents the cosine similarity function between two vectors; k represents the kth negative sample, k = 1, 2...N, N is the total number of negative samples.

[0108] The construction of positive sample pairs can be achieved through sample enhancement, which is to generate positive sample pairs by randomly inserting, randomly deleting and repeating words in the original samples. Negative sample pairs can use all other samples in the same batch as negative sample pairs. Figure 6 , Figure 6 It is a schematic diagram of the training process of the natural language processing model provided by an embodiment of the present invention, in which the BERT models of the two paths share parameters, and the BERT model after training can be directly used to extract the vectors of the model answer and the reference answer, and after extraction, the average cosine similarity score is calculated as a quantitative indicator of the answer reference consistency ability of the large language model to be tested. Furthermore, the BERT model obtained after training can also be used in the retrieval model under the specific retrieval framework mentioned later.

[0109] In an embodiment of the present invention, after obtaining the answer of the large language model to be tested, the semantic similarity between the model answer of the large language model and the reference answer is calculated, and the semantic similarity calculation is used as a quantitative evaluation indicator. After vector extraction by the natural language processing model, the different meanings of words in different contexts can be understood, thereby providing more accurate semantic similarity.

[0110] Specifically, the consistency evaluation of the model answer and the reference answer includes the consistency evaluation of the model answer and the reference answer under a specific search framework and without a specific search framework, and the corresponding second similarity index includes a second professional similarity index Score (res spe ,ans) and the second common similarity index Score(res nor ,ans).

[0111] It should be noted that in order to solve the problem that the large language model lacks knowledge in specific fields and the knowledge cannot be updated in real time after training is completed, the prior art often uses a specific retrieval framework to plug in a specific knowledge base to solve the problem. For example, the specific retrieval framework is the RAG (Retrieval-Augmented Generation) framework. The RAG framework is a technology that combines the retrieval model and the generation model together. Therefore, evaluating the answer effect of the large language model under a specific retrieval framework is also the focus of the present invention. Under a specific retrieval framework, the quality of the answer depends on the quality of the retrieval and the ability of the large language model to integrate information. When evaluating the ability of the large language model, assuming that the same retrieval model is selected and the retrieved content is consistent, then the quality of the answer depends on the ability of the generation model to extract, synthesize and generate coherent answers from these fragments. At this time, the focus of the evaluation is whether the large language model can effectively extract useful information from multiple information sources, and perform logical integration to generate accurate and meaningful answers.

[0112] See also Figure 7 , Figure 7 is a flowchart of generating a second professional similarity index provided by an embodiment of the present invention. Under a specific search framework, the data input into the large language model includes an original question and corresponding simulated search content, and the simulated search content includes the following three search fragments:

[0113] 1.1) Search snippets that are completely relevant to the original question;

[0114] 1.2) Retrieval segments that are partially related to the original question; that is, some retrieval segments are related to the original question, while some retrieval segments are retrieved incorrectly and are irrelevant to the original question, thus becoming interference content;

[0115] 1.3) Retrieval fragments that are irrelevant to the original question.

[0116] The expected large language model corresponding to the above retrieval segments 1.1) to 1.3) should perform as follows in the corresponding scenarios:

[0117] 2.1) Correctly and relevantly integrate all search fragment information to output answers;

[0118] 2.2) Filter irrelevant search fragment information and generate answers based on the search fragment content related to the question;

[0119] 2.3) Refuse to answer, indicating that no corresponding evidence is found in the knowledge base.

[0120] Therefore, in order to evaluate the ability of the large language model under a specific retrieval framework, the above three simulated retrieval contents should be set for each original question, and a corresponding reference answer should be set for each retrieval result. In actual testing, the original question and the simulated retrieval content are input into the large language model to be tested. After obtaining the model's answer, the semantic similarity between the model answer and the reference answer in the three cases is calculated to obtain the second professional similarity index. The specific calculation method of the second professional similarity index is the same as the above formula (5), satisfying the following formula:

[0121]

[0122] Among them, res spe represents the answer of the model under a specific retrieval framework (such as the RAG framework); f(res spe ) and f(ans) represent the model answer and reference answer in a specific framework. The question vector and the second answer vector generated by the BERT-like model after optimization training, ||f(res spe )|| and ||f(ans)|| represent the modulus of the corresponding vectors respectively.

[0123] In the embodiment of the present invention, in order to solve the problem that the large language model lacks specific domain knowledge and the knowledge cannot be updated in real time after training, a specific search framework is often used to plug in a specific knowledge base in the prior art. Therefore, the embodiment of the present invention also evaluates the answer effect of the large language model in a specific search framework, which can more comprehensively evaluate the answer accuracy of the large language model and improve the reliability of the large language model evaluation. In addition, by setting multiple search fragments, the ability of the large language model in a specific search framework can be better evaluated.

[0124] See also Figure 8 , Figure 8 is a flow chart of generating an answer accuracy index provided by an embodiment of the present invention, wherein the ability assessment also includes an objective question answering ability assessment; then, the ability assessment of the large language model is performed to obtain corresponding assessment indicators, including:

[0125] S31, inputting objective question cases and corresponding prompts in the data set into the large language model, so that the large language model outputs a reasoning process based on each objective question case, and determines a corresponding answer based on the reasoning process;

[0126] S32, recording the number of successful outputs of the large language model for correct answers;

[0127] S33. The proportion of the number of successes to the total number of answers is used as an indicator of the answer accuracy of the large language model.

[0128] It should be noted that when evaluating the objective answering ability of a large language model, there are already many standard data sets, such as the C-Eval data set. In order to automatically extract the answers of the large language model to the multiple-choice questions in the standard data set, the existing method limits the large language model to output only one token, that is, one of the four options ABCD, by means of prompt or fine-tuning. However, this method will damage the reasoning ability of the large language model. This is because the principle of the large language model determines that the generation of each token it outputs depends on the input information and the previously generated content. By putting the summary and reasoning in front, the large language model needs to fully understand the text first, and the output reasoning chain is displayed. The large language model will consider the reasoning process when making a judgment, and directly require the large language model to give the final answer, which may lead to a decrease in the accuracy of reasoning. In order not to damage the reasoning ability of the large language model, the evaluation method proposed in the present invention realizes the fully automated extraction of the answer derived by the large language model without damaging the reasoning ability of the large language model. In the present invention, a one-shot prompt is used to first let the large language model to be tested output the reasoning process, and then let the large language model to be tested make a judgment based on the reasoning process, output the answer, and output it in a standard JSON format.

[0129] For example, the specific prompt design is as follows:

[0130] After analyzing the question stem and options, please answer the multiple-choice questions according to the following examples:

[0131] topic:

[0132] The more the Lorenz curve is convex toward the horizontal axis ()

[0133] A. The larger the Gini coefficient, the greater the income inequality

[0134] B. The larger the Gini coefficient, the more equal the income

[0135] C. The smaller the Gini coefficient, the greater the income inequality

[0136] D. The smaller the Gini coefficient, the more equal the income

[0137] Answer format

[0138] {

[0139] "Reasoning": "The Lorenz curve is a graphical tool for measuring the inequality of income or wealth distribution. The more the Lorenz curve is convex toward the horizontal axis, the greater the gap between the actual income distribution and the completely equal distribution, that is, the more unequal the income distribution is. The Gini coefficient is a commonly used indicator to measure the income or consumption gap within a country or region. Its value ranges from 0 to 1. The larger the value, the higher the degree of inequality.",

[0140] "Answer": "D"

[0141] }

[0142] Therefore, regular expressions can be used to extract the answers of the large language model. For multiple-choice questions, the correct answer rate indicator of the large language model's ability to answer general multiple-choice questions can be intuitively obtained using accuracy, that is, the correct answer rate indicator is the proportion of correctly classified samples to the total samples.

[0143] In an embodiment of the present invention, for the evaluation of a large language model in objective question-answering, the large language model needs to complete reasoning during the evaluation process so that the evaluation process does not damage the reasoning ability of the large language model, and uses objective questions for automated evaluation, thereby reducing manual intervention and improving evaluation efficiency and accuracy.

[0144] Specifically, the objective answering ability assessment includes objective answering ability assessment in general fields and professional fields, and the corresponding answer accuracy index includes the general field accuracy index Score ans.nor And professional field accuracy index Score ans.spe The data set corresponding to the general field is a general data set, and the data set corresponding to the professional field is a dedicated data set.

[0145] In the embodiment of the present invention, the objective answering ability in general and special fields is evaluated separately, which can more comprehensively evaluate the answer accuracy of the large language model and improve the reliability of the large language model evaluation. In addition, a special data set is used to evaluate the answering ability of the large language model in the professional field, which improves the evaluation reliability of the answering ability of the large language model in the professional field.

[0146] It should be noted that compared with the general field multiple-choice question answering ability assessment, the assessors can use the standard multiple-choice question data set, while the professional field multiple-choice question assessment data generally only has the professional field original document data, lacking the standard professional field data set. Therefore, the present invention proposes a method for automatically constructing a dedicated data set from professional field documents. Fig. 9 , Fig. 9It is a flowchart for constructing objective question cases in a dedicated dataset provided by an embodiment of the present invention. The method for constructing objective question cases in the dedicated dataset includes:

[0147] S311. Obtain at least one target keyword from at least one document in the professional field;

[0148] S312. Extract the original text containing the target keyword in the document, and hide the target keyword in the original text;

[0149] S313. Use a natural language processing model to predict the original text with the target keyword hidden, and obtain at least one predicted answer;

[0150] S314. Construct an objective question case for the dedicated dataset with the original text with the target keyword hidden as the question stem, and the target keyword and at least one predicted answer different from the target keyword as options.

[0151] Specifically, in step S311, the obtaining at least one target keyword from at least one document in the professional field includes: obtaining several candidate keywords from at least one document in the professional field; for each candidate keyword, calculating the occurrence frequency of the candidate keyword in the corresponding document, and calculating the document proportion of the documents containing the candidate keyword in the document set; calculating the importance degree value of each candidate keyword in the corresponding document according to the occurrence frequency and the document proportion; sorting the candidate keywords of each document in the document set according to the importance degree value, and extracting the top K candidate keywords in each document as the target keyword; where K≥1 and K is an integer.

[0152] Exemplarily, first use a tokenizer (such as the jieba tokenizer) to tokenize the document text in the professional field to facilitate the subsequent extraction of candidate keywords using the TF-IDF algorithm. Before tokenization, data cleaning is also required to remove irrelevant characters such as special symbols and numbers. After tokenization, in order to avoid the interference of some common words such as "de" (of), "zai" (in) which are irrelevant to the professional field, the generated word list needs to be filtered to remove common words. Then use the TF-IDF method to calculate the word frequency of candidate keywords in the professional field documents. TF-IDF is a statistical method used to evaluate the importance of a word in a document relative to the entire corpus. The calculated TF value represents the relative importance of a word in a single document, and the larger the value, the more important the word is in the document; the calculated IDF value reflects the global importance of a word, and less common words have higher IDF values.

[0153] For each candidate keyword, calculate the occurrence frequency of the candidate keyword in the corresponding document, that is, the TF value, which satisfies the following formula:

[0154]

[0155] Among them, f w,d The total number of times the representative word candidate keyword w appears in document d, n d Represents the total number of words in document d.

[0156] Calculate the proportion of documents containing the candidate keyword in the document collection, that is, the IDF value, which satisfies the following formula:

[0157]

[0158] Where D represents the total number of documents in the document collection; n w,D Represents the number of documents in the document collection that contain the candidate keyword w.

[0159] The importance value of each candidate keyword in the corresponding document is calculated according to the occurrence frequency TF(w, d) and the document proportion IDF(w, D), that is, the TF-IDF score is calculated, which satisfies the following formula:

[0160] TF-IDF=TF(w,d)·IDF(w,D) (10);

[0161] It can be understood that the calculation of TF-IDF combines the term frequency (TF) and the inverse document frequency (IDF) to determine the overall importance of a term in a document. The candidate keywords of each document in the document set are sorted according to the importance value, and the first K candidate keywords in each document are extracted as target keywords; where K≥1, and K is an integer.

[0162] Specifically, in steps S312 to S314, see Fig.10 , Fig.10 This is another flowchart of an example of constructing an objective question case in a dedicated data set provided by an embodiment of the present invention. The original text containing K target keywords is extracted from the document, such as combining the sentence containing these target keywords with the sentences above and below it into a question stem. The extracted target keywords are used as the correct options for the multiple-choice questions. The extracted target keywords are hidden in the original text, that is, these target keywords are replaced with special mask marks, and [mask] is used as a special mark. Then, a natural language processing model (such as a BERT model) is used to predict the hidden part to obtain at least one predicted answer. If the natural language processing model has been pre-trained as shown in formula (6) above, the pre-trained dedicated natural language model can be used to predict hidden words. If the natural language processing model has not been pre-trained, the natural language processing model commonly used in the prior art can be used to predict hidden words. Fig.11As shown in the figure, "CLS" indicates the start symbol, "SEP" is used as the middle separator between two sentences, and "[M]" indicates a special mark, which is used to hide the target keyword. After the natural language processing model, the predicted answer can be output. Then, the original text with the target keyword hidden is used as the question stem, and the target keyword and at least one predicted answer different from the target keyword are used as options to construct an objective question case of a dedicated data set.

[0163] It should be noted that another feature of the BERT model is that its structure consists of self-attention layers. One of the main advantages of self-attention is its bidirectionality and position independence. Unlike traditional unidirectional language models, this means that it can consider the context of a word to predict the masked word at the same time, and can capture long-distance dependencies to better understand the context of the word in the sentence. This bidirectional context understanding makes the BERT model more accurate than the unidirectional model when predicting masked words. The principle of the self-attention mechanism is as follows:

[0164]

[0165] Where X is the input sequence, XW Q , XW K and XW v are the weight matrices of query, key and value vectors, respectively, d k is the dimension of the key vector, and T represents the transposed matrix.

[0166] In actual use, the first four words with the highest probability of being the masked content predicted by the BERT model are taken. If the word with the highest probability coincides with the original correct option, the 2nd to 4th words with the highest probability are used as interference items. If they do not coincide, the 1st to 3rd words with the highest probability are used as interference items. Fig.11 For example, for the hidden target keyword "FG" (the original correct option), assume that the top 4 values ​​with larger predicted answer probability values ​​output by the BERT model are "FG", "EF", "FH" and "GH". Since the original correct option is included at this time, "EF", "FH" and "GH" are taken as interference items, and together with "FG" they form the options for the objective question case. Alternatively, assume that the top 4 values ​​with larger predicted answer probability values ​​output by the BERT model are "GH", "EF", "FH" and "KL". Since the original correct option is not included at this time, "GH", "EF" and "FH" are taken as interference items, and together with "FG" they form the options for the objective question case. At the same time, in order to increase the difficulty of the BERT model to generate interference items, you can use the document set D in the professional field, and continue to pre-train the BERT model with the MLM loss function before generating interference items.

[0167] In the embodiment of the present invention, a dedicated data set is automatically constructed from documents in professional fields to evaluate the answering ability of the large language model in professional fields, which makes up for the defect in the prior art that only original document data can be used for evaluation when evaluating the large language model in professional fields, so that the evaluation of the large language model is more in line with the actual needs of professional field applications. In addition, by screening out important keywords in professional field documents as objective question cases, the generated objective question cases are more in line with the meaning expressed by professional field documents, thereby making the evaluation of the large language model more in line with the actual needs of professional field applications.

[0168] See also Fig.12 , see Fig.12 , Fig.12 is a flow chart of tool calling indicators provided by an embodiment of the present invention, wherein the capability evaluation also includes tool calling capability evaluation; then, the capability evaluation of the large language model is performed to obtain corresponding evaluation indicators, including:

[0169] S41, inputting a plurality of questions containing reference keywords into the large language model; wherein each reference keyword corresponds to the call of at least one query tool;

[0170] S42, recording the effective number of successful calls to the tool corresponding to the reference keyword;

[0171] S43. The ratio of the effective number of times to the total number of times of calling is used as a tool calling index of the large language model.

[0172] It should be noted that this capability mainly evaluates the ability to call models and use external tools (such as API, database query, etc.). The model needs to recognize the intention to use the correct tool and the parameters required to call the tool. Register the tool function according to the requirements of the large language model to be evaluated. The optional tools include: weather query API call, numerical calculation, stock query, etc.

[0173] For example, the large language model must correctly recognize what tool function to use and the parameters required by the tool function in order to pass. For example, if you input: "I want to know what the weather is like in New York today?", the large language model recognizes that the user's intention is to query the weather, and chooses to call the weather query tool. The large language model correctly passes the parameters (city name, date), indicating that the tool has been successfully called. This evaluation only focuses on whether the call is successful or not, and does not focus on the effectiveness of the model's answer after the tool is called. Therefore, the tool call indicator is used to characterize the accuracy of calling the tool, that is, recording the effective number of successful tool calls, and then using the ratio of the effective number of calls to the total number of calls as the tool call indicator Socre tool .

[0174] In an embodiment of the present invention, the tool calling capability of the large language model is also added to the capability evaluation of the large language model. The tool calling capability can directly reflect whether the model's answer is correct. Therefore, by evaluating the tool calling capability of the large language model, the capability evaluation process of the large language model is improved, and the answer accuracy of the large language model can be more comprehensively evaluated, thereby improving the reliability of the large language model evaluation.

[0175] Furthermore, for the first similarity index Score(query,ans) and the second general similarity index Score(res nor ,ans), the second professional similarity index Score (res spe ,ans), general field accuracy index Score ans.nor , Professional field accuracy index Score ans.sps And tool call indicator Score tool For these six evaluation indicators, users can choose which indicators to use and set the weights of the indicators according to the needs of the target application. Under the guidance of the present invention, they can prepare training data to automatically complete the evaluation of any large language model to be tested, and select a suitable large language model as the basis for the application of large language models in professional fields according to the evaluation results. Considering that the evaluation system proposed in the present invention is mainly to evaluate the application ability of large language models in professional fields, based on the principle of specialization as the main, generalization as the auxiliary, and combination of generalization and specialization, the present invention provides two systematic evaluation ideas, and uses the application of all six evaluation indicators as an example for explanation.

[0176] In the first implementation, the bottom-line thinking method is adopted. The large language models whose first similarity index, second general similarity index and general field accuracy index all meet the corresponding minimum threshold are preferentially selected as candidate models, and then these candidate models are weighted and summed according to the second professional similarity index, professional field accuracy index and tool call index to obtain the score of the candidate large language model, and this score is used as the evaluation result.

[0177] It should be noted that the general ability evaluation of the large language model is the bottom-line evaluation indicator for whether to be included in the selection. The current mainstream large language models generally have a correct score of more than 0.7 in answering the c-eval general multiple-choice question dataset. At this time, the large language model needs to first meet the following constraints:

[0178] (1) Considering that continuing to pre-train or fine-tune the base model will often damage the general ability of the model while improving the professional ability of the model, it is required that the general multiple-choice ability answering accuracy score of the large language model included in the selection should not be less than 0.7, that is, the general domain accuracy index Score ans.nor Not less than 0.7.

[0179] (2) Under the specific retrieval framework, the cosine similarity score of the model answer and the reference answer is not less than 0.8, that is, the second general similarity index Score (res) nor ,ans) shall not be less than 0.8.

[0180] (3) Since the semantic gap between model questions and model answers cannot be completely eliminated, the score requirement for the consistency between model questions and model answers can be appropriately lowered compared to the consistency requirement between model answers and reference answers. For example, the threshold of the cosine similarity score indicator for the consistency between model questions and model answers can be set to 0.75, that is, the first similarity indicator Score(query, ans) is not less than 0.75.

[0181] Furthermore, considering that specific retrieval frameworks and tool calling capabilities are more in line with the actual application scenarios of large language models in professional fields, these two capabilities can be given higher weights. The candidate models that meet the bottom line requirements are calculated and ranked according to the following formula:

[0182] Scores um1 =ω1*Score ans.spe +ω2*Score tool +ω3*Score(res spe ,ans) (12);

[0183] Among them, Score sum1 is the total score calculated by the first method; ω1 is the weight corresponding to the professional field accuracy index; ω2 is the weight corresponding to the tool call index; ω3 is the weight corresponding to the second professional similarity index, which can satisfy: ω1>ω2=ω3; such as ω1=0.2, ω2=ω3=0.4.

[0184] In the second implementation, all evaluation indicators are directly weighted and summed to obtain the evaluation results. In this case, all evaluation indicators are taken into account and ranked in the form of total scores, but professional ability is given a higher weight, satisfying the following formula:

[0185] Score sum2 =ω1*Score ans.spe +ω2*Score tool +ω3*Score(res spe ,ans)+ω4*Score(res nor ,ans)+ω5*Score ans.nor +ω6*Score(query,ans) (13);

[0186] Among them, Score sum2is the total score calculated by the second method; ω4 is the weight corresponding to the second general similarity indicator; ω5 is the weight corresponding to the general field accuracy indicator; ω6 is the weight corresponding to the first similarity indicator, which can satisfy: ω2 and ω3 are set larger; such as ω2=ω3=0.3, ω1=ω4=ω5=ω6=0.1.

[0187] It should be noted that these two evaluation ideas are only a recommended systematic evaluation method. Users can dynamically adjust the weights and introduce additional evaluation indicators according to actual needs, and the present invention will not go into details.

[0188] Compared with the prior art, the large language model evaluation method provided by the present invention has the following beneficial effects:

[0189] 1. Comprehensiveness: This paper proposes a systematic indicator system that can comprehensively evaluate the application capabilities of large language models in different fields, including the ability to answer multiple-choice questions in general and professional fields, tool calling capabilities, etc.

[0190] 2. Automation: When evaluating objective answering ability, multiple-choice questions are used for automated evaluation without compromising the reasoning ability of the large language model, which reduces manual intervention and improves evaluation efficiency. When evaluating subjective answering ability, quantitative evaluation indicators that can be automatically calculated are used to evaluate the ability of the large language model to be tested, which also reduces manual intervention and improves evaluation efficiency.

[0191] 3. Semantic quality assessment: A new evaluation mechanism based on semantic understanding is introduced. It no longer relies solely on the overlap of words or phrases, but more accurately measures the semantic quality of the text, which is crucial for understanding the actual meaning of the text generated by the model.

[0192] 4. Applicability to professional fields: Special attention is paid to key indicators such as the tool calling capability of large language models in professional fields and the consistency between answers and prompt words, so that the evaluation is more in line with the actual needs of professional field applications.

[0193] 5. Advanced technical means: It adopts the BERT-type model optimization training method based on contrastive learning, as well as the consistency evaluation of model answers and reference answers under a specific retrieval framework. These advanced technical means improve the accuracy and reliability of the evaluation.

[0194] See also Fig.13 , Fig.13 is a structural block diagram of a large language model evaluation device 100 provided in an embodiment of the present invention, wherein the language model evaluation device 100 comprises:

[0195] A capability assessment module 11 is used to conduct capability assessment on the large language model to obtain corresponding assessment indicators; wherein the capability assessment includes subjective question answering capability assessment, and the subjective question answering capability assessment includes at least one of consistency assessment between model questions and model answers and consistency assessment between model answers and reference answers;

[0196] The evaluation result generating module 12 is used to calculate the evaluation score of the large language model according to at least one evaluation indicator, and use the evaluation score as the evaluation result of the large language model.

[0197] Specifically, when the subjective question-answering ability assessment includes a consistency assessment of model questions and model answers, the ability assessment module 11 is used to: use a natural language processing model to extract vectors of model questions and model answers to obtain a question vector and a first answer vector; and calculate a first similarity index between the model question and the model answer based on the question vector and the first answer vector.

[0198] Specifically, when the subjective question-answering ability assessment includes a consistency assessment of the model answer and the reference answer, the ability assessment module 11 is used to: use a natural language processing model to extract the vectors of the model answer and the reference answer to obtain a second answer vector and an answer vector; and calculate a second similarity index between the model answer and the reference answer based on the answer vector and the second answer vector.

[0199] Specifically, the ability assessment also includes objective answering ability assessment; then, the ability assessment module 11 is used to: input the objective question cases and corresponding prompts in the data set into the large language model, so that the large language model outputs the reasoning process based on each objective question case, and determines the corresponding answer based on the reasoning process; records the number of successful answers output by the large language model; and uses the proportion of the successful number of times in the total number of answers as the answer accuracy indicator of the large language model.

[0200] Specifically, the capability assessment also includes tool calling capability assessment; then, the capability assessment module 11 is used to: input a number of questions containing reference keywords in the large language model; wherein each reference keyword corresponds to the call of at least one query tool; record the effective number of successful calls to the tool corresponding to the reference keyword; and use the ratio of the effective number to the total number of calls as the tool calling index of the large language model.

[0201] It is worth noting that the working process of each module in the large language model evaluation device 100 provided in the embodiment of the present invention can refer to the working process of the large language model evaluation method described in the above embodiment, and will not be repeated here.

[0202] See also Fig.14 , Fig.142 is a structural block diagram of a large language model evaluation device 200 provided in an embodiment of the present invention, and the large language model evaluation device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned large language model evaluation method embodiments are implemented, such as steps S1-S2, S11-S12, S21-S22, S31-S33, S311-S314, and S41-S43.

[0203] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the large language model evaluation device 200.

[0204] The large language model evaluation device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the large language model evaluation device 200 and does not constitute a limitation on the large language model evaluation device 200. The large language model evaluation device 200 may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the large language model evaluation device 200 may also include input and output devices, network access devices, buses, and the like.

[0205] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the large language model evaluation device 200, and uses various interfaces and lines to connect various parts of the entire large language model evaluation device 200.

[0206] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the large language model evaluation device 200 by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0207] Among them, if the module / unit integrated in the large language model evaluation device 200 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0208] An embodiment of the present invention further provides a computer program product, including computer instructions, which, when executed by a processor, implement the large language model evaluation method as described in the above embodiment.

[0209] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A large language model evaluation method, characterized in that: include: Performing capability evaluation on the large language model to obtain corresponding evaluation indicators; wherein the capability evaluation includes subjective question answering capability evaluation, and the subjective question answering capability evaluation includes at least one of consistency evaluation between model questions and model answers and consistency evaluation between model answers and reference answers; An evaluation score of the large language model is calculated according to at least one evaluation indicator, and the evaluation score is used as an evaluation result of the large language model.

2. The large language model evaluation method according to claim 1, characterized in that: When the subjective answering ability evaluation includes the consistency evaluation of the model questions and the model answers, the ability evaluation of the large language model is performed to obtain the corresponding evaluation indicators, including: The natural language processing model is used to extract the vectors of the model question and the model answer to obtain the question vector and the first answer vector; A first similarity index between the model question and the model answer is calculated based on the question vector and the first answer vector.

3. The large language model evaluation method according to claim 1, characterized in that: When the subjective answering ability evaluation includes consistency evaluation of the model answer and the reference answer, the ability evaluation of the large language model is performed to obtain corresponding evaluation indicators, including: The natural language processing model is used to extract the vectors of the model answer and the reference answer to obtain the second answer vector and the answer vector; A second similarity index between the model answer and the reference answer is calculated based on the answer vector and the second answer vector.

4. The large language model evaluation method according to claim 3, characterized in that: The consistency evaluation of the model answer and the reference answer includes the consistency evaluation of the model answer and the reference answer under a specific retrieval framework and without a specific retrieval framework, and the corresponding second similarity index includes a second professional similarity index and a second general similarity index.

5. The large language model evaluation method according to claim 4, characterized in that: In a specific search framework, the data input into the large language model includes the original question and the corresponding simulated search content, and the simulated search content includes the following three search segments: Search snippets that are completely relevant to the original question; A search fragment partially related to the original question; Retrieve snippets that are not relevant to the original question.

6. The large language model evaluation method according to claim 1, characterized in that: The ability assessment also includes objective question-answering ability assessment; then, the ability assessment of the large language model is performed to obtain corresponding assessment indicators, including: Inputting objective question cases and corresponding prompts in the data set into the large language model, so that the large language model outputs a reasoning process based on each objective question case, and determines a corresponding answer based on the reasoning process; Recording the number of successful outputs of the large language model for correct answers; The proportion of the number of successes to the total number of answers is used as the answer accuracy indicator of the large language model.

7. The large language model evaluation method according to claim 6, characterized in that: The objective question-answering ability assessment includes objective question-answering ability assessment in general fields and professional fields, and the corresponding answer accuracy index includes a general field accuracy index and a professional field accuracy index.

8. The large language model evaluation method according to claim 7, characterized in that: The data set corresponding to the general field is a general data set, and the data set corresponding to the professional field is a dedicated data set.

9. The large language model evaluation method according to claim 8, characterized in that: The method for constructing objective question cases in the dedicated data set includes: Acquire at least one target keyword from at least one document in a professional field; Extracting original text containing the target keyword from the document, and hiding the target keyword in the original text; Using a natural language processing model to predict the original text in which the target keyword is hidden, to obtain at least one predicted answer; An objective question case of a dedicated data set is constructed by taking the original text with the target keyword hidden as the question stem, and taking the target keyword and at least one predicted answer different from the target keyword as options.

10. The large language model evaluation method according to claim 9, characterized in that: The acquiring at least one target keyword from at least one document in a professional field includes: Acquire a number of candidate keywords from at least one document in a professional field; For each candidate keyword, calculate the frequency of occurrence of the candidate keyword in the corresponding document, and calculate the proportion of the number of documents containing the candidate keyword in the document collection; Calculate the importance value of each candidate keyword in the corresponding document according to the occurrence frequency and the document proportion; The candidate keywords of each document in the document set are sorted according to the importance value, and the first K candidate keywords in each document are extracted as target keywords; wherein K≥1, and K is an integer.

11. The large language model evaluation method according to claim 1, characterized in that: The capability assessment also includes tool call capability assessment; then, the capability assessment of the large language model is performed to obtain corresponding assessment indicators, including: Inputting a plurality of questions containing reference keywords into the large language model; wherein each reference keyword corresponds to the call of at least one query tool; Record the effective number of successful calls to the tool corresponding to the reference keyword; The ratio of the effective number of times to the total number of calls is used as the tool call index of the large language model.

12. A large language model evaluation device, characterized in that: include: A capability assessment module, used to conduct capability assessment on the large language model and obtain corresponding assessment indicators; wherein the capability assessment includes subjective question answering capability assessment, and the subjective question answering capability assessment includes at least one of consistency assessment between model questions and model answers and consistency assessment between model answers and reference answers; An evaluation result generating module is used to calculate an evaluation score of the large language model according to at least one evaluation indicator, and use the evaluation score as the evaluation result of the large language model.

13. A large language model evaluation device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the large language model evaluation method as described in any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model evaluation method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the large language model evaluation method as described in any one of claims 1 to 11.

Images