Quality evaluation method and device of generative large model, storage medium and equipment

By generating text grades and converting them into scores, the problem of inaccurate quality assessment of generative large models is solved, the stability and accuracy of the assessment are improved, the training process is simplified, and dataset expansion is supported.

CN119005132AActive Publication Date: 2024-11-22GUANGDONG UCAP INTERNET INFORMATION TECH

Patent Information

Application Number
CN202410989589.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-22
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

Existing generative large model quality assessment methods lack a holistic semantic understanding of the assessment answers, leading to inaccurate assessments.

Method used

By obtaining triples and using a pre-trained quality assessment model, the prompts for each assessment dimension are processed to generate text levels and convert them into scores. Finally, a comprehensive quality score is calculated, and the Log-Softmax function is used to improve training efficiency and accuracy.

Benefits of technology

It enables scoring from multiple evaluation dimensions, improving the stability and accuracy of generative large model quality evaluation, simplifying the training process, and supporting dataset expansion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005132A_ABST
    Figure CN119005132A_ABST
Patent Text Reader

Abstract

The invention discloses a quality evaluation method and device of a generative large model, a storage medium and equipment, and belongs to the technical field of deep learning. Obtaining a triple, wherein the triple comprises a question, a standard answer and a to-be-evaluated answer generated by the generative large model; for each evaluation dimension in n evaluation dimensions, generating a cue word for the triple according to the cue word template corresponding to the evaluation dimension, n being greater than or equal to 2; processing the cue word corresponding to each evaluation dimension by using a quality evaluation model, generating a text grade corresponding to each evaluation dimension, and converting the text grade into a corresponding score; the n scores are comprehensively calculated, the quality score of the to-be-evaluated answer is obtained, and the quality score is used for reflecting the quality of the answer generated by the generative large model. According to the invention, scoring can be carried out from multiple evaluation dimensions, so that the final quality evaluation is more stable and comprehensive, and the accuracy of quality evaluation is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a quality assessment method, device, storage medium and equipment for a generative large model. Background Art

[0002] With the rapid development of natural language processing technology, large language model technology is becoming more and more common. Large language models are generative large models trained with large data sets. They have powerful functions, including recognition, summarization, translation, prediction, and generation of text content. Generative large models have demonstrated outstanding capabilities in many tasks. However, in actual applications, how to evaluate the quality of generative large models is also crucial for us.

[0003] In the related technology, a computer device can calculate evaluation indicators based on the standard answer and the answer to be evaluated, such as the Bilingual Evaluation Understudy (BLEU) indicator, the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) indicator, etc., and then calculate the score based on the evaluation indicators. However, this type of evaluation lacks a grasp of the overall semantics of the answer to be evaluated, resulting in inaccurate evaluation. Summary of the invention

[0004] The present application provides a quality assessment method, device, storage medium and equipment for a generative large model, which is used to solve the problem that the scores calculated according to traditional evaluation indicators lack the grasp of the overall semantics of the answers to be evaluated, resulting in inaccurate evaluation. The technical solution is as follows:

[0005] According to a first aspect of the present application, a quality assessment method for a generative large model is provided, the method comprising:

[0006] Acquire a triplet, wherein the triplet includes a question, a standard answer to the question, and an answer to be evaluated generated by a generative large model according to the question;

[0007] For each evaluation dimension of n evaluation dimensions, generating a prompt word for the triple according to a prompt word template corresponding to the evaluation dimension, n≥2;

[0008] Using a pre-trained quality assessment model to process the prompt words corresponding to each assessment dimension, respectively, generate a text grade corresponding to each assessment dimension, and convert the text grade into a corresponding score, wherein the text grade is one of a plurality of grades divided in a region from excellent to poor and described in words;

[0009] The n scores are comprehensively calculated to obtain a quality score of the answer to be evaluated, and the quality score is used to reflect the quality of the answer generated by the generative large model.

[0010] In a possible implementation, the using of a pre-trained quality assessment model to process the prompt words corresponding to each assessment dimension respectively includes:

[0011] Using the quality assessment model to obtain the assessment rules of the corresponding assessment dimension from each prompt word;

[0012] The quality assessment model is used to process the triples corresponding to the assessment dimensions according to the assessment rules.

[0013] In a possible implementation, converting the text grade into a corresponding score includes:

[0014] Using the quality assessment model to obtain a preset mapping relationship, the mapping relationship includes a corresponding relationship between each text level and each score;

[0015] The quality assessment model is used to search the mapping relationship for a score corresponding to the generated text grade.

[0016] In a possible implementation, the method further includes:

[0017] Acquire a training data set, wherein each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels;

[0018] For each evaluation dimension of the n evaluation dimensions, generating a prompt word for the triple according to a prompt word template corresponding to the evaluation dimension;

[0019] Create quality assessment models;

[0020] For each training sample, the quality assessment model is used to process n prompt words corresponding to the training sample to obtain n prediction scores;

[0021] A preset loss function is used to perform loss calculation on the n actual scores and the n predicted scores, and the quality assessment model is fine-tuned and trained according to the calculation results.

[0022] In a possible implementation, the using the quality assessment model to process the n prompt words corresponding to the training sample to obtain n prediction scores includes:

[0023] Using the quality assessment model to process the n prompt words corresponding to the training sample, to obtain n predicted text grades corresponding to the n assessment dimensions;

[0024] The n predicted text levels are converted into corresponding n prediction scores.

[0025] In one possible implementation, the loss function is Log-Softmax.

[0026] According to a second aspect of the present application, a quality assessment device for a generative large model is provided, the device comprising:

[0027] An acquisition module, used to acquire a triple, wherein the triple includes a question, a standard answer to the question, and an answer to be evaluated generated by a generative large model according to the question;

[0028] A generating module, for generating a prompt word for the triplet according to a prompt word template corresponding to the evaluation dimension for each evaluation dimension of n evaluation dimensions, where n≥2;

[0029] A scoring module, for processing the prompt words corresponding to each evaluation dimension respectively using a pre-trained quality assessment model, generating a text grade corresponding to each evaluation dimension, and converting the text grade into a corresponding score, wherein the text grade is one of a plurality of grades divided in a region from excellent to poor and described in words;

[0030] An evaluation module is used to perform comprehensive calculations on the n scores to obtain a quality score of the answer to be evaluated, wherein the quality score is used to reflect the quality of the answer generated by the generative large model.

[0031] In a possible implementation, the acquisition module is further used to acquire a training data set, each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels;

[0032] The generating module is further configured to generate a prompt word for the triplet according to a prompt word template corresponding to the evaluation dimension for each of the n evaluation dimensions;

[0033] Create a module for creating a quality assessment model;

[0034] A scoring module, for processing, for each training sample, n prompt words corresponding to the training sample using the quality assessment model to obtain n prediction scores;

[0035] The training module is used to perform loss calculation on the n actual scores and the n predicted scores using a preset loss function, and to perform fine-tuning training on the quality assessment model according to the calculation results.

[0036] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the quality assessment method of the generative large model as described above.

[0037] According to a fourth aspect of the present application, a computer device is provided, wherein the computer device includes the quality assessment device of the above-mentioned generative large model.

[0038] The beneficial effects of the technical solution provided by this application include at least:

[0039] Fine-tuning is training based on the pre-trained large oracle model. Since the quality assessment model is a large language model, fine-tuning the quality assessment model can fully utilize the capabilities of the large oracle model, simplify the training process, and improve training efficiency. In addition, when fine-tuning the quality assessment model based on the labeled training data set, the training data set can be expanded according to user needs, thereby realizing the expansion of the quality assessment model.

[0040] The quality assessment model can generate n scores for n assessment dimensions, and then comprehensively calculate the n scores to obtain the quality score, thus realizing scoring from multiple assessment dimensions, making the final quality assessment more stable and comprehensive, and improving the accuracy of quality assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a method for training a quality assessment model according to an embodiment of the present application;

[0043] Figure 2 is a flow chart of a quality assessment method for a generative large model provided by an embodiment of the present application;

[0044] Figure 3 is a flow chart of a quality assessment method for a generative large model provided by an embodiment of the present application;

[0045] Figure 4It is a structural block diagram of a quality assessment device for a generative large model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, it shows a method flow chart of a quality assessment method of a generative large model provided by an embodiment of the present application, and the quality assessment method of the generative large model can be applied to a computer device. The quality assessment method of the generative large model may include:

[0048] Step 101, obtaining a training data set, each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels.

[0049] A generative big model is a model used to answer input questions. In order to evaluate the quality of a generative big model, we can use the answer generated by the generative big model as the answer to be evaluated, form a triple of the question, the standard answer, and the answer to be evaluated, and use the quality assessment model to score the quality of the answer to be evaluated. If the quality score of the answer to be evaluated is high, it is determined that the quality of the corresponding generative big model is high; if the quality score of the answer to be evaluated is low, it is determined that the quality of the corresponding generative big model is poor. In this way, users can choose the generative big model they need according to the quality of each generative big model.

[0050] The triples in the training samples can include any questions. For example, the question is "What is the relationship between Xiao Ming and Xiao Gang?" The standard answer is "Xiao Ming and Xiao Gang are the chairmen of two well-known domestic game companies." The answer to be evaluated by a generative large model is "Xiao Ming and Xiao Gang are cousins."

[0051] In addition to triples, the training samples also include n annotated actual scores. Where n is the number of evaluation dimensions, and n≥2. The content and number of evaluation dimensions can be set and adjusted according to actual business needs. For example, n=2, and the two evaluation dimensions include the fluency of the sentence and whether there are factual errors in the sentence; for another example, n=5, and the five evaluation dimensions include the fluency of the sentence, whether there are factual errors in the sentence, whether the sentence is complete, whether the sentence is timely, and whether the sentence is irrelevant.

[0052] The computer device can set multiple scores for each evaluation dimension, and the number of scores can be set according to business needs. After a large number of experiments, we found that the quality assessment model is not particularly accurate in judging numbers. It may not be able to distinguish the meaning of specific numbers 1, 2, 3, 4, and 5. This results in the final scoring stage. The scores output by the quality assessment model may be somewhat arbitrary, such as 5 points for low-quality sentences and 1 point for high-quality sentences. In order to solve this problem, this application maps the specific scores to the text level described in words when training the quality assessment model. That is, one evaluation dimension corresponds to m scores, and m scores correspond to m text levels. For example, m is 5, 1 point corresponds to poor, 2 points are poor, 3 points correspond to general, 4 points correspond to good, and 5 points correspond to excellent. The quality assessment model trained in this way will not give specific numbers when scoring, but will give corresponding text levels, and then convert the text levels into corresponding scores, which can effectively avoid the problem of hallucination of numbers by the quality assessment model and make the scoring more accurate.

[0053] For each triple, the computer device can annotate the text grade and score of the answer to be evaluated from each evaluation dimension, so a triple can be annotated with n text grades and n actual scores. Still taking the question "What is the relationship between Xiao Ming and Xiao Gang?", the standard answer is "Xiao Ming and Xiao Gang are the chairmen of two well-known domestic game companies.", and the answer to be evaluated is "Xiao Ming and Xiao Gang are cousins." For example, in the evaluation dimension of "sentence fluency", the text grade can be annotated as excellent, and the actual score is 5 points; in the evaluation dimension of "whether there are factual errors in the sentence", the text grade can be annotated as poor, and the actual score is 1 point.

[0054] In this embodiment, assuming that n is 5, a training sample may include a triple and the following content:

[0055] 1. Evaluation dimension 1, text level 1, actual score 1;

[0056] 2. Evaluation dimension 2, text level 2, actual score 2;

[0057] 3. Evaluation dimension 3, text level 3, actual score 3;

[0058] 4. Evaluation dimension 4, text level 4, actual score 4;

[0059] 5. Evaluation dimension 5, text level 5, actual score 5.

[0060] Step 102: for each of the n evaluation dimensions, generate a prompt word for the triple according to the prompt word template corresponding to the evaluation dimension.

[0061] Each evaluation dimension corresponds to a prompt word template to prompt the quality evaluation model from which dimension to score the answer to be evaluated. When there are n evaluation dimensions, the computer device can generate n prompt words.

[0062] Taking the evaluation dimension of "whether the statement contains factual errors" as an example, the generated prompt words can be: Please evaluate the quality score of the answer to be evaluated 2 from the perspective of "whether the statement contains factual errors" based on the question and standard answer 1. The score range of each item is an integer score from 0 to 5 points. The scoring standard reference is: {scoring standard for factual errors}. Below are the question, standard answer 1, and answer to be evaluated 2. Question: {question_i}\nStandard answer 1: {answer_base_i}\nAnswer to be evaluated 2: {answer_llm_i}. \nPlease give the score in the format of: x points.

[0063] Step 103: Create a quality assessment model.

[0064] The large quality assessment model is pre-trained into a large language model, and the pre-training is unsupervised training.

[0065] Step 104: for each training sample, the quality assessment model is used to process the n prompt words corresponding to the training sample to obtain n prediction scores.

[0066] Each training sample corresponds to n prompt words, and the quality assessment model generates a prediction score for each prompt word, and finally n prediction scores can be obtained.

[0067] Among them, using the quality assessment model to process the n prompt words corresponding to the training sample to obtain n prediction scores can include: using the quality assessment model to process the n prompt words corresponding to the training sample to obtain n prediction text levels corresponding to n evaluation dimensions; converting the n prediction text levels into corresponding n prediction scores.

[0068] For example, in the evaluation dimension of "sentence fluency", if the predicted text grade generated by the quality assessment model is excellent, then the corresponding prediction score is determined to be 5 points.

[0069] Step 105: Calculate the losses of the n actual scores and the n predicted scores using a preset loss function, and fine-tune the quality assessment model according to the calculation results.

[0070] The raw output of a large language model (usually an unprocessed score or log-odds) needs to be converted into an interpretable probability distribution. This conversion is usually achieved through the loss function Softmax, which can convert any real-valued vector into a probability distribution so that the value of each element is between (0,1) and the sum of all elements is 1. However, when it comes to large-scale models and high-dimensional outputs, directly using Softmax may face problems of numerical instability and low computational efficiency. To solve this problem, we set the loss function of the quality assessment model to Log-Softmax. The Log-Softmax function provides an effective way to directly calculate the logarithm of the probability, thereby enhancing the numerical stability and computational efficiency of the large language model when processing large-scale data.

[0071] The definition of the Log-Softmax function is as follows:

[0072]

[0073] Among them, z i is the i-th element of the input vector and K is the dimension of the vector.

[0074] like Figure 2 As shown, it shows a method flow chart of a quality assessment method of a generative large model provided by an embodiment of the present application, and the quality assessment method of the generative large model can be applied to a computer device. The quality assessment method of the generative large model may include:

[0075] Step 201, obtain a triple, which includes a question, a standard answer to the question, and an answer to be evaluated generated by the generative large model according to the question.

[0076] Triples can include any questions. For example, the question is "What is the relationship between Xiao Ming and Xiao Gang?", the standard answer is "Xiao Ming and Xiao Gang are the chairmen of two well-known domestic game companies.", and the answer to be evaluated by a generative large model is "Xiao Ming and Xiao Gang are cousins.".

[0077] Step 202: for each evaluation dimension of the n evaluation dimensions, generate a prompt word for the triple according to the prompt word template corresponding to the evaluation dimension, where n≥2.

[0078] Each evaluation dimension corresponds to a prompt word template to prompt the quality evaluation model from which dimension to score the answer to be evaluated. When there are n evaluation dimensions, the computer device can generate n prompt words.

[0079] Taking the evaluation dimension of "whether the statement contains factual errors" as an example, the generated prompt words can be: Please evaluate the quality score of the answer to be evaluated 2 from the perspective of "whether the statement contains factual errors" based on the question and standard answer 1. The score range of each item is an integer score from 0 to 5 points. The scoring standard reference is: {scoring standard for factual errors}. Below are the question, standard answer 1, and answer to be evaluated 2. Question: {question_i}\nStandard answer 1: {answer_base_i}\nAnswer to be evaluated 2: {answer_llm_i}. \nPlease give the score in the format of: x points.

[0080] Step 203, using the pre-trained quality assessment model to process the prompt words corresponding to each assessment dimension respectively, generate a text grade corresponding to each assessment dimension, and convert the text grade into a corresponding score. The text grade is one of a plurality of grades divided in an area from excellent to poor and described in words.

[0081] The quality assessment model in this embodiment can be used Figure 1 The method shown is trained.

[0082] For example, in the evaluation dimension of "sentence fluency", if the predicted text grade generated by the quality assessment model is excellent, then the corresponding prediction score is determined to be 5 points.

[0083] Step 204, comprehensively calculate the n scores to obtain a quality score of the answer to be evaluated, and the quality score is used to reflect the quality of the answer generated by the generative large model.

[0084] The comprehensive calculation may be a weighted calculation or other calculation, which is not limited here.

[0085] In this embodiment, the quality score is positively correlated with the quality of the generative large model, that is, the higher the quality score, the better the quality of the generative large model; the lower the quality score, the worse the quality of the generative large model.

[0086] In summary, the quality assessment method of the generative large model provided in the embodiment of the present application is fine-tuned based on the pre-trained large prediction model. Since the quality assessment model is a large language model, the ability of the large prediction model can be fully utilized by fine-tuning the quality assessment model, simplifying the training process and improving the training efficiency. In addition, when fine-tuning the quality assessment model according to the labeled training data set, the training data set can be expanded according to user needs, thereby realizing the expansion of the quality assessment model.

[0087] The quality assessment model can generate n scores for n assessment dimensions, and then comprehensively calculate the n scores to obtain the quality score, thus realizing scoring from multiple assessment dimensions, making the final quality assessment more stable and comprehensive, and improving the accuracy of quality assessment.

[0088] like Figure 3 As shown, it shows a flow chart of a quality assessment method of a generative large model provided by an embodiment of the present application, and the quality assessment method of the generative large model can be applied to a computer device. The quality assessment method of the generative large model may include:

[0089] Step 301, obtain a triple, which includes a question, a standard answer to the question, and an answer to be evaluated generated by the generative large model according to the question.

[0090] Triples can include any questions. For example, the question is "What is the relationship between Xiao Ming and Xiao Gang?", the standard answer is "Xiao Ming and Xiao Gang are the chairmen of two well-known domestic game companies.", and the answer to be evaluated by a generative large model is "Xiao Ming and Xiao Gang are cousins.".

[0091] Step 302: for each evaluation dimension of the n evaluation dimensions, generate a prompt word for the triple according to the prompt word template corresponding to the evaluation dimension, where n≥2.

[0092] Each evaluation dimension corresponds to a prompt word template to prompt the quality evaluation model from which dimension to score the answer to be evaluated. When there are n evaluation dimensions, the computer device can generate n prompt words.

[0093] Taking the evaluation dimension of "whether the statement contains factual errors" as an example, the generated prompt words can be: Please evaluate the quality score of the answer to be evaluated 2 from the perspective of "whether the statement contains factual errors" based on the question and standard answer 1. The score range of each item is an integer score from 0 to 5 points. The scoring standard reference is: {scoring standard for factual errors}. Below are the question, standard answer 1, and answer to be evaluated 2. Question: {question_i}\nStandard answer 1: {answer_base_i}\nAnswer to be evaluated 2: {answer_llm_i}. \nPlease give the score in the format of: x points.

[0094] Step 303, using the quality assessment model to obtain the assessment rules of the corresponding assessment dimensions from each prompt word; using the quality assessment model to process the triples corresponding to the assessment dimensions according to the assessment rules, generate a text grade corresponding to each assessment dimension, and convert the text grade into a corresponding score. The text grade is one of a plurality of grades divided in a region from excellent to poor and described in words.

[0095] One evaluation dimension corresponds to m scores, and m scores correspond to m text levels. For example, if m is 5, 1 point corresponds to poor, 2 points to poor, 3 points to average, 4 points to good, and 5 points to excellent.

[0096] The evaluation rule is the scoring standard reference mentioned in step 302. The quality evaluation model can process the triples according to the scoring standard corresponding to each evaluation dimension, and obtain n text grades corresponding to n evaluation standards. Then, the quality evaluation model needs to convert the text grades into scores.

[0097] Specifically, converting the text level into a corresponding score may include: using a quality assessment model to obtain a preset mapping relationship, the mapping relationship including a correspondence between each text level and each score; using the quality assessment model to search the mapping relationship for a score corresponding to the generated text level.

[0098] For example, in the evaluation dimension of "sentence fluency", if the predicted text grade generated by the quality assessment model is excellent, then the corresponding prediction score is determined to be 5 points.

[0099] Step 304, comprehensively calculate the n scores to obtain a quality score of the answer to be evaluated, and the quality score is used to reflect the quality of the answer generated by the generative large model.

[0100] The comprehensive calculation may be a weighted calculation or other calculation, which is not limited here.

[0101] In this embodiment, the quality score is positively correlated with the quality of the generative large model, that is, the higher the quality score, the better the quality of the generative large model; the lower the quality score, the worse the quality of the generative large model.

[0102] In summary, the quality assessment method of the generative large model provided in the embodiment of the present application is fine-tuned based on the pre-trained large prediction model. Since the quality assessment model is a large language model, the ability of the large prediction model can be fully utilized by fine-tuning the quality assessment model, simplifying the training process and improving the training efficiency. In addition, when fine-tuning the quality assessment model according to the labeled training data set, the training data set can be expanded according to user needs, thereby realizing the expansion of the quality assessment model.

[0103] The quality assessment model can generate n scores for n assessment dimensions, and then comprehensively calculate the n scores to obtain the quality score, thus realizing scoring from multiple assessment dimensions, making the final quality assessment more stable and comprehensive, and improving the accuracy of quality assessment.

[0104] like Figure 4As shown, it shows a structural block diagram of a quality assessment device for a generative large model provided by an embodiment of the present application, and the quality assessment device for a generative large model can be applied to a computer device. The quality assessment device for a generative large model may include:

[0105] An acquisition module 410 is used to acquire a triple, where the triple includes a question, a standard answer to the question, and an answer to be evaluated generated by the generative large model according to the question;

[0106] A generating module 420 is used to generate a prompt word for each of the n evaluation dimensions according to the prompt word template corresponding to the evaluation dimension for the triple, where n≥2;

[0107] Scoring module 430, used to process the prompt words corresponding to each evaluation dimension respectively using the pre-trained quality assessment model, generate a text grade corresponding to each evaluation dimension, and convert the text grade into a corresponding score, where the text grade is one of a plurality of grades divided in a region from excellent to poor and described in words;

[0108] The evaluation module 440 is used to perform comprehensive calculations on the n scores to obtain a quality score of the answer to be evaluated. The quality score is used to reflect the quality of the answer generated by the generative large model.

[0109] In an optional embodiment, the scoring module 430 is further configured to:

[0110] The quality assessment model is used to obtain the assessment rules of the corresponding assessment dimensions from each prompt word;

[0111] The quality assessment model is used to process the triples corresponding to the assessment dimensions according to the assessment rules.

[0112] In an optional embodiment, the scoring module 430 is further configured to:

[0113] A preset mapping relationship is obtained by using a quality assessment model, wherein the mapping relationship includes a correspondence between each text level and each score;

[0114] The quality assessment model is used to find the score corresponding to the generated text level in the mapping relationship.

[0115] In an optional embodiment, the acquisition module 410 is further used to acquire a training data set, each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels;

[0116] The generating module 420 is further configured to generate a prompt word for the triple according to the prompt word template corresponding to the evaluation dimension for each evaluation dimension of the n evaluation dimensions;

[0117] Create a module for creating a quality assessment model;

[0118] Scoring module 430, for processing n prompt words corresponding to each training sample using the quality assessment model to obtain n prediction scores;

[0119] The training module is used to calculate the loss of n actual scores and n predicted scores using a preset loss function, and to fine-tune the quality assessment model based on the calculation results.

[0120] In an optional embodiment, the scoring module 430 is further configured to:

[0121] The quality assessment model is used to process the n prompt words corresponding to the training samples to obtain n predicted text grades corresponding to the n assessment dimensions;

[0122] Convert n predicted text levels into corresponding n predicted scores.

[0123] In an optional embodiment, the loss function is Log-Softmax.

[0124] In summary, the quality assessment device of the generative large model provided in the embodiment of the present application is fine-tuned based on the pre-trained large prediction model. Since the quality assessment model is a large language model, the ability of the large prediction model can be fully utilized by fine-tuning the quality assessment model, simplifying the training process and improving the training efficiency. In addition, when fine-tuning the quality assessment model according to the labeled training data set, the training data set can be expanded according to user needs, thereby realizing the expansion of the quality assessment model.

[0125] The quality assessment model can generate n scores for n assessment dimensions, and then comprehensively calculate the n scores to obtain the quality score, thus realizing scoring from multiple assessment dimensions, making the final quality assessment more stable and comprehensive, and improving the accuracy of quality assessment.

[0126] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the quality assessment method of the generative large model as described above.

[0127] An embodiment of the present application provides a computer device, which includes a quality assessment device for any of the above-mentioned generative large models.

[0128] It should be noted that: the quality assessment device for the generative large model provided in the above embodiment only uses the division of the above functional modules as an example when performing the quality assessment of the generative large model. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the quality assessment device for the generative large model is divided into different functional modules to complete all or part of the functions described above. In addition, the quality assessment device for the generative large model provided in the above embodiment and the quality assessment method embodiment of the generative large model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0129] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0130] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A quality assessment method for a generative large model, characterized in that: The method comprises: Acquire a triplet, wherein the triplet includes a question, a standard answer to the question, and an answer to be evaluated generated by a generative large model according to the question; For each evaluation dimension of n evaluation dimensions, generating a prompt word for the triple according to a prompt word template corresponding to the evaluation dimension, n≥2; Using a pre-trained quality assessment model to process the prompt words corresponding to each assessment dimension, respectively, generate a text grade corresponding to each assessment dimension, and convert the text grade into a corresponding score, wherein the text grade is one of a plurality of grades divided in a region from excellent to poor and described in words; The n scores are comprehensively calculated to obtain a quality score of the answer to be evaluated, and the quality score is used to reflect the quality of the answer generated by the generative large model.

2. The quality assessment method of a generative large model according to claim 1, characterized in that: The method of using the pre-trained quality assessment model to process the prompt words corresponding to each assessment dimension respectively includes: Using the quality assessment model to obtain the assessment rules of the corresponding assessment dimension from each prompt word; The quality assessment model is used to process the triples corresponding to the assessment dimensions according to the assessment rules.

3. The quality assessment method of a generative large model according to claim 1, characterized in that: The converting the text grade into a corresponding score comprises: Using the quality assessment model to obtain a preset mapping relationship, the mapping relationship includes a corresponding relationship between each text level and each score; The quality assessment model is used to search the mapping relationship for a score corresponding to the generated text grade.

4. The quality assessment method of a generative large model according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a training data set, wherein each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels; For each evaluation dimension of the n evaluation dimensions, generating a prompt word for the triple according to a prompt word template corresponding to the evaluation dimension; Create quality assessment models; For each training sample, the quality assessment model is used to process n prompt words corresponding to the training sample to obtain n prediction scores; A preset loss function is used to perform loss calculation on the n actual scores and the n predicted scores, and the quality assessment model is fine-tuned and trained according to the calculation results.

5. The quality assessment method of a generative large model according to claim 4, characterized in that: The process of using the quality assessment model to process the n prompt words corresponding to the training sample to obtain n prediction scores includes: Using the quality assessment model to process the n prompt words corresponding to the training sample, to obtain n predicted text grades corresponding to the n assessment dimensions; The n predicted text levels are converted into corresponding n prediction scores.

6. The quality assessment method of a generative large model according to claim 4, characterized in that: The loss function is Log-Softmax.

7. A quality assessment device for a generative large model, characterized in that: The device comprises: An acquisition module, used to acquire a triple, wherein the triple includes a question, a standard answer to the question, and an answer to be evaluated generated by a generative large model according to the question; A generating module, for generating a prompt word for the triplet according to a prompt word template corresponding to the evaluation dimension for each evaluation dimension of n evaluation dimensions, where n≥2; A scoring module, for processing the prompt words corresponding to each evaluation dimension respectively using a pre-trained quality assessment model, generating a text grade corresponding to each evaluation dimension, and converting the text grade into a corresponding score, wherein the text grade is one of a plurality of grades divided in a region from excellent to poor and described in words; An evaluation module is used to perform comprehensive calculations on the n scores to obtain a quality score of the answer to be evaluated, wherein the quality score is used to reflect the quality of the answer generated by the generative large model.

8. The quality assessment device for a generative large model according to claim 7, characterized in that: The acquisition module is further used to acquire a training data set, each training sample in the training data set includes a triple and n actual scores, the triple includes a question, a standard answer to the question and an answer to be evaluated, and the n actual scores correspond to n actual text levels; The generating module is further configured to generate a prompt word for the triplet according to a prompt word template corresponding to the evaluation dimension for each of the n evaluation dimensions; Create a module for creating a quality assessment model; A scoring module, for processing, for each training sample, n prompt words corresponding to the training sample using the quality assessment model to obtain n prediction scores; The training module is used to perform loss calculation on the n actual scores and the n predicted scores using a preset loss function, and to perform fine-tuning training on the quality assessment model according to the calculation results.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the quality assessment method of the generative large model as described in any one of claims 1 to 6.

10. A computer device, characterized in that: The computer device includes: a quality assessment device for a generative large model as described in claim 7 or 8.

Citation Information

Patent Citations

  • Content and form diversity fused Chinese question generation method and system

    CN114970563A

  • Quality evaluation method, system and equipment of multi-modal large model and storage medium

    CN117909702A

  • Large-model-oriented multi-dimensional question and answer pair generation task evaluation method and system

    CN118093344A

  • Generative large language model selection question evaluation method and system

    CN118113845A

  • System and method for automatic quality management and coaching

    US20180096278A1

Cited By

  • Training method and device for credit evaluation model of specific person, medium and equipment

    CN119722099A

  • Dialogue model evaluation and dialogue remodeling method, equipment and medium

    CN120012946A

  • Data processing method and device, electronic equipment and computer readable storage medium

    CN121051405A