Large model generation content evaluation method and device, medium and equipment

Abstract

The embodiment of the invention provides a large model generation content evaluation method, device and equipment. A specific embodiment of the method comprises the steps of obtaining a target cue word, and inputting the target cue word into a target large model to obtain first generation content; auxiliary prompts are constructed, the auxiliary prompts comprise a first prompt and a second prompt, the first prompt indicates a scoring standard for the generated content, and the second prompt indicates a reference answer for the target prompt word; constructing a request scoring prompt word according to the first generation content, the target prompt word and the auxiliary prompt; and inputting the request scoring cue word into the large evaluation model to obtain a first score for the first generation content. Through the method, the generation content of the large model can be accurately evaluated in an automatic mode, the evaluation efficiency is improved, and the evaluation cost is reduced.

Description

Technical Field

[0001] This disclosure relates to the field of large model technology, and in particular to a method, apparatus, medium and device for evaluating the content generated by large models. Background Technology

[0002] With the development of large model technology, content generated by large models has been widely used across various industries. To determine the quality of content generated by large models, evaluation can be conducted. However, due to the often divergent and unstable nature of large model-generated content, accurate evaluation using conventional automated content parsing methods is difficult. Therefore, the evaluation of large model-generated content currently relies on manual methods. However, manual evaluation not only requires a significant amount of human labor, but the accuracy of the evaluation results is also constrained by subjective factors such as the experience and knowledge of the personnel. Summary of the Invention

[0003] This disclosure describes a method, apparatus, medium, and device for evaluating large model-generated content.

[0004] Based on the first aspect, a method for evaluating content generated by large models is provided, including:

[0005] Obtain target prompt words, input the target prompt words into the target large model to obtain the first generated content; construct auxiliary prompts, the auxiliary prompts include a first prompt and a second prompt, the first prompt indicating the scoring criteria for the generated content, and the second prompt indicating the reference answer for the target prompt words;

[0006] Based on the first generated content, the target prompt word, and the auxiliary prompt, a request rating prompt word is constructed; the request rating prompt word is input into the evaluation model to obtain a first rating for the first generated content.

[0007] According to the second aspect, an evaluation device for large model-generated content is provided, comprising:

[0008] The construction unit is configured to: acquire target prompt words, input the target prompt words into a target large model to obtain first generated content; and construct auxiliary prompts, wherein the auxiliary prompts include a first prompt and a second prompt, the first prompt indicating a scoring criterion for the generated content and the second prompt indicating a reference answer for the target prompt words.

[0009] The scoring unit is configured to construct a request scoring prompt based on the first generated content, the target prompt, and the auxiliary prompt; and input the request scoring prompt into the evaluation model to obtain a first score for the first generated content.

[0010] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0011] According to a fourth aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the first aspect.

[0012] The present disclosure provides an apparatus, device, and medium. First, a target prompt word can be obtained and input into a target large-scale model to obtain first generated content. Then, auxiliary prompts are constructed, including a first prompt and a second prompt. The first prompt indicates the scoring criteria for the generated content, and the second prompt indicates a reference answer for the target prompt word. Next, based on the first generated content, the target prompt word, and the auxiliary prompts, a request-score prompt word is constructed. This request-score prompt word is input into the evaluation large-scale model to obtain a first score for the first generated content. This method allows for accurate and automated evaluation of the generated content of the large-scale model, improving evaluation efficiency and reducing evaluation costs. Attached Figure Description

[0013] Figure 1 A schematic diagram of the evaluation scheme for content generated by a large model is shown;

[0014] Figure 2 A schematic diagram of an evaluation method for large model-generated content according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A flowchart illustrating an evaluation method for large model-generated content according to an embodiment of the present disclosure is shown.

[0016] Figure 4 A schematic diagram illustrating scoring using multiple evaluation sub-models according to an embodiment of this disclosure is shown;

[0017] Figure 5 A schematic diagram illustrating a method for obtaining the scoring weights of the evaluation sub-model according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic block diagram of a training data determination apparatus for a large model for generating prompt words according to an embodiment of the present disclosure is shown;

[0019] Figure 7 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown;

[0020] Figure 8 A schematic diagram of the structure of a storage medium suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0021] The technical solutions provided in this specification will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0022] In the description of the implementations disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one / an implementation" or "the implementation" should be understood as "at least one / an implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below.

[0023] As mentioned earlier, with the rapid development of large model technology, content generated by large models has been widely used across various industries. To determine the quality of content generated by large models, it is necessary to evaluate it. However, due to the often divergent and unstable nature of large model-generated content, accurate evaluation using conventional automated content parsing methods is difficult. Therefore, the evaluation of large model-generated content currently relies on manual methods. Figure 1 A schematic diagram of the evaluation scheme for content generated by a large model is shown. For example... Figure 1 As shown, prompts can typically be input into a large model to obtain the generated content, which is then evaluated by humans to obtain a score for the content generated by the large model.

[0024] However, manual evaluation has the following problems: First, it consumes a large amount of manual work and is time-consuming for each evaluation. Second, it relies heavily on the subjective judgment of the evaluators. Even with pre-set evaluation standards, different evaluators often have different understandings of these standards. For the same generated content, different evaluators may produce significantly different results, making it difficult to objectively unify the evaluation results. Third, due to the "illusion problem" of large models—that is, large models often generate unfounded or factually inaccurate content—accurately judging the generated false content often depends heavily on the evaluators' knowledge and judgment, further increasing the difficulty of manual evaluation. Fourth, the rapid development of large model technology leads to rapid iteration of the models themselves. After each iteration, the model often needs to be re-evaluated, further increasing the total cost of frequent manual evaluation.

[0025] To address the aforementioned technical problems, this disclosure provides a method for evaluating the content generated by a large model.

[0026] Figure 2 A schematic diagram of an evaluation method for large model-generated content according to an embodiment of the present disclosure is shown. Figure 2 As shown, in some embodiments, for example, the target prompt word can be input into the target large model being tested to obtain the output generated content. Then, based on the target prompt word, the generated content, and auxiliary prompts, a request evaluation prompt word is constructed. The auxiliary prompts include scoring criteria for the generated content and a reference answer for the target prompt word. Then, the request evaluation prompt word is input into the evaluation large model to obtain the output quality score for the generated content. In another embodiment, the auxiliary prompts may also include historical generated content related to the target prompt word and historical scores for that historical generated content, as well as one or more of the question categories to which the target prompt word belongs.

[0027] The advantages of this method are as follows: First, it can automatically evaluate the content generated by the target large model through a large evaluation model, which greatly improves the evaluation efficiency of generated content compared to manual evaluation, and reduces the labor and time costs. Second, by adding auxiliary prompts, such as scoring criteria for the generated content and reference answers for the target prompts, to the evaluation request prompts input to the large evaluation model, the accuracy and stability of the quality scores output by the large evaluation model can be improved, especially the ability to evaluate false content generated by the large model. Third, in some embodiments, multiple scores can be generated for the same generated content based on multiple evaluation sub-models and multiple prompt models, and the multiple scores can be weighted and summed to obtain a total score, further improving the accuracy and generalization ability of the generated content evaluation.

[0028] The following describes the detailed process of this method.

[0029] Figure 3 A flowchart illustrating an evaluation method for large model-generated content according to an embodiment of this disclosure is shown. Figure 3 As shown, the method includes at least the following steps:

[0030] Step S301: Obtain target prompt words, input the target prompt words into the target large model to obtain first generated content; construct auxiliary prompts, the auxiliary prompts include a first prompt and a second prompt, the first prompt indicating the scoring criteria for the generated content, and the second prompt indicating the reference answer for the target prompt words;

[0031] Step S303: Construct a request rating prompt word based on the first generated content, the target prompt word, and the auxiliary prompt; input the request rating prompt word into the evaluation model to obtain a first rating for the first generated content.

[0032] First, in step S301, target prompt words are obtained and input into the target large model to obtain first generated content. Then, auxiliary prompts are constructed, including a first prompt and a second prompt. The first prompt indicates the scoring criteria for the generated content, and the second prompt indicates a reference answer for the target prompt words.

[0033] A large model typically refers to an artificial intelligence model with hundreds of millions or more parameters, pre-trained on massive datasets. A prompt is the text information input to the large model, intended to guide it in generating corresponding content based on the prompt. In different embodiments, the target large model can be of different specific types or with different neural network structures, and the target prompt can also be different specific prompts; this specification does not impose any limitations on this.

[0034] In this step, the target prompt word can be input into the target large model to obtain the first generated content. In different embodiments, depending on the target prompt word, the first generated content can be different specific content in different modalities. In one embodiment, the modality of the first generated content can be, for example, one of the following: text modality, image modality, video modality, and audio modality. In one example, the target prompt word could be, for example, "In which dynasty was Li Si born?", and the first generated content could be, for example, "Li Si was a famous poet during the Tang Dynasty of China. Therefore, Li Si was born in the Tang Dynasty."

[0035] Auxiliary prompts are used to input the evaluation model in subsequent steps to improve the evaluation effect of the model's output. The auxiliary prompts may include a first prompt and a second prompt, wherein the first prompt indicates the scoring criteria for the generated content. In different embodiments, the specific scoring criteria indicated by the first prompt may differ. In one embodiment, the first prompt may be, for example, "5 points: logically correct, accurate and fluent expression; 4 points: logically basically correct, expression without errors; 3 points: logically basically correct, but key content is not responded to; 2 points: logical flaws, part of the result is incorrect; 1 point: logical flaws, part of the content is misleading; 0 points: false content."

[0036] The second prompt indicates a reference answer for the target prompt word. The parameter answer is the correct answer for the target prompt word. In different embodiments, the specific reference answer indicated by the second prompt may also differ depending on the target prompt word. In different embodiments, the number of reference answers indicated by the second prompt may also be one or more. In one embodiment, the target prompt word may be, for example, "In which dynasty was Li Si born?", and the second prompt may be, for example, "Reference Answer 1: Li Si was born in the Tang Dynasty. Reference Answer 2: Li Si was born in 701 AD, therefore he was born in the Tang Dynasty."

[0037] In one embodiment, the auxiliary prompt may further include a third prompt, which is obtained through the following steps: based on the target prompt word, retrieving historical prompt words related to the target prompt word, second generated content for the historical prompt words, and a second rating for the second generated content from a preset historical prompt knowledge base; and constructing the third prompt based on the historical prompt words, the second generated content, and the second rating. In different embodiments, the construction method or data storage method of the historical prompt knowledge base may differ, and this specification does not limit this. In different embodiments, depending on the target prompt word, the historical prompt words, second generated content, and second rating retrieved from the historical prompt knowledge base may differ. In different embodiments, the number of combinations of historical prompt words, second generated content, and second rating indicated by the second prompt may also be one or more. In one embodiment, the target prompt could be, for example, "In which dynasty was Li Si born?", and the second prompt could be, for example, "[Similar Case 1] Question: In which dynasty did event x occur? Answer: Event x occurred in the Tang Dynasty of China. Score: 5 points, logically correct, precise and fluent expression. [Similar Case 2] Question: What are the characteristics of Li Si's poetry? Answer: Li Si's poetry is characterized by its unrestrained imagination, ornate language, and passionate expression, showcasing the feelings and perspectives of people from different countries. Score: 1 point, Note: Logically incorrect." Through this method, in subsequent steps, the evaluation model can be guided to generate a first score for the first generated content based on historical prompts related to the target prompt, historical generated content (second generated content) for the historical prompt, and historical scores for the historical content (second score), thereby improving the accuracy of the first score.

[0038] In one embodiment, the auxiliary prompt may further include a fourth prompt. The fourth prompt may indicate the question category corresponding to the target prompt word. In different embodiments, depending on the target prompt word, the specific question category indicated by the fourth prompt word may vary. In one embodiment, the target prompt word may be, for example, "In which dynasty was Li Si born?", and the fourth prompt may be, for example, "Historical knowledge Q&A". Through this method, in subsequent steps, the evaluation model can be guided to generate a first score for the first generated content based on the question category corresponding to the target prompt word, further improving the accuracy of the first score.

[0039] Then, in step S303, a request rating prompt word can be constructed based on the first generated content, target prompt word, and auxiliary prompt obtained in step S301. In different embodiments, the specific content of the constructed request rating prompt word may vary depending on the first generated content, target prompt word, and auxiliary prompt.

[0040] In one embodiment, the constructed request scoring prompt is, for example, "Please refer to the scoring criteria, historical scoring cases, and the standard answer that the model should return to score the current output of the model."

[0041] Scoring criteria: 5 points: Logically correct, accurate and fluent expression; 4 points: Basically logically correct, accurate expression; 3 points: Basically logically correct, but fails to respond to key content; 2 points: Logically flawed, some results are incorrect; 1 point: Obvious logical flaws, some content is misleading; 0 points: False content.

[0042] Historical Scoring Examples: [Similar Example 1] Question: In which dynasty did event x occur? Answer: Event x occurred in the Tang Dynasty of China. Score: 5 points, logically correct, precise and fluent expression. [Similar Example 2] Question: What are the characteristics of Li Si's poetry? Answer: Li Si's poetry features unrestrained imagination, ornate language, and passionate emotions, showcasing the feelings and perspectives of people from different countries. Score: 1 point, Note: Logical error;

[0043] Suggested Answers: Answer 1: Li Si was born in the Tang Dynasty. Answer 2: Li Si was born in 701 AD, therefore he was born in the Tang Dynasty.

[0044] Question type: Historical knowledge question;

[0045] Question: In which dynasty was Li Si born?

[0046] Model output: Li Si was a famous poet during the Tang Dynasty in China. Therefore, Li Si was born in the Tang Dynasty.

[0047] After constructing the request rating prompt, the request rating prompt can be input into the evaluation model to obtain a first rating for the first generated content. In different embodiments, the evaluation model can also be a large model of different specific types or with different neural network structures, and this specification does not limit it.

[0048] In one embodiment, the request rating prompt words can be input into multiple evaluation sub-models to obtain multiple third ratings output by multiple evaluation sub-models. The first rating is determined based on the multiple third ratings and the rating weights corresponding to the multiple evaluation sub-models in this evaluation. Figure 4 A schematic diagram illustrating scoring using multiple evaluation sub-models according to embodiments of the present disclosure is shown. For example... Figure 4 As shown, the request rating prompt can be input into multiple evaluation sub-models (e.g., evaluation sub-model 1, evaluation sub-model 2, and evaluation sub-model 3). Each evaluation sub-model can be a different large model used for evaluating the generated content. The resulting scores are S1 (from evaluation sub-model 1), S2 (from evaluation sub-model 2), and S3 (from evaluation sub-model 3). The first score S is obtained by weighting S1, S2, and S3 as w1·S1 + w2·S2 + w3·S3. Since different evaluation sub-models often have different evaluation levels for different target prompts and generated content in actual generation environments, using multiple evaluation sub-models to score based on the request rating prompt and determining the first score by weighting the scores from each evaluation sub-model results in better generalization performance compared to relying on a single evaluation model. This improves the accuracy of scoring generalized target prompts and generated content.

[0049] In different embodiments, the specific methods for determining the scoring weights corresponding to each of the multiple evaluation sub-models in this evaluation can differ. In one embodiment, the scoring weights corresponding to each evaluation sub-model in this evaluation are obtained through the following steps: inputting the identifiers of each evaluation sub-model, the identifiers of the prompt word templates corresponding to the request scoring prompt words input to each evaluation sub-model in this evaluation, and the identifiers of the question categories corresponding to the target prompt words into a pre-trained weight allocation model to obtain the scoring weights corresponding to each evaluation sub-model in this evaluation. In a specific embodiment, before inputting the identifiers of each evaluation sub-model, the identifiers of the prompt word templates corresponding to the request scoring prompt words input to each evaluation sub-model in this evaluation, and the identifiers of the question categories corresponding to the target prompt words into the weight allocation model, they can be vectorized.

[0050] Specifically, Figure 5 A schematic diagram illustrating a method for obtaining the scoring weights of the evaluation sub-model according to an embodiment of this disclosure is shown. Figure 5As shown, for example, the generated content of the target large model can be evaluated using multiple evaluation sub-models (e.g., evaluation sub-model 1, evaluation sub-model 2, and evaluation sub-model 3). The identifier of evaluation sub-model 1 (e.g., LLMID1), the identifier of the prompt word template corresponding to the request scoring prompt word input to evaluation sub-model 1 in this evaluation (e.g., MID2), and the identifier of the question category corresponding to the target prompt word (e.g., PYID1) can be input into the weight allocation model to obtain the scoring weight w1 of evaluation sub-model 1. Similarly, the identifier of evaluation sub-model 2 (e.g., LLMID2), the identifier of the prompt word template corresponding to the request scoring prompt word input to evaluation sub-model 1 in this evaluation (e.g., MID1), and the identifier of the question category corresponding to the target prompt word (PYID1) can be input into the weight allocation model to obtain the scoring weight w2 of evaluation sub-model 2. The scoring weight w3 of evaluation sub-model 2 can be obtained in a similar manner.

[0051] A cue word template is a template for the statements or instructions contained within a cue word. Due to the richness of natural language, cue words with the same semantic meaning can often have multiple different cue word templates, meaning they can be represented by various different specific statements or instructions. Therefore, inputting different evaluation sub-models or inputting the same evaluation request cue word into the same evaluation sub-model in different evaluation rounds can also correspond to different specific cue word templates, i.e., different specific statements. Since different evaluation sub-models often have different evaluation levels for different question types and different cue word templates in real-world production scenarios, a pre-trained weight allocation model can be used to assign weights to each evaluation sub-model based on the combination of each evaluation sub-model with different question types and cue word templates in this evaluation. This can improve the accuracy of the scoring weights assigned to each evaluation sub-model in this evaluation, and thus improve the accuracy of the first score obtained after weighted calculation.

[0052] In different embodiments, the weight allocation model can be different specific types of neural network models. In one embodiment, the weight allocation model can be a Gradient Boosting Decision Tree (GBDT) model. Predicting the rating weights of the rating sub-model using a gradient boosting decision tree has the advantages of high prediction accuracy and strong robustness.

[0053] In different embodiments, the pre-training method of the weight allocation model can vary. In one embodiment, training samples are input into the weight allocation model to obtain the first predicted weights output by the model. The training samples include an identifier of any prompt word template corresponding to the training prompt words used to evaluate the content generated by the large model, an identifier of the question category corresponding to the prompt words on which the large model generates the content, and an identifier of any evaluation sub-model among the multiple evaluation sub-models. Based on the first predicted weights and the pre-acquired weight labels for the training samples, the weight difference is determined, and the parameters of the weight allocation model are updated based on the weight difference. In this way, the weight allocation model can be trained with only a small amount of pre-labeling resources (weight labels), thereby automatically obtaining the weights of the evaluation sub-models in subsequent content generation evaluations through the trained weight allocation model, reducing the workload of manual weight labeling and improving evaluation efficiency.

[0054] In one embodiment, the weight allocation model can also be pre-trained through the following steps: Training prompts indicating the evaluation of content generated by the large model are input into multiple evaluation sub-models to obtain multiple fourth scores; for each evaluation sub-model, the identifier of the prompt template corresponding to the current input of the training prompts, the identifier of the question category corresponding to the prompts on which the large model's content is based, and the identifier of the evaluation sub-model are input into the weight allocation model to obtain the second predicted weights corresponding to the evaluation sub-model; a predicted score is determined based on the multiple fourth scores and the second predicted weights corresponding to each evaluation sub-model; the score difference is determined based on the predicted score and the pre-acquired score labels; and the parameters of the weight allocation model are updated based on the score difference. In this way, the weight allocation model can be trained using only a small amount of another pre-labeled resource (score labels) without the need for weight labeling. In production scenarios, score labels are often easier to obtain than weight labels, thus further reducing the overall workload of manual labeling.

[0055] Figure 6 A schematic block diagram of an evaluation apparatus for large model-generated content according to an embodiment of the present disclosure is shown. This apparatus is used to perform, for example... Figure 3 The method shown. (As shown) Figure 6 As shown, the device 600 includes:

[0056] The construction unit 601 is configured to: acquire target prompt words; input the target prompt words into a target large model to obtain first generated content; and construct auxiliary prompts, wherein the auxiliary prompts include a first prompt and a second prompt, the first prompt indicating a scoring criterion for the generated content and the second prompt indicating a reference answer for the target prompt words.

[0057] The scoring unit 602 is configured to construct a request scoring prompt word based on the first generated content, the target prompt word, and the auxiliary prompt; and input the request scoring prompt word into the evaluation model to obtain a first score for the first generated content.

[0058] This disclosure also provides an electronic device, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements, for example... Figure 3 The method shown.

[0059] The following can also be referenced Figure 7 It shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiments of this application. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0060] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701. The aforementioned processing device 701 may be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 702 or a program loaded from storage device 708 into random access memory (RAM) 703. RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing device 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0061] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0062] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the evaluation method for large model generation content provided in embodiments of this application.

[0063] This disclosure also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed in a computer, it causes the computer to perform the functions provided in the embodiments of this application. Figure 3 This illustrates a method for evaluating content generated by a large model. Figure 8 This is a schematic diagram illustrating a storage medium for implementing an embodiment of this application. For example, such as... Figure 8 As shown, the storage medium 800 can be a non-transitory computer-readable storage medium used to store non-transitory computer-executable instructions 801. When the non-transitory computer-executable instructions 801 are executed by a processor, they can implement the evaluation method for large model generation content provided in the embodiments of this application. For example, when the non-transitory computer-executable instructions 801 are executed by a processor, one or more steps in the evaluation method for large model generation content provided in the embodiments of this application can be performed. For example, the storage medium 800 can be applied in the above-mentioned electronic device. For example, the storage medium 800 can include the memory in the electronic device. The description of the storage medium 800 can be found in the description of the memory in the embodiments of the electronic device, and will not be repeated here. The specific functions and technical effects of the storage medium 800 can be found in the description of the evaluation method for large model generation content provided in the embodiments of this application, and will not be repeated here.

[0064] It should be noted that the computer-readable medium in the embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a memory card of a smartphone, a storage component of a tablet computer, a portable computer disk, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein computer-readable program code is carried. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0065] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the server, cause the electronic device to implement the evaluation method for large model generation content provided in the embodiments of this application.

[0066] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. The units described in the embodiments of the present disclosure may be implemented in software or hardware. The names of the units do not necessarily constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), Systems-on-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0068] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0069] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this disclosure. Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0070] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of the present invention. Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating content generated by a large model, comprising: Obtain the target prompt words, input the target prompt words into the target large model, and obtain the first generated content; Construct auxiliary prompts, which include a first prompt and a second prompt, wherein the first prompt indicates the scoring criteria for the generated content and the second prompt indicates the reference answer for the target prompt word; Based on the first generated content, the target prompt word, and the auxiliary prompt, construct a request rating prompt word; Input the requested rating prompt into the evaluation model to obtain the first rating for the first generated content.

2. The method according to claim 1, wherein, The auxiliary prompts also include a third prompt; The third prompt is obtained through the following steps: based on the target prompt word, retrieve historical prompt words related to the target prompt word, second generated content for the historical prompt words, and a second score for the second generated content from a preset historical prompt knowledge base; A third prompt is constructed based on the historical prompt words, the second generated content, and the second score.

3. The method according to claim 1, wherein, The auxiliary prompts also include a fourth prompt, which indicates the question category corresponding to the target prompt word.

4. The method according to claim 1, wherein, The evaluation model includes multiple evaluation sub-models; Inputting the requested rating prompts into the large evaluation model yields a first rating for the first generated content, including: The request rating prompts are input into multiple evaluation sub-models to obtain multiple third ratings output by the multiple evaluation sub-models. Based on the multiple third ratings and the rating weights corresponding to each evaluation sub-model in this evaluation, the first rating is determined.

5. The method according to claim 4, wherein, The scoring weights of each evaluation sub-model for this evaluation are obtained through the following steps: the identifiers of each evaluation sub-model, the identifiers of the prompt word templates corresponding to the request scoring prompt words input to each evaluation sub-model for this evaluation, and the identifiers of the question categories corresponding to the target prompt words are input into the pre-trained weight allocation model to obtain the scoring weights of each evaluation sub-model for this evaluation.

6. The method according to claim 5, wherein, The weight allocation model is a gradient boosting decision tree model.

7. The method according to claim 5, wherein, The weight allocation model is pre-trained through the following steps: The training samples are input into the weight allocation model to obtain the first predicted weights output by the weight allocation model. The training samples include the identifier of any prompt word template corresponding to the training prompt word that indicates the evaluation of the content generated by the large model, the identifier of the question category corresponding to the prompt word on which the content generated by the large model is based, and the identifier of any evaluation sub-model among the multiple evaluation sub-models. Based on the first predicted weight and the pre-acquired weight labels for the training samples, the weight difference is determined, and the parameters of the weight allocation model are updated based on the weight difference.

8. The method according to claim 5, further comprising: The training prompts that indicate the evaluation of the content generated by the large model are input into multiple evaluation sub-models to obtain multiple fourth scores. For each evaluation sub-model The training prompts are input into the evaluation sub-model, along with the prompt template identifier, the question category identifier, the prompts used by the large model to generate content, and the evaluation sub-model identifier. These are then input into the weight allocation model to obtain the second prediction weight corresponding to the evaluation sub-model. Based on the multiple fourth scores and the second prediction weights corresponding to each evaluation sub-model, a predicted score is determined; based on the predicted score and the pre-acquired score labels, a score difference is determined; and based on the score difference, the parameters of the weight allocation model are updated.

9. An evaluation device for large model-generated content, comprising: The construction unit is configured to acquire target prompt words, input the target prompt words into the target large model, and obtain the first generated content; Construct auxiliary prompts, which include a first prompt and a second prompt, wherein the first prompt indicates the scoring criteria for the generated content and the second prompt indicates the reference answer for the target prompt word; The scoring unit is configured to construct a request scoring prompt based on the first generated content, the target prompt word, and the auxiliary prompt; Input the requested rating prompt into the evaluation model to obtain the first rating for the first generated content.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.

Images