Large model content generation capability evaluation method and device

Abstract

The invention provides a large model content generation ability evaluation method and device, and the method comprises the steps: obtaining a target data set for evaluating a target large model based on the multi-dimensional reasoning ability of the target large model, and evaluating the different-dimensional reasoning ability of the target large model based on the target data set, and obtaining a score of the reasoning ability of each dimension, obtaining a corresponding weighted average score based on the score of the reasoning ability of each dimension of the target large model and the use scene of the target large model, and determining the weighted average score as a first total score of the multi-dimensional reasoning ability of the target large model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method and device for evaluating the content generation capability of a large model. Background Art

[0002] With the rapid development of artificial intelligence, AI big models have also been widely used. The development of various AI big models involves testing the capabilities of the big models themselves in order to compare the capabilities of different big models.

[0003] At present, large models are generally tested through large model evaluation tools. Specifically, large model evaluation tools use standardized test cases and data sets and clear performance indicators to measure the accuracy, efficiency, and generalization ability of the model.

[0004] However, the current technology of big models is updated and iterated very quickly. The existing big model evaluation methods are difficult to be compatible with the big models that have been released not long ago. In addition, the test data sets are not independently designed and maintained, and it is difficult to flexibly adjust the question set and evaluation process according to the application scenarios of the big models to be tested. In addition, there are few evaluations of the performance of big models in the vertical field of insurance. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a method and device for evaluating the content generation capability of a big model, by evaluating the reasoning capability of different dimensions of a target big model through a target data set reflecting insurance field data, and calling the interface of the target big model for evaluation. Since the capability of the big model is tested directly by calling the big model service interface, there is no need to pay attention to the iteration of the underlying technology of the big model, thus avoiding the incompatibility of the evaluation scheme with the latest big model. At the same time, since the big model is tested through an independently constructed target data set, the target data set and evaluation process of the big model can be flexibly adjusted. In addition, the big model can also be evaluated for the vertical field of insurance.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating the content generation capability of a large model, the method comprising:

[0007] Based on the multi-dimensional reasoning ability of the target big model, a target data set for evaluating the target big model is obtained; wherein the target data set includes a plurality of evaluation question sets; and the target data set reflects insurance field data;

[0008] Based on the target data set, the reasoning ability of different dimensions of the target large model is evaluated to obtain a score of the reasoning ability of each dimension;

[0009] Based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, a corresponding weighted average score is obtained, and the weighted average score is determined as the first total score of the multi-dimensional reasoning ability of the target large model.

[0010] In a possible implementation, the reasoning ability of different dimensions of the target large model is evaluated based on the target data set to obtain a score of the reasoning ability of each dimension, including:

[0011] For the reasoning ability of one dimension of the target large model, a corresponding evaluation question is selected from the target data set based on the reasoning ability of the dimension, and an answer of the target large model is obtained based on the evaluation question;

[0012] Determine a scoring system corresponding to the reasoning ability of the dimension, and score the answer based on the scoring system to obtain a score for the reasoning ability of each dimension; wherein the scoring system includes multiple scoring dimensions, and each scoring dimension corresponds to a weight.

[0013] In a possible implementation, scoring the answers based on the scoring system to obtain a score for reasoning ability in each dimension includes:

[0014] For each scoring dimension of the scoring system, a weight and a grade standard corresponding to each scoring dimension are set; wherein the grade standard includes multiple grades, and different grades correspond to different scoring intervals; the scoring dimension includes multiple first-level dimensions; each first-level dimension corresponds to a weight; and each first-level dimension includes multiple second-level dimensions;

[0015] A referee macromodel for scoring is determined, and based on the referee macromodel and according to the weight and grade standard of each scoring dimension, a score of reasoning ability of each dimension is obtained.

[0016] In a possible implementation, obtaining the score of the reasoning ability of each dimension based on the weight and grade standard of each scoring dimension includes:

[0017] Based on the weight and grade standard of each scoring dimension, determine the score of the scoring dimension for the answer of the target macro model;

[0018] Based on the score of each scoring dimension for the target large model and the weight of each scoring dimension, a score of the reasoning ability of each dimension is obtained.

[0019] In a possible implementation, determining the score of the answer of the target macro model for the scoring dimension based on the weight and grade standard of each scoring dimension includes:

[0020] Scoring the content quality of the answer of the target macro model based on the grade standard of each scoring dimension and each secondary dimension included in each scoring dimension to obtain a secondary score of each secondary dimension;

[0021] Based on the secondary score of the secondary dimension, the corresponding primary score of the primary dimension is obtained, and based on the primary score of each primary dimension and the weight of each scoring dimension, the score of the scoring dimension for the answer to the target large model is obtained.

[0022] In a possible implementation, the score of the reasoning ability of each dimension of the target large model and the usage scenario of the target large model are used to obtain a corresponding weighted average score, including:

[0023] Set corresponding weights for the reasoning ability scores of each dimension based on the usage scenarios of the target large model;

[0024] Based on the score and weight of the reasoning ability of each dimension, a weighted average score of the target large model is obtained.

[0025] In a possible implementation, the large model content generation capability evaluation method of the present application further includes:

[0026] Obtaining a second total score obtained by evaluating the multi-dimensional reasoning ability of the target large model by the target user; wherein the second total score represents the total score of the artificial evaluation of the multi-dimensional reasoning ability of the target large model;

[0027] The weights of the second total score and the first total score are determined respectively based on the usage scenarios of the target big model, and based on the weights of the second total score and the first total score, and the second total score and the first total score, a third total score of the reasoning ability of each dimension of the target big model is obtained.

[0028] In a second aspect, the embodiment of the present application further provides a large model content generation capability evaluation device, the device comprising:

[0029] A first acquisition module is used to acquire a target data set for evaluating the target large model based on the multi-dimensional reasoning ability of the target large model; wherein the target data set includes a plurality of evaluation question sets; and the target data set reflects insurance field data;

[0030] A second acquisition module is used to evaluate the reasoning ability of different dimensions of the target large model based on the target data set to obtain a score of the reasoning ability of each dimension;

[0031] The third acquisition module is used to obtain the corresponding weighted average score based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, and determine the weighted average score as the first total score of the multi-dimensional reasoning ability of the target large model.

[0032] In a possible implementation manner, the second acquisition module is specifically configured to:

[0033] For the reasoning ability of one dimension of the target large model, a corresponding evaluation question is selected from the target data set based on the reasoning ability of the dimension, and an answer of the target large model is obtained based on the evaluation question;

[0034] Determine a scoring system corresponding to the reasoning ability of the dimension, and score the answer based on the scoring system to obtain a score for the reasoning ability of each dimension; wherein the scoring system includes multiple scoring dimensions, and each scoring dimension corresponds to a weight.

[0035] In a possible implementation manner, the second acquisition module is specifically configured to:

[0036] For each scoring dimension of the scoring system, a weight and a grade standard corresponding to each scoring dimension are set; wherein the grade standard includes multiple grades, and different grades correspond to different scoring intervals; the scoring dimension includes multiple first-level dimensions; each first-level dimension corresponds to a weight; and each first-level dimension includes multiple second-level dimensions;

[0037] A referee macromodel for scoring is determined, and based on the referee macromodel and according to the weight and grade standard of each scoring dimension, a score of reasoning ability of each dimension is obtained.

[0038] In a possible implementation manner, the second acquisition module is specifically configured to:

[0039] Based on the weight and grade standard of each scoring dimension, determine the score of the scoring dimension for the answer of the target macro model;

[0040] Based on the score of each scoring dimension for the target large model and the weight of each scoring dimension, a score of the reasoning ability of each dimension is obtained.

[0041] In a possible implementation manner, the second acquisition module is further configured to:

[0042] Scoring the content quality of the answer of the target macro model based on the grade standard of each scoring dimension and each secondary dimension included in each scoring dimension to obtain a secondary score of each secondary dimension;

[0043] Based on the secondary score of the secondary dimension, the corresponding primary score of the primary dimension is obtained, and based on the primary score of each primary dimension and the weight of each scoring dimension, the score of the scoring dimension for the answer to the target large model is obtained.

[0044] In a possible implementation manner, the third acquisition module is specifically configured to:

[0045] Set corresponding weights for the reasoning ability scores of each dimension based on the usage scenarios of the target large model;

[0046] Based on the score and weight of the reasoning ability of each dimension, a weighted average score of the target large model is obtained.

[0047] In a possible implementation manner, the large model content generation capability evaluation device of the present application further includes:

[0048] A fourth acquisition module is used to obtain a second total score obtained by the target user through evaluation of the multi-dimensional reasoning ability of the target large model; wherein the second total score represents the total score of the artificial evaluation of the multi-dimensional reasoning ability of the target large model;

[0049] The fifth acquisition module is used to determine the weights of the second total score and the first total score based on the usage scenario of the target large model, and obtain the third total score of the reasoning ability of each dimension of the target large model based on the weights of the second total score and the first total score, and the second total score and the first total score.

[0050] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the large model content generation capability assessment method as described in any one of the first aspects.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large model content generation capability assessment method described in any one of the first aspects are executed.

[0052] The embodiment of the present application provides a method and device for evaluating the content generation capability of a large model. Based on the multi-dimensional reasoning capability of the target large model, a target data set for evaluating the target large model is obtained. Based on the target data set, the reasoning capability of the target large model in different dimensions is evaluated to obtain the score of the reasoning capability of each dimension. Based on the score of the reasoning capability of each dimension of the target large model and the usage scenario of the target large model, the corresponding weighted average score is obtained, and the weighted average score is determined as the first total score of the multi-dimensional reasoning capability of the target large model. In the present application, the reasoning capability of the target large model in different dimensions is evaluated by a target data set reflecting insurance field data, and the interface of the target large model is called for evaluation. Since the capability of the large model is tested directly by calling the large model service interface, there is no need to pay attention to the iteration of the underlying technology of the large model, and the incompatibility of the evaluation scheme with the latest large model is avoided. At the same time, since the large model is tested by the self-built target data set, the target data set and evaluation process of the large model can be flexibly adjusted. In addition, the large model can also be evaluated for the vertical field of insurance.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a flow chart of a method for evaluating the large model content generation capability according to an embodiment of the present application;

[0056] Figure 2 It is a schematic diagram of the target large model content generation capability evaluation process;

[0057] Figure 3 It is a schematic diagram of the multi-dimensional reasoning capabilities of the target large model;

[0058] Figure 4 It is a schematic diagram of the composition of the target data set;

[0059] Figure 5 is a flow chart of a method for evaluating the large model content generation capability according to another embodiment of the present application;

[0060] Figure 6 is a flow chart of a method for evaluating the large model content generation capability according to another embodiment of the present application;

[0061] Figure 7 It is a schematic diagram of the scoring system for large model evaluation;

[0062] Figure 8 It is a structural schematic diagram of a large model content generation capability evaluation device according to an embodiment of the present application;

[0063] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0065] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0066] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0067] Taking into account the rapid development of artificial intelligence, AI big models have also been widely used. The development of various AI big models involves testing the capabilities of the big models themselves in order to compare the capabilities of different big models.

[0068] At present, big models are generally tested through big model evaluation tools. Specifically, big model evaluation tools use standardized test cases and data sets and clear performance indicators to measure the accuracy, efficiency and generalization ability of the model. However, the current big model technology is updated and iterated quickly. The existing big model evaluation methods are difficult to be compatible with the big models that have just been released. In addition, the test data sets are not independently designed and maintained, and it is difficult to flexibly adjust the problem set and evaluation process according to the application scenarios of the big models to be tested. In addition, there are few evaluations of the performance of big models in the vertical field of insurance.

[0069] To address this problem, the present application provides a method and device for evaluating the content generation capability of a big model. The reasoning capability of different dimensions of a target big model is evaluated by a target data set that reflects insurance field data, and the interface of the target big model is called for evaluation. Since the capability of the big model is tested directly by calling the big model service interface, there is no need to pay attention to the iteration of the underlying technology of the big model, thus avoiding the incompatibility of the evaluation scheme with the latest big model. At the same time, since the big model is tested by an independently constructed target data set, the target data set and evaluation process of the big model can be flexibly adjusted. In addition, the big model can also be evaluated for the vertical field of insurance.

[0070] Figure 1 1 is a flow chart of a method for evaluating the large model content generation capability according to an embodiment of the present application. Figure 1 As shown, the large model content generation capability evaluation method of the embodiment of the present application may specifically include:

[0071] S101. Based on the multi-dimensional reasoning capability of the target large model, a target data set for evaluating the target large model is obtained.

[0072] S102. Evaluate the reasoning ability of different dimensions of the target large model based on the target data set to obtain a score for the reasoning ability of each dimension.

[0073] S103. Based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, a corresponding weighted average score is obtained, and the weighted average score is determined as a first total score of the multi-dimensional reasoning ability of the target large model.

[0074] In the above-mentioned big model content generation capability evaluation method, the reasoning capability of different dimensions of the target big model is evaluated through the target data set reflecting the insurance field data, and the interface of the target big model is called for evaluation. Since the ability of the big model is tested directly by calling the big model service interface, there is no need to pay attention to the iteration of the underlying technology of the big model, which avoids the incompatibility of the evaluation scheme with the latest big model. At the same time, since the big model is tested through the independently built target data set, the target data set and evaluation process of the big model can be flexibly adjusted. In addition, the big model can also be evaluated for the vertical field of insurance.

[0075] The above exemplary steps of the embodiment of the present application are described below with reference to specific examples:

[0076] S101, based on the multi-dimensional reasoning ability of the target large model, obtain a target data set for evaluating the target large model.

[0077] In the embodiment of the present application, the target large model is the AI ​​(Artificial Intelligence) large model to be tested, for example, Figure 2 As shown in the figure, the large models to be tested include chatglm3-6b, chatglm-66b and other large models. The multi-dimensional reasoning ability represents the content generation ability of the large model (that is, the general reasoning ability), that is, the content generation ability of the large model includes multi-dimensional reasoning ability, which includes nine dimensions of reasoning ability, including calculation, logical reasoning, code, knowledge encyclopedia, language comprehension, long text, generation creation, role-playing, and financial ability. Among them, the reasoning ability of the dimension of financial ability includes two sub-dimensions: financial knowledge assessment and other financial knowledge assessment. For example, Figure 3 As shown; the target data set is a data set for evaluating the content generation capability of the large model, wherein the target data set includes multiple evaluation question sets, and the target data set reflects insurance field data. For example, the target data set includes a mathematics test set, a knowledge question and answer test set, a code evaluation set, a professional business ability test set, an industry general ability test set, and a comprehensive test set in the insurance field. For example, Figure 4 As shown; based on the multi-dimensional reasoning ability of the target large model, a target data set for evaluating the target large model is obtained for subsequent processing.

[0078] It can be seen that since the target data set includes a comprehensive test set in the insurance field, etc., the large model can be evaluated in the vertical field of insurance.

[0079] It should be noted that part of the data in the target dataset comes from public information, and part comes from internal policy data and customer information in the insurance industry. It can not only comprehensively test the reasoning ability of the big model, but also test the question-answering generation ability of the big model in the insurance field.

[0080] S102, based on the target data set, the reasoning ability of the target large model in different dimensions is evaluated to obtain the reasoning ability score of each dimension. In the embodiment of the present application, after calling the interface of the target large model, the reasoning ability of the target large model in different dimensions is evaluated based on the target data set in step S101 to obtain the reasoning ability score of each dimension for subsequent processing.

[0081] It should be noted that after calling the interface of the target large model, the reasoning ability of the target large model in different dimensions is evaluated based on the target data set. Figure 2 As shown, the large model interface is called to evaluate the large model.

[0082] Therefore, because the reasoning ability of the big model can be tested directly on the application side by calling the big model service interface, there is no need to pay attention to the iteration of the underlying technology of the big model, thus avoiding the incompatibility of the evaluation solution with the latest big model.

[0083] S103, based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, obtain the corresponding weighted average score, and determine the weighted average score as the first total score of the multi-dimensional reasoning ability of the target large model.

[0084] In the embodiment of the present application, the usage scenario of the target big model is the purpose of the target big model, such as the purpose of copywriting generation, code generation, mathematical calculations, etc., which can be used by a machine to objectively evaluate the quality of generated content, etc. The weighted average score is the total score of the multi-dimensional reasoning ability of the target big model obtained by weighted averaging according to the purpose of the target big model. Based on the reasoning ability score of each dimension of the target big model obtained in step S102 and the usage scenario of the target big model, the corresponding weighted average score is obtained, and the weighted average score is determined as the first total score of the multi-dimensional reasoning ability of the target big model.

[0085] The large model content generation capability evaluation method provided in the embodiment of the present application obtains a target data set for evaluating the target large model based on the multi-dimensional reasoning capability of the target large model, evaluates the reasoning capability of the target large model in different dimensions based on the target data set, obtains the score of the reasoning capability of each dimension, obtains the corresponding weighted average score based on the score of the reasoning capability of each dimension of the target large model and the usage scenario of the target large model, and determines the weighted average score as the first total score of the multi-dimensional reasoning capability of the target large model. The large model content generation capability evaluation method of the present application evaluates the reasoning capability of the target large model in different dimensions through a target data set reflecting insurance field data, and calls the interface of the target large model for evaluation. Since the ability of the large model is tested directly by calling the large model service interface, there is no need to pay attention to the iteration of the underlying technology of the large model, and the incompatibility of the evaluation scheme with the latest large model is avoided. At the same time, since the large model is tested through the self-built target data set, the target data set and evaluation process of the large model can be flexibly adjusted. In addition, the large model can also be evaluated for the vertical field of insurance.

[0086] Further, such as Figure 5 As shown, the step S102 in the above embodiment of “evaluating the reasoning ability of different dimensions of the target large model based on the target data set to obtain a score of the reasoning ability of each dimension” may specifically include the following steps:

[0087] S501, for the reasoning ability of one dimension of the target large model, select corresponding evaluation questions from the target data set based on the reasoning ability of the dimension, and obtain the answer of the target large model based on the evaluation questions.

[0088] In the embodiment of the present application, the reasoning ability of one dimension of the target large model is evaluated, and the corresponding evaluation questions are selected from the target data set based on the reasoning ability of this dimension, and the answers of the target large model are obtained based on the evaluation questions for subsequent processing. It should be noted that the present application takes the reasoning ability of the dimension of computing ability as an example for description.

[0089] It should be noted that, for the reasoning ability of each dimension of the big model, a certain number of representative questions are selected from the relevant test set, and these questions are asked to the target big model to obtain the answer of the big model.

[0090] Optionally, a test model is determined based on the target data set and different dimensions of the reasoning ability of the target data set; the evaluation questions are input into the test model to obtain the answers of the target large model to the evaluation questions.

[0091] S502, determining the scoring system corresponding to the reasoning ability of the dimension, and scoring the answers based on the scoring system to obtain the score of the reasoning ability of each dimension.

[0092] In an embodiment of the present application, the scoring system includes multiple scoring dimensions, each scoring dimension corresponds to a weight, and the scoring system corresponding to the reasoning ability of this dimension to be evaluated is determined, and the answer of the target large model in step S501 is scored based on the scoring system to obtain the score of the reasoning ability of each dimension for subsequent processing.

[0093] It should be noted that there are corresponding scoring systems for different dimensions of reasoning ability. The scoring systems corresponding to each dimension of reasoning ability are shown in Tables 1 to 3 below:

[0094] Table 1

[0095]

[0096]

[0097] Table 2

[0098]

[0099] Table 3

[0100]

[0101]

[0102] For example, taking computing power as an example, the target large model's answer accuracy (weighted 35%), complex problem handling ability (weighted 35%), robustness (weighted 10%), user-friendliness (weighted 10%), and creativity (weighted 10%) are tested.

[0103] Further, such as Figure 6 As shown, the step S502 in the above embodiment of “scoring the answers based on the scoring system to obtain the score of the reasoning ability of each dimension” may specifically include the following steps:

[0104] S601, for each scoring dimension of the scoring system, set the weight and grade standard corresponding to each scoring dimension.

[0105] In an embodiment of the present application, the grade standard represents the score of the content quality of the answer of the large model. The grade standard includes multiple grades, and different grades correspond to different scoring intervals. For example, the grades are full marks, higher, medium, lower and irrelevant. The scoring interval corresponding to each grade is full marks (100 points), higher (75-99 points), medium (50-74 points), lower (25-49 points) and irrelevant (0-24 points), that is, for each secondary dimension, the score will be set to five levels according to the content quality of the answer of the large model; the scoring dimension includes multiple first-level dimensions, each first-level dimension corresponds to a weight, that is, the scoring dimension is the first-level dimension, so the nine scoring dimensions of the present application also represent nine first-level dimensions, the weight of the above-mentioned scoring dimension is the weight of the first-level dimension, and the sum of the weights of the first-level dimension (scoring dimension) is 1, and each first-level dimension includes multiple (generally 4 to 8) second-level dimensions. For example, for the first-level dimension of mathematical ability, the second-level dimensions of accuracy, complex problem handling and other abilities will be focused on; for each scoring dimension of the scoring system, the weight and grade standard corresponding to each scoring dimension are set for subsequent processing. It should be noted that for each scoring dimension, a grade standard will be set. For example, when testing the accuracy of the answer, if the large model is completely correct and error-free when performing numerical calculations and solving complex mathematical problems, it will be given full marks; if the model is mostly correct in numerical calculations and complex mathematical problems, with occasional small errors, it will be given a higher score (75-99 points), and so on. Figure 7 As shown, the scoring system for large model evaluation is represented.

[0106] Specifically, taking computing ability as an example, the scoring dimensions include five first-level dimensions: accuracy, complexity handling capability, robustness, user-friendliness, and creativity, and the scoring dimensions are described in detail.

[0107] For the first-level dimension accuracy (weighted at 35%), it includes two second-level dimensions: the accuracy of numerical calculations and the solution of complex mathematical problems. The accuracy of numerical calculations indicates the accuracy of the model in performing basic arithmetic operations (addition, subtraction, and division), and the solution of complex mathematical problems indicates the accuracy of the model in solving complex mathematical problems (such as calculus, linear algebra, probability statistics, etc.). The corresponding grade standards are full marks, higher, medium, lower and irrelevant. Among them, the full mark (100 points) indicates that the model is completely correct and error-free when performing numerical calculations and solving complex mathematical problems; higher (75-99 points) indicates that the model is mostly correct in numerical calculations and complex mathematical problems, with occasional small errors; medium (50-74 points) indicates that the model is partially correct in numerical calculations and complex mathematical problems, but there are obvious errors; lower (25-49 points) indicates that the model is mostly incorrect in numerical calculations and complex mathematical problems, with a large number of errors; irrelevant (0-24 points) indicates that the model is completely wrong or irrelevant in numerical calculations and complex mathematical problems. The other first-level dimensions of complexity handling capability, robustness, user-friendliness, and creativity are similar and will not be elaborated here.

[0108] S602, determine the referee macro model used for scoring, and obtain the score of reasoning ability of each dimension based on the referee macro model and according to the weight and grade standard of each scoring dimension.

[0109] In the embodiment of the present application, the referee big model is a big model used for scoring, for example, Figure 2 As shown, the referee model can be ChatGPT, etc. After the referee model is determined, based on the referee model, according to the weight and grade standard of each scoring dimension in step S601, the score of the reasoning ability of each dimension is obtained. Optionally, the referee model interface can be called to determine the referee model used for scoring, for example, Figure 2 shown.

[0110] As a possible implementation method, when obtaining the reasoning ability score of each dimension based on the weight and grade standard of each scoring dimension, the score of the scoring dimension for the answer of the target large model is determined based on the weight and grade standard of each scoring dimension; based on the score of each scoring dimension for the target large model and the weight of each scoring dimension, the reasoning ability score of each dimension is obtained.

[0111] Optionally, when determining the score of the target large model's answer for the scoring dimension based on the weight and grading standard of each scoring dimension, the content quality of the target large model's answer is scored based on the grading standard of each scoring dimension and each secondary dimension included in each scoring dimension to obtain a secondary score for each secondary dimension; based on the secondary score of the secondary dimension, the corresponding first-level score of the first-level dimension is obtained, and based on the first-level score of each first-level dimension and the weight of each scoring dimension, the reasoning ability score of each dimension is obtained.

[0112] Furthermore, when the corresponding weighted average score is obtained based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, the corresponding weight is set for the reasoning ability score of each dimension based on the usage scenario of the target large model; based on the reasoning ability score and weight of each dimension, the weighted average score of the target large model is obtained. For example, as mentioned above, when the usage scenario of the target large model is code generation, mathematical calculation, etc., a larger weight can be set for computing power, which will not be repeated here.

[0113] Furthermore, a second total score is obtained by evaluating the multi-dimensional reasoning ability of the target large model by the target user; the weights of the second total score and the first total score are determined based on the usage scenario of the target large model, and the third total score of the reasoning ability of each dimension of the target large model is obtained based on the weights of the second total score and the first total score, as well as the second total score and the first total score. Among them, the second total score represents the total score of the artificial evaluation of the multi-dimensional reasoning ability of the target large model; the target user is the user who manually evaluates the multi-dimensional reasoning ability of the target large model, and the target user can be an expert review team.

[0114] Specifically, after obtaining the first total score (i.e., the machine score) based on the above-mentioned referee big model, manual scoring will also be performed. The scoring system of manual scoring is the same as the above-mentioned machine scoring, thereby obtaining the manual score, i.e., the second total score. The first total score and the second total score are given corresponding weights according to the usage scenario of the target big model. The sum of the weights of the second total score and the first total score is 1. The first total score and the second total score are combined with their corresponding circles to obtain the score, which is the final third total score of the target big model's content generation capability. For example, the final score of the big model's content generation capability = w1*machine score + w2*manual score, the total score is 100, w1+w2=1, the specific values ​​of w1 and w2 depend on the usage scenario of the big model. If the big model is mainly used for code generation, mathematical calculations, and other tasks that can be objectively evaluated by the machine to generate content quality, w1=0.7, if the big model is mainly used for copywriting generation and other tasks, given that the quality of such content is more subjective, w2=0.7.

[0115] It is understandable that, for example, Figure 2As shown, the overall evaluation process of this application is to write a large model automated test script based on the test question set and the test dimension design plan (i.e., related to the scoring system) to obtain the answers of the large model to be tested to the given questions, and then use a scoring method that combines the requirements of the specified scoring system to score the reasoning ability of each dimension of the large model (the large model review team will also refer to the scoring system) to draw a final conclusion on the content generation ability of a large model.

[0116] Figure 8 is a flow chart of a large model content generation capability evaluation device according to an embodiment of the present application, such as Figure 8 As shown, the large model content generation capability evaluation device 800 of the embodiment of the present application may specifically include:

[0117] The first acquisition module 801 is used to acquire a target data set for evaluating the target big model based on the multi-dimensional reasoning ability of the target big model; wherein the target data set includes multiple evaluation question sets; and the target data set reflects insurance field data.

[0118] The second acquisition module 802 is used to evaluate the reasoning ability of different dimensions of the target large model based on the target data set to obtain the score of the reasoning ability of each dimension.

[0119] The third acquisition module 803 is used to obtain the corresponding weighted average score based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, and determine the weighted average score as the first total score of the multi-dimensional reasoning ability of the target large model.

[0120] In a possible implementation manner, the second acquisition module is specifically configured to:

[0121] For the reasoning ability of one dimension of the target large model, select the corresponding evaluation questions from the target data set based on the reasoning ability of this dimension, and obtain the answer of the target large model based on the evaluation questions;

[0122] Determine the scoring system corresponding to the reasoning ability of the dimension, and score the answers based on the scoring system to obtain the score of the reasoning ability of each dimension; wherein the scoring system includes multiple scoring dimensions, and each scoring dimension corresponds to a weight.

[0123] In a possible implementation manner, the second acquisition module is specifically configured to:

[0124] For each scoring dimension of the scoring system, a weight and grade standard corresponding to each scoring dimension are set; wherein the grade standard includes multiple grades, and different grades correspond to different scoring intervals; the scoring dimension includes multiple first-level dimensions; each first-level dimension corresponds to a weight; each first-level dimension includes multiple second-level dimensions;

[0125] Determine the referee model used for scoring, and based on the referee model and according to the weight and grade standard of each scoring dimension, obtain the score of reasoning ability of each dimension.

[0126] In a possible implementation manner, the second acquisition module is specifically configured to:

[0127] Based on the weight and grade standard of each scoring dimension, determine the score of the scoring dimension for the answer of the target large model;

[0128] Based on the score of each scoring dimension for the target large model and the weight of each scoring dimension, the score of the reasoning ability of each dimension is obtained.

[0129] In a possible implementation manner, the second acquisition module is further used to:

[0130] Score the content quality of the target macromodel's answer based on the grading standard of each scoring dimension and each secondary dimension included in each scoring dimension to obtain a secondary score for each secondary dimension;

[0131] Based on the secondary score of the secondary dimension, the corresponding primary score of the primary dimension is obtained, and based on the primary score of each primary dimension and the weight of each scoring dimension, the score of the scoring dimension for the answer to the target large model is obtained.

[0132] In a possible implementation manner, the third acquisition module is specifically configured to:

[0133] Set corresponding weights for the reasoning ability scores of each dimension based on the usage scenarios of the target large model;

[0134] Based on the score and weight of the reasoning ability of each dimension, the weighted average score of the target large model is obtained.

[0135] In a possible implementation manner, the large model content generation capability evaluation device of the present application further includes:

[0136] The fourth acquisition module is used to obtain a second total score obtained by the target user through evaluation of the multi-dimensional reasoning ability of the target large model; wherein the second total score represents the total score of the human evaluation of the multi-dimensional reasoning ability of the target large model;

[0137] The fifth acquisition module is used to determine the weights of the second total score and the first total score based on the usage scenario of the target large model, and obtain the third total score of the reasoning ability of each dimension of the target large model based on the weights of the second total score and the first total score, and the second total score and the first total score.

[0138] The large model content generation capability evaluation device provided in the embodiment of the present application obtains a target data set for evaluating the target large model based on the multi-dimensional reasoning capability of the target large model, evaluates the reasoning capability of the target large model in different dimensions based on the target data set, obtains the score of the reasoning capability of each dimension, obtains the corresponding weighted average score based on the score of the reasoning capability of each dimension of the target large model and the usage scenario of the target large model, and determines the weighted average score as the first total score of the multi-dimensional reasoning capability of the target large model. The large model content generation capability evaluation device of the present application evaluates the reasoning capability of the target large model in different dimensions through a target data set reflecting insurance field data, and calls the interface of the target large model for evaluation. Since the ability of the large model is tested directly by calling the large model service interface, there is no need to pay attention to the iteration of the underlying technology of the large model, and the incompatibility of the evaluation scheme with the latest large model is avoided. At the same time, since the large model is tested through the self-built target data set, the target data set and evaluation process of the large model can be flexibly adjusted. In addition, the large model can also be evaluated for the vertical field of insurance.

[0139] like Fig. 9 As shown, an electronic device 900 provided in an embodiment of the present application includes: a processor 901, a memory 902 and a bus, wherein the memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 through the bus, and the processor 901 executes the machine-readable instructions to perform the steps of the above-mentioned large model content generation capability assessment method.

[0140] Specifically, the above-mentioned memory 902 and processor 901 can be general-purpose memory and processor, which are not specifically limited here. When the processor 901 runs the computer program stored in the memory 902, it can execute the above-mentioned large model content generation capability evaluation method.

[0141] Corresponding to the above-mentioned large model content generation capability evaluation method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large model content generation capability evaluation method are executed.

[0142] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0143] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0144] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the deployment method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0145] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for evaluating the content generation capability of a large model, characterized in that: The method comprises: Based on the multi-dimensional reasoning ability of the target big model, a target data set for evaluating the target big model is obtained; wherein the target data set includes a plurality of evaluation question sets; and the target data set reflects insurance field data; Based on the target data set, the reasoning ability of different dimensions of the target large model is evaluated to obtain a score of the reasoning ability of each dimension; Based on the score of the reasoning ability of each dimension of the target large model and the usage scenario of the target large model, a corresponding weighted average score is obtained, and the weighted average score is determined as the first total score of the multi-dimensional reasoning ability of the target large model.

2. The method according to claim 1, characterized in that: The reasoning ability of different dimensions of the target large model is evaluated based on the target data set to obtain the reasoning ability score of each dimension, including: For the reasoning ability of one dimension of the target large model, a corresponding evaluation question is selected from the target data set based on the reasoning ability of the dimension, and an answer of the target large model is obtained based on the evaluation question; Determine a scoring system corresponding to the reasoning ability of the dimension, and score the answer based on the scoring system to obtain a score for the reasoning ability of each dimension; wherein the scoring system includes multiple scoring dimensions, and each scoring dimension corresponds to a weight.

3. The method according to claim 2, characterized in that Scoring the answers based on the scoring system to obtain a score for reasoning ability in each dimension includes: For each scoring dimension of the scoring system, a weight and a grade standard corresponding to each scoring dimension are set; wherein the grade standard includes multiple grades, and different grades correspond to different scoring intervals; the scoring dimension includes multiple first-level dimensions; each first-level dimension corresponds to a weight; and each first-level dimension includes multiple second-level dimensions; A referee macromodel for scoring is determined, and based on the referee macromodel and according to the weight and grade standard of each scoring dimension, a score of reasoning ability of each dimension is obtained.

4. The method according to claim 3, characterized in that The reasoning ability score of each dimension is obtained based on the weight and grade standard of each scoring dimension, including: Based on the weight and grade standard of each scoring dimension, determine the score of the scoring dimension for the answer of the target macro model; Based on the score of each scoring dimension for the target large model and the weight of each scoring dimension, a score of the reasoning ability of each dimension is obtained.

5. The method according to claim 4, characterized in that The step of determining the score of the answer of the target macro model for the scoring dimension based on the weight and grade standard of each scoring dimension includes: Scoring the content quality of the answer of the target macro model based on the grade standard of each scoring dimension and each secondary dimension included in each scoring dimension to obtain a secondary score of each secondary dimension; Based on the secondary score of the secondary dimension, the corresponding primary score of the primary dimension is obtained, and based on the primary score of each primary dimension and the weight of each scoring dimension, the score of the scoring dimension for the answer to the target large model is obtained.

6. The method according to claim 3, characterized in that The corresponding weighted average score is obtained based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, including: Set corresponding weights for the reasoning ability scores of each dimension based on the usage scenarios of the target large model; Based on the score and weight of the reasoning ability of each dimension, a weighted average score of the target large model is obtained.

7. The method according to claim 2, characterized in that: The method further comprises: Obtaining a second total score obtained by evaluating the multi-dimensional reasoning ability of the target large model by the target user; wherein the second total score represents the total score of the artificial evaluation of the multi-dimensional reasoning ability of the target large model; The weights of the second total score and the first total score are determined respectively based on the usage scenarios of the target big model, and based on the weights of the second total score and the first total score, and the second total score and the first total score, a third total score of the reasoning ability of each dimension of the target big model is obtained.

8. A large model content generation capability evaluation device, characterized in that: The device comprises: A first acquisition module is used to acquire a target data set for evaluating the target large model based on the multi-dimensional reasoning ability of the target large model; wherein the target data set includes a plurality of evaluation question sets; and the target data set reflects insurance field data; A second acquisition module is used to evaluate the reasoning ability of different dimensions of the target large model based on the target data set to obtain a score of the reasoning ability of each dimension; The third acquisition module is used to obtain the corresponding weighted average score based on the reasoning ability score of each dimension of the target large model and the usage scenario of the target large model, and determine the weighted average score as the first total score of the multi-dimensional reasoning ability of the target large model.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the large model content generation capability assessment method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the large model content generation capability assessment method as described in any one of claims 1 to 7.

Images