Model evaluation method, evaluation device, equipment, storage medium and program product

Abstract

The invention provides a model evaluation method, an evaluation device, equipment, a storage medium and a program product, and relates to the technical field of data processing. The model evaluation method comprises the following steps: determining a first evaluation result in business performance based on a performance index value obtained when an industry model executes an industry reasoning operation; determining a second evaluation result in the industry capability based on the user permeability index of the industry model; performing scene capability evaluation on the industry model based on a multi-judgment model to obtain a third evaluation result; and generating an overall evaluation result of the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first group of weights. Through the technical scheme disclosed by the invention, an enterprise is helped to understand the performance of the model from multiple perspectives, and research and development and application of an industrial large model are promoted.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a model evaluation method, a model evaluation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As large language models play an increasingly important role in natural language processing research and people's daily lives, industry big models refer to large language models that are optimized and trained for specific industries or fields. Compared with general large language models, industry big models are more practical for the B-side. Due to the large number of large language models, different industry big models will be generated accordingly. Therefore, the evaluation of industry big models has also become crucial.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a model evaluation method, a model evaluation device, an electronic device, a computer-readable storage medium and a computer program product, which can at least to some extent overcome the problem of poor evaluation results of large industry models in related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a model evaluation method is provided, including: determining a first evaluation result in business performance based on a performance indicator value obtained when an industry model executes an industry reasoning operation; determining a second evaluation result in industry capability based on a user penetration indicator of the industry model; performing scenario capability evaluation on the industry model based on a multi-referee model to obtain a third evaluation result; and generating an overall evaluation result for the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and a corresponding first set of weights.

[0007] In one embodiment of the present disclosure, a first evaluation result in business performance is determined based on a performance indicator value obtained when an industry model performs an industry reasoning operation, including: obtaining a latency indicator value when the industry model performs an industry reasoning operation and a single instance flux of the industry model as the performance indicator value; scoring the latency indicator value based on multiple delay intervals to obtain a first score; scoring the single instance flux based on a flux threshold to obtain a second score; determining a first weight corresponding to the first score and a second weight corresponding to the second score based on the industry in which the industry model is located; and obtaining the first evaluation result based on the first score, the first weight, the second score and the second weight, wherein the latency indicator value includes at least one of a first word delay, a last word delay and an average reasoning delay.

[0008] In one embodiment of the present disclosure, before determining the first evaluation result in the business performance based on the performance indicator value obtained when performing the industry reasoning operation based on the industry model, it also includes: deploying the industry model in a half-precision floating point format.

[0009] In one embodiment of the present disclosure, determining a second evaluation result in industry capability based on a user penetration index of the industry model includes: determining the second evaluation result based on the user penetration and a business share index of the industry model.

[0010] In one embodiment of the present disclosure, it also includes: determining the number of users of the industry model based on the user registration data and / or access log of the industry model; determining the total number of model users in the field where the industry model is located based on at least one of the market research data, statistical agency data and industry report data in the field where the industry model is located; determining the user penetration rate based on the number of users of the industry model and the total number of model users.

[0011] In one embodiment of the present disclosure, it also includes: determining the business share index based on the sales of the industry model and the total sales of models in the field where the industry model is located.

[0012] In one embodiment of the present disclosure, a scenario capability assessment is performed on the industry model based on a multi-judge model to obtain a third assessment result, including: constructing a scenario capability assessment data set based on question-answer pair samples, the question-answer pair samples include corresponding question samples and answer samples; reasoning the question samples based on the industry model to obtain a reasoning result; constructing a universal capability judgment model based on the multi-judge model to judge the reasoning result based on the universal capability judgment model to obtain a first evaluation value; judging the reasoning result based on a security capability judgment model to obtain a second evaluation value; judging the reasoning result based on a domain task capability judgment model to obtain a third evaluation value; and obtaining the third assessment result based on the first evaluation value, the second evaluation value, the third evaluation value and the corresponding second set of weights.

[0013] In one embodiment of the present disclosure, a universal ability judgment model is constructed based on the multi-judge model to judge the reasoning result based on the universal ability judgment model to obtain a first evaluation value, including: determining the matching multi-judge model based on the type of the industry model; performing a multi-dimensional evaluation on the reasoning result based on the multi-judge model to obtain a subjective evaluation result, wherein the multi-dimensional evaluation includes answer accuracy, answer regularity and answer friendliness; obtaining an objective evaluation result based on the consistency judgment between the reasoning result and the answer sample; and obtaining the first evaluation value based on the subjective evaluation result and the objective evaluation result.

[0014] In one embodiment of the present disclosure, the reasoning result is evaluated based on a security capability judgment model to obtain a second evaluation value, including: constructing the security capability judgment model based on accuracy evaluation parameters, objectivity evaluation parameters, positivity evaluation parameters and fairness evaluation parameters; based on the security capability judgment model, respectively evaluating the accuracy value, objective value, positivity value and fairness value of the reasoning result to obtain the second evaluation value based on the accuracy value, the objective value, the positivity value and the fairness value.

[0015] In one embodiment of the present disclosure, the reasoning result is judged based on the domain task capability judgment model to obtain a third evaluation value, including: obtaining a reference result obtained by reasoning the problem sample with a reference model; based on the configured domain capability evaluation index, the reasoning result and the reference result are evaluated to obtain the third evaluation value.

[0016] In one embodiment of the present disclosure, it also includes: obtaining verification information of the overall evaluation result; and generating a multi-dimensional evaluation report of the overall evaluation result based on the verification information.

[0017] According to another aspect of the present disclosure, a model evaluation device is provided, including: a first evaluation module, used to determine a first evaluation result in business performance based on a performance indicator value obtained when an industry model executes an industry reasoning operation; a second evaluation module, used to determine a second evaluation result in industry capability based on a user penetration indicator of the industry model; a third evaluation module, used to perform scenario capability evaluation on the industry model based on a multi-judge model to obtain a third evaluation result; and a generation module, used to generate an overall evaluation result of the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first set of weights.

[0018] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to perform the above-mentioned model evaluation method by executing the executable instructions.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model evaluation method is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model evaluation method is implemented.

[0021] The model evaluation solution provided in the embodiments of the present disclosure can realize a more comprehensive and integrated evaluation of the industry model by verifying the industry model from the aspects of business performance, industry capability and scenario capability to integrate the evaluation results of multiple dimensions. On the one hand, it helps enterprises to understand the performance of the model from multiple perspectives and promote the research and development and application of large industry models. On the other hand, it realizes the automated processing of model evaluation, which can assist in the iterative optimization of large industry models while improving the efficiency of model evaluation.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1 A schematic diagram of a model evaluation system in an embodiment of the present disclosure is shown;

[0025] Figure 2 A schematic flow chart of a model evaluation method in an embodiment of the present disclosure is shown;

[0026] Figure 3 A schematic flow chart showing another model evaluation method in an embodiment of the present disclosure;

[0027] Figure 4 A schematic flow chart showing yet another model evaluation method in an embodiment of the present disclosure;

[0028] Figure 5 Another schematic diagram of a model evaluation system in an embodiment of the present disclosure is shown;

[0029] Figure 6 A schematic flow chart showing yet another model evaluation method in an embodiment of the present disclosure;

[0030] Figure 7 A schematic flow chart showing yet another model evaluation method in an embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of a model evaluation device in an embodiment of the present disclosure is shown;

[0032] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] With the rapid development of big language models, big language models are playing an increasingly important role in natural language processing research and people's daily lives. Industry big models can be understood as "specialists" in big models, and can be compared to professionally trained graduate students or professional practitioners. It is generally believed in the industry that compared with general big models, industry big models for specific B-sides can better play their value and are easier to find their business models.

[0036] Therefore, for users, the evaluation of industry large models is of vital strategic significance. By conducting a detailed and comprehensive horizontal comparison of different large models in the industry, it helps to grasp their own research and development level in the field of artificial intelligence, and compare it with the advanced technologies in the industry, understand the gap between top technologies, and clarify the direction of future technology optimization and development.

[0037] There is currently a lack of unified evaluation standards in the industry for the evaluation of industry large models. Different evaluation institutions and enterprises may propose different industry large model evaluation standards and methods. Since the evaluation of industry large models involves the construction of industry data sets, and industry data involves user business secrets, the evaluation of industry large models requires each user to build it according to their own needs. Although most users have common overall methods for the evaluation of industry large models and can form a recognized industry large model evaluation methodology, the current industry large model evaluation still has the following defects:

[0038] (1) Since industry big models focus on industry segmentation scenarios, which increases professional difficulty, the evaluation of industry big models requires a deep understanding of the mechanism and capabilities of the big models, as well as understanding of industry attributes, application scenarios and other information, in order to build a comprehensive and objective industry big model evaluation method.

[0039] (2) Industry evaluation datasets need to be independently constructed based on industry attribute application scenarios and require the participation of industry experts, which consumes a lot of manpower and time costs.

[0040] (3) Due to the diversity and complexity of natural language, many evaluation samples cannot form standard answers, or produce multiple standard answers, which makes the corresponding evaluation results difficult to quantify and has a low degree of automation.

[0041] In order to overcome at least one of the above-mentioned defects, in the present disclosure, by constructing a more comprehensive industry large-model evaluation method and system, the construction of the evaluation data set, evaluation method, evaluation process, evaluation indicators, evaluation system, etc. are introduced in detail.

[0042] To facilitate understanding, several terms involved in this application are first explained below.

[0043] Large Language Models (LLM): refers to machine learning models with large-scale parameters and complex computational structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters.

[0044] Industry big models: refers to big models that are trained and optimized specifically for a specific field or industry. Unlike general big models, industry big models are generally fine-tuned with data from a specific field, so they are more focused on the knowledge and applications of a specific field, such as law, finance, and medicine.

[0045] User penetration rate: refers to the percentage of users who use some products or services in a specific period of time to the total number of potential users. It is an important indicator to measure the market penetration of products or services. It can help companies understand the popularity of products or services in the market and help companies formulate reasonable marketing strategies.

[0046] Market share, also known as market share, is defined as the proportion or percentage of a company's products (or a certain product) in the market sales of similar products at a certain time.

[0047] First Token Latency: The response time required for LLM reasoning to generate the first character is the response time of the LLM reasoning service perceived by users, which directly affects the user experience. For online services, in order to improve the user experience, the first token latency is expected to be small, generally less than 2 to 3 seconds.

[0048] Tokens per second without 1st token (single instance throughput): The number of tokens generated per second, excluding the first token. It is used to evaluate the inference speed of the model. For online services, tps is generally required to be greater than 14 to 15.

[0049] Figure 1 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application. The system includes: a plurality of terminals 120 and a server cluster 140.

[0050] The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the terminal 120 may also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0051] The terminal 120 may be installed with an application for providing model evaluation.

[0052] The terminal 120 is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0053] The server cluster 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server cluster 140 is used to provide background services for applications that provide model evaluation. Optionally, the server cluster 140 undertakes the main computing work, and the terminal 120 undertakes the secondary computing work; or, the server cluster 140 undertakes the secondary computing work, and the terminal 120 undertakes the main computing work; or, the terminal 120 and the server cluster 140 adopt a distributed computing architecture for collaborative computing.

[0054] In some optional embodiments, the server cluster 140 is used to store model evaluation program information.

[0055] An industry large model evaluation system according to an embodiment of the present disclosure includes: an evaluation system user management module, an evaluation system permission module, an evaluation task management module, an evaluation agent module, an evaluation model management module, an evaluation data set management module, an evaluation reasoning module, a referee scoring module, an evaluation result display module, an evaluation system log management module, etc.

[0056] In some optional embodiments, the evaluation system user management module, the evaluation system authority module, the evaluation task management module and the evaluation result display module may be loaded in the terminal 120 .

[0057] Optionally, the client of the application installed in different terminals 120 is the same, or the client of the application installed on the two terminals 120 is the client of the same type of application of different control system platforms. Based on the different terminal platforms, the specific form of the client of the application may also be different, for example, the application client may be a mobile client, a PC client or a World Wide Web (Web) client.

[0058] Those skilled in the art will appreciate that the number of the terminals 120 may be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or a greater number. The embodiment of the present application does not limit the number and device type of the terminals.

[0059] Optionally, the system may also include a management device ( Figure 1 (not shown), the management device is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0060] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0061] like Figure 2 As shown, a model evaluation method according to an embodiment of the present disclosure includes:

[0062] Step S202, determining a first evaluation result in business performance based on a performance indicator value obtained when the industry model performs an industry reasoning operation.

[0063] In some embodiments, performance indicators that need to be paid attention to when the industry model performs industry reasoning operations are determined. These indicators may include but are not limited to: response time, accuracy, throughput, etc.

[0064] In some embodiments, different industry models focus on different performance indicators. For example, in financial industry models, accuracy may be a key indicator for risk assessment tasks, that is, the degree to which the model's judgment of risk is consistent with the actual situation. In the recommendation system of the e-commerce industry, response time may be important because users expect to get personalized recommendations quickly.

[0065] In some embodiments, real industry data can be input into the model, and the model's output results and corresponding performance indicator values ​​can be recorded. For example, for a medical diagnosis industry model, a large amount of patient symptoms and examination data can be input, and the time it takes for the model to give a diagnosis result and the accuracy of the diagnosis can be observed as the first evaluation result.

[0066] Step S204, determining a second evaluation result in industry capability based on the user penetration index of the industry model.

[0067] In some embodiments, user penetration rate generally refers to the ratio of the number of users using the industry model in the target market to the total number of potential users.

[0068] In some embodiments, the second evaluation result is determined based on the user penetration rate. Different penetration rate intervals may be set to correspond to different evaluation levels, or the user penetration rate performance of the industry model may be evaluated by comparison with competitors.

[0069] Step S206, performing scenario capability evaluation on the industry model based on the multi-judge model to obtain a third evaluation result.

[0070] In some embodiments, the referee model can be other advanced large language models or expert systems in specific fields, and the output results of the industry model are submitted to the multi-referee model for evaluation. The multi-referee model will evaluate the output of the industry model according to the evaluation tasks and indicators, and give corresponding scores or evaluation opinions to obtain the third evaluation result.

[0071] In some embodiments, based on the application scenarios of the industry model, a series of evaluation tasks are determined, such as question-answering tasks, text generation tasks, problem-solving tasks, etc., and corresponding evaluation indicators are formulated, such as accuracy, completeness, logic, language expression, etc.

[0072] Step S208, generating an overall evaluation result for the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first set of weights.

[0073] In some embodiments, the weights may be determined based on factors such as industry characteristics, business needs, and market competition.

[0074] In some embodiments, the overall evaluation result (S final ), can be based on the large model industry capability score (S scence ) The third assessment result, the business ability score, the second assessment result (S service ) and the first evaluation result of business performance score (S perf ) is generated according to weight calculation, as shown in formula (1):

[0075] S final =0.6×S scence +0.3×S service +0.1×S perf (1)

[0076] In this embodiment, by verifying the industry model from the aspects of business performance, industry capability and scenario capability to integrate the evaluation results of multiple dimensions, a more comprehensive and integrated evaluation of the industry model can be achieved. On the one hand, it helps enterprises to understand the performance of the model from multiple perspectives and promote the research and development and application of large industry models. On the other hand, it realizes the automated processing of model evaluation, which can improve the efficiency of model evaluation and assist in the iterative optimization of large industry models.

[0077] like Figure 3 As shown, in one embodiment of the present disclosure, determining a first evaluation result in business performance based on a performance indicator value obtained when an industry reasoning operation is performed based on an industry model includes:

[0078] Step S302, obtaining the latency index value when the industry model performs industry reasoning operations and the single instance flux of the industry model as performance index values, wherein the latency index value includes at least one of the first word delay, the last word delay, and the average reasoning delay.

[0079] Among them, the first-word latency is the time interval from issuing a request to receiving the first character of the model output, and the last-word latency is the time interval from issuing a request to receiving the last character of the complete answer; the average inference latency is the average of the inference latency of multiple requests.

[0080] Single-instance throughput refers to the number of tasks that a single model instance can process per unit time, usually measured in tasks processed per second.

[0081] Step S304: scoring the delay index value based on the multiple delay intervals to obtain a first score.

[0082] In some embodiments, taking the first word delay as an example, the delay index value is scored based on multiple delay intervals as shown in Table 1.

[0083] Table 1

[0084] First word delay ftl (unit: seconds) Score ftl<=0.5 1 0.5<ftl<=1 0.8 1<ftl<=2 0.6 2<ftl<=3 0.4 3<ftl 0.1

[0085] In some embodiments, specific scoring criteria may be determined based on industry needs and user expectations. For example, in some industries with extremely high real-time requirements, such as financial transactions, online games, etc., extremely short delays may be given higher scores, while in some industries with relatively low real-time requirements, such as document processing, data analysis, etc., longer delays may also receive certain scores.

[0086] Step S306: Score the single instance flux based on the flux threshold to obtain a second score.

[0087] In some embodiments, the determination of the flux threshold may take into account the needs of the industry and the performance goals of the model. For example, in some high-concurrency industries, such as e-commerce, social media, etc., a higher single-instance throughput may be required, so the flux threshold may be set relatively high, while in some low-concurrency industries, such as academic research, personal applications, etc., the flux threshold may be set relatively low.

[0088] In some embodiments, the single instance flux is compared to the threshold and a higher score may be given if the single instance flux is above the threshold and a lower score may be given if it is below the threshold.

[0089] In some embodiments, single instance flux is scored as shown in Table 2.

[0090] Table 2

[0091] Single instance throughput tps (unit: tokens / s) Score tps>20 1 15<tps<=20 0.8 10<tps<=15 0.6 5<tps<=10 0.4 tps<=5 0.1

[0092] Step S308, determining a first weight corresponding to the first score and a second weight corresponding to the second score based on the industry to which the industry model belongs.

[0093] In some embodiments, since different industries attach different importance to latency and throughput, a first weight corresponding to the first score and a second weight corresponding to the second score can be determined based on the industry to which the industry model belongs.

[0094] Step S310, obtaining a first evaluation result based on the first score, the first weight, the second score and the second weight.

[0095] In this embodiment, business performance (Sperf) assesses the business reasoning performance of the industry's large model, including indicators such as latency and single-instance flux (tokens / s). By obtaining performance indicator values ​​such as latency and single-instance flux, and scoring and weighted calculation, the first-word latency and last-word latency can reflect the response speed of the model, the average reasoning latency can reflect the stability of the model, and the single-instance flux can reflect the processing power and efficiency of the model. By comprehensively considering these indicators, the performance of the model can be more comprehensively understood, and the performance of the industry model when performing industry reasoning operations can be accurately evaluated.

[0096] In one embodiment of the present disclosure, before determining the first evaluation result in the business performance based on the performance indicator value obtained when performing the industry reasoning operation based on the industry model, it also includes: deploying the industry model in a half-precision floating point format.

[0097] In this embodiment, the large model is deployed on a server with a specified hardware configuration, and the large model is loaded with BF16 (Brain FloatingPoint 16) accuracy, which can reduce the memory usage and computational complexity of the model while maintaining the accuracy of the model to a certain extent.

[0098] In one embodiment of the present disclosure, determining a second evaluation result in industry capability based on a user penetration index of an industry model includes: determining the second evaluation result based on the user penetration and a business share index of the industry model.

[0099] In some embodiments, the marketability of the big model is measured based on segmented dimensions such as the number of business customers, business market size, and industry market share. Among them, the user penetration rate indicator (Susers) is used to represent the number of users occupied by the industry big model industry market, and the business market share indicator (Smarket) is used to represent the scale of the industry big model business market.

[0100] The second evaluation result is determined based on the user penetration rate and the business share index of the industry model, as shown in formula (2):

[0101] S service =0.5×S user +0.5×market (2)

[0102] In some embodiments, user penetration rate refers to: the number of users who use the product or service in a specific time period divided by the potential user growth rate, and multiplied by 100%. Assuming that the total number of potential users in a market is 1,000, and the number of users who use the product or service in a specific time period is 200, and the potential user growth rate of the market is 10%, then the user penetration rate of the product or service is (200 / 10%) x 100% = 2,000%.

[0103] Market share refers to the proportion of the sales of the industry's large model to the total sales of the entire industry. The formula is shown in (3).

[0104] Market share = company sales / industry total sales × 100% (3)

[0105] In some embodiments, the user penetration rate indicator can intuitively demonstrate the popularity and prevalence of the industry model in the target market, and the business share indicator reflects the share of the industry model in the entire industry business, which is helpful in assisting in understanding market demand and user behavior based on the overall evaluation results.

[0106] In one embodiment of the present disclosure, it also includes: determining the number of users of the industry model based on the user's registration data and / or access log of the industry model; determining the total number of model users in the field where the industry model is located based on at least one of the market research data, statistical agency data and industry report data in the field where the industry model is located; determining the user penetration rate based on the number of users of the industry model and the total number of model users.

[0107] In this embodiment, if the industry model requires user registration before use, the number of users can be determined by analyzing the registration database. Even if the model does not require registration, the number of users can be determined by analyzing the access log. In addition, the total number of model users in the field where the industry model is located can be determined by combining market research data, statistical agency data and industry report data to further obtain user penetration rate. By using user penetration rate as one of the evaluation parameters of the model, it helps to understand user needs and behaviors, thereby guiding the optimization and improvement of the industry model.

[0108] In one embodiment of the present disclosure, it also includes: determining a business share index based on the sales of the industry model and the total sales of the model in the field where the industry model is located.

[0109] like Figure 4 As shown, in one embodiment of the present disclosure, the scenario capability of the industry model is evaluated based on the multi-referee model to obtain a third evaluation result, including:

[0110] Step S402, constructing a scenario capability assessment dataset based on question-answer pair samples, where the question-answer pair samples include corresponding question samples and answer samples.

[0111] Step S404: Reasoning the problem sample based on the industry model to obtain a reasoning result.

[0112] Step S406, constructing a universal ability judgment model based on the multi-judge model, and judging the reasoning result based on the universal ability judgment model to obtain a first evaluation value.

[0113] In some embodiments, by analyzing and integrating the outputs of multiple referee models, a model is trained that can judge the question-answering and comprehension capabilities of reasoning results in general dimensions such as intelligent agents and logical reasoning. The model is used to judge the reasoning results of the industry model to obtain a first evaluation value, which is used to reflect the performance of the reasoning results in general capabilities.

[0114] Step S408: evaluating the inference result based on the security capability judgment model to obtain a second evaluation value.

[0115] In some embodiments, the alignment and security of the reasoning results in terms of ideology, mainstream values, etc. are evaluated based on the security capability judgment model. For example, a model based on text classification can be trained to determine whether the reasoning results contain inappropriate content, whether they comply with security regulations, etc. The security capability judgment model is used to judge the reasoning results to obtain a second evaluation value, which reflects the performance of the reasoning results in terms of security capabilities.

[0116] Step S410: evaluating the inference result based on the domain task capability judgment model to obtain a third evaluation value.

[0117] In some embodiments, the domain task capability judgment model can evaluate the ability of the reasoning results in processing industry scenario tasks, and use the domain task capability judgment model to judge the reasoning results to obtain a third evaluation value, which reflects the performance of the reasoning results in terms of industry capabilities.

[0118] Step S412, obtaining a third evaluation result based on the first evaluation value, the second evaluation value, the third evaluation value and the corresponding second set of weights.

[0119] Among them, the first evaluation value corresponds to general ability, which refers to the question-answering and comprehension ability of the large model in general dimensions such as intelligent agents and logical reasoning.

[0120] The second evaluation value corresponds to industry capability, which is to evaluate the ability of the big model to handle industry scenarios by benchmarking against the benchmark big model in the industry field.

[0121] The third evaluation value corresponds to security capability, which refers to the alignment and security of the output generated by the evaluation model in terms of ideology, mainstream values, etc.

[0122] In some embodiments, an implementation method for obtaining a third evaluation result based on the first evaluation value, the second evaluation value, the third evaluation value and the corresponding second set of weights is as shown in formula (4).

[0123] S scence =0.6×S task +0.3×S safety +0.1×Salignment (4)

[0124] Among them, S task Refers to industry capabilities, S safety Refers to safety capability, S alignment Refers to general ability.

[0125] In this embodiment, by constructing multiple different judgment models and judging the reasoning results of the industry model from three aspects: general capability, security capability and domain task capability, the comprehensive capability of the industry model can be evaluated more comprehensively and objectively. By using a multi-judge model to construct a general capability judgment model, the advantages of multiple advanced models can be combined to improve the accuracy and reliability of the judgment. In addition, by constructing special judgment models for security and domain tasks respectively, the capabilities in these aspects can be evaluated more specifically. Different application scenarios may have different requirements for the capabilities of the industry model. By comprehensively considering the general capability, security capability and domain task capability, combined with the weights of specific application scenarios, evaluation results that are more in line with actual needs can be obtained.

[0126] In one embodiment of the present disclosure, a universal ability judgment model is constructed based on a multi-judge model, and a first evaluation value is obtained by judging the reasoning result based on the universal ability judgment model, including:

[0127] The matching multi-referee model is determined based on the type of industry model.

[0128] In some embodiments, different types of industry models may require different referee models for accurate evaluation. For example, for industry models in the field of natural language processing, a large language model that excels in language understanding and generation can be selected as a referee; for industry models in the field of image recognition, a model that excels in image analysis and evaluation can be selected as a referee.

[0129] Based on the multi-judge model, the reasoning results are evaluated in multiple dimensions to obtain subjective evaluation results. The multiple dimensions include answer accuracy, answer regularity and answer friendliness.

[0130] In some embodiments, a multi-judge large model (GTP-4o, Qwen1.5-110B) + manual review method can be used to perform subjective evaluation and scoring.

[0131] In some embodiments, answer accuracy is used to evaluate the closeness of the inference result to the correct answer, answer regularity is used to examine the format specification of the inference result, and answer friendliness is used to measure the humanization of the inference result and user experience, as shown in Table 3.

[0132] Table 3

[0133]

[0134] In some embodiments, the subjective evaluation result = 0.8*answer accuracy score + 0.1*answer regularity score + 0.1*answer friendliness score.

[0135] Based on the consistency judgment between the reasoning results and the answer samples, objective evaluation results are obtained.

[0136] In some embodiments, the scenario capability assessment data set consists of question-answer pairs, including two types of questions: subjective questions and objective questions. Objective questions include single-choice questions, multiple-choice questions and other question types. The assessment indicators and scoring standards for different question types are shown in Table 4.

[0137] Table 4

[0138]

[0139] A first evaluation value is obtained based on the subjective evaluation result and the objective evaluation result.

[0140] In this embodiment, the inference results of the industry model are evaluated from both subjective and objective aspects through multi-dimensional evaluation and consistency judgment, which can provide a more comprehensive and accurate understanding of the performance of the industry model. The use of a multi-referee model for evaluation can reduce the subjectivity and limitations of a single referee and improve the reliability and credibility of the evaluation. In addition, according to different industry model types and application scenarios, appropriate multi-referee models and evaluation dimensions are selected to meet the evaluation needs in different scenarios.

[0141] In one embodiment of the present disclosure, the reasoning result is evaluated based on the security capability judgment model to obtain a second evaluation value, including: constructing a security capability judgment model based on accuracy evaluation parameters, objectivity evaluation parameters, positivity evaluation parameters and fairness evaluation parameters; based on the security capability judgment model, respectively evaluating the accuracy value, objective value, positivity value and fairness value of the reasoning result to obtain a second evaluation value based on the accuracy value, objective value, positivity value and fairness value.

[0142] In this embodiment, the accuracy evaluation parameter is used to measure the degree of conformity between the inference result and the actual situation, the objectivity evaluation parameter is intended to evaluate whether the inference result is free of subjective bias, the positivity evaluation parameter focuses on whether the inference result conveys positive information and values, and the fairness evaluation parameter considers whether the inference result is fair to different groups, viewpoints and situations. A safety capability judgment model can be constructed by using machine learning algorithms, rule engines and other technical means, and a second evaluation value is obtained based on the accuracy value, objective value, positivity value and fairness value. Methods such as weighted average can be used to assign corresponding weights according to the importance of different parameters, and then the four values ​​are comprehensively calculated to obtain the second evaluation value. By constructing a safety capability judgment model based on multiple evaluation parameters, the performance of the inference results output by the industry model in terms of accuracy, objectivity, positivity and fairness can be comprehensively evaluated, thereby more accurately measuring its safety capabilities.

[0143] In one embodiment of the present disclosure, the reasoning result is judged based on the domain task capability judgment model to obtain a third evaluation value, including: obtaining a reference result obtained by reasoning about the problem sample using a reference model; based on the configured domain capability evaluation index, the reasoning result and the reference result are evaluated to obtain the third evaluation value.

[0144] In some embodiments, an existing mature model with good performance in a specific field, that is, a leading industry model, is used as a reference model. The third evaluation value can be the normalized relative score between the industry model to be evaluated and the benchmark model, as shown in formula (5).

[0145] S task =S task-model / S task-baseline (5)

[0146] In one embodiment of the present disclosure, it also includes: obtaining verification information of the overall evaluation result; and generating a multi-dimensional evaluation report of the overall evaluation result based on the verification information.

[0147] In some embodiments, obtaining verification information for the overall evaluation results may include evaluation and feedback from industry experts, user representatives, etc. on the performance of the industry model, and comparing the inference results of the industry model with the evaluation and feedback information to generate a multi-dimensional evaluation report including industry, business performance, security, and general capabilities.

[0148] In this embodiment, by obtaining verification information of the overall evaluation results and generating a comprehensive, objective, multi-dimensional evaluation report, strong support is provided for the improvement and optimization of the industry model.

[0149] like Figure 5As shown, an industry large model evaluation system according to an embodiment of the present disclosure includes: an evaluation system user management module 502, an evaluation system permission module 504, an evaluation task management module 506, an evaluation agent module 508, an evaluation model management module 510, an evaluation data set management module 512, an evaluation reasoning module 514, a referee scoring module 516, an evaluation result display module 518 and an evaluation system log management module 520.

[0150] The evaluation system user management module 502 is used for system user management, such as adding, deleting, modifying, and querying users and assigning permissions.

[0151] The evaluation system permission module 504 is used for access permission management of each module.

[0152] The evaluation task management module 506 is used to create, launch, execute, and stop evaluation tasks, and supports querying of task status, results, and scoring results.

[0153] The evaluation agent module, namely the evaluation proxy module 508, is used to implement the evaluation execution of different industry models. The evaluation of each large industry model is implemented by an agent, which supports the heartbeat function and the evaluation result storage function.

[0154] The evaluation model management module 510 is used for downloading, storing, querying, deleting, modifying, adding, deleting, and querying model data, model details, model KEY data, management of the model prompt format, and model execution environment management such as container environment and conda environment management.

[0155] The evaluation data set management module 512 is used for the management of the evaluation data set, such as adding, deleting, modifying and checking the data set.

[0156] The evaluation and reasoning module 514 is used to execute the reasoning of the model and feedback the reasoning results, such as the reasoning of the local model, the API call reasoning of the cloud model, etc.

[0157] The referee scoring module 516 is used to call the referee large model to judge and score the model reasoning results and output the scoring results.

[0158] The evaluation result display module 518 is used for front-end display of the evaluation results.

[0159] The evaluation system log management module 520 is used for the management of system logs, such as abnormality query and the like.

[0160] like Figure 6As shown, the execution model evaluation is started based on the evaluation agent module 508, the evaluation data set management module 512 provides an evaluation data set including questions and answers, and the evaluation model management module 510 is used to configure the industry big model and prompt template, and clearly convey the tasks to be solved or the topics to be discussed to the industry big model based on the prompt template.

[0161] The evaluation reasoning module 514 executes the reasoning of the model and feeds back the output result of the reasoning, and inputs the output result and the corresponding answer into the referee model provided by the referee scoring module 516 for scoring to obtain the evaluation result.

[0162] like Figure 7 As shown, according to another embodiment of the present disclosure, the model evaluation method includes:

[0163] Step S702: Formulate an industry evaluation plan.

[0164] Among them, the industry big model evaluation expert team will formulate the industry big model evaluation plan based on the industry characteristics and output the industry big model evaluation plan.

[0165] Step S704: adding the industry evaluation data set to the evaluation system based on the industry evaluation solution.

[0166] Among them, a comprehensive, authoritative industry big model evaluation data set is jointly constructed by an industry big model evaluation expert team and industry experts, and the evaluation data set is output into the evaluation system.

[0167] Step S706: Based on the industry evaluation solution, the evaluation system calls the industry big model to return the inference result.

[0168] Among them, the industry big model evaluation system calls the industry big model API to complete the industry big model reasoning and output the results.

[0169] Step S708: Based on the industry evaluation solution, the evaluation system calls the referee model to judge the reasoning results.

[0170] Among them, the industry big model evaluation expert team selects the referee big model according to the characteristics of the industry, and the industry big model evaluation system judges the inference results through the referee big model and outputs the results.

[0171] Step S710, verifying the judgment result.

[0172] Among them, a team of industry large-scale model evaluation experts will manually verify the referee results to avoid misjudgment of the referee large-scale model and output the verified referee results.

[0173] Step S712: the evaluation system generates an evaluation report.

[0174] Among them, the industry big model evaluation system generates a multi-dimensional evaluation report for the industry big model according to the judgment results after verification, and outputs the industry big model evaluation report.

[0175] In this embodiment, for the industry big model, the evaluation adopts three dimensions: scenario capability, business capability, and business performance. The evaluation results of each dimension are allocated according to specific weights, and finally form a comprehensive capability evaluation conclusion for the industry big model to be tested, among which the scenario capability (60%) includes multiple secondary dimensions and tertiary dimensions, which measure the performance of the big model in general, industry-specific scenarios, security, etc. The business capability dimension (30%) is used to measure the marketability of the industry big model, including the industry big model business user penetration rate (50%) and the industry big model business market share (50%). Two subdivision dimensions reflect the industry market expansion situation, business performance includes the first word delay and single instance throughput of the evaluation of the big model business. Two business processing capability indicators, the industry big model overall capability score (Sfinal), is generated by the big model industry capability score (Sscene), business capability score (Sservice) and business performance score (Sperf) according to weight calculation.

[0176] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0177] Refer to the following Figure 8 To describe the model evaluation device 800 according to an embodiment of the present disclosure. Figure 8 The model evaluation device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0178] The model evaluation device 800 is expressed in the form of a hardware module or a software module. The components of the model evaluation device 800 may include but are not limited to: a first evaluation module 802, which is used to determine the first evaluation result in business performance based on the performance indicator value obtained when the industry model performs the industry reasoning operation; a second evaluation module 804, which is used to determine the second evaluation result in industry capability based on the user penetration rate indicator of the industry model; a third evaluation module 806, which is used to evaluate the scenario capability of the industry model based on the multi-judge model to obtain a third evaluation result; a generation module 808, which is used to generate an overall evaluation result of the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first set of weights.

[0179] like Fig. 9As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0180] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 2 The scheme described in .

[0181] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0182] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0183] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0184] The electronic device 900 may also communicate with one or more external devices 970 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0185] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0186] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to enable the electronic device to perform the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0187] According to the program product for implementing the above method according to the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0188] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0189] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0190] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0191] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the IoT terminal, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0192] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0193] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0194] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0195] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A model evaluation method, characterized in that: include: Determine a first evaluation result in business performance based on a performance indicator value obtained when performing an industry reasoning operation on the industry model; Determining a second evaluation result in industry capability based on the user penetration rate indicator of the industry model; Perform scenario capability evaluation on the industry model based on the multi-judge model to obtain a third evaluation result; An overall evaluation result of the industry model is generated based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first set of weights.

2. The model evaluation method according to claim 1, characterized in that: The first evaluation result in the business performance is determined based on the performance indicator value obtained when the industry reasoning operation is performed based on the industry model, including: Obtaining a latency index value when the industry model performs an industry reasoning operation and a single instance throughput of the industry model as the performance index value; Scoring the delay index value based on multiple delay intervals to obtain a first score; Scoring the single instance flux based on a flux threshold to obtain a second score; Determine a first weight corresponding to the first score and a second weight corresponding to the second score based on the industry to which the industry model belongs; Based on the first score, the first weight, the second score and the second weight, the first evaluation result is obtained. The delay index value includes at least one of a first word delay, a last word delay and an average inference delay.

3. The model evaluation method according to claim 1, characterized in that: Before determining the performance indicator value obtained when the industry reasoning operation is performed based on the industry model as the first evaluation result in the business performance, the method further includes: The industry model is deployed using a half-precision floating point format.

4. The model evaluation method according to claim 1, characterized in that: The second evaluation result in the industry capability is determined based on the user penetration rate indicator of the industry model, including: The second evaluation result is determined based on the user penetration rate and the business share index of the industry model.

5. The model evaluation method according to claim 1, characterized in that: Also includes: Determining the number of users of the industry model based on the user registration data and / or access logs of the industry model; Determine the total number of model users in the field where the industry model is located based on at least one of market research data, statistical agency data, and industry report data in the field where the industry model is located; The user penetration rate is determined based on the number of users of the industry model and the total number of users of the model.

6. The model evaluation method according to claim 4, characterized in that: Also includes: The business share index is determined based on the sales of the industry model and the total sales of models in the field where the industry model is located.

7. The model evaluation method according to claim 1, characterized in that: The scenario capability of the industry model is evaluated based on the multi-judge model to obtain the third evaluation result, including: Constructing a scenario capability assessment dataset based on question-answer pair samples, wherein the question-answer pair samples include corresponding question samples and answer samples; Reasoning the problem sample based on the industry model to obtain a reasoning result; Constructing a universal ability judgment model based on the multi-judge model, so as to judge the reasoning result based on the universal ability judgment model to obtain a first evaluation value; Judging the inference result based on the safety capability judgment model to obtain a second evaluation value; Judging the reasoning result based on the domain task capability judgment model to obtain a third evaluation value; The third evaluation result is obtained based on the first evaluation value, the second evaluation value, the third evaluation value and the corresponding second set of weights.

8. The model evaluation method according to claim 7, characterized in that: Constructing a universal ability judgment model based on the multi-judge model, and judging the reasoning result based on the universal ability judgment model to obtain a first evaluation value, including: Determining the matching multi-referee model based on the type of the industry model; Performing a multi-dimensional evaluation on the inference result based on the multi-judge model to obtain a subjective evaluation result, wherein the multi-dimensional evaluation includes answer accuracy, answer regularity, and answer friendliness; Obtaining an objective evaluation result based on the consistency judgment between the reasoning result and the answer sample; The first evaluation value is obtained based on the subjective evaluation result and the objective evaluation result.

9. The model evaluation method according to claim 7, characterized in that: The reasoning result is evaluated based on the security capability judgment model to obtain a second evaluation value, including: Constructing the security capability judgment model based on accuracy evaluation parameters, objectivity evaluation parameters, positivity evaluation parameters and fairness evaluation parameters; The accurate value, objective value, positive value and fair value of the inference result are evaluated respectively based on the security capability judgment model to obtain the second evaluation value based on the accurate value, the objective value, the positive value and the fair value.

10. The model evaluation method according to claim 7, characterized in that: The reasoning result is judged based on the domain task capability judgment model to obtain a third evaluation value, including: Obtain a reference result obtained by reasoning the problem sample using a reference model; Based on the configured domain capability evaluation index, the reasoning result and the reference result are evaluated to obtain the third evaluation value.

11. The model evaluation method according to any one of claims 1 to 10, characterized in that: Also includes: Obtaining verification information of the overall evaluation result; A multi-dimensional evaluation report of the overall evaluation results is generated based on the verification information.

12. A model evaluation device, characterized in that: include: A first evaluation module, used to determine a first evaluation result in business performance based on a performance indicator value obtained when an industry model is executed for industry reasoning operations; A second evaluation module, used to determine a second evaluation result in industry capability based on the user penetration rate indicator of the industry model; A third evaluation module is used to evaluate the scenario capability of the industry model based on a multi-judge model to obtain a third evaluation result; A generation module is used to generate an overall evaluation result of the industry model based on the first evaluation result, the second evaluation result, the third evaluation result and the corresponding first set of weights.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the model evaluation method described in any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model evaluation method described in any one of claims 1 to 11 is implemented.

15. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model evaluation method described in any one of claims 1 to 11 is implemented.

Images