A large model evaluation system, large model evaluation method and product

Abstract

The embodiment of the present application provides a large model evaluation system, a large model evaluation method and a product, which belongs to the technical field of large models. The evaluation system includes an evaluation interaction module and an evaluation chain. The evaluation chain includes multiple blockchain nodes, and any blockchain node among the multiple blockchain nodes corresponds to an evaluation party agent module of an evaluation party, wherein: the evaluation interaction module is used to respond to the target operation of the large model user and initiate an evaluation request of the target large model to the evaluation chain; the evaluation chain is used to respond to the evaluation request of the target large model and determine multiple target blockchain nodes according to the type of the target large model; the evaluation party agent modules corresponding to each of the multiple target blockchain nodes evaluate the target large model respectively, and the evaluation data generated by any evaluation party agent module is synchronized to all blockchain nodes of the evaluation chain. The embodiment of the present application aims to generate more accurate evaluation results for any large model.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of large models, and specifically, to a large model evaluation system, a large model evaluation method, and a product. Background Art

[0002] Artificial intelligence models were initially trained for specific application scenarios, that is, small models, which have the problem of poor versatility and need to be retrained for new scenarios. With the development of AI technology, data, computing power and algorithms have been improved, and small models have gradually evolved into large models. Large models are pre-trained on massive and broad data, have large scale and generalization, and are key technologies for realizing general intelligence.

[0003] As the foundation of AI applications, big models have become an important tool. How to choose a suitable big model among a large number of big models has become a major challenge for users to build AI applications, and evaluating the capabilities of big models is a key issue.

[0004] At present, the evaluation method for large models is that any evaluation organization or evaluation party first defines an evaluation benchmark, and then tests the large model to be tested through a test data set, and scores each indicator according to the test results; the evaluation benchmark consists of a set of standardized test sets or tasks, such as test sets or tasks covering multiple fields such as text generation, text classification, and question-answering systems, in order to evaluate the performance of large models in processing natural language tasks; the evaluation benchmarks are roughly divided into three categories, namely: general benchmarks, specific benchmarks and multi-modal benchmarks.

[0005] However, the current method of evaluating large models based on any evaluation organization or evaluation party is that the evaluation results of the evaluator are not public data, lack multi-party witnesses and consensus, and the evaluation processes of different evaluators are different, resulting in a lack of universal and accurate evaluation results for any large model. Summary of the invention

[0006] The embodiments of the present application provide a large model evaluation system, a large model evaluation method and a product, aiming to generate more accurate evaluation results for any large model.

[0007] In a first aspect, an embodiment of the present application provides a large model evaluation system, the evaluation system comprising an evaluation interaction module and an evaluation chain, the evaluation chain comprising a plurality of blockchain nodes, any one of the plurality of blockchain nodes corresponding to an evaluation party agent module of an evaluation party, wherein:

[0008] The evaluation interaction module is used to respond to the target operation of the large model user and initiate an evaluation request of the target large model to the evaluation chain;

[0009] The evaluation chain is used to respond to the evaluation request of the target large model and determine multiple target blockchain nodes according to the type of the target large model;

[0010] The evaluation agent modules corresponding to the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation agent module is synchronized to all blockchain nodes in the evaluation chain.

[0011] Optionally, the evaluation chain is used to:

[0012] Determine a target evaluation benchmark corresponding to the target large model according to the type of the target large model, and select multiple candidate blockchain nodes corresponding to multiple evaluation parties having the target evaluation benchmark;

[0013] Selecting a calibrated number of the plurality of target blockchain nodes from the plurality of candidate blockchain nodes based on a consensus algorithm, and determining a primary blockchain node from the plurality of target blockchain nodes, wherein the evaluation party agent module corresponding to the primary blockchain node is used to execute an evaluation strategy on the target large model;

[0014] When the evaluation agent module corresponding to the main blockchain node completes the execution of the evaluation strategy for the target large model, the next target blockchain node is selected as the main blockchain node among the multiple target blockchain nodes until the multiple target blockchain nodes have completed the evaluation strategy for the target large model.

[0015] Optionally, when any target blockchain node serves as the main blockchain node, the evaluation party agent module corresponding to the target blockchain node executes the evaluation strategy of the target large model. If the evaluation data for the target large model is not generated within the target time, the evaluation chain is used to:

[0016] The target blockchain node is eliminated from the multiple target blockchain nodes, the next target blockchain node is selected as the main blockchain node, and a candidate blockchain node is selected from the multiple candidate blockchain nodes as the target blockchain node.

[0017] Optionally, any evaluator agent module includes an evaluation service execution unit. When any evaluator agent module executes the evaluation strategy on the target large model, the evaluation service execution unit in the evaluator agent module is used to:

[0018] Acquiring model information of the target large model on the evaluation chain;

[0019] When the model information of the target large model exists in the evaluation chain and the target large model has not completed the evaluation, selecting a target number of evaluation problems in the evaluation data set corresponding to the target evaluation benchmark;

[0020] The target large model is called according to the service calling interface of the target large model included in the model information of the target large model, and the target large model is evaluated according to the evaluation questions of the target quantity to generate evaluation data corresponding to the target large model.

[0021] Optionally, any evaluation party agent module further includes a data chain interaction unit. When the evaluation service execution unit of the evaluation party agent module generates evaluation data corresponding to the target large model, the data chain interaction unit is used to:

[0022] The evaluation data of the target large model is uploaded to the chain and stored in the corresponding blockchain node.

[0023] Optionally, when the evaluation data corresponding to the target large model generated by any evaluation party agent module is stored in the corresponding blockchain node, the blockchain node will broadcast the block containing the evaluation data of the target large model to all blockchain nodes in the evaluation chain.

[0024] Optionally, the same on-chain data copies are maintained on multiple blockchain nodes of the evaluation chain, and the on-chain data copies include: model information, evaluation data and evaluation records of any large model;

[0025] The evaluation data is the evaluation result generated by any evaluation agent module for the large model; the evaluation record is the comprehensive evaluation result determined by the evaluation chain based on the evaluation data generated by multiple evaluation agent modules for the large model.

[0026] Optionally, the on-chain data copy also includes information corresponding to different evaluation benchmarks and the evaluation benchmark corresponding to any evaluation party, and the information of any evaluation benchmark includes the evaluation benchmark name, evaluation benchmark type, evaluation benchmark description and evaluation benchmark version.

[0027] Optionally, an evaluation execution contract is deployed on the evaluation chain, the evaluation execution contract includes a large model registration interface, and the evaluation interaction module includes:

[0028] An evaluation registration module, for responding to a registration operation of a large model user, calling the large model registration interface of the evaluation execution contract, and registering the model information of any large model on the evaluation chain;

[0029] The model information includes the service call interface, name, version number and type of any large model.

[0030] Optionally, the evaluation execution contract includes an interface for obtaining evaluation data and an interface for obtaining evaluation records, and the evaluation interaction module also includes an evaluation result query module for:

[0031] In response to the first query operation of the large model user, calling the evaluation data acquisition interface of the evaluation execution contract to obtain the evaluation data of any evaluation party agent module for any large model on the evaluation chain;

[0032] In response to the second query operation of the large model user, the evaluation record acquisition interface of the evaluation execution contract is called to obtain the evaluation record of any large model on the evaluation chain. The evaluation record is a comprehensive evaluation result determined by the evaluation chain based on the evaluation data generated by multiple evaluation party agent modules for the large model.

[0033] Optionally, the evaluation interaction module further includes:

[0034] The evaluation interface verification module is used to verify whether the service call interface in the model information of any large model is correct.

[0035] Optionally, an evaluation execution contract is deployed on the evaluation chain, and the evaluation execution contract includes an interface for obtaining model information and an interface for writing evaluation data. Any evaluation party agent module is used to:

[0036] Call the model information acquisition interface of the evaluation execution contract to obtain the model information of any large model on the evaluation chain;

[0037] The write evaluation data interface of the evaluation execution contract is called to write the evaluation data corresponding to any generated large model into the evaluation chain.

[0038] Optionally, any evaluator agent module further includes:

[0039] The evaluation benchmark management unit is used to manage various evaluation benchmarks of the evaluation party agent module.

[0040] Optionally, an evaluation benchmark management contract is deployed on the evaluation chain, and the evaluation benchmark management contract includes an evaluation benchmark acquisition interface, and the evaluation benchmark management unit is used to:

[0041] In response to the evaluation benchmark acquisition operation, calling the evaluation benchmark acquisition interface of the evaluation benchmark management contract to acquire any evaluation benchmark on the evaluation chain;

[0042] In response to the evaluation benchmark adding operation, calling the evaluation benchmark adding interface of the evaluation benchmark management contract to add a new evaluation benchmark to the evaluation chain;

[0043] In response to the evaluation benchmark modification operation, the evaluation benchmark modification interface of the evaluation benchmark management contract is called to modify any evaluation benchmark on the evaluation chain.

[0044] Optionally, any evaluator agent module further includes an evaluation data set management unit, which is used to:

[0045] In response to the data set adding operation, adding any data set to the evaluator agent module;

[0046] In response to the data set modification operation, any data set is modified in the evaluator agent module.

[0047] Optionally, the target operation includes a registration operation of a large model user for a target large model or an evaluation request operation of a large model user for a target large model;

[0048] When the target operation is a registration operation of the large model user for the target large model, the evaluation interaction module is used to respond to the registration operation of the large model user, register the model information of the target large model on the evaluation chain, and initiate an evaluation request of the target large model to the evaluation chain;

[0049] When the target operation is an evaluation request operation of the large model user for the target large model, the evaluation request module in the evaluation interaction module responds to the evaluation request operation of the large model user and initiates an evaluation request for the target large model to the evaluation chain.

[0050] In a second aspect, an embodiment of the present application provides a large model evaluation method, the method comprising:

[0051] Constructing a large model evaluation system, wherein the large model evaluation system includes an evaluation interaction module and an evaluation chain, wherein the evaluation chain includes multiple blockchain nodes, and any blockchain node among the multiple blockchain nodes corresponds to an evaluation party agent module of an evaluator;

[0052] The evaluation interaction module initiates an evaluation request of the target large model to the evaluation chain in response to the target operation of the large model user;

[0053] The evaluation chain responds to the evaluation request of the target large model and determines a plurality of target blockchain nodes according to the type of the target large model;

[0054] The evaluation agent modules corresponding to the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation agent module is synchronized to all blockchain nodes in the evaluation chain.

[0055] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the large model evaluation method described in the second aspect of the embodiment is executed.

[0056] In a fourth aspect, an embodiment of the present application provides a non-volatile readable storage medium, wherein the non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the large model evaluation method described in the second aspect of the embodiment is performed.

[0057] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the large model evaluation method described in the second aspect of the embodiment.

[0058] Beneficial effects:

[0059] The evaluation system includes an evaluation interaction module and an evaluation chain. The evaluation chain includes multiple blockchain nodes. Any blockchain node among the multiple blockchain nodes corresponds to an evaluation party agent module of an evaluator. The evaluation interaction module responds to the target operation of the large model user and initiates an evaluation request of the target large model to the evaluation chain; the evaluation chain responds to the evaluation request of the target large model and determines multiple target blockchain nodes according to the type of the target large model; the evaluation party agent modules corresponding to each of the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation party agent module is synchronized to all blockchain nodes of the evaluation chain.

[0060] Based on blockchain, any large model is evaluated. Any blockchain node in the blockchain network of the evaluation chain represents an evaluator. The evaluation chain determines the evaluators participating in the evaluation, and the evaluation data of the large model generated by the evaluation agent of any evaluator is synchronized to all blockchain nodes on the evaluation chain. The consensus algorithm of blockchain technology supports multiple parties to participate in the evaluation of the large model, making the evaluation results of the generated large model more accurate and credible. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0062] Figure 1 A schematic diagram of the architecture of a large model evaluation system provided in an embodiment of the present application is shown;

[0063] Figure 2 A flowchart of the steps of the large model evaluation method provided in an embodiment of the present application is shown;

[0064] Figure 3 A schematic diagram of the evaluation process provided by an embodiment of the present application is shown;

[0065] Figure 4 A schematic diagram of the evaluation process provided by an embodiment of the present application is shown;

[0066] Figure 5 A schematic diagram of the evaluation process provided by an embodiment of the present application is shown;

[0067] Figure 6 An interactive flow chart of the evaluation process provided by an embodiment of the present application is shown;

[0068] Figure 7 A schematic diagram of a computer device provided in an embodiment of the present application is shown;

[0069] Figure 8 A schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application is shown;

[0070] Fig. 9 A schematic diagram of a computer program product provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0072] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, each embodiment of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in each embodiment of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not constitute any limitation, and the various embodiments can be combined with each other and referenced to each other without contradiction.

[0073] LLM: Large Language Model, large model or large language model;

[0074] AI: Artificial Intelligence.

[0075] Artificial intelligence models are initially trained for specific application scenarios, that is, small models, such as different machine learning models. However, small models have the problem of poor versatility and can only be adapted to specific application scenarios and undergo targeted model training. Retraining is required for new scenarios, and the training process involves parameter adjustment, optimization, and cost. Model training requires a large amount of labeled data. Insufficient data will lead to high training costs, low efficiency, and unsatisfactory accuracy.

[0076] With the development of AI technology, data, computing power and algorithms have been improved, and small models have gradually evolved into large models. Large models are pre-trained on massive and broad data, and have large scale and generalization capabilities. They are the key technology for realizing general intelligence.

[0077] There are three technical routes for the development of large language models: Bert mode (Bidirectional Encoder Representations from Transformers, a deep learning model based on the Transformer architecture), GPT mode (Generative Pre-trained Transformer, a generative pre-trained model), and hybrid mode; according to the task type, large models can be divided into single-task, multi-task, understanding type, and generation type large models.

[0078] The Bert mode is more suitable for single tasks such as understanding and a specific scenario. It uses two-way language model pre-training and task fine-tuning, and is "specialized but light"; the GPT mode performs best in generation tasks. It uses unidirectional language model pre-training and zero-sample prompt word technology, which is suitable for generation tasks or multi-tasks, and is "heavy but general". Compared with Bert, the GPT model has a larger number of parameters and achieves more general performance. As the number of parameters increases, the effect of the GPT model becomes better and better; the hybrid mode combines the two methods, using unidirectional language model pre-training and task fine-tuning.

[0079] As the foundation of AI applications, big models have become an important tool. Among the numerous big models, how to choose the right big model to apply has become a major challenge for users to build AI applications. Therefore, making a reasonable evaluation or assessment of the capabilities of big models is a key issue.

[0080] The large model evaluation method usually defines an evaluation benchmark first, and then tests the large model to be tested through an evaluation set, and scores each indicator according to the evaluation results; the evaluation benchmark consists of a set of standardized test sets or tasks, such as test sets or tasks covering multiple fields such as text generation, text classification, and question-answering systems, to evaluate the performance of large models in processing natural language tasks; current evaluation benchmarks are roughly divided into three categories, namely: general benchmarks, specific benchmarks, and multi-modal benchmarks.

[0081] A general evaluation benchmark is a comprehensive evaluation method used to test the performance of large models in various aspects. It generally includes a variety of natural language processing tasks, such as sentiment analysis, question and answer, and textual implications. A segmented domain evaluation benchmark is a specialized evaluation method used to test the performance of large models on specific tasks. It generally only includes one or several specific evaluation tasks. A multimodal evaluation benchmark is a comprehensive evaluation method used to evaluate the ability of large models to process multimodal data. It includes multiple types of evaluation tasks.

[0082] However, the current method of evaluating large models based on any evaluation benchmark usually uses conventional test problems or tasks to test large models. However, the excellent performance of a large model in a single task does not represent the global capability of the large model, which is prone to evaluation misleading. In addition, based on commonly used test problems, there are problems such as targeted training of large models to obtain higher evaluation results. When any evaluation organization evaluates a large model, the evaluation results are not public data, and lack multi-party witnesses and consensus, resulting in a lack of credibility in the evaluation process of any large model.

[0083] In order to solve the above problems, an embodiment of the present application provides a large model evaluation system.

[0084] Reference Figure 1 , showing a schematic diagram of the architecture of a large model evaluation system provided in an embodiment of the present application, wherein the evaluation system includes an evaluation interaction module and an evaluation chain.

[0085] The evaluation interaction module can be a decentralized application for data interaction between any large model user and the evaluation chain, that is, the evaluation interaction module can adopt DAPP (Decentralized Application). The evaluation interaction module is mainly used for large model users to register large models, trigger large model evaluation and obtain evaluation results. The evaluation interaction module can also provide a user interface, and any large model user can operate based on the user interface on the evaluation interaction module.

[0086] Specifically, the evaluation chain is a peer-to-peer network composed of multiple blockchain nodes. The evaluation chain includes multiple blockchain nodes. Any blockchain node among the multiple blockchain nodes corresponds to an evaluation party agent module of an evaluation party. The evaluation party agent module is an evaluation service program associated with the evaluation party and the blockchain node, which completes the evaluation process of any large model on behalf of any evaluation party. The same on-chain data copy is maintained on multiple blockchain nodes on the evaluation chain, and the on-chain data copy includes: model information of any large model, evaluation data, evaluation records, information corresponding to different evaluation benchmarks, and evaluation benchmarks corresponding to any evaluation party.

[0087] Different assessors may have different assessment benchmarks. The information corresponding to any assessment benchmark includes the assessment benchmark type, assessment benchmark name, assessment benchmark description and assessment benchmark version information. For example, the assessment benchmark types include: general assessment benchmark, segmented field assessment benchmark and multimodal assessment benchmark; the assessment benchmark corresponding to each assessor is recorded on the assessment chain. For example, the assessors using the general assessment benchmark include assessor 1 and assessor 3, the assessors using the segmented field assessment benchmark include assessor 1 and assessor 2, and the assessors using the multimodal assessment benchmark include assessor 4 and assessor 5. In the actual implementation process, the assessors in the assessment system can be expanded by expanding new blockchain nodes in the network structure of the assessment chain, the new blockchain nodes are associated with the assessor agent module of the new assessor, and the information of the assessment benchmark of the new assessor is uploaded to the assessment chain.

[0088] Smart contracts are also deployed on the evaluation chain. Smart contracts are computing programs that can execute contract terms and are used to operate the data on the evaluation chain's ledger. Smart contracts are used to complete the reading and writing of key data on the evaluation chain, thereby completing the automated evaluation process of large models without requiring excessive human intervention. In this embodiment, smart contracts include evaluation execution contracts and evaluation benchmark management contracts.

[0089] In a feasible implementation, the evaluation interaction module includes an evaluation registration module, which is used to respond to the registration operation of the large model user, call the large model registration interface of the evaluation execution contract, and register the model information of any large model on the evaluation chain.

[0090] The model information includes the service call interface, name, version number and type of any large model, including general type, special type and multimodal type; the model information can also include model description, registration time and evaluation status, such as waiting in line, being evaluated and completed evaluation.

[0091] In the actual implementation process, the service call interface of any large model can be an evaluation interface execution script for the large model, so that any evaluator's agent module can call and execute it. The service call interface can also provide key access parameters of any large model. The evaluator's agent module can use an integrated SDK, such as LangChain, to complete the service call to the large model based on the key access parameters.

[0092] The evaluation interaction module also includes an evaluation interface verification module, which is used to verify whether the service call interface in the model information of any large model is correct. That is, in the process of registering any large model, the evaluation interface verification module on the evaluation interaction module first verifies the service call interface provided by the large model user to determine whether the provided service call interface can call the large model normally.

[0093] The evaluation interaction module also includes an evaluation result query module, which is used to respond to the first query operation of the large model user, call the evaluation data acquisition interface of the evaluation execution contract, and obtain the evaluation data of any evaluation party agent module for any large model on the evaluation chain.

[0094] The evaluation data is the evaluation result generated by any evaluation party agent module for the big model. The evaluation data generated by any evaluation party agent module for the big model includes: all evaluation questions evaluated by the evaluation party agent module for the big model, the evaluation answer returned by the big model for any evaluation question, and the score of any evaluation answer; in the actual implementation process, any evaluation party agent module can also calculate a total evaluation score based on the score corresponding to each evaluation question, such as using the sum of the scores of all evaluation questions as the total score.

[0095] In response to the second query operation of the large model user, the evaluation record acquisition interface of the evaluation execution contract is called to obtain the evaluation record of any large model on the evaluation chain.

[0096] The evaluation record is a comprehensive evaluation result determined by the evaluation chain based on the evaluation data generated by multiple evaluation party agent modules for any large model. By way of example, the evaluation record includes the evaluation start and end time, the comprehensive evaluation score and the list of evaluation parties participating in the evaluation for any large model; by way of example, the comprehensive evaluation score can be the data obtained by summing up the total evaluation scores of each evaluation party agent module for the large model.

[0097] In the actual implementation process, the evaluation party agent module generates a total evaluation score for any large model, and the evaluation chain generates a comprehensive evaluation score based on the total evaluation scores of multiple evaluation party agent modules. The calculation method can be set according to the needs of actual application, and is not limited in this embodiment.

[0098] In a feasible implementation, the evaluation agent module of any evaluation party includes an evaluation benchmark management unit, an evaluation data set management unit, an evaluation service execution unit, and a data chain interaction unit. Specifically:

[0099] The evaluation benchmark management unit is used to manage various evaluation benchmarks of the evaluation party agent module. For example, the evaluation benchmark management unit can be used to perform any of the following:

[0100] In response to the evaluation benchmark acquisition operation, calling the evaluation benchmark acquisition interface of the evaluation benchmark management contract to acquire any evaluation benchmark on the evaluation chain;

[0101] In response to the evaluation benchmark adding operation, calling the evaluation benchmark adding interface of the evaluation benchmark management contract to add a new evaluation benchmark to the evaluation chain;

[0102] In response to the evaluation benchmark modification operation, the evaluation benchmark modification interface of the evaluation benchmark management contract is called to modify any evaluation benchmark on the evaluation chain.

[0103] For example, the current evaluation benchmark of evaluation party 1 corresponding to blockchain node 1 is a general evaluation benchmark. When evaluation party 1 expands its evaluation business and adds an application segmented field evaluation benchmark, it can add a new evaluation benchmark to the evaluation chain through the evaluation benchmark addition operation and the evaluation benchmark addition interface of the evaluation benchmark management contract: a segmented field evaluation benchmark. The on-chain data copy of each node on the evaluation chain will also record the evaluation benchmarks currently owned by evaluation party 1 corresponding to blockchain node 1, including the general evaluation benchmark and the segmented field evaluation benchmark; similarly, any evaluation benchmark can also be modified or deleted.

[0104] The evaluation data set management unit is used to add any data set in the evaluation party agent module in response to a data set adding operation; or to modify any data set in the evaluation party agent module in response to a data set modifying operation.

[0105] Any evaluation party can add evaluation data sets that meet its own evaluation benchmarks based on the evaluation data set management unit, so that it can add, delete, modify and query data sets according to the needs of actual applications.

[0106] The evaluation service execution unit of the evaluator agent module is used to evaluate any large model, and the data chain interaction unit is used to store the generated evaluation data on the chain.

[0107] Reference Figure 2 , shows a flowchart of the steps of the large model evaluation method provided in the embodiment of the present application. In the large model evaluation method, the large model evaluation system described in this embodiment is first constructed, and then based on the constructed large model evaluation system, the large model evaluation method includes the following steps:

[0108] S101: The evaluation interaction module initiates an evaluation request of a target large model to the evaluation chain in response to a target operation of a large model user.

[0109] For example, the target operation includes a registration operation of a large model user for a target large model or an evaluation request operation of a large model user for a target large model.

[0110] When the target operation is a registration operation of a large model user for a target large model, the evaluation interaction module is used to respond to the registration operation of the large model user, register the model information of the target large model on the evaluation chain, and initiate an evaluation request for the target large model to the evaluation chain; that is, after registering a new large model on the evaluation chain, the evaluation interaction module can automatically initiate an evaluation request for the large model to the evaluation chain.

[0111] When the target operation is an evaluation request operation of a large model user for a target large model, the evaluation request module in the evaluation interaction module responds to the evaluation request operation of the large model user and initiates an evaluation request for the target large model to the evaluation chain; after the registration of the target large model is completed, an evaluation request operation can be performed on the evaluation interaction module according to the actual needs of the large model user, thereby starting the evaluation of the large model.

[0112] S102: The evaluation chain responds to the evaluation request of the target large model and determines multiple target blockchain nodes according to the type of the target large model.

[0113] Since different evaluation benchmarks are maintained on the evaluation chain, and the evaluation benchmarks corresponding to multiple evaluators are also different, when it is necessary to evaluate the target large model, it is first necessary to determine multiple evaluators that can have evaluation benchmarks that can evaluate the target large model.

[0114] Specifically, a target evaluation benchmark corresponding to the target large model is determined according to the type of the target large model, and multiple candidate blockchain nodes corresponding to multiple evaluation parties having the target evaluation benchmark are selected.

[0115] For example, when the type of the target large model is a general type, the corresponding target evaluation benchmark is a general evaluation benchmark; when the type of the target large model is a special type, the corresponding target evaluation benchmark is a segmented field evaluation benchmark; when the type of the target large model is a multimodal type, the corresponding target evaluation benchmark is a multimodal evaluation benchmark.

[0116] Then, according to the evaluation benchmarks corresponding to each evaluation party recorded on the evaluation chain, the candidate blockchain nodes corresponding to the evaluation party with the target evaluation benchmark are determined. For example, the evaluation benchmark of evaluation party 1 corresponding to blockchain node 1 includes a general evaluation benchmark, the evaluation benchmark of evaluation party 2 corresponding to blockchain node 2 includes a general evaluation benchmark and a segmented field evaluation benchmark, and the evaluation benchmark of evaluation party 3 corresponding to blockchain node 3 includes a multimodal evaluation benchmark. If the type of target large model 1 is a general type, the target evaluation benchmark corresponding to target large model 1 is a general evaluation benchmark. At this time, the evaluation parties with a general evaluation benchmark are evaluation party 1 and evaluation party 2, and blockchain node 1 and blockchain node 2 can be selected as candidate blockchain nodes.

[0117] Then, the evaluation chain selects a calibrated number of the multiple target blockchain nodes from the multiple candidate blockchain nodes based on the consensus algorithm. For example, there may be many evaluation parties with a target evaluation benchmark, so there are also many candidate blockchain nodes. In the actual implementation process, a calibrated number can be set, and the target large model can be evaluated using the calibrated number of target blockchain nodes. For example, the calibrated number can be set to 10, and the size of the calibrated number can be customized according to the needs of the actual application.

[0118] Then, a main blockchain node is determined among the multiple target blockchain nodes, and the evaluation party agent module corresponding to the main blockchain node is used to execute the evaluation strategy on the target large model.

[0119] When the evaluation agent module corresponding to the main blockchain node completes the execution of the evaluation strategy for the target large model, the next target blockchain node is selected as the main blockchain node among the multiple target blockchain nodes until the multiple target blockchain nodes have completed the evaluation strategy for the target large model.

[0120] When any target blockchain node serves as the main blockchain node, the evaluation party agent module corresponding to the target blockchain node executes the evaluation strategy of the target large model. When the evaluation data for the target large model is not generated within the target time, the evaluation chain can eliminate the target blockchain node from the multiple target blockchain nodes, select the next target blockchain node as the main blockchain node, and select a candidate blockchain node from the multiple candidate blockchain nodes as the target blockchain node, thereby ensuring that a calibrated number of target blockchain nodes complete the evaluation of the target large model.

[0121] S103: The evaluation agent modules corresponding to the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation agent module is synchronized to all blockchain nodes in the evaluation chain.

[0122] Any evaluator agent module includes an evaluation service execution unit. When any evaluator agent module executes an evaluation strategy on the target large model, the evaluation service execution unit in the evaluator agent module calls the model information acquisition interface of the evaluation execution contract to obtain the model information of the target large model on the evaluation chain.

[0123] When the model information of the target large model exists in the evaluation chain and the target large model has not completed the evaluation, a target number of evaluation problems are selected from the evaluation data set corresponding to the target evaluation benchmark; for example, N evaluation problems can be randomly selected from the evaluation data set corresponding to the target evaluation benchmark.

[0124] The target large model is called according to the service calling interface of the target large model included in the model information of the target large model, and the target large model is evaluated according to the evaluation questions of the target quantity to generate evaluation data corresponding to the target large model.

[0125] The data on-chain interaction unit then calls the write-in evaluation data interface of the evaluation execution contract to write the evaluation data corresponding to any generated large model into the evaluation chain. Specifically, after the evaluation party agent module stores the evaluation data of the target large model on the chain in the corresponding blockchain node, the blockchain node broadcasts the block containing the evaluation data of the target large model to all blockchain nodes in the evaluation chain, thereby reaching a multi-party consensus on the evaluation results of the target large model and recording them.

[0126] Reference Figure 3 , shows a schematic diagram of the evaluation processing flow provided by an embodiment of the present application. During the evaluation process of multiple target blockchain nodes, a main blockchain node is first selected, and the evaluation party agent module corresponding to the main blockchain node obtains the model information of the target large model on the evaluation chain. Then, the evaluation party agent module corresponding to the main blockchain node evaluates the target large model, and stores the evaluation data on the chain. Then, it is determined whether a calibrated number of target blockchain nodes have completed the evaluation. If a calibrated number of target blockchain nodes have completed the evaluation, the evaluation data of each evaluation party is summarized, and an evaluation record is generated and uploaded to the chain; if a calibrated number of target blockchain nodes have not completed the evaluation, the next main blockchain node is selected to continue the evaluation.

[0127] In this embodiment, in order to ensure that the evaluation results are recognized by all evaluators, a consensus algorithm based on blockchain is implemented to support multiple evaluators to jointly evaluate the target large model, and the selection of the evaluator is determined by the consensus algorithm.

[0128] The consensus algorithm can adopt an algorithm with a Leader mechanism, such as the Raft (Replication and FaultTolerant) algorithm or DPoS (Delegated Proof-of-Stake).

[0129] The Raft algorithm is a consistency protocol that allows network partitions. The main strategy is to first select a Leader from all candidates, and the Leader is responsible for all subsequent decisions. Raft requires that there is at most one Leader at any time. During normal operation, blockchain nodes are divided into Leaders and Followers. The Raft algorithm will limit a term for the Leader, and a new Leader will be re-elected at the end of the term. During the term, the Leader is fully responsible for taking transactions from the transaction pool, packaging transactions into blocks, and putting blocks on the chain; Followers will synchronize based on the Leader node, and elect a new Leader node when the Leader node fails.

[0130] Reference Figure 4 , shows a schematic diagram of the evaluation process provided by the embodiment of the present application, and the multi-party evaluation process of the large model implemented based on the Raft consensus algorithm includes:

[0131] A1: Select a main blockchain node (Leader node) from multiple target blockchain nodes.

[0132] A2: The evaluation agent module associated with the main blockchain node obtains the model information of the target large model being evaluated or waiting to be evaluated from the evaluation chain.

[0133] If there is no model information of the target large model on the evaluation chain, no operation will be performed; if there is model information of the target large model, N evaluation questions will be randomly selected from the evaluation data set corresponding to the local target evaluation benchmark, and the service call interface in the model information of the target large model will be used to complete the evaluation interaction, and the evaluation data will be stored on the chain.

[0134] A3: If the main blockchain node does not generate a new block in this round of consensus cycle, the main blockchain node will fail and a new main blockchain node will be elected. If no new block containing evaluation data is generated in the consensus cycle, it means that there is no model information of the target large model on the evaluation chain or the evaluation cannot be completed normally in this round.

[0135] A4: When the main blockchain node generates a new block within this round of consensus cycle, the main blockchain node broadcasts the new block to other nodes for verification. After other nodes pass the verification, the new block is written to the end of the blockchain data structure to become a new chain tail block.

[0136] The new block is the block that contains the evaluation data of this round.

[0137] The DPoS algorithm is the Delegated Proof of Stake. In the DPoS algorithm, the witness mechanism is used to solve the consensus problem of decentralized multi-party participation. It uses stakeholders to select any number of witnesses in a fair and democratic way through voting to generate blocks; witnesses are generated by voting by the entities using the blockchain network. Each witness is required to obtain at least 50% of the votes. The selected witnesses take turns to be responsible for block generation, that is, in each round of consensus, a witness node is selected as the main node to generate blocks and broadcast them to other witness nodes for verification.

[0138] Reference Figure 5 , shows a schematic diagram of the evaluation process provided by an embodiment of the present application, and the multi-party evaluation process of a large model implemented based on the DPoS consensus algorithm includes:

[0139] B1: The blockchain nodes corresponding to all evaluation parties act as equity nodes to vote on the candidate blockchain nodes and select the witness node set as the target blockchain node set.

[0140] For example, candidate nodes that can vote more than 50% will be selected as witness nodes.

[0141] B2: Randomly sort the target blockchain node set, and each target blockchain node is selected as the primary blockchain node in turn.

[0142] Assume that the current witness i is selected as the Leader node.

[0143] B3: The evaluation agent module corresponding to the main blockchain node obtains the model information of the target large model being evaluated or waiting to be evaluated from the evaluation chain.

[0144] When there is no model information of the target large model on the evaluation chain, no operation will be performed; if there is model information of the target large model, N evaluation questions will be randomly selected from the evaluation data set corresponding to the local target evaluation benchmark, and the service call interface in the model information of the target large model will be used to complete the evaluation interaction, and the evaluation data will be stored on the chain.

[0145] B4: When the main blockchain node fails to generate a new block within the current consensus cycle, the main blockchain node becomes invalid and a new main blockchain node is selected.

[0146] For example, the witness i++ is used as the main blockchain node.

[0147] B5: When the main blockchain node generates a new block within this round of consensus cycle, the main blockchain node broadcasts the new block to other nodes for verification. After other nodes pass the verification, the new block is written to the end of the blockchain data structure to become a new chain tail block.

[0148] Reference Figure 6 , showing an interactive flow chart of the evaluation process provided in an embodiment of the present application, the big model management user calls the big model registration interface of the evaluation execution contract through the evaluation interaction module, registers the model information of the big model on the evaluation chain and requests to evaluate the big model.

[0149] Then the evaluation chain determines multiple target blockchain node sets according to the type of the target large model, and selects any main blockchain node. The evaluation service execution unit of the evaluation party agent module of the main blockchain node calls the evaluation execution contract to obtain model information interface, and obtains the model information of the target large model on the evaluation chain. After the evaluation service execution unit of the evaluation party agent module completes the evaluation strategy, it calls the evaluation execution contract to write evaluation data interface, and stores the generated evaluation data corresponding to the target large model on the chain.

[0150] The evaluation result query module on the evaluation interaction module responds to the first query operation of the large model user, calls the evaluation data acquisition interface of the evaluation execution contract, and obtains the evaluation data of any evaluation party agent module for the target large model on the evaluation chain; or responds to the second query operation of the large model user, calls the evaluation record acquisition interface of the evaluation execution contract, and obtains the evaluation record of any large model on the evaluation chain.

[0151] In addition, any evaluation party agent module can also call the evaluation benchmark management contract to obtain the evaluation benchmark interface to obtain any evaluation benchmark on the evaluation chain, or call the evaluation benchmark management contract to add the evaluation benchmark interface and modify the evaluation benchmark interface to add a new evaluation benchmark or modify any evaluation benchmark on the evaluation chain.

[0152] The large model evaluation system and evaluation method provided in this embodiment mainly solve the decentralized multi-party evaluation problem for large models. The method can be extended to other multi-party evaluation systems to promote consensus among all parties on the evaluation process and increase the credibility of the evaluation results.

[0153] The specific beneficial effects include the following:

[0154] 1. Based on blockchain, decentralized multi-party evaluation of large language models, i.e. the capabilities of large models, is achieved, effectively solving the problems of arbitrary evaluation and partial generalization in traditional methods where evaluation is completed by a single evaluation agency;

[0155] 2. The decentralized multi-party evaluation is combined with the consensus algorithm of the blockchain. The evaluation process is agreed upon by all parties involved, which enhances the credibility of the evaluation results.

[0156] 3. Smart contracts for multi-party evaluation are designed, including evaluation benchmark management contracts and evaluation execution contracts, to ensure that the evaluation process is programmed and executed efficiently without human intervention.

[0157] 4. The key data structure stored on the chain is designed, including: model information of large model registration, evaluation data, evaluation records and evaluation benchmarks, etc. The key data of the evaluation process is stored through the blockchain to ensure that the evaluation process is fair, open and just.

[0158] 5. At the same time, the large model evaluation records are stored on the chain, which can effectively prevent evaluation by brushing questions or stealing questions.

[0159] Reference Figure 7 , shows a schematic diagram of a computer device provided in an embodiment of the present application, the computer device comprising: at least one processor 701, and a memory 702, the memory 702 storing a computer program that can be run on the processor 701, wherein the processor 701 executes the large model evaluation method described in this embodiment when executing the computer program.

[0160] Reference Figure 8 , showing a schematic diagram of a non-volatile readable storage medium provided in an embodiment of the present application, wherein the non-volatile readable storage medium 800 stores a computer program 801, wherein the computer program 801, when executed by a processor, executes the large model evaluation method described in this embodiment.

[0161] Reference Fig. 9 , showing a schematic diagram of a computer program product provided in an embodiment of the present application, the computer program product 900 includes a computer program / instruction 901, and when the computer program / instruction 901 is executed by a processor, the large model evaluation method described in the embodiment is implemented.

[0162] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0163] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0168] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0169] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A large model evaluation system, characterized in that: The evaluation system includes an evaluation interaction module and an evaluation chain. The evaluation interaction module is used for data interaction between any large model user and the evaluation chain. The evaluation chain includes multiple blockchain nodes, and any blockchain node among the multiple blockchain nodes corresponds to an evaluation party agent module of an evaluation party, wherein: The evaluation interaction module is used to respond to the target operation of the large model user and initiate an evaluation request of the target large model to the evaluation chain; The evaluation chain is used to respond to the evaluation request of the target large model, determine multiple target blockchain nodes according to the type of the target large model, and the evaluation party corresponding to each of the multiple target blockchain nodes has a target evaluation benchmark corresponding to the type of the target large model; The evaluation party agent modules corresponding to each of the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation party agent module is synchronized to all blockchain nodes in the evaluation chain. In the process of any evaluation party agent module evaluating the target large model, the target large model is called according to the service call interface of the target large model in the model information of the target large model obtained on the evaluation chain, and the target large model is evaluated by selecting a target number of evaluation questions in the evaluation data set corresponding to the target evaluation benchmark.

2. The evaluation system according to claim 1, characterized in that: The evaluation chain is used to: Determine a target evaluation benchmark corresponding to the target large model according to the type of the target large model, and select multiple candidate blockchain nodes corresponding to multiple evaluation parties having the target evaluation benchmark; Selecting a calibrated number of the plurality of target blockchain nodes from the plurality of candidate blockchain nodes based on a consensus algorithm, and determining a primary blockchain node from the plurality of target blockchain nodes, wherein the evaluation party agent module corresponding to the primary blockchain node is used to execute an evaluation strategy on the target large model; When the evaluation agent module corresponding to the main blockchain node completes the execution of the evaluation strategy for the target large model, the next target blockchain node is selected as the main blockchain node among the multiple target blockchain nodes until the multiple target blockchain nodes have completed the evaluation strategy for the target large model.

3. The evaluation system according to claim 2, characterized in that: When any target blockchain node is used as the main blockchain node, the evaluation party agent module corresponding to the target blockchain node executes the evaluation strategy of the target large model. If the evaluation data for the target large model is not generated within the target time, the evaluation chain is used to: The target blockchain node is eliminated from the multiple target blockchain nodes, the next target blockchain node is selected as the main blockchain node, and a candidate blockchain node is selected from the multiple candidate blockchain nodes as the target blockchain node.

4. The evaluation system according to claim 2, characterized in that: Any evaluator agent module includes an evaluation service execution unit. When any evaluator agent module executes the evaluation strategy on the target large model, the evaluation service execution unit in the evaluator agent module is used to: Acquiring model information of the target large model on the evaluation chain; When the model information of the target large model exists in the evaluation chain and the target large model has not completed the evaluation, selecting a target number of evaluation problems in the evaluation data set corresponding to the target evaluation benchmark; The target large model is called according to the service calling interface of the target large model included in the model information of the target large model, and the target large model is evaluated according to the evaluation questions of the target quantity to generate evaluation data corresponding to the target large model.

5. The evaluation system according to claim 4, characterized in that: Any evaluation party agent module further includes a data chain interaction unit. When the evaluation service execution unit of the evaluation party agent module generates evaluation data corresponding to the target large model, the data chain interaction unit is used to: The evaluation data of the target large model is uploaded to the chain and stored in the corresponding blockchain node.

6. The evaluation system according to claim 5, characterized in that: When the evaluation data corresponding to the target large model generated by any evaluation party agent module is stored in the corresponding blockchain node, the blockchain node will broadcast the block containing the evaluation data of the target large model to all blockchain nodes in the evaluation chain.

7. The evaluation system according to claim 1, characterized in that: The same on-chain data copies are maintained on multiple blockchain nodes of the evaluation chain, and the on-chain data copies include: model information, evaluation data and evaluation records of any large model; The evaluation data is the evaluation result generated by any evaluation agent module for the large model; the evaluation record is the comprehensive evaluation result determined by the evaluation chain based on the evaluation data generated by multiple evaluation agent modules for the large model.

8. The evaluation system according to claim 7, characterized in that: The on-chain data copy also includes information corresponding to different evaluation benchmarks and the evaluation benchmark corresponding to any evaluation party. The information of any evaluation benchmark includes the evaluation benchmark name, evaluation benchmark type, evaluation benchmark description and evaluation benchmark version.

9. The evaluation system according to claim 1, characterized in that: The evaluation chain is deployed with an evaluation execution contract, which includes a large model registration interface. The evaluation interaction module includes: An evaluation registration module, for responding to a registration operation of a large model user, calling the large model registration interface of the evaluation execution contract, and registering the model information of any large model on the evaluation chain; The model information includes the service call interface, name, version number and type of any large model.

10. The evaluation system according to claim 9, characterized in that: The evaluation execution contract includes an interface for obtaining evaluation data and an interface for obtaining evaluation records. The evaluation interaction module also includes an evaluation result query module for: In response to the first query operation of the large model user, calling the evaluation data acquisition interface of the evaluation execution contract to obtain the evaluation data of any evaluation party agent module for any large model on the evaluation chain; In response to the second query operation of the large model user, the evaluation record acquisition interface of the evaluation execution contract is called to obtain the evaluation record of any large model on the evaluation chain. The evaluation record is a comprehensive evaluation result determined by the evaluation chain based on the evaluation data generated by multiple evaluation party agent modules for the large model.

11. The evaluation system according to claim 10, characterized in that: The evaluation interaction module also includes: The evaluation interface verification module is used to verify whether the service call interface in the model information of any large model is correct.

12. The evaluation system according to claim 1, characterized in that: The evaluation chain is deployed with an evaluation execution contract, which includes an interface for obtaining model information and an interface for writing evaluation data. Any evaluation party agent module is used to: Call the model information acquisition interface of the evaluation execution contract to obtain the model information of any large model on the evaluation chain; The write evaluation data interface of the evaluation execution contract is called to write the evaluation data corresponding to any generated large model into the evaluation chain.

13. The evaluation system according to claim 1, characterized in that: Any evaluator agent module also includes: The evaluation benchmark management unit is used to manage various evaluation benchmarks of the evaluation party agent module.

14. The evaluation system according to claim 13, characterized in that: The evaluation chain is deployed with an evaluation benchmark management contract, which includes an evaluation benchmark acquisition interface, and the evaluation benchmark management unit is used to: In response to the evaluation benchmark acquisition operation, calling the evaluation benchmark acquisition interface of the evaluation benchmark management contract to acquire any evaluation benchmark on the evaluation chain; In response to the evaluation benchmark adding operation, calling the evaluation benchmark adding interface of the evaluation benchmark management contract to add a new evaluation benchmark to the evaluation chain; In response to the evaluation benchmark modification operation, the evaluation benchmark modification interface of the evaluation benchmark management contract is called to modify any evaluation benchmark on the evaluation chain.

15. The evaluation system according to claim 13, characterized in that: Any evaluation party agent module also includes an evaluation data set management unit for: In response to the data set adding operation, adding any data set to the evaluator agent module; In response to the data set modification operation, any data set is modified in the evaluator agent module.

16. The evaluation system according to claim 1, characterized in that: The target operation includes a registration operation of a large model user for a target large model or an evaluation request operation of a large model user for a target large model; When the target operation is a registration operation of the large model user for the target large model, the evaluation interaction module is used to respond to the registration operation of the large model user, register the model information of the target large model on the evaluation chain, and initiate an evaluation request of the target large model to the evaluation chain; When the target operation is an evaluation request operation of the large model user for the target large model, the evaluation request module in the evaluation interaction module responds to the evaluation request operation of the large model user and initiates an evaluation request for the target large model to the evaluation chain.

17. A large model evaluation method, characterized in that: The method comprises: Constructing a large model evaluation system, the large model evaluation system comprising an evaluation interaction module and an evaluation chain, the evaluation interaction module being used for data interaction between any large model user and the evaluation chain; the evaluation chain comprising a plurality of blockchain nodes, any of the plurality of blockchain nodes corresponding to an evaluation party agent module of an evaluation party; The evaluation interaction module initiates an evaluation request of the target large model to the evaluation chain in response to the target operation of the large model user; The evaluation chain responds to the evaluation request of the target large model, determines multiple target blockchain nodes according to the type of the target large model, and the evaluation parties corresponding to each of the multiple target blockchain nodes have target evaluation benchmarks corresponding to the type of the target large model; The evaluation party agent modules corresponding to each of the multiple target blockchain nodes respectively evaluate the target large model, and the evaluation data generated by any evaluation party agent module is synchronized to all blockchain nodes in the evaluation chain. In the process of any evaluation party agent module evaluating the target large model, the target large model is called according to the service call interface of the target large model in the model information of the target large model obtained on the evaluation chain, and the target large model is evaluated by selecting a target number of evaluation questions in the evaluation data set corresponding to the target evaluation benchmark.

18. A computer device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the large model evaluation method of claim 17 is performed when the processor executes the computer program.

19. A non-volatile readable storage medium, characterized in that: The non-volatile readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the large model evaluation method of claim 17.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the large model evaluation method described in claim 17 is implemented.

Images