Method and device for evaluating large model privatization deployment scheme

Abstract

The invention provides a large model privatization deployment scheme evaluation method and device, and the method comprises the steps: pre-determining a target problem and a target business document for a to-be-tested large model, and obtaining an insurance capability score of the to-be-tested large model based on the target problem and the target business document; deploying the to-be-tested large model on preset to-be-tested hardware, inputting the target problem into the to-be-tested large model to obtain multi-dimensional performance index data when the to-be-tested large model works on the to-be-tested hardware, and obtaining a hardware performance test score of the to-be-tested large model based on the performance index data; and testing the multi-dimensional reasoning ability of the to-be-tested large models to obtain reasoning ability scores of the to-be-tested large models, and obtaining deployment scores of the to-be-tested large models when the to-be-tested large models are deployed on the to-be-tested hardware platforms based on the reasoning ability scores, the insurance ability scores and the hardware performance test scores.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method and device for evaluating a large-scale model privatization deployment solution. Background Art

[0002] With the rapid development of artificial intelligence, AI (Artificial Intelligence) big models have also been widely used. The development of various AI big models involves the evaluation of big models to compare the capabilities of different big models.

[0003] At present, existing traditional large model evaluation tools mainly focus on the generation capabilities of the large models themselves. Generally, they use standardized test cases and data sets and clear performance indicators to measure the accuracy, efficiency and generalization ability of the models.

[0004] However, traditional big model evaluation tools only test the capabilities of the big model itself, but lack attention to the application scenarios of the big model service when it is deployed. For example, performance requirements and compatibility stability during deployment make it difficult to ensure that the big model performs well in dealing with business scenarios in the insurance industry. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a method and device for evaluating the privatization deployment scheme of a large model, testing the insurance capability score of the large model to be tested through target problems and target business documents, obtaining a hardware performance test score by deploying the large model to be tested on the hardware to be tested, and obtaining a deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning capability score, the insurance capability score and the hardware performance test score. Thus, a deployment scheme score can be given according to the performance of the large model as a reference for selecting the deployment scheme, thereby achieving attention to the large model service during deployment, and by scoring the deployment scheme based on insurance field issues, it can be ensured that the large model has good performance when dealing with insurance industry business scenarios.

[0006] In a first aspect, an embodiment of the present application provides a method for evaluating a large model privatization deployment solution, the method comprising:

[0007] Predetermine the target question and target business document for the large model to be tested, and obtain the insurance capability score of the large model to be tested based on the target question and the target business document; wherein the target question is a professional question in the insurance field with an objective answer;

[0008] Deploy the large model to be tested on the preset hardware to be tested, input the target problem into the large model to be tested, obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtain the hardware performance test score of the large model to be tested based on the performance indicator data;

[0009] The multi-dimensional reasoning ability of the large model to be tested is tested to obtain a reasoning ability score of the large model to be tested, and the deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning ability score, the insurance ability score and the hardware performance test score.

[0010] In a possible implementation, obtaining the insurance capability score of the large model to be tested based on the target question and the target business document includes:

[0011] Acquire an answer of the large model to be tested to the target question based on the target question and the target business document, and obtain a retrieval enhancement generation capability score of the large model to be tested based on the answer;

[0012] Determine a corresponding target complex business scenario for the large model to be tested, process the target complex business scenario based on the large model to be tested, and obtain a plug-in calling capability score of the large model to be tested;

[0013] The insurance capability score of the large model to be tested is calculated based on the retrieval enhancement generation capability score and the plug-in calling capability score.

[0014] In a possible implementation, obtaining an answer of the large model to be tested to the target question based on the target question and the target business document, and obtaining a retrieval enhancement generation capability score of the large model to be tested based on the answer, includes:

[0015] Obtaining a first answer of the large model to be tested to the target question, and transferring the target business document to the large model to be tested, to obtain a second answer of the large model to be tested to the target question;

[0016] For the large model to be tested, corresponding review experts are determined, and evaluation results of the review experts for the first answer and the second answer are obtained respectively, and a retrieval enhancement generation capability score of the large model to be tested is obtained based on the evaluation results.

[0017] In a possible implementation, the processing of the target complex business scenario based on the large model to be tested to obtain a plug-in calling capability score of the large model to be tested includes:

[0018] Obtaining a call success rate and a call time of the large model to be tested calling target business data on the target complex business scenario; wherein the target business data is business data obtained by the large model to be tested calling multiple interfaces on the target complex business scenario;

[0019] The evaluation results of the review experts on the call success rate and the call time are obtained, and the plug-in call capability score of the large model to be tested is obtained based on the evaluation results.

[0020] In a possible implementation, the large model to be tested is deployed on a preset hardware to be tested, and a preset second target problem is input into the large model to be tested, so as to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware platform to be tested, including:

[0021] Determine all large models to be tested that are compatible with the hardware to be tested, and deploy and run all large models to be tested on the hardware to be tested in sequence;

[0022] Determine the target concurrency number of the large model to be tested, input the second target problem of the target problem length into the large model to be tested, and obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested.

[0023] In a possible implementation manner, obtaining a hardware performance test score of the large model to be tested based on the performance indicator data includes:

[0024] Based on the performance indicator data of all the large models to be tested, the ranking of the performance indicator data of each dimension of each large model to be tested is obtained;

[0025] Based on the ranking of the performance indicator data of each dimension of each large model to be tested, a performance indicator score of the performance indicator data of each dimension of the large model to be tested is obtained;

[0026] The performance indicator scores of each dimension of the large model to be tested are added together to obtain the hardware performance test score of the large model to be tested.

[0027] In a possible implementation, obtaining the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score, and the hardware performance test score includes:

[0028] Determine a first weight of the reasoning ability score and a second weight of the insurance ability score based on the business scenario of the large model to be tested;

[0029] The product of the reasoning ability score and the first weight, the product of the insurance ability score and the second weight, and the sum of the hardware performance test score are calculated to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested.

[0030] In a second aspect, the embodiment of the present application further provides a large model privatization deployment solution evaluation device, the device comprising:

[0031] A first acquisition module is used to predetermine a target question and a target business document for the large model to be tested, and obtain an insurance capability score of the large model to be tested based on the target question and the target business document; wherein the target question is a professional question in the insurance field with an objective answer;

[0032] A second acquisition module is used to deploy the large model to be tested on a preset hardware to be tested, and input the target problem into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and to obtain a hardware performance test score of the large model to be tested based on the performance indicator data;

[0033] The third acquisition module is used to test the multi-dimensional reasoning ability of the large model to be tested, obtain the reasoning ability score of the large model to be tested, and obtain the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score.

[0034] In a possible implementation manner, the first acquisition module is specifically configured to:

[0035] Acquire an answer of the large model to be tested to the target question based on the target question and the target business document, and obtain a retrieval enhancement generation capability score of the large model to be tested based on the answer;

[0036] Determine a corresponding target complex business scenario for the large model to be tested, process the target complex business scenario based on the large model to be tested, and obtain a plug-in calling capability score of the large model to be tested;

[0037] The insurance capability score of the large model to be tested is calculated based on the retrieval enhancement generation capability score and the plug-in calling capability score.

[0038] In a possible implementation manner, the first acquisition module is specifically configured to:

[0039] Obtaining a first answer of the large model to be tested to the target question, and transferring the target business document to the large model to be tested, to obtain a second answer of the large model to be tested to the target question;

[0040] For the large model to be tested, corresponding review experts are determined, and evaluation results of the review experts for the first answer and the second answer are obtained respectively, and a retrieval enhancement generation capability score of the large model to be tested is obtained based on the evaluation results.

[0041] In a possible implementation manner, the first acquisition module is specifically configured to:

[0042] Obtaining a call success rate and a call time of the large model to be tested calling target business data on the target complex business scenario; wherein the target business data is business data obtained by the large model to be tested calling multiple interfaces on the target complex business scenario;

[0043] The evaluation results of the review experts on the call success rate and the call time are obtained, and the plug-in call capability score of the large model to be tested is obtained based on the evaluation results.

[0044] In a possible implementation manner, the second acquisition module is further configured to:

[0045] Determine all large models to be tested that are compatible with the hardware to be tested, and deploy and run all large models to be tested on the hardware to be tested in sequence;

[0046] Determine the target concurrency number of the large model to be tested, input the second target problem of the target problem length into the large model to be tested, and obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested.

[0047] In a possible implementation manner, the second acquisition module is specifically configured to:

[0048] Based on the performance indicator data of all the large models to be tested, the ranking of the performance indicator data of each dimension of each large model to be tested is obtained;

[0049] Based on the ranking of the performance indicator data of each dimension of each large model to be tested, a performance indicator score of the performance indicator data of each dimension of the large model to be tested is obtained;

[0050] The performance indicator scores of each dimension of the large model to be tested are added together to obtain the hardware performance test score of the large model to be tested.

[0051] In a possible implementation manner, the third acquisition module is specifically configured to:

[0052] Determine a first weight of the reasoning ability score and a second weight of the insurance ability score based on the business scenario of the large model to be tested;

[0053] The product of the reasoning ability score and the first weight, the product of the insurance ability score and the second weight, and the sum of the hardware performance test score are calculated to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the large-model privatization deployment solution evaluation method as described in any one of the first aspects.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large-model privatization deployment solution evaluation method described in any one of the first aspects are executed.

[0056] The embodiment of the present application provides a method and device for evaluating a large model privatization deployment solution, which predetermines a target problem and a target business document for the large model to be tested, and obtains an insurance capability score of the large model to be tested based on the target problem and the target business document; deploys the large model to be tested on a preset hardware to be tested, and inputs the target problem into the large model to be tested, obtains multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtains a hardware performance test score of the large model to be tested based on the performance indicator data; tests the multi-dimensional reasoning ability of the large model to be tested, obtains a reasoning ability score of the large model to be tested, and obtains a deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score. In this application, the insurance capability score of the big model to be tested is tested through target problems and target business documents, the hardware performance test score is obtained by deploying the big model to be tested on the hardware to be tested, and the deployment score of each big model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning capability score, the insurance capability score and the hardware performance test score. Therefore, a deployment plan score can be given according to the performance of the big model as a reference for selecting the deployment plan, thereby achieving attention to the big model service during deployment and implementation, and by scoring the deployment plan based on insurance field issues, it can be ensured that the big model has good performance when dealing with insurance industry business scenarios.

[0057] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 is a flow chart of a method for evaluating a large model privatization deployment solution according to an embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of the evaluation process when the large model to be tested is deployed on the hardware to be tested;

[0061] Figure 3 is a flow chart of a method for evaluating a large model privatization deployment solution according to another embodiment of the present application;

[0062] Figure 4 is a flow chart of a method for evaluating a large model privatization deployment solution according to another embodiment of the present application;

[0063] Figure 5 It is a schematic diagram of the hardware performance test of the large model to be tested;

[0064] Figure 6 is a flow chart of a method for evaluating a large model privatization deployment solution according to another embodiment of the present application;

[0065] Figure 7 is a flow chart of a method for evaluating a large model privatization deployment solution according to another embodiment of the present application;

[0066] Figure 8 It is a structural schematic diagram of a large model privatization deployment solution evaluation device according to an embodiment of the present application;

[0067] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0069] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0070] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0071] Taking into account the rapid development of artificial intelligence, AI (Artificial Intelligence) big models have also been widely used. The development of various AI big models involves the evaluation of big models to compare the capabilities of different big models.

[0072] At present, the existing traditional big model evaluation tools mainly focus on the generation capabilities of the big model itself, generally through standardized test cases and data sets, using clear performance indicators to measure the accuracy, efficiency and generalization capabilities of the model. However, traditional big model evaluation tools only test the capabilities of the big model itself, and lack attention to the application scenarios of the big model service when it is deployed and implemented, such as performance requirements and compatibility stability during deployment and implementation, making it difficult to ensure the good performance of the big model in dealing with business scenarios in the insurance industry.

[0073] To address this problem, the present application provides a method and device for evaluating a large model privatization deployment solution. The insurance capability score of the large model to be tested is tested through target problems and target business documents. The hardware performance test score is obtained by deploying the large model to be tested on the hardware to be tested. The deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning capability score, the insurance capability score and the hardware performance test score. Thus, a deployment solution score can be given according to the performance of the large model as a reference for selecting the deployment solution, thereby achieving attention to the large model service during deployment and implementation. Moreover, by scoring the deployment solution based on insurance field issues, it can be ensured that the large model has good performance when dealing with insurance industry business scenarios.

[0074] Figure 1 1 is a flow chart of a method for evaluating a large model privatization deployment solution according to an embodiment of the present application. Figure 1 As shown, the large model privatization deployment solution evaluation method of the embodiment of the present application may specifically include:

[0075] S101. Predetermine target questions and target business documents for the large model to be tested, and obtain an insurance capability score of the large model to be tested based on the target questions and target business documents.

[0076] S102. Deploy the large model to be tested on the preset hardware to be tested, and input the target problem into the large model to be tested, obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtain the hardware performance test score of the large model to be tested based on the performance indicator data.

[0077] S103, testing the multi-dimensional reasoning ability of the large model to be tested to obtain the reasoning ability score of the large model to be tested, and obtaining the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score.

[0078] In the above-mentioned large model privatization deployment solution evaluation method, the insurance capability score of the big model to be tested is tested through target problems and target business documents, the hardware performance test score is obtained by deploying the big model to be tested on the hardware to be tested, and the deployment score of each big model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning capability score, insurance capability score and hardware performance test score. Therefore, a deployment solution score can be given according to the performance of the big model as a reference for selecting the deployment solution, which realizes the attention to the big model service during deployment and implementation, and by scoring the deployment solution based on insurance field issues, it can ensure that the big model has good performance when dealing with insurance industry business scenarios.

[0079] The above exemplary steps of the embodiment of the present application are described below with reference to specific examples:

[0080] S101, predetermine the target problem and target business document for the large model to be tested, and obtain the insurance capability score of the large model to be tested based on the target problem and the target business document.

[0081] In the embodiment of the present application, the large model to be tested is the AI ​​(Artificial Intelligence) large model to be tested, for example, Figure 2 As shown, the big model to be tested includes multiple big models, namely big model A, big model B, and big model C; the target question is a professional question in the insurance field with an objective answer, and the target business document is a business document (within the industry or the company) that helps the big model understand and answer the target question; the insurance capability score is the score of the insurance capability test of the big model to be tested, and the target question and target business document for the big model to be tested are determined, and the insurance capability score of the big model to be tested is obtained based on the target question and the target business document for subsequent processing.

[0082] S102, deploying the large model to be tested on the preset hardware to be tested, and inputting the target problem into the large model to be tested, obtaining multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtaining the hardware performance test score of the large model to be tested based on the performance indicator data.

[0083] In the embodiment of the present application, the hardware to be tested is a pre-set hardware platform to be tested, and the multi-dimensional performance indicator data includes at least computing core utilization, video memory occupancy, throughput, and first word latency. The performance indicator data characterizes the performance of various performance indicators of the large model to be tested, and the hardware performance test score is the score obtained by performing a hardware performance test on the large model to be tested; the large model to be tested is deployed on the preset hardware to be tested, and the target problem is input into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and based on the performance indicator data, the hardware performance test score of the large model to be tested is obtained for subsequent processing.

[0084] S103, testing the multi-dimensional reasoning ability of the large model to be tested to obtain the reasoning ability score of the large model to be tested, and obtaining the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score.

[0085] In the embodiment of the present application, the multi-dimensional reasoning ability of the large model to be tested is tested to obtain a reasoning ability score of the large model to be tested. Based on the reasoning ability score, the insurance ability score obtained in step S101 and the hardware performance test score obtained in step S102, a deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained.

[0086] It should be noted that the content generation capability score, that is, the reasoning capability score, of the large model to be tested is obtained through the content generation capability test solution, which specifically tests the more common problems in the insurance field.

[0087] The large model privatization deployment scheme evaluation method provided in the embodiment of the present application predetermines the target problem and target business document for the large model to be tested, and obtains the insurance capability score of the large model to be tested based on the target problem and the target business document; the large model to be tested is deployed on the preset hardware to be tested, and the target problem is input into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtains the hardware performance test score of the large model to be tested based on the performance indicator data; the multi-dimensional reasoning ability of the large model to be tested is tested to obtain the reasoning ability score of the large model to be tested, and the deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning ability score, the insurance ability score and the hardware performance test score. The big model privatization deployment scheme evaluation method of the present application tests the insurance capability score of the big model to be tested through target problems and target business documents, obtains the hardware performance test score by deploying the big model to be tested on the hardware to be tested, and obtains the deployment score of each big model to be tested when deployed on each hardware platform to be tested based on the reasoning capability score, insurance capability score and hardware performance test score. Therefore, a deployment scheme score can be given according to the performance of the big model as a reference for selecting the deployment scheme, thereby achieving attention to the big model service during deployment and implementation, and by scoring the deployment scheme based on insurance field issues, it can ensure that the big model has good performance when dealing with insurance industry business scenarios.

[0088] Further, such as Figure 3 As shown, the step S101 in the above embodiment of "obtaining the insurance capability score of the large model to be tested based on the target question and the target business document" may specifically include the following steps:

[0089] S301, based on the target question and the target business document, obtain the answer of the large model to be tested to the target question, and obtain the retrieval enhancement generation capability score of the large model to be tested based on the answer.

[0090] In the embodiment of the present application, the retrieval-augmented generation capability score is the score of the retrieval-augmented generation (1RAG) capability test of the large model to be tested. The answer of the large model to be tested to the target question is obtained based on the target question and the target business document, and the retrieval-augmented generation capability score of the large model to be tested is obtained based on the answer for subsequent processing.

[0091] It can be understood by those skilled in the art that RAG capability is the key to the big model's ability to accurately handle professional issues in vertical fields. Therefore, this application conducts RAG capability testing on the big model based on relevant issues in the insurance field.

[0092] It should be noted that the present application does not impose too many restrictions on the specific method of obtaining the retrieval enhancement generation capability score of the large model to be tested based on the target question and the target business document, and can be set according to actual conditions.

[0093] As a possible implementation method, the first answer of the large model to be tested to the target question is obtained, and the target business document is passed into the large model to be tested, and the second answer of the large model to be tested to the target question is obtained; for the large model to be tested, the corresponding review experts are determined, and the evaluation results of the review experts for the first answer and the second answer are obtained respectively, and the retrieval enhancement generation ability score of the large model to be tested is obtained based on the evaluation results. Among them, the evaluation results represent the content quality generated by the large model to be tested, the professionalism of the generated content, the accuracy of the answer, and the response time; the first answer represents the answer of the large model to be tested to the target question before the target business document is passed in, and the second answer represents the answer of the large model to be tested to the target question after the target business document is passed in.

[0094] S302, determining a corresponding target complex business scenario for the large model to be tested, processing the target complex business scenario based on the large model to be tested, and obtaining a plug-in calling capability score of the large model to be tested.

[0095] In an embodiment of the present application, the target complex business scenario is a complex business scenario involving multiple interface calls, for example, "If a user wants the big model to recommend insurance products for him, the big model will obtain the user's profile data, browsing history, and the main products promoted in the user's area by calling multiple interfaces, and give the user specific recommendations based on this information. At the same time, it will push user information to the insurance business manager who represents the product in the user's area to facilitate marketing personnel to develop customers." For this complex business scenario, the plug-in calling capability score is the score of the plug-in calling capability test of the big model to be tested, determines the target complex business scenario corresponding to the big model to be tested, and processes the target complex business scenario based on the big model to be tested to obtain the plug-in calling capability score of the big model to be tested for subsequent processing.

[0096] As a possible implementation method, the call success rate and call time of the large model to be tested calling the target business data in the target complex business scenario are obtained; the evaluation results of the evaluation experts on the call success rate and call time are obtained, and the plug-in call capability score of the large model to be tested is obtained based on the evaluation results. Among them, the target business data is the business data obtained by the large model to be tested calling multiple interfaces in the target complex business scenario, the call success rate is the success rate of the large model to be tested calling the data in the target complex business scenario, and the call time is the time required for the large model to be tested to call the data in the target complex business scenario.

[0097] S303, calculating the insurance capability score of the large model to be tested based on the search enhancement generation capability score and the plug-in calling capability score.

[0098] Optionally, the sum of the retrieval enhancement generation capability score and the plug-in calling capability score is calculated, and the sum is used as the insurance capability score.

[0099] In the embodiment of the present application, the insurance capability score of the large model to be tested is calculated based on the search enhancement generation capability score obtained in step S301 and the plug-in calling capability score obtained in step S302. For example, the review expert gives a score as the insurance capability score by combining the search enhancement generation capability score and the plug-in calling capability score of the large model to be tested.

[0100] Further, such as Figure 4 As shown, the step S102 in the above embodiment of "deploying the large model to be tested on the preset hardware to be tested, and inputting the target problem into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested" may specifically include the following steps:

[0101] S401, determine all large models to be tested that are compatible with the hardware to be tested, and deploy and run all the large models to be tested on the hardware to be tested in sequence.

[0102] In the embodiment of the present application, the hardware to be tested is first selected, then all the large models to be tested that are compatible with the hardware to be tested are determined, and all the large models to be tested are deployed and run on the hardware to be tested in sequence (one by one in a queue, not run simultaneously) for subsequent processing. Figure 5 As shown, it can be seen that there are multiple large models to be tested and multiple hardware to be tested.

[0103] S402, determining the target concurrency of the large model to be tested, inputting a second target problem of the target problem length into the large model to be tested, and obtaining multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested.

[0104] In the embodiment of the present application, the target problem length is the length of the set target problem, for example, 500 tokens; the target concurrency is the concurrency of the large model to be tested, for example, 50; the target concurrency of the large model to be tested is determined, and the second target problem of the target problem length is input into the large model to be tested, and the large model to be tested is processed to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested. It should be noted that both the target concurrency and the target problem length will affect the reasoning speed of the large model, so the variables must be controlled. For example, the performance of each performance indicator of the large model when the target concurrency is 50 and the input target problem length is 500 tokens, that is, the performance indicator data; Figure 2 As shown, the parameters to be measured represent the multi-dimensional performance indicators of the present application, and the obtained parameter indicators are the multi-dimensional performance indicator data of the present application, and continue as Figure 5 As shown, the test content also represents the multi-dimensional performance indicator data of the present application, such as video memory occupancy, throughput, first word delay, etc.

[0105] In general, if Figure 2 As shown in the figure, the hardware performance test is independent of the reasoning capability test. A service instance of the selected big model is deployed on the hardware to be tested, and then a script is used to concurrently call the big model interface so that the big model can generate answers to preset questions and collect performance indicator data such as computing core utilization, video memory occupancy, throughput, and first-word latency when the big model is working.

[0106] Further, such as Figure 6 As shown, the “obtaining a hardware performance test score of the large model to be tested based on the performance indicator data” in step S102 in the above embodiment may specifically include the following steps:

[0107] S601, obtaining a ranking of performance indicator data of each dimension of each large model to be tested based on the performance indicator data of all large models to be tested.

[0108] In the embodiment of the present application, based on the performance indicator data of all the large models to be tested obtained in the above embodiment, the ranking of the performance indicator data of each dimension of each large model to be tested is determined for subsequent processing. For example, as shown in the following table, m large models to be tested are deployed in sequence on a certain hardware to be tested and the performance is tested, where the performance indicator of one dimension of chatglm3-6b, namely the throughput, ranks nth among the m models.

[0109] S602, obtaining a performance indicator score of the performance indicator data of each dimension of each large model to be tested based on the ranking of the performance indicator data of each dimension of the large model to be tested.

[0110] In the embodiment of the present application, based on the ranking of the performance indicator data of each dimension of each large model to be tested obtained in step S601, the performance indicator score of the performance indicator data of each dimension of the large model to be tested can be obtained. For example, to continue, the throughput of chatglm3-6b ranks nth among m models, then the throughput score of chatglm3-6b on the hardware to be tested is m-n+1. Similarly, the scores of performance indicators such as the first word delay, video memory occupancy, and core utilization of chatglm3-6b when deployed on the hardware to be tested can be obtained. Thus, the performance indicator score of the performance indicator data of each dimension of each large model to be tested can be obtained, for example, as shown in Table 1 below:

[0111] Table 1

[0112]

[0113]

[0114] S603, adding up the performance index scores of each dimension of the large model to be tested to obtain a hardware performance test score of the large model to be tested.

[0115] In the embodiment of the present application, the performance index scores of each dimension of the large model to be tested obtained in step S602 are added together to obtain the hardware performance test score of the large model to be tested when deployed on the hardware to be tested, thereby obtaining the hardware performance test score of each large model to be tested when deployed on each hardware to be tested. For example, adding all the scores of the various indicators of the above chatglm3-6b is the hardware performance test score of chatglm3-6b when deployed on the hardware to be tested, as shown in Table 1 above.

[0116] Further, such as Figure 7 As shown, in step S103 in the above embodiment, "obtaining a deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score" may specifically include the following steps:

[0117] S701, determining a first weight of the reasoning ability score and a second weight of the insurance ability score based on the business scenario of the large model to be tested.

[0118] S702, calculate the sum of the product of the reasoning ability score and the first weight, the product of the insurance ability score and the second weight, and the hardware performance test score to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested.

[0119] In the embodiment of the present application, the business scenario is the business scenario of the large model to be tested. The first weight of the reasoning ability score and the second weight of the insurance ability score are determined based on the business scenario. The sum of the product of the reasoning ability score and the first weight obtained in the above embodiment, the product of the insurance ability score and the second weight, and the hardware performance test score are calculated to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested, thereby obtaining the deployment score of each large model to be tested when deployed on each hardware to be tested. For example, the reasoning ability score is α, the insurance ability score is β, the first weight is w1, the second weight is β, and the hardware performance test score is γ, then the deployment score = w1*α+w2*β+γ.

[0120] Therefore, by using the deployment score of each large model to be tested when deployed on each hardware to be tested, the deployment plan of the large model to be tested is tested, and the deployment plan with good deployment effect can be screened out. For example, the large model to be tested and the hardware to be tested with the highest deployment score are selected as the deployment plan.

[0121] Figure 8 is a flow chart of a large model privatization deployment solution evaluation device according to an embodiment of the present application, such as Figure 8 As shown, the large model privatization deployment solution evaluation device 800 of the embodiment of the present application may specifically include:

[0122] The first acquisition module 801 is used to predetermine the target questions and target business documents for the large model to be tested, and obtain the insurance capability score of the large model to be tested based on the target questions and target business documents; wherein the target questions are professional questions in the insurance field and have objective answers.

[0123] The second acquisition module 802 is used to deploy the large model to be tested on the preset hardware to be tested, and input the target problem into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and to obtain the hardware performance test score of the large model to be tested based on the performance indicator data.

[0124] The third acquisition module 803 is used to test the multi-dimensional reasoning ability of the large model to be tested, obtain the reasoning ability score of the large model to be tested, and obtain the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score.

[0125] In a possible implementation manner, the first acquisition module is specifically configured to:

[0126] Based on the target question and the target business document, the answer of the large model to be tested to the target question is obtained, and based on the answer, the retrieval enhancement generation capability score of the large model to be tested is obtained;

[0127] Determine the corresponding target complex business scenario for the large model to be tested, process the target complex business scenario based on the large model to be tested, and obtain the plug-in calling capability score of the large model to be tested;

[0128] The insurance capability score of the large model to be tested is calculated based on the retrieval enhancement generation capability score and the plug-in calling capability score.

[0129] In a possible implementation manner, the first acquisition module is specifically configured to:

[0130] Obtain the first answer of the large model to be tested to the target question, and transfer the target business document to the large model to be tested to obtain the second answer of the large model to the target question;

[0131] For the large model to be tested, the corresponding review experts are determined, the review experts' evaluation results for the first answer and the second answer are obtained respectively, and the retrieval enhancement generation capability score of the large model to be tested is obtained based on the evaluation results.

[0132] In a possible implementation manner, the first acquisition module is specifically configured to:

[0133] Obtain the call success rate and call time of the large model to be tested calling the target business data in the target complex business scenario; wherein the target business data is the business data obtained by the large model to be tested calling multiple interfaces in the target complex business scenario;

[0134] Obtain the evaluation results of the review experts on the call success rate and call time, and obtain the plug-in call capability score of the large model to be tested based on the evaluation results.

[0135] In a possible implementation manner, the second acquisition module is further used to:

[0136] Determine all large models to be tested that are compatible with the hardware to be tested, and deploy and run all large models to be tested on the hardware to be tested in sequence;

[0137] Determine the target concurrency of the large model to be tested, input a second target problem of the target problem length into the large model to be tested, and obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested.

[0138] In a possible implementation manner, the second acquisition module is specifically configured to:

[0139] Based on the performance indicator data of all the large models to be tested, the ranking of the performance indicator data of each dimension of each large model to be tested is obtained;

[0140] Based on the ranking of the performance indicator data of each dimension of each large model to be tested, a performance indicator score of the performance indicator data of each dimension of the large model to be tested is obtained;

[0141] The performance indicator scores of each dimension of the large model to be tested are added together to obtain the hardware performance test score of the large model to be tested.

[0142] In a possible implementation manner, the third acquisition module is specifically configured to:

[0143] Determine the first weight of the reasoning ability score and the second weight of the insurance ability score based on the business scenario of the large model to be tested;

[0144] The product of the reasoning ability score and the first weight, the product of the insurance ability score and the second weight, and the sum of the hardware performance test score are calculated to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested.

[0145] The large model privatization deployment solution evaluation device provided in the embodiment of the present application predetermines the target problem and target business document for the large model to be tested, and obtains the insurance capability score of the large model to be tested based on the target problem and the target business document; deploys the large model to be tested on the preset hardware to be tested, and inputs the target problem into the large model to be tested, obtains multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtains the hardware performance test score of the large model to be tested based on the performance indicator data; tests the multi-dimensional reasoning ability of the large model to be tested, obtains the reasoning ability score of the large model to be tested, and obtains the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score. The big model privatization deployment plan evaluation device of the present application tests the insurance capability score of the big model to be tested through target problems and target business documents, obtains the hardware performance test score by deploying the big model to be tested on the hardware to be tested, and obtains the deployment score of each big model to be tested when deployed on each hardware platform to be tested based on the reasoning capability score, insurance capability score and hardware performance test score. Therefore, a deployment plan score can be given according to the performance of the big model as a reference for selecting the deployment plan, thereby realizing the attention to the big model service during deployment and implementation, and by scoring the deployment plan based on insurance field issues, it can ensure that the big model has good performance when dealing with insurance industry business scenarios.

[0146] like Fig. 9 As shown, an electronic device 900 provided in an embodiment of the present application includes: a processor 901, a memory 902 and a bus, wherein the memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 through the bus, and the processor 901 executes the machine-readable instructions to perform the steps of the above-mentioned large-model privatization deployment solution evaluation method.

[0147] Specifically, the above-mentioned memory 902 and processor 901 can be general-purpose memory and processor, which are not specifically limited here. When the processor 901 runs the computer program stored in the memory 902, it can execute the above-mentioned large-model privatization deployment solution evaluation method.

[0148] Corresponding to the above-mentioned large-model privatization deployment solution evaluation method, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned large-model privatization deployment solution evaluation method are executed.

[0149] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0150] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the deployment method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0152] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for evaluating a large model privatization deployment solution, characterized in that: The method comprises: Predetermine the target question and target business document for the large model to be tested, and obtain the insurance capability score of the large model to be tested based on the target question and the target business document; wherein the target question is a professional question in the insurance field with an objective answer; Deploy the large model to be tested on the preset hardware to be tested, input the target problem into the large model to be tested, obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and obtain the hardware performance test score of the large model to be tested based on the performance indicator data; The multi-dimensional reasoning ability of the large model to be tested is tested to obtain a reasoning ability score of the large model to be tested, and the deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning ability score, the insurance ability score and the hardware performance test score.

2. The method according to claim 1, characterized in that The step of obtaining the insurance capability score of the large model to be tested based on the target question and the target business document includes: Acquire an answer of the large model to be tested to the target question based on the target question and the target business document, and obtain a retrieval enhancement generation capability score of the large model to be tested based on the answer; Determine a corresponding target complex business scenario for the large model to be tested, process the target complex business scenario based on the large model to be tested, and obtain a plug-in calling capability score of the large model to be tested; The insurance capability score of the large model to be tested is calculated based on the retrieval enhancement generation capability score and the plug-in calling capability score.

3. The method according to claim 2, characterized in that The step of obtaining an answer of the large model to be tested to the target question based on the target question and the target business document, and obtaining a retrieval enhancement generation capability score of the large model to be tested based on the answer, includes: Obtaining a first answer of the large model to be tested to the target question, and transferring the target business document to the large model to be tested, to obtain a second answer of the large model to be tested to the target question; For the large model to be tested, corresponding review experts are determined, and evaluation results of the review experts for the first answer and the second answer are obtained respectively, and a retrieval enhancement generation capability score of the large model to be tested is obtained based on the evaluation results.

4. The method according to claim 3, characterized in that The processing of the target complex business scenario based on the large model to be tested to obtain a plug-in calling capability score of the large model to be tested includes: Obtaining a call success rate and a call time of the large model to be tested calling target business data on the target complex business scenario; wherein the target business data is business data obtained by the large model to be tested calling multiple interfaces on the target complex business scenario; The evaluation results of the review experts on the call success rate and the call time are obtained, and the plug-in call capability score of the large model to be tested is obtained based on the evaluation results.

5. The method according to claim 1, characterized in that The step of deploying the large model to be tested on the preset hardware to be tested, and inputting the preset second target problem into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware platform to be tested, includes: Determine all large models to be tested that are compatible with the hardware to be tested, and deploy and run all large models to be tested on the hardware to be tested in sequence; Determine the target concurrency number of the large model to be tested, input the second target problem of the target problem length into the large model to be tested, and obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested.

6. The method according to claim 1, characterized in that The step of obtaining the hardware performance test score of the large model to be tested based on the performance indicator data includes: Based on the performance indicator data of all the large models to be tested, the ranking of the performance indicator data of each dimension of each large model to be tested is obtained; Based on the ranking of the performance indicator data of each dimension of each large model to be tested, a performance indicator score of the performance indicator data of each dimension of the large model to be tested is obtained; The performance indicator scores of each dimension of the large model to be tested are added together to obtain the hardware performance test score of the large model to be tested.

7. The method according to claim 1, characterized in that The deployment score of each large model to be tested when deployed on each hardware platform to be tested is obtained based on the reasoning ability score, the insurance ability score and the hardware performance test score, including: Determine a first weight of the reasoning ability score and a second weight of the insurance ability score based on the business scenario of the large model to be tested; The product of the reasoning ability score and the first weight, the product of the insurance ability score and the second weight, and the sum of the hardware performance test score are calculated to obtain the deployment score of the large model to be tested when deployed on the hardware to be tested.

8. A large-scale model privatization deployment scheme evaluation device, characterized in that: The device comprises: A first acquisition module is used to predetermine a target question and a target business document for the large model to be tested, and obtain an insurance capability score of the large model to be tested based on the target question and the target business document; wherein the target question is a professional question in the insurance field with an objective answer; A second acquisition module is used to deploy the large model to be tested on a preset hardware to be tested, and input the target problem into the large model to be tested, to obtain multi-dimensional performance indicator data of the large model to be tested when working on the hardware to be tested, and to obtain a hardware performance test score of the large model to be tested based on the performance indicator data; The third acquisition module is used to test the multi-dimensional reasoning ability of the large model to be tested, obtain the reasoning ability score of the large model to be tested, and obtain the deployment score of each large model to be tested when deployed on each hardware platform to be tested based on the reasoning ability score, the insurance ability score and the hardware performance test score.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the large-model privatization deployment solution evaluation method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the large-model privatization deployment solution evaluation method as described in any one of claims 1 to 7.

Images