Large model reasoning service evaluation method and device, electronic equipment and medium

Abstract

The invention provides a large model reasoning service evaluation method and device, electronic equipment and a medium. The method comprises the following steps: receiving an interface mode selection instruction; under the condition that the interface mode selection instruction indicates to select the interface mode of the large model platform, determining an internal reasoning service interface corresponding to a target reasoning service in the large model platform as a to-be-evaluated interface; under the condition that the interface mode selection instruction indicates to select a user-defined interface mode, a user-defined external reasoning service interface is determined as a to-be-evaluated interface, and the external reasoning service interface corresponds to an external reasoning service; performing interface detection on the to-be-evaluated interface; and if the to-be-evaluated interface passes the interface detection, evaluating the reasoning service corresponding to the to-be-evaluated interface by using the test set to obtain an evaluation result.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to evaluation methods, devices, electronic devices and media for large model reasoning services. Background Art

[0002] Large models refer to deep learning models with a large number of parameters and computational complexity. These models have excellent performance in handling complex tasks, such as natural language processing, image recognition, speech recognition, etc. As computer hardware performance continues to improve and deep learning algorithms are rapidly optimized, large models are developing at a rapid pace. How to evaluate large models has become an urgent problem to be solved. Summary of the invention

[0003] To overcome the problems existing in the related art, this specification provides an evaluation method, device, electronic device and medium for large model reasoning services.

[0004] According to a first aspect of an embodiment of the present specification, a method for evaluating a large model reasoning service is provided, the method comprising: receiving an interface mode selection instruction; when the interface mode selection instruction indicates selecting a large model platform interface mode, determining an internal reasoning service interface corresponding to a target reasoning service in the large model platform as an interface to be evaluated; when the interface mode selection instruction indicates selecting a custom interface mode, determining a custom external reasoning service interface as an interface to be evaluated, the external reasoning service interface corresponding to an external reasoning service; performing interface detection on the interface to be evaluated; if the interface to be evaluated passes the interface detection, using a test set to evaluate the reasoning service corresponding to the interface to be evaluated, to obtain an evaluation result.

[0005] According to a second aspect of an embodiment of the present specification, there is provided an evaluation device for a large model reasoning service, comprising: a mode selection module for receiving an interface mode selection instruction; a first mode determination module for determining, when the interface mode selection instruction indicates the selection of a large model platform interface mode, an internal reasoning service interface corresponding to a target reasoning service in the large model platform as an interface to be evaluated; a second mode determination module for determining, when the interface mode selection instruction indicates the selection of a custom interface mode, a custom external reasoning service interface as an interface to be evaluated, the external reasoning service interface corresponding to an external reasoning service; an interface detection module for performing interface detection on the interface to be evaluated; and an evaluation module for evaluating, if the interface to be evaluated passes the interface detection, using a test set to evaluate the reasoning service corresponding to the interface to be evaluated, to obtain an evaluation result.

[0006] According to a third aspect of the embodiments of this specification, there is provided an electronic device, including:

[0007] processor;

[0008] a memory for storing processor-executable instructions;

[0009] The processor is configured to execute the evaluation method of the large model reasoning service of the first aspect or any corresponding embodiment thereof.

[0010] According to the fourth aspect of the embodiments of this specification, there is provided an evaluation device for a large model reasoning service, including: a computer program product, including computer instructions, the computer instructions are used to enable a computer to execute the evaluation method for a large model reasoning service of the first aspect or any corresponding implementation method thereof.

[0011] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0012] The embodiments of this specification are applicable to evaluating reasoning services that have been published on a large model platform, and are also applicable to evaluating reasoning services on other platforms, and are quick and convenient to execute.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0015] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0016] Figure 2 It is a flowchart of an evaluation method of a large model reasoning service according to an exemplary embodiment of this specification.

[0017] Figure 3 The invention is a flowchart showing interface delay adaptation according to an exemplary embodiment.

[0018] Figure 4A The following is a schematic diagram of an interface mode selection interface according to an embodiment of the present specification.

[0019] Figure 4B A schematic diagram of a service selection interface according to an embodiment of the present specification is schematically shown.

[0020] Figure 4C The figure is a schematic diagram of selecting a service interface when a detection passes according to an exemplary embodiment.

[0021] Figure 4DIt is a schematic diagram of an evaluation file execution interface according to an exemplary embodiment.

[0022] Figure 4E The figure is a schematic diagram showing the selection of a test set in an evaluation file execution interface according to an exemplary embodiment.

[0023] Figure 4F It is a schematic diagram of an evaluation file execution interface when an evaluation is executed according to an exemplary embodiment of the present specification.

[0024] Figure 4G It is a schematic diagram of an evaluation file execution interface after the evaluation is completed according to an exemplary embodiment of the present specification.

[0025] Figure 4H It is a schematic diagram of an evaluation result display interface according to an exemplary embodiment of the present specification.

[0026] Figure 5 It is a schematic diagram of a test set shown in this specification according to an exemplary embodiment.

[0027] Figure 6 It is a hardware structure diagram of the computer device where the evaluation device of the large model reasoning service in the embodiment of this specification is located.

[0028] Figure 7 It is a block diagram of an evaluation device for a large model reasoning service according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0030] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] Next, the embodiments of this specification are described in detail.

[0033] The following combination Figure 1 The system architecture of the evaluation method and device that can apply the large model reasoning service in the embodiments of this specification is described. It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of this specification can be applied, to help those skilled in the art understand the technical content of this specification, but it does not mean that the embodiments of this specification cannot be used in other devices, systems, environments or scenarios.

[0034] Figure 1 It is a schematic diagram of a system architecture shown in this specification according to an exemplary embodiment.

[0035] like Figure 1 As shown, the system architecture may include, for example, an evaluation system, a network, a large model platform, and an external reasoning service platform. The network is used to provide a medium for a communication link between the evaluation system, the large model platform, and the external reasoning service platform. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use the evaluation system to interact with the large model platform and external reasoning service platform through the network to receive or send messages, etc.

[0037] The evaluation system can be used to evaluate the reasoning services provided by the big model platform and the reasoning services provided by the external reasoning service platform. The evaluation system can be integrated into the big model platform or be independent of the big model platform.

[0038] The large model platform can be a platform for developing, training, deploying and managing large models. For example, it can include the Baiye Lingxi large model platform. The external reasoning service platform can be a platform that can provide reasoning services other than the large model platform. The external reasoning service platform can provide a reasoning service interface. The reasoning service interface can be directly called to use the corresponding reasoning service.

[0039] The following is a detailed description of the evaluation method of the large model reasoning service provided by the embodiment of this specification. Figure 2 As shown, Figure 2This is a flowchart of an evaluation method for a large model reasoning service shown in this specification according to an exemplary embodiment. The method can be applied to an evaluation system. The evaluation method for a large model reasoning service provided in the embodiment of this specification may include the following steps.

[0040] In step 210, an interface mode selection instruction is received.

[0041] According to an embodiment of the present specification, the interface mode may include a large model platform interface mode and a custom interface mode. In the large model platform interface mode, the evaluation system may evaluate the reasoning service inside the large model platform. In the custom interface mode, the evaluation system may evaluate the custom reasoning service outside the large model platform.

[0042] According to an embodiment of the present specification, the user may select the interface mode by inputting an interface mode selection instruction to the evaluation system.

[0043] In step 220, when the interface mode selection instruction indicates to select the large model platform interface mode, the internal reasoning service interface corresponding to the target reasoning service in the large model platform is determined as the interface to be evaluated.

[0044] According to an embodiment of the present specification, the evaluation system may, for example, query the internal reasoning services in an available state in the large model platform to obtain a set of available reasoning services.

[0045] The user can select an internal reasoning service to be evaluated from the set of available reasoning services by inputting an internal reasoning service selection instruction to the evaluation system.

[0046] The evaluation system receives an internal reasoning service selection instruction and determines an internal reasoning service indicated by the internal reasoning service selection instruction in a set of available reasoning services as a target reasoning service.

[0047] In step 230, when the interface mode selection instruction indicates to select the custom interface mode, a custom external reasoning service interface is determined as the interface to be evaluated, and the external reasoning service interface corresponds to the external reasoning service.

[0048] According to an embodiment of the present specification, the external reasoning service interface may be preset by a user, and the evaluation system may use the corresponding external reasoning service by calling the external reasoning service interface.

[0049] In step 240, interface detection is performed on the interface to be evaluated.

[0050] According to an embodiment of the present specification, the evaluation system may, for example, send an inference test request to the interface to be evaluated based on the network address of the interface to be evaluated. If a response message of the interface to be evaluated is received, and the response message includes a status identifier indicating that the request has been successful, it is determined that the interface to be evaluated has passed the interface detection. Otherwise, it is determined that the interface to be evaluated has not passed the interface detection. For example, the status identifier may include a status code, and a status code of 200 may indicate that the request has been successful. If the status code in the response message is 200, it can be determined that the interface to be evaluated has passed the interface detection.

[0051] Optionally, when the internal reasoning service interface corresponding to the target reasoning service in the large model platform is determined as the interface to be evaluated, the evaluation system can, for example, obtain the network address of the interface to be evaluated from the large model platform. When a customized external reasoning service interface is determined as the interface to be evaluated, the user can input customized interface information to the evaluation system, and the customized interface information can include information such as the network address of the interface to be evaluated. The evaluation system can receive the customized interface information, thereby obtaining the network address of the interface to be evaluated contained in the customized interface information.

[0052] In step 250, if the interface to be evaluated passes the interface detection, the reasoning service corresponding to the interface to be evaluated is evaluated using the test set to obtain an evaluation result.

[0053] The evaluation method of the large model reasoning service according to the embodiment of this specification is applicable to the evaluation of the reasoning service published on the large model platform, and is also applicable to the evaluation of the reasoning service of other platforms, and the execution is fast and convenient. The method can be applied to the testing of the large model related platform, the development of large model applications, and other scenarios, and can improve the work efficiency in these scenarios.

[0054] Optionally, the test set may include a knowledge classification, a question stem, answer options, and an option list of at least one assessment question. The knowledge classification may represent the evaluation dimension of the current assessment question. The question stem may represent the question stem portion of the current assessment question. The answer option may represent the option number of the correct answer to the current assessment question. The option list may represent the specific content of inputting different options.

[0055] Based on this, step 250 may include, for example, the following steps.

[0056] In step 251, the interface to be evaluated is called to infer the stem and option list of each evaluation question in the test set to obtain the output answer corresponding to each evaluation question.

[0057] According to an embodiment of the present specification, for example, a reasoning request may be sent to the interface to be evaluated, the reasoning request including the stem of the evaluation question and a list of options. The reasoning service corresponding to the interface to be evaluated may select an option matching the stem from the list of options as the output answer corresponding to the evaluation question.

[0058] In step 252, the output answer corresponding to each evaluation question is compared with the answer options of the evaluation question to obtain the answer result of the reasoning service for each evaluation question, and the answer result includes correct or wrong.

[0059] According to an embodiment of the present specification, if the output answer corresponding to the evaluation question is consistent with the answer option of the evaluation question, it can be determined that the answer result of the reasoning service to the evaluation question is correct. Otherwise, it is determined that the answer result of the reasoning service to the evaluation question is wrong.

[0060] In step 253, the answer results of each evaluation question are counted according to the knowledge classification of the evaluation question, and the correct rate of each knowledge classification is obtained as the evaluation result.

[0061] Optionally, the test set can be recorded in a preset format. The preset format can include, for example, a table file. The table file can include at least one table item, each table item including a knowledge classification, a question stem, answer options, and an option list of an assessment question. Based on this, when the test set is needed, the table file can be imported and parsed to obtain the test set.

[0062] When the evaluation system calls the inference service interface to evaluate the corresponding inference service, some inference service interfaces may be limited, setting an upper limit on the number of interface calls or the number of tokens per unit time, resulting in inference failure during the evaluation process. Since the current limiting rules set by the interface are unknown, in this embodiment, the interface delay can be adapted by automatic retry and automatic adjustment of the interface delay, so that during the evaluation process, the inference service interface request interval automatically matches the current limiting rules set by the interface, avoiding evaluation failure due to interface current limiting.

[0063] Based on this, during the evaluation of the inference service corresponding to the interface to be evaluated, if calling the interface to be evaluated fails and the current retry count is less than or equal to the retry count threshold, the interface to be evaluated can be called again after waiting for the delay time.

[0064] If the interface to be evaluated is successfully called again, the preset time adjustment parameter is increased, and the integer part of the quotient of the time adjustment parameter divided by the retry count threshold is calculated, and the sum of the calculated delay time initial value is used as the new delay time;

[0065] If the call to the interface to be evaluated fails again, and the current number of retries is less than or equal to the retry threshold, the operation of calling the interface to be evaluated again after the waiting delay time is returned;

[0066] If the call to the evaluated interface fails again and the current retry count is greater than the retry count threshold, a call failure prompt message is generated and the duration adjustment parameter is reduced.

[0067] like Figure 3 FIG. 1 is a flowchart of interface delay adaptation according to an exemplary embodiment.

[0068] In this embodiment, a global variable Ft may be defined to represent a time adjustment parameter, delay to represent a delay time, and Try to represent a retry count. Initially, Ft=0, Delay=N, where N may be, for example, 10, in seconds, Try=0, and the retry count threshold may be 3.

[0069] When the first call to the inference service interface fails (for example, the returned status code is not 200), the inference service interface will be called again after sleeping for 10 seconds after the first call. If the second call is successful, it means that the inference service is unstable and the call to the inference service interface may fail. At this time, let Ft add 1, and let Delay = N + Ft / / 3, where / / represents the integer division operator.

[0070] If the call fails again, the number of retries Try is increased by 1. If Try is less than or equal to the retry threshold, the inference service interface is called again after waiting for 10 seconds. If Try is greater than the retry threshold, Ft is reduced by 1. In this embodiment, the retry threshold can be, for example, 3 times.

[0071] In an inference request, if it succeeds within 3 retries, Ft will increase by 1.

[0072] If an inference request fails after three retries, the retry is exited and the call fails. Ft is reduced by 1, that is, the time adjustment parameter is reduced to avoid frequent retries when the service status is very poor, which will extend the test time.

[0073] During the test of the entire test set, if the number of failures increases, it means that the service status is not good, and the delay time Delay of the inference request retry needs to be extended. According to the formula Delay = N + Ft / / 3, every time Ft increases by 3, Delay increases by 1s.

[0074] Optionally, the evaluation system can support custom construction of large model evaluation system. The evaluation logic can be implemented by Python coding and encapsulated as an .exe executable program, which can run on Windows operating system. Users can directly execute the program on Windows operating system to evaluate the specified reasoning interface through the program, and a built-in evaluation set is provided.

[0075] The evaluation system can provide main business functions and main business functions.

[0076] The main business functions may include, for example:

[0077] 1) Set the interface mode. The supported interface modes may include, for example, a large model platform interface mode and a custom interface mode.

[0078] 2) The evaluation set supports automatic execution and can also support the generation of custom evaluation sets.

[0079] 3) Evaluation parameter adjustment. For example, it can support multi-round settings, temperature value settings, etc.

[0080] The main business functions may include, for example:

[0081] 1) Historical environment records. For example, the environment in which historical evaluations were performed can be automatically recorded, and environment login parameters can be automatically saved.

[0082] 2) Inference interface delay setting. For example, it can support custom delay setting, inference interface delay adaptation, etc.

[0083] 3) Evaluation result display. For example, it can support dynamic display of evaluation progress and scores, searchable evaluation result files, display of evaluation results by dimension, automatic ranking of evaluation results, etc.

[0084] Figure 4A The following is a schematic diagram of an interface mode selection interface according to an embodiment of the present specification.

[0085] like Figure 4A As shown, the evaluation system can display an interface mode selection interface. The interface mode selection interface includes a selection component for selecting a large model platform interface and a custom interface, a login information input box, an interface information input box, etc. The selection component can be used to select an interface mode for the test, and the interface mode can include a "large model platform interface" mode and a "custom interface" mode. The selection component can include, for example, a single-select form component.

[0086] The login information input box can be used to input login information for the large model platform, such as IP address, port number, user name, password, etc. The interface information input box can be used to input relevant information of the custom interface, such as URL (Uniform Resource Locator), model name, etc.

[0087] In the "Large Model Platform Interface" mode, you can log in to the large model platform according to the login information, obtain the login token, and then call the reasoning service interface in the large model platform environment. If the login fails, the user is prompted to re-enter the login information. In the "Custom Interface" mode, you can call the custom interface according to the URL and model name of the custom interface, and the calling process does not require a token.

[0088] Optionally, a login record file login_records.txt can be created in the local file directory Dataset. When the large model platform login is successful, or the custom interface detection is passed, the current login parameters can be recorded in this file. The next time the evaluation system is opened, the parameters of the last successful login will be automatically filled into the corresponding input box.

[0089] Optionally, after selecting the "Large Model Platform Interface" mode, the login information input box can be set to be available, and the interface information input box can be set to be unavailable. After selecting the "Custom Interface" mode, the login information input box can be set to be unavailable, and the interface information input box can be set to be available.

[0090] When the mode is selected as the large model platform login in the previous step and the login is successful, the service interface selection interface is displayed.

[0091] Figure 4B A schematic diagram of a service selection interface according to an embodiment of the present specification is schematically shown.

[0092] like Figure 4B As shown, the service interface selection interface displays a list of all model services in the current large model platform environment that are in "service" status, and provides radio button options. After selecting a service, the interface is tested. If the test is successful, proceed to the next step.

[0093] For example, all services can be obtained through the service query interface of the large model platform, and then the services whose status field value is 3 can be filtered. The value of the status field is 3, which means that the status of the service is "in service".

[0094] Then, you can send a detection request to the selected service interface. Figure 4CFIG. 1 is a schematic diagram of selecting a service interface when the detection is passed according to an exemplary embodiment. If the received response status code is 200, it is determined that the detection is passed, and a prompt message "Interface detection successful" can be displayed.

[0095] If the previously selected mode is "Custom Interface", there is no need to display the service interface selection interface, and the interface detection can be performed directly. If the response status code received is 200, it is determined that the detection has passed, and the prompt "Interface detection successful" can be displayed.

[0096] After the service interface is successfully detected, the evaluation file execution interface will be displayed. Figure 4D FIG. 1 is a schematic diagram of an evaluation file execution interface according to an exemplary embodiment. The functions of the interface may include selecting a test set, setting execution parameters, displaying execution progress, and viewing results after evaluation is completed.

[0097] Execution parameters may include, for example, Temperature (temperature value), interface request delay, prompt (prompt word) setting, execution rounds, etc.

[0098] In this embodiment, a drop-down option box can be set to set Temperature. The supported setting value can be 0-1.0, and the step size can be 0.1, which is convenient for comparing the evaluation results under different temperature conditions.

[0099] The temperature value in the large model can be used to adjust the output probability distribution of the model, thereby affecting the randomness and diversity of the model. The temperature value can change the probability distribution of the model's predictions by affecting the output of the Softmax function. Lower temperature values ​​can make the generated text more coherent and accurate, while higher temperature values ​​make the text more diverse and creative.

[0100] The temperature value can have the following effects:

[0101] Controlling randomness: Higher temperature values ​​will make the Softmax distribution smoother, making the model more likely to generate more diverse and random text. Conversely, lower temperature values ​​will make the distribution more spiky, generating more deterministic and consistent text.

[0102] Adjust the entropy of the output probability distribution: The temperature value can adjust the entropy of the output probability distribution, increasing or decreasing randomness. When the temperature value is close to 0, the output becomes very deterministic; when the temperature value is greater than 1, the output becomes more random.

[0103] In this embodiment, it is possible to support setting execution rounds. Based on this, the execution rounds can be set to multiple times, that is, multiple rounds of execution evaluation are supported. In this embodiment, when using the custom interface mode, the reasoning interface in some environments has a limit on the number of reasoning times per unit time. Therefore, the interface request delay can be set to avoid the situation where the reasoning request is rejected due to excessive request frequency. If the user does not set it, the interface request delay can be a default value, and the default value can be 0, for example.

[0104] In some scenarios, when the user calls a custom interface to evaluate the model, it is not clear whether the interface has limited flow. Based on this, in this embodiment, based on the custom interface delay, the logic of adaptive delay can be added in the background. The specific logic is as follows:

[0105] When the delay is 0, if the inference request fails, wait for 10 seconds and resubmit. If the inference request succeeds after resubmission, the retry count will be increased by 1. When the retry count reaches 2, it can be determined that the previous inference request failed due to interface current limiting. Accordingly, the delay can be changed to 6S and the count can be cleared. 6s is the first preset value.

[0106] When the delay is not equal to 0, the above situation occurs and the count reaches 2, then the delay is increased by 3S. 3s is the second preset value. If the delay is not equal to 0, it means that the number of retries has increased and the delay has been extended. If the call fails again for 2 times, it means that the current default delay cannot guarantee the success of the interface retry, then the default delay can be fixedly increased by 3s.

[0107] like Figure 4E , is a schematic diagram of selecting a test set in an evaluation file execution interface according to an exemplary embodiment. In this embodiment, a drop-down option box can be set to select a test set in a test set storage directory, and the test set can be a table file (e.g., .xlsx file). The designated storage directory of the test set can be pre-set, for example, it can be \Dataset\evaluate.

[0108] Once the parameters and test sets are set, the user can click the "Evaluation Execution" button to start the evaluation. Figure 4F FIG. 1 is a schematic diagram of an evaluation file execution interface during evaluation execution according to an exemplary embodiment of the present specification. Figure 4F As shown, when the assessment is executed, the execution progress can be displayed in real time on the assessment file execution interface.

[0109] Figure 4G FIG. 1 is a schematic diagram of an evaluation file execution interface after the evaluation is completed according to an exemplary embodiment of the present specification. Figure 4GAs shown, after the evaluation is completed, the execution progress shown in the evaluation file execution interface example becomes 100%, and the execution results are displayed, and the user can view the results.

[0110] Figure 4H is a schematic diagram of an evaluation result display interface according to an exemplary embodiment of this specification. Figure 4H As shown, the evaluation result display interface can display the evaluation data of each evaluation dimension and the overall evaluation data, for example, it may include the number of executions, the number of correct answers, the number of errors, the score rate, the evaluation, the ranking, etc.

[0111] According to an embodiment of the present specification, for example, a corresponding evaluation result record file can be created for each test set in the background, and the file format can be, for example, .xlsx. When the evaluation task is completed (progress 100%), the current evaluation result can be recorded in the file, and the evaluation result can include, for example, the score of each evaluation dimension and the total score.

[0112] The scores of each evaluation dimension and the total score can be ranked separately, and the ranking of the scores of each evaluation dimension in the existing test records can be displayed on the result display interface.

[0113] According to an embodiment of the present specification, the assessment questions in the test set may be objective questions. In this embodiment, a certain option number may be set for each objective question, and each objective question may have multiple options.

[0114] Figure 5 FIG. 1 is a schematic diagram of a test set according to an exemplary embodiment of the present specification. Figure 5 As shown, the test set can support customization. The customized test set can uniformly adopt a preset format, and the preset format can be, for example, in the form of a spreadsheet file (such as .xlsx).

[0115] The table headers in the spreadsheet file may include "knowledge classification", "question stem", "reference answer", "answer options" and "option content".

[0116] Knowledge classification can be a required item, indicating the evaluation dimension of the current assessment question. When the assessment results are counted, they will be counted and ranked separately according to the knowledge classification.

[0117] The question stem can be a mandatory item, which indicates the question stem part of the current assessment question. When evaluating the model, the question stem needs to be submitted to the model.

[0118] The reference answer can be a non-mandatory item, indicating the correct answer to the current assessment question. The correct answer can be text content.

[0119] Answer options can be mandatory and represent the option number of the correct answer to the current assessment question. The backend can determine whether the model is correct by comparing the options output by the model with the answer options.

[0120] The option list can be a required item, which means you need to enter the specific content of different options. The specific number of options can be determined according to the actual question.

[0121] Based on this, users can fill in the "knowledge classification", "question stem", "reference answer", "answer options" and "option content" of each assessment question in the table in accordance with the above table format according to specific scenario requirements, so as to obtain a customized test set. Alternatively, users can also modify the test set template that comes with the assessment system to obtain a customized test set.

[0122] In this solution, after obtaining a customized test set, it can be saved to a specified file directory. For example, the test set can be saved to the \Dataset\evaluate\ directory of the directory where the evaluation program file is located. When the evaluation system executes the test set selection step, it can automatically read all the test set files in the specified file directory and generate a list of options in the drop-down menu based on the name of the test set file.

[0123] Corresponding to the embodiments of the aforementioned methods, this specification also provides embodiments of an evaluation device for a large model reasoning service and a terminal to which it is applied.

[0124] The embodiment of the evaluation device for the large model reasoning service of this specification can be applied to a computer device, such as a server or a terminal device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 6 As shown, it is a hardware structure diagram of the computer device where the evaluation device of the large model reasoning service of the embodiment of this specification is located, except Figure 6 In addition to the processor 610, memory 630, network interface 620, and non-volatile memory 640 shown, the server or electronic device where the device 631 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0125] like Figure 7 As shown, Figure 7 1 is a block diagram of an evaluation device for a large model reasoning service according to an exemplary embodiment of the present specification, the device comprising:

[0126] The mode selection module 710 is used to receive an interface mode selection instruction;

[0127] The first mode determination module 720 is used to determine the internal reasoning service interface corresponding to the target reasoning service in the large model platform as the interface to be evaluated when the interface mode selection instruction indicates to select the large model platform interface mode;

[0128] A second mode determination module 730 is used to determine a customized external reasoning service interface as the interface to be evaluated when the interface mode selection instruction indicates to select a customized interface mode, and the external reasoning service interface corresponds to the external reasoning service;

[0129] An interface detection module 740 is used to perform interface detection on the interface to be evaluated;

[0130] The evaluation module 750 is used to evaluate the reasoning service corresponding to the interface to be evaluated by using the test set if the interface to be evaluated passes the interface detection, and obtain an evaluation result.

[0131] Optionally, the test set may include a knowledge classification, a question stem, answer options, and an option list of at least one assessment question; and the assessment module may include:

[0132] The calling submodule is used to call the interface to be evaluated to infer the stem and option list of each evaluation question in the test set, and obtain the output answer corresponding to each evaluation question;

[0133] The comparison submodule is used to compare the output answer corresponding to each evaluation question with the answer options of the evaluation question to obtain the answer result of the reasoning service for each evaluation question, and the answer result includes correct or wrong;

[0134] The statistical submodule is used to count the answer results of each assessment question according to the knowledge classification of the assessment question, and obtain the correct rate of each knowledge classification as the assessment result.

[0135] Optionally, the device may further include:

[0136] The delay module is used to call the interface to be evaluated again after waiting for the delay time if the call to the interface to be evaluated fails and the current retry count is less than or equal to the retry count threshold during the evaluation of the reasoning service corresponding to the interface to be evaluated.

[0137] A delay duration increasing module, configured to increase a preset duration adjustment parameter if the interface to be evaluated is successfully called again, and calculate the sum of the integer part of the quotient of the duration adjustment parameter divided by the retry count threshold and the initial value of the delay duration as a new delay duration;

[0138] A return module is used to return the operation of calling the interface to be evaluated again after waiting for a delay time if the call to the interface to be evaluated fails again and the current retry count is less than or equal to the retry count threshold;

[0139] The delay duration reduction module is used to generate a call failure prompt message and reduce the duration adjustment parameter if the interface to be evaluated fails to be called again and the current retry count is greater than the retry count threshold.

[0140] Optionally, the device may further include:

[0141] The import module is used to import and parse the table file to obtain the test set. The table file includes at least one table item, and each table item includes the knowledge classification, question stem, answer options and option list of an assessment question.

[0142] Optionally, the device may further include:

[0143] A query module, used to query the internal reasoning services in the large model platform in an available state to obtain a set of available reasoning services when the interface mode selection instruction indicates to select the large model platform interface mode;

[0144] A service selection module, used for receiving an internal reasoning service selection instruction;

[0145] The service determination module is used to determine the internal reasoning service indicated by the internal reasoning service selection instruction in the available reasoning service set as the target reasoning service.

[0146] Optionally, the detection module may include:

[0147] A test request sending submodule, used to send an inference test request to the interface to be evaluated according to the network address of the interface to be evaluated;

[0148] The response analysis submodule is used to determine that the interface to be evaluated passes the interface detection if response information of the interface to be evaluated is received and the response information includes a status identifier indicating that the request has been successful.

[0149] Optionally, the device may further include:

[0150] A first acquisition module is used to acquire the network address of the interface to be evaluated from the large model platform when the internal reasoning service interface corresponding to the target reasoning service in the large model platform is determined as the interface to be evaluated;

[0151] The second acquisition module is used to receive the custom interface information when the custom external reasoning service interface is determined as the interface to be evaluated, and obtain the network address of the interface to be evaluated contained in the custom interface information.

[0152] The evaluation device for the large model reasoning service according to the embodiment of this specification is suitable for evaluating the reasoning service published on the large model platform, and is also suitable for evaluating the reasoning service of other platforms, and the execution is fast and convenient. The device can be used in scenarios such as testing of large model related platforms and large model application development, and can improve the work efficiency in these scenarios.

[0153] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: receive an interface mode selection instruction; when the interface mode selection instruction indicates to select a large model platform interface mode, determine an internal reasoning service interface corresponding to a target reasoning service in the large model platform as an interface to be evaluated; when the interface mode selection instruction indicates to select a custom interface mode, determine a custom external reasoning service interface as an interface to be evaluated, the external reasoning service interface corresponding to an external reasoning service; perform interface detection on the interface to be evaluated; if the interface to be evaluated passes the interface detection, use a test set to evaluate the reasoning service corresponding to the interface to be evaluated, and obtain an evaluation result.

[0154] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0155] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0156] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0158] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0159] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A method for evaluating a large model reasoning service, characterized in that: The method comprises: Receive interface mode selection instruction; In the case where the interface mode selection instruction indicates to select the large model platform interface mode, determining the internal reasoning service interface corresponding to the target reasoning service in the large model platform as the interface to be evaluated; In the case where the interface mode selection instruction indicates to select a custom interface mode, determining a custom external reasoning service interface as the interface to be evaluated, the external reasoning service interface corresponding to the external reasoning service; Performing interface detection on the interface to be evaluated; If the interface to be evaluated passes the interface detection, the reasoning service corresponding to the interface to be evaluated is evaluated using the test set to obtain an evaluation result.

2. The method according to claim 1, characterized in that The test set includes a knowledge classification, a question stem, answer options, and an option list of at least one evaluation question; the test set is used to evaluate the reasoning service corresponding to the interface to be evaluated to obtain an evaluation result, including: Calling the to-be-evaluated interface to infer the stem and option list of each evaluation question in the test set to obtain an output answer corresponding to each evaluation question; Compare the output answer corresponding to each evaluation question with the answer options of the evaluation question to obtain the answer result of the reasoning service for each evaluation question, wherein the answer result includes correct or wrong; The answer results of each evaluation question are counted according to the knowledge classification of the evaluation question to obtain the correct rate of each knowledge classification as the evaluation result.

3. The method according to claim 1, characterized in that The method further comprises: In the process of evaluating the reasoning service corresponding to the interface to be evaluated, if calling the interface to be evaluated fails and the current retry count is less than or equal to the retry count threshold, calling the interface to be evaluated again after waiting for the delay time; If the interface to be evaluated is successfully called again, the preset time adjustment parameter is increased, and the integer part of the quotient of the time adjustment parameter divided by the retry number threshold is calculated, and the sum of the initial value of the delay time is used as the new delay time; If calling the interface to be evaluated again fails, and the current number of retries is less than or equal to the retry number threshold, then returning to the operation of calling the interface to be evaluated again after the waiting delay time; If calling the interface to be evaluated fails again, and the current number of retries is greater than the retry number threshold, a call failure prompt message is generated, and the duration adjustment parameter is reduced.

4. The method according to claim 2, characterized in that: The method further comprises: Import and parse a table file to obtain a test set, wherein the table file includes at least one table item, and each table item includes a knowledge classification, a question stem, answer options, and an option list of an assessment question.

5. The method according to claim 1, characterized in that In the case where the interface mode selection instruction indicates to select the large model platform interface mode, the method further includes: Querying the internal reasoning services in the large model platform that are in an available state to obtain a set of available reasoning services; receiving an internal reasoning service selection instruction; An internal reasoning service indicated by the internal reasoning service selection instruction in the available reasoning service set is determined as the target reasoning service.

6. The method according to claim 1, characterized in that The performing interface detection on the interface to be evaluated comprises: Sending a reasoning test request to the interface to be evaluated according to the network address of the interface to be evaluated; If response information of the interface to be evaluated is received, and the response information includes a status identifier indicating that the request has been successful, it is determined that the interface to be evaluated passes the interface detection.

7. The method according to claim 6, characterized in that The method further comprises: When the internal reasoning service interface corresponding to the target reasoning service in the large model platform is determined as the interface to be evaluated, the network address of the interface to be evaluated is obtained from the large model platform; When a customized external reasoning service interface is determined as the interface to be evaluated, customized interface information is received, and a network address of the interface to be evaluated contained in the customized interface information is obtained.

8. An evaluation device for a large model reasoning service, characterized in that: The device comprises: A mode selection module, used for receiving an interface mode selection instruction; A first mode determination module is used to determine, when the interface mode selection instruction indicates to select the large model platform interface mode, an internal reasoning service interface corresponding to the target reasoning service in the large model platform as the interface to be evaluated; A second mode determination module is used to determine a customized external reasoning service interface as the interface to be evaluated when the interface mode selection instruction indicates to select a customized interface mode, wherein the external reasoning service interface corresponds to an external reasoning service; An interface detection module, used for performing interface detection on the interface to be evaluated; The evaluation module is used to evaluate the reasoning service corresponding to the interface to be evaluated by using the test set if the interface to be evaluated passes the interface detection, so as to obtain an evaluation result.

9. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images