An algorithm evaluation method, system, apparatus, client and server

Abstract

The embodiment of the application provides an algorithm evaluation method, system, device, client and server, relates to the technical field of algorithm testing, and the method comprises the following steps: obtaining index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; determining the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the obtained test data of the specified evaluation dimension index; determining a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; and generating a business conclusion of the to-be-tested algorithm for a business scene based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation mode indicated by the target algorithm evaluation strategy. The efficiency of algorithm evaluation can be improved by the scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of algorithm testing, in particular to an algorithm evaluation method, system, device, client and server. BACKGROUND

[0002] Algorithm evaluation refers to evaluating the running situation of an algorithm in a certain business scenario to provide data support for subsequent business.

[0003] In order to determine the running situation of an algorithm in a certain business scenario, the related art mainly adopts an artificial method to evaluate the algorithm. For example, in the algorithm training process, the algorithm after training needs to be tested by using a validation data set to determine whether the accuracy, recall rate and the like of the trained algorithm meet the running requirements in the business scenario. In the related art, a human being needs to analyze the test data for algorithm testing to obtain the final validation result.

[0004] Since the related art needs to evaluate the algorithm artificially, the efficiency of algorithm evaluation is low. SUMMARY

[0005] The embodiments of the present application aim to provide an algorithm evaluation method, system, device, client and server to improve the efficiency of algorithm evaluation. The specific technical solutions are as follows.

[0006] In a first aspect, the embodiments of the present application provide an algorithm evaluation method, which comprises:

[0007] Obtaining index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; wherein the specified test strategy is a test strategy matched with a business scenario of the to-be-tested algorithm, the specified test strategy indicates that the to-be-tested algorithm is tested under at least one test configuration, and the specified evaluation index is at least one of various evaluation indexes;

[0008] Determining a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the obtained test data of the specified evaluation index;

[0009] Determining a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation index;

[0010] Generating a business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation way indicated by the target algorithm evaluation strategy.

[0011] Optionally, the index data of the specified evaluation index obtained by the to-be-tested algorithm according to the specified test strategy comprises:

[0012] an algorithm test task or an offline evaluation task is acquired; wherein the algorithm test task comprises an algorithm identifier, a strategy identifier and a specified evaluation index; the offline evaluation task comprises storage address information; the storage address information indicates an acquisition address of offline index data, and the offline index data is index data of the specified evaluation index obtained by pre-testing a to-be-tested algorithm according to a specified test strategy;

[0013] in the case of acquiring the algorithm test task, a to-be-tested algorithm indicated by the algorithm identifier is tested according to a specified test strategy indicated by the strategy identifier, to obtain index data of the to-be-tested algorithm under the specified evaluation index;

[0014] in the case of acquiring the offline evaluation task, the offline index data is read from the acquisition address indicated by the storage address information.

[0015] Optionally, the algorithm test task or the offline evaluation task is acquired, comprising:

[0016] a task generation interface is displayed; wherein the task generation interface comprises a task type selection box;

[0017] a task selection operation on the task type selection box is received;

[0018] in the case of the task type selected by the task selection operation being a test type, a first sub-interface is displayed; wherein the first sub-interface comprises a test algorithm selection box, a test strategy selection box and an evaluation index selection box; selection operations on the test algorithm selection box, the test strategy selection box and the evaluation index selection box are respectively received, to determine the selected algorithm identifier, strategy identifier and specified evaluation index; and an algorithm test task is generated based on the selected algorithm identifier, strategy identifier and specified evaluation index;

[0019] in the case of the task type selected by the task selection operation being an offline type, a second sub-interface is displayed; wherein the second sub-interface comprises a position selection box; a selection operation on the position selection box is received, to determine the selected storage address information; and an offline evaluation task is generated based on the selected storage address information.

[0020] Optionally, the test configuration comprises a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration; the terminal hardware configuration indicates terminal information deployed when the to-be-tested algorithm is tested, and the data type configuration indicates type information of data input when the to-be-tested algorithm is tested.

[0021] The corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration is determined based on the obtained test data of the specified evaluation dimension, comprising:

[0022] Based on the obtained test data of the specified evaluation dimension, a change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, when the to-be-tested algorithm is in the first dimension configuration.

[0023] Optionally, the terminal hardware configuration comprises at least one of a terminal configuration and a hardware configuration; wherein the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained by the terminal when the to-be-tested algorithm is tested.

[0024] The data type configuration comprises at least one of a data source configuration, a task number configuration, and a data format configuration; wherein the data source configuration indicates a data source to which input data of the to-be-tested algorithm belongs when the to-be-tested algorithm is tested; the task number configuration indicates a maximum number of tasks processed synchronously by the to-be-tested algorithm when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of the input data of the to-be-tested algorithm when the to-be-tested algorithm is tested.

[0025] Optionally, the terminal configuration comprises at least one of a terminal number, a terminal type, and a terminal version; the hardware configuration comprises at least one of a number of graphics cards, a size of memory, and a processor computing power; the data source configuration comprises at least one of a scenario to which a data source belongs, a data source type, and a number of data in the data source; and the data format configuration comprises at least one of resolution, frame rate, and type.

[0026] Optionally, the determination of the preset algorithm evaluation strategy associated with the specified test strategy comprises:

[0027] At least one algorithm evaluation strategy is displayed; wherein any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index.

[0028] receive a selection operation for the at least one algorithm evaluation strategy, and select, as the algorithm evaluation strategy associated with the specified test strategy, an algorithm evaluation strategy selected by the selection operation from the at least one algorithm evaluation strategy.

[0029] Optionally, after the business conclusion of the to-be-tested algorithm for the business scenario is generated in the business conclusion generation manner indicated by the target algorithm evaluation strategy based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, the method further comprises:

[0030] display an evaluation display interface; wherein the evaluation display interface comprises a target legend corresponding to the specified evaluation index and / or the generated business conclusion; wherein the target legend is a legend generated based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0031] Optionally, the target algorithm evaluation strategy comprises an algorithm evaluation template; the algorithm evaluation template comprises a condition field and a blank field to be filled with content; the condition field comprises a filtering condition for filtering a test configuration and / or an evaluation index;

[0032] The business conclusion of the to-be-tested algorithm for the business scenario is generated in the business conclusion generation manner indicated by the target algorithm evaluation strategy based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, comprising:

[0033] According to the filtering condition contained in the condition field in the algorithm evaluation template, the test configuration when the to-be-tested algorithm is tested and / or the index data of the specified evaluation index obtained by testing the to-be-tested algorithm are filtered to obtain filtered data;

[0034] From the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, determine the index data and / or the test configuration corresponding to the filtered data;

[0035] Fill the determined index data and / or test configuration into the blank field in the algorithm evaluation template to obtain the business conclusion of the to-be-tested algorithm for the business scenario.

[0036] Optionally, the evaluation indexes comprise at least one of a hardware evaluation index and an algorithm evaluation index;

[0037] The hardware evaluation index comprises at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value;

[0038] The algorithm evaluation index includes at least one of algorithm throughput, algorithm accuracy, and response time.

[0039] In a second aspect, an embodiment of the present application provides an algorithm evaluation method applied to a client in an algorithm evaluation system, the algorithm evaluation system further comprising a server, and the method comprising:

[0040] displaying a task generation interface;

[0041] receiving a configuration operation on the task generation interface to generate an algorithm evaluation task, wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy;

[0042] sending the algorithm evaluation task to the server, so that the server acquires the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task after receiving the algorithm evaluation task, determines a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the acquired test data of the specified evaluation dimension, determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation index, and generates a business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation way indicated by the target algorithm evaluation strategy.

[0043] In a third aspect, an embodiment of the present application provides an algorithm evaluation method applied to a server in an algorithm evaluation system, the algorithm evaluation system further comprising a client, and the method comprising:

[0044] receiving an algorithm evaluation task generated by the client, wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy;

[0045] acquiring the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task;

[0046] determining a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the acquired test data of the specified evaluation dimension;

[0047] determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index;

[0048] generate, according to a business conclusion generation manner indicated by the target algorithm evaluation strategy, a business conclusion of the to-be-tested algorithm for the business scenario based on a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0049] In a fourth aspect, an embodiment of the present application provides an algorithm evaluation system, which comprises a client and a server.

[0050] The client is configured to display a task generation interface, receive a configuration operation on the task generation interface, generate an algorithm evaluation task, wherein the algorithm evaluation task indicates a data acquisition strategy of acquiring index data of a specified evaluation index obtained by a to-be-tested algorithm according to a specified test strategy, and send the algorithm evaluation task to the server.

[0051] The server is configured to receive the algorithm evaluation task sent by the client, acquire the index data of the specified evaluation index obtained by the to-be-tested algorithm according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task, determine a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the acquired test data of the specified evaluation dimension, determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index, and generate a business conclusion of the to-be-tested algorithm for the business scenario according to a business conclusion generation manner indicated by the target algorithm evaluation strategy based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0052] Optionally, the task generation interface comprises a task type selection box, and the configuration operation on the task generation interface comprises a task selection operation on the task type selection box.

[0053] The client receives the configuration operation on the task generation interface to generate the algorithm evaluation task, which comprises:

[0054] In a case where the task type selected by the task selection operation is a test type, a first sub-interface is displayed; the first sub-interface includes a test algorithm selection box, a test strategy selection box, and an evaluation index selection box; selection operations on the test algorithm selection box, the test strategy selection box, and the evaluation index selection box are received respectively, and selected algorithm identification, strategy identification, and specified evaluation indexes are determined; based on the selected algorithm identification, strategy identification, and specified evaluation indexes, an algorithm test task is generated as an algorithm evaluation task;

[0055] In a case where the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; the second sub-interface includes a location selection box; a selection operation on the location selection box is received, and selected storage address information is determined; based on the selected storage address information, an offline evaluation task is generated as an algorithm evaluation task; the selected storage address information indicates an acquisition address of offline index data, and the offline index data is index data of specified evaluation indexes obtained by testing a to-be-tested algorithm according to a specified test strategy.

[0056] Optionally, the server acquires, based on a data acquisition strategy indicated by the algorithm evaluation task, index data of the specified evaluation indexes obtained by testing the to-be-tested algorithm according to the specified test strategy, including:

[0057] In a case where the acquired task is the algorithm test task, a to-be-tested algorithm indicated by the algorithm identification is tested according to a specified test strategy indicated by the strategy identification, to obtain index data of the to-be-tested algorithm under the specified evaluation indexes;

[0058] In a case where the acquired task is the offline evaluation task, the offline index data is read from an acquisition address indicated by the selected storage address information.

[0059] Optionally, the client further displays at least one algorithm evaluation strategy after displaying the task generation interface; any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; a selection operation on the at least one algorithm evaluation strategy is received, and an algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy is sent to the server;

[0060] The server determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, including:

[0061] The algorithm evaluation strategy sent by the client is received as the target algorithm evaluation strategy.

[0062] In a fifth aspect, an embodiment of the present application provides an algorithm evaluation device, the device comprising:

[0063] an index data acquisition module configured to acquire index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; wherein the specified test strategy is a test strategy matched with a business scenario of the to-be-tested algorithm, the specified test strategy instructs the to-be-tested algorithm to be tested under at least one test configuration, and the specified evaluation index is at least one of each evaluation index;

[0064] a corresponding relationship determination module configured to determine a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the acquired test data of the specified evaluation index;

[0065] an evaluation strategy determination module configured to determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index;

[0066] a business conclusion generation module configured to generate a business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0067] In a sixth aspect, an embodiment of the present application provides an algorithm evaluation device applied to a client in an algorithm evaluation system, the algorithm evaluation system further comprising a server; the device comprising:

[0068] an interface display module configured to display a task generation interface;

[0069] an algorithm evaluation task generation module configured to receive a configuration operation for the task generation interface and generate an algorithm evaluation task; wherein the algorithm evaluation task indicates a data acquisition strategy of acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy;

[0070] The algorithm evaluation task sending module is configured to send the algorithm evaluation task to the server, so that the server, after receiving the algorithm evaluation task, acquires the index data of the specified evaluation indicator obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; determines the correspondence between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration based on the acquired test data of the specified evaluation dimension; determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation indicator; and generates a business conclusion of the to-be-tested algorithm for the business scenario according to the correspondence between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration based on the business conclusion generation way indicated by the target algorithm evaluation strategy.

[0071] In a seventh aspect, an embodiment of the present application provides an algorithm evaluation device applied to a server in an algorithm evaluation system, the algorithm evaluation system further comprising a client; the device comprising:

[0072] A task receiving module is configured to receive an algorithm evaluation task generated by the client; the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation indicator obtained by a to-be-tested algorithm in a test according to a specified test strategy;

[0073] An index acquiring module is configured to acquire the index data of the specified evaluation indicator obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task;

[0074] A relationship determining module is configured to determine the correspondence between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration based on the acquired test data of the specified evaluation dimension;

[0075] A strategy determining module is configured to determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation indicator;

[0076] A conclusion generating module is configured to generate a business conclusion of the to-be-tested algorithm for the business scenario according to the correspondence between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration based on the business conclusion generation way indicated by the target algorithm evaluation strategy.

[0077] The embodiment of the present application further provides a client, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor, and the processor is prompted by the machine executable instructions to implement the steps of the algorithm evaluation method applied to the client.

[0078] The embodiment of the present application further provides a server, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor, and the processor is prompted by the machine executable instructions to implement the steps of the algorithm evaluation method applied to the server.

[0079] The embodiment of the present application further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0080] The memory is used for storing a computer program.

[0081] The processor is used for executing the program stored on the memory, and the steps of the algorithm evaluation method of the first aspect are implemented.

[0082] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the algorithm evaluation method.

[0083] The embodiment of the present application further provides a computer program product containing instructions, when the computer program product is run on a computer, the computer program product makes the computer execute the algorithm evaluation method.

[0084] The embodiment of the present application has the following beneficial effects:

[0085] The algorithm test method provided by the embodiment of the present application comprises the following steps: obtaining index data of a specified evaluation index of a to-be-tested algorithm, which is obtained by testing the to-be-tested algorithm according to a specified test strategy, the specified test strategy being a test strategy matched with a business scenario of the to-be-tested algorithm, and the to-be-tested algorithm being tested under at least one test configuration; determining a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration; and generating a business conclusion of the to-be-tested algorithm for the business scenario according to a business conclusion generation manner indicated by a target algorithm evaluation strategy, based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration. Since the index data of the specified evaluation index of the to-be-tested algorithm is obtained by testing the to-be-tested algorithm according to the specified test strategy, the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, the corresponding relationship can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the corresponding relationship according to the business conclusion generation manner indicated by the target algorithm evaluation strategy. Therefore, the present application can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0086] Of course, practicing any of the products or methods of the present application need not necessarily result in all of the above advantages. BRIEF DESCRIPTION OF DRAWINGS

[0087] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0088] Figure 1 The flowchart of the algorithm evaluation method provided by the embodiment of the present application;

[0089] Figure 2 The full life cycle schematic diagram of the model in the embodiment of the present application;

[0090] Figure 3 The change function schematic diagram in the algorithm evaluation method provided by the embodiment of the present application;

[0091] Figure 4 Another flowchart of the algorithm evaluation method provided by the embodiment of the present application;

[0092] Figure 5 The schematic diagram of the first sub-interface in the algorithm evaluation method provided by the embodiment of the present application;

[0093] Figure 6A schematic diagram of a page of a task list in the algorithm evaluation method provided by the embodiment of the present application;

[0094] Figure 7 Another schematic diagram of a change function in the algorithm evaluation method provided by the embodiment of the present application

[0095] Figure 8 A schematic diagram of a template selection page in the algorithm evaluation method provided by the embodiment of the present application;

[0096] Figure 9 Another schematic diagram of a template selection page in the algorithm evaluation method provided by the embodiment of the present application;

[0097] Figure 10 Another flowchart of the algorithm evaluation method provided by the embodiment of the present application;

[0098] Figure 11 Another flowchart of the algorithm evaluation method provided by the embodiment of the present application;

[0099] Figure 12 Another flowchart of the algorithm evaluation method provided by the embodiment of the present application;

[0100] Figure 13 A structural schematic diagram of the algorithm evaluation system provided by the embodiment of the present application;

[0101] Figure 14 A structural schematic diagram of the algorithm evaluation device provided by the embodiment of the present application;

[0102] Figure 15 A structural schematic diagram of the client provided by the embodiment of the present application;

[0103] Figure 16 A structural schematic diagram of the server provided by the embodiment of the present application;

[0104] Figure 17 A structural schematic diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0105] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0106] To improve the efficiency of algorithm evaluation, the embodiments of the present application provide an algorithm evaluation method, system, device, client and server. The method can be applied to various electronic devices, such as personal computers, servers, mobile phones and other devices with data processing capabilities. In one implementation, the method can be applied to an algorithm evaluation system, and the method can be implemented by software, hardware or a combination of software and hardware.

[0107] In one implementation, the execution subject of the algorithm evaluation method provided by the embodiments of the present application can be deployed in an algorithm provider and / or an algorithm demander, that is, the algorithm evaluation method provided by the embodiments of the present application can be applied to electronic devices deployed in an algorithm provider and / or an algorithm deployer. The algorithm provider can be a service provider providing services such as construction, development, verification and deployment of algorithms and / or models, and the algorithm demander can be a demander with algorithm and / or model deployment requirements.

[0108] The algorithm evaluation method provided by the embodiments of the present application will be introduced first as follows. The method can include the following steps:

[0109] Obtaining index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; wherein the specified test strategy is a test strategy matched with a business scenario of the to-be-tested algorithm, the specified test strategy indicates that the to-be-tested algorithm is tested under at least one test configuration, and the specified evaluation index is at least one of various evaluation indexes;

[0110] Based on the obtained test data of the specified evaluation index, determining a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration;

[0111] Determining a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation index;

[0112] Generating a business conclusion of the to-be-tested algorithm for a business scenario based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation way indicated by the target algorithm evaluation strategy.

[0113] In this embodiment, by acquiring the indicator data of the specified evaluation indicators obtained from testing the algorithm under test according to a specified testing strategy—a testing strategy that matches the business scenario of the algorithm under test and instructs the algorithm to be tested under at least one test configuration—the correspondence between the indicator data of the specified evaluation indicators of the algorithm under test and the test configuration is determined. Then, according to the business conclusion generation method indicated by the target algorithm evaluation strategy, based on the correspondence between the indicator data of the specified evaluation indicators of the algorithm under test and the test configuration, a business conclusion for the business scenario is generated for the algorithm under test. Since the indicator data of the specified evaluation indicators obtained from testing the algorithm under test according to the specified testing strategy can be obtained, the correspondence between the indicator data of the specified evaluation indicators of the algorithm under test and the test configuration can be determined. This correspondence can reflect how the indicator data of the specified evaluation indicators of the algorithm under test changes with the test configuration. Therefore, the business conclusion for the business scenario can be generated using the business conclusion generation method indicated by the target algorithm evaluation strategy and this correspondence. It is evident that this solution can automatically generate business conclusions for the business scenario using the acquired indicator data, thereby improving the efficiency of algorithm evaluation.

[0114] The algorithm evaluation method provided by the embodiments of the present invention will be described exemplarily below with reference to the accompanying drawings.

[0115] like Figure 1 As shown, the algorithm evaluation method provided in this embodiment of the invention may include the following steps:

[0116] S101, Obtain the indicator data of the specified evaluation indicator obtained by testing the algorithm under test according to the specified test strategy; wherein, the specified test strategy is a test strategy that matches the business scenario of the algorithm under test, and the specified test strategy instructs the algorithm under test to be tested under at least one test configuration; the specified evaluation indicator is at least one of the evaluation indicators.

[0117] According to the difference of the electronic device deployment scenarios to which the embodiments of the present application are applied, the way in which the embodiments of the present application obtain the index data of the specified evaluation indicators obtained by the to-be-tested algorithm according to the specified test strategy can be different. In one implementation, when the algorithm evaluation method provided by the embodiments of the present application is applied to the electronic device deployed at the algorithm provider, the electronic device can obtain the index data of the specified evaluation indicators obtained by the to-be-tested algorithm according to the specified test strategy after receiving the evaluation request sent by the algorithm demander. The evaluation request can be an instruction requesting to perform the algorithm test task or the offline evaluation task. Alternatively, when the algorithm evaluation method provided by the embodiments of the present application is applied to the electronic device deployed at the algorithm demander, the electronic device can obtain the index data of the specified evaluation indicators obtained by the to-be-tested algorithm according to the specified test strategy from the algorithm provider. Optionally, the algorithm provider can also send the to-be-tested algorithm to the algorithm demander while sending the index data. The electronic device deployed at the algorithm demander can perform subsequent algorithm evaluation steps after receiving the index data of the specified evaluation indicators obtained by the to-be-tested algorithm according to the specified test strategy. The algorithm provider can send the index data and / or the to-be-tested algorithm to the algorithm demander in various ways, for example, by copying to the electronic device deployed at the algorithm demander through a storage hard disk, or by sending to the electronic device deployed at the algorithm demander through online transmission, and the embodiments of the present application do not make specific limitations in this regard.

[0118] The to-be-tested algorithm can be an algorithm developed for any business scenario, which can be one or more AI (Artificial Intelligence) models or other types of algorithms, and the types of the one or more algorithms can be the same or different. For example, the to-be-tested algorithm can be a face recognition algorithm for a face recognition scenario, a speech recognition algorithm for a speech recognition scenario, an OCR (Optical Character Recognition) recognition algorithm for a character recognition scenario, or any other type of neural network algorithm. Taking the to-be-tested algorithm as an AI model as an example, the full life cycle process of model construction, development, deployment and operation and maintenance management can be as shown in Figure 2 The model needs to go through five stages of data processing, model construction, model development, model verification and model deployment. In the model verification stage, the algorithm evaluation needs to be performed on the model, so that the business conclusion for the scenario can be obtained according to the algorithm evaluation in the model deployment stage, and the optimal model deployment method can be selected.

[0119] Any test strategy indicates an algorithm to test under at least one test configuration. Different test strategies can be pre-configured according to different business scenarios of the algorithm. For example, when the algorithm to be tested is an image recognition algorithm, the test strategy indicates the algorithm to be tested to obtain input data from a specified image data source during testing, and / or indicates the algorithm to be tested to be deployed in a terminal device containing a camera. When the algorithm to be tested is a speech recognition algorithm, the test strategy indicates the algorithm to be tested to obtain input data from a specified speech data source during testing, and / or indicates the algorithm to be tested to be deployed in a terminal device containing a microphone.

[0120] The test configuration can include a deployed terminal of the algorithm to be tested and / or input data of the algorithm to be tested during testing.

[0121] Since a terminal deploying the algorithm to be tested and data serving as input of the algorithm during testing are required when testing the algorithm, the test configuration can include a deployed terminal of the algorithm to be tested and / or input data of the algorithm to be tested during testing. The terminal can be a server, an edge box, a terminal, a board card, or other electronic devices, and the terminal is provided with a computing chip such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and / or a NPU (Neural-Network Processing Unit) to support running of the algorithm. The input data of the algorithm to be tested can be video data, image data, audio data, or any other type of data requiring algorithm processing.

[0122] Therefore, the test configuration can be a terminal hardware configuration and / or a data type configuration for testing the algorithm to be tested. The terminal hardware configuration can be used to indicate terminal information of the algorithm to be tested during testing. The data type configuration can be used to indicate type information of data input of the algorithm to be tested during testing. That is, when testing the algorithm to be tested, the terminal hardware is determined according to the terminal hardware configuration in the test configuration, and / or the data input to the algorithm to be tested during testing is determined according to the type information of the data indicated by the data type configuration, and then the algorithm to be tested is tested to generate index data of a specified evaluation index of the algorithm to be tested during testing according to the specified test strategy.

[0123] The evaluation index is a dimension required for algorithm evaluation, and can be at least one of a hardware evaluation index and an algorithm evaluation index. The hardware evaluation index can include at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value. The algorithm evaluation index can include at least one of an algorithm throughput, an algorithm accuracy, and a response time. The algorithm accuracy can be at least one of an algorithm accuracy rate and an algorithm recall rate. The specified evaluation index can be at least one of the evaluation indexes. For example, the specified evaluation index is the hardware resource utilization rate and the algorithm accuracy.

[0124] According to different requirements, the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy can be different. In an implementation, the to-be-tested algorithm can be tested according to the specified test strategy, so as to obtain the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy. Alternatively, the to-be-tested algorithm can be an algorithm that has been tested in advance. In this case, the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy can be obtained by obtaining offline data.

[0125] In this case, in an implementation, an algorithm test task or an offline evaluation task can be obtained. The algorithm test task includes an algorithm identifier, a strategy identifier, and a specified evaluation index. The offline evaluation task includes storage address information. The storage address information indicates an address for obtaining offline index data. The offline index data is index data of the specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy in advance.

[0126] The algorithm identifier included in the algorithm test task is an identifier of the to-be-tested algorithm. The strategy identifier is an identifier of the specified test strategy. When the obtained task is the algorithm test task, the to-be-tested algorithm indicated by the algorithm identifier can be tested according to the specified test strategy indicated by the strategy identifier, to obtain index data of the to-be-tested algorithm under the specified evaluation index. When the obtained task is the offline evaluation task, offline index data can be read from the address indicated by the storage address information.

[0127] As can be seen, in the present solution, the to-be-tested algorithm indicated by the algorithm identifier can be tested according to the specified test strategy indicated by the strategy identifier, to obtain index data of the to-be-tested algorithm under the specified evaluation index, by obtaining an algorithm test task. When the index data of the to-be-tested algorithm has been generated in advance, the offline index data can be read from the address indicated by the storage address information, by obtaining an offline evaluation task. The flexibility of the algorithm evaluation method provided in the present embodiment is improved.

[0128] S102, determine a corresponding relationship between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration based on the obtained test data of the specified evaluation indicator;

[0129] Since the specified test strategy indicates that the to-be-tested algorithm is tested under at least one test configuration, after testing the to-be-tested algorithm according to the specified test strategy, the index data corresponding to the test configuration can be obtained for each test configuration. Therefore, when the test data of the specified evaluation indicator is obtained, the test configuration corresponding to the test data can be determined, and a corresponding relationship between the index data and the test configuration is established, so that the specified evaluation indicator of the to-be-tested algorithm under different test configurations is obtained, which provides a basis for generating a business conclusion.

[0130] The corresponding relationship between the index data and the test configuration can be recorded in the form of a data table, and can also be recorded in the form of a statistical chart, such as a bar chart, a line chart, a pie chart, etc.

[0131] S103, determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation indicator;

[0132] In an implementation manner, different algorithm evaluation strategies can be set for algorithms in different business scenarios, and the set algorithm evaluation strategies are associated with the test strategies matched with the business scenarios of the algorithms. The association relationship can also be recorded in the form of a data table. When algorithm evaluation needs to be performed on the to-be-tested algorithm, the algorithm evaluation strategy associated with the specified test strategy can be determined as the target algorithm evaluation strategy.

[0133] In another implementation manner, at least one algorithm evaluation strategy can also be displayed; for example, an algorithm evaluation strategy selection interface can be displayed, and the interface can include each to-be-selected algorithm evaluation strategy, wherein any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation indicator. Then, a selection operation for at least one algorithm evaluation strategy is received, and the algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy is determined as the algorithm evaluation strategy associated with the specified test strategy. In this implementation manner, the algorithm evaluation strategy associated with the specified test strategy is determined by performing a selection operation on at least one algorithm evaluation strategy, which can improve user experience. Further, the displayed algorithm evaluation strategy can also be a recommended algorithm evaluation strategy, which can be a preset algorithm evaluation strategy for the business scenario, or a popular algorithm evaluation strategy for the business scenario, i.e., an algorithm evaluation strategy with a usage time and / or a usage frequency reaching a preset threshold.

[0134] S104, generate the business conclusion of the to-be-tested algorithm for the business scenario according to the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, and the business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0135] In this step, after determining the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, and the target algorithm evaluation strategy, the business conclusion of the to-be-tested algorithm for the business scenario is generated according to the corresponding relationship and the business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0136] For example, when the test configuration is multiple and each test configuration is a terminal hardware configuration, the manner of generating the business conclusion of the to-be-tested algorithm for the business scenario can be: determining the test configuration corresponding to the test data satisfying the preset index condition in each test data, and then generating the business conclusion of the to-be-tested algorithm for the business scenario according to the test data satisfying the preset index condition and the test configuration corresponding thereto. In one implementation, when the purpose of algorithm evaluation is to determine the test configuration corresponding to the test data satisfying the preset index condition, the business conclusion can be the test configuration corresponding to the test data satisfying the preset index condition.

[0137] When the test configuration is single and the test configuration is a terminal hardware configuration, the purpose of algorithm evaluation is to determine the specified evaluation index of the to-be-tested algorithm when tested according to the terminal hardware configuration, and the manner of generating the business conclusion of the to-be-tested algorithm for the business scenario can be: taking the obtained index data of the specified evaluation index as the business conclusion for the business scenario.

[0138] Further, in some scenarios, multiple different algorithms need to be configured for the same terminal, for example, a face recognition algorithm and a voice recognition algorithm are configured for the same camera at the same time, then the tests of multiple algorithms can be performed in the terminal at the same time, and the business conclusion of each algorithm of the terminal is determined according to the above method, and the business conclusion of the to-be-tested algorithm for the business scenario is generated based on the business conclusion of each algorithm of the terminal, for example, the business conclusion of the to-be-tested algorithm for the business scenario can be obtained by summarizing and counting the business conclusion of each algorithm of the terminal. In addition, there are scenarios in which multiple different algorithms are configured in multiple terminals respectively, then the tests of multiple algorithms in each terminal can be performed at the same time, and the business conclusion of each algorithm in each terminal is determined according to the above method, and the business conclusion of the to-be-tested algorithm for the business scenario is generated based on the business conclusion of each algorithm in each terminal.

[0139] In addition, accumulated historical data can be obtained through the way of burying points in each stage of algorithm evaluation, which can be used for user behavior analysis, pre-evaluation of algorithm upgrade strategy, hardware resource budget before product planning, cost investment pre-evaluation, adjustment of algorithm processing strategy selection evaluation, algorithm data migration, traffic migration evaluation, algorithm model lightweight, acceleration processing effect evaluation and other product behaviors to generate greater data value.

[0140] In the embodiment, after obtaining the index data of the specified evaluation index of the to-be-tested algorithm according to the specified test strategy, the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, the corresponding relationship can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scene can be generated by using the business conclusion generation mode indicated by the target algorithm evaluation strategy and using the corresponding relationship. It can be seen that the present scheme can automatically generate the business conclusion for the business scene by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0141] Optionally, in another embodiment of the present application, the test configuration can include a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of terminal hardware configuration and data type configuration; the terminal hardware configuration indicates the terminal information deployed when the to-be-tested algorithm is tested, and the data type configuration indicates the type information of the data input when the to-be-tested algorithm is tested.

[0142] In an implementation manner, the terminal hardware configuration includes at least one of terminal configuration and hardware configuration; wherein the terminal configuration indicates the terminal deployment information when the to-be-tested algorithm is tested, for example, the terminal configuration can include at least one of terminal quantity, terminal type, terminal version, etc. The hardware configuration indicates the hardware information contained in the terminal when the to-be-tested algorithm is tested; for example, the hardware configuration can include at least one of GPU quantity, memory size, processor computing power, etc.

[0143] The data type configuration can include at least one of a data source configuration, a task quantity configuration, and a data format configuration. The data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested. For example, the data source configuration can include at least one of a data source belonging scene, a data source type, and a data quantity in the data source. The task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested. When the number of tasks processed synchronously when the to-be-tested algorithm is tested is a plurality, input of each task can be a plurality of channels drawn from the same data source, or can be different data sources. The data format configuration indicates a data format of input data when the to-be-tested algorithm is tested. For example, when the data source type is an image, the data format configuration can include at least one of a resolution, an image type, and the like. When the data source type is a video, the data format configuration can include at least one of a resolution, a frame rate, a video type, and the like.

[0144] Based on the obtained test data of the specified evaluation dimension, a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration is determined, including:

[0145] Based on the obtained test data of the specified evaluation dimension, a change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0146] In this embodiment, the first dimension configuration can be fixed, and different second dimension configurations can be set, so as to obtain the change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration. The first dimension configuration and the second dimension configuration can be set according to business requirements.

[0147] For example, it is necessary to determine the index data of the specified evaluation index of the to-be-tested algorithm for different task quantities. The first dimension configuration can be a terminal hardware configuration, and the data source configuration and the data format configuration in the data type configuration. The second dimension configuration can be the task quantity configuration, so that the change function of the index data of the specified evaluation index with respect to the change of the task quantity can be determined.

[0148] In the embodiment, the first dimension configuration, the second dimension configuration and the specified evaluation index can also be set according to the algorithm service characteristics. For example, if the algorithm to be tested is a face recognition algorithm, the second dimension configuration can be the data quantity, the first dimension configuration can be other data type configurations and terminal hardware configurations, and the specified evaluation index can be the algorithm accuracy, recognition speed and model computing power. Then, the change function of the index data of the specified evaluation index of the algorithm to be tested with respect to the data quantity can be determined. In actual application, different data quantity configurations can be realized by selecting face databases of different image data orders, such as face databases of 1000 orders, 10,000 orders, 100,000 orders, 300,000 orders and 1,000,000 orders. If the algorithm to be tested is a speech recognition algorithm, the second dimension configuration can be the data source type, such as at least two of the types of speech in a financial scenario, the types of speech in an artistic scenario and the types of speech in an IT (Internet Technology) scenario, the first dimension configuration can be other data type configurations and terminal hardware configurations, and the specified evaluation index can be the algorithm accuracy. Then, the change function of the accuracy of the algorithm to be tested with respect to the data source type can be determined. If the algorithm to be tested is an OCR recognition algorithm, the second dimension configuration can also be the data source type, such as at least two of the types of electronic business cards, the types of ID cards and the types of business licenses, the first dimension configuration can be other data type configurations and terminal hardware configurations, and the specified evaluation index can be the algorithm accuracy.

[0149] It should be noted that the first dimension configuration can also be adjusted to determine the change function of the index data of the specified evaluation index of the algorithm to be tested with respect to the second dimension configuration under a plurality of first dimension configurations. For example, in the case where the first dimension configuration is the resolution in the data format configuration and the second dimension configuration is the frame rate, the change function of the index data of the specified evaluation index of the algorithm to be tested with respect to the frame rate under different task quantities can be determined by changing the task quantity configuration. In the case where the first dimension configuration is the frame rate in the data format configuration and the second dimension configuration is the resolution, the change function of the index data of the specified evaluation index of the algorithm to be tested with respect to the resolution under different task quantities can be determined by changing the task quantity configuration.

[0150] In an implementation manner, the algorithm to be tested can include multiple versions of the algorithm, the second dimension configuration can also be the version of the algorithm, the specified index value can be the algorithm accuracy, and thus a change function of the algorithm accuracy of the algorithm to be tested with respect to the version is obtained. Further, the change function of the algorithm accuracy of the algorithm to be tested with respect to the version under different data source types can also be obtained by changing the data source type in the first dimension configuration. The change function in the implementation manner can be plotted as shown in Figure 3 Figure 3 FIG. 2, in which the algorithm version is taken as the horizontal coordinate and the algorithm accuracy is taken as the vertical coordinate, and the change functions of the algorithm accuracy with respect to the version under three data source types are plotted, and each data source type is represented by a broken line graph.

[0151] In the embodiment, the efficiency of algorithm evaluation can be improved. Further, the test configuration includes the first dimension configuration and the second dimension configuration, and based on the obtained test data of the specified evaluation dimension, the change function of the index data of the specified evaluation index of the algorithm to be tested with respect to the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the algorithm to be tested and the test configuration, which can reflect the change relationship of the index data of the specified evaluation index of the algorithm to be tested with respect to the configuration of different dimensions, enrich the basis for generating the business conclusion of the algorithm to be tested for the business scenario, and thus the algorithm to be tested can be more comprehensively evaluated.

[0152] Optionally, in another embodiment of the present application, as shown in Figure 4 the above obtaining the algorithm test task or the offline evaluation task can include:

[0153] S401, a task generation interface is displayed; wherein the task generation interface includes a task type selection box;

[0154] The above task generation interface can be a man-machine interaction interface, and the task generation interface includes a task type selection box, wherein the task type includes an algorithm test task or an offline evaluation task.

[0155] S402, a task selection operation for the task type selection box is received;

[0156] In this step, the task selection operation can be performed by using the above task type to select the algorithm test task or the offline evaluation task, and the task selection operation can be performed by a test personnel or a developer. The task selection operation can be in the form of clicking or inputting text, and can also be not limited to this.

[0157] ​After receiving the task selection operation on the task type selection box, if the task type selected by the task selection operation is the test type, step S403 is performed, and if the task type selected by the task selection operation is the offline type, step S404 is performed.

[0158] S403, a first sub-interface is displayed; the first sub-interface includes a test algorithm selection box, a test strategy selection box, and an evaluation index selection box; selection operations on the test algorithm selection box, the test strategy selection box, and the evaluation index selection box are received respectively to determine selected algorithm identification, strategy identification, and specified evaluation indexes; and an algorithm test task is generated based on the selected algorithm identification, the strategy identification, and the specified evaluation indexes.

[0159] In an implementation manner, the first sub-interface can be as shown in Figure 5 The test algorithm selection box is "select model" in Figure 5 The test algorithm selection box can select algorithm identification, for example, a human head detection model; the test strategy selection box can select strategy identification, for example, the test strategy selection box can be "select dataset" in Figure 5 The test strategy selection box includes a front face dataset, a large angle dataset, an indoor dataset, and an outdoor dataset; and the evaluation index selection box can select specified evaluation indexes, for example, "select evaluation index item" in Figure 5 The evaluation index selection box includes accuracy, average processing time, and GPU utilization. In addition, the first sub-interface can further include a task name box for inputting a task name, for example, a human head detection V1.1 model regression test; a task type selection box for selecting a task type; a task execution frequency selection box including options such as immediate execution, periodic execution, trigger execution, and no need to execute; and a result notification box for inputting a contact address such as an email address for receiving a feedback business conclusion.

[0160] S404, a second sub-interface is displayed; the second sub-interface includes a position selection box; a selection operation on the position selection box is received to determine selected storage address information; and an offline evaluation task is generated based on the selected storage address information.

[0161] If the task type selected by the task selection operation is the offline type, it indicates that the to-be-tested algorithm to be evaluated this time has been pre-tested to generate index data of specified evaluation indexes. In this case, the second sub-interface includes a position selection box for selecting storage address information of the index data of the specified evaluation indexes. Then, the index data of the specified evaluation indexes can be read based on the selected storage address information to generate an offline evaluation task to perform subsequent evaluation operations.

[0162] After the algorithm test task or the offline evaluation task is generated, as shown in Figure 6The page displaying the task list can display the task name, task type, creation mode, task state, creation time and operation of each task in different columns of the task list. In the operation column, different operations can be selected, including starting, for executing the evaluation task; details, for viewing the evaluation result of the test task; and deleting, for deleting the test task.

[0163] Further, in an implementation, after generating the business conclusion of the to-be-tested algorithm for the business scenario, an evaluation display interface can also be displayed; wherein the evaluation display interface includes the target legend corresponding to the specified evaluation index and / or the generated business conclusion; wherein the target legend is a legend generated based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0164] The target legend can be any type of graph, which can be used to display the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration. For example, the change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the second dimension configuration under the first dimension configuration of the to-be-tested algorithm can be displayed, realizing the visualization of the index. For example, Figure 7 as shown, Figure 7 The change function of each evaluation index of the face recognition algorithm with respect to the processing number is displayed in the form of a graph; the change of the accuracy of different face database algorithms with respect to the order of magnitude of different face databases when the face recognition algorithm has 10 input channels; and the change function of the average processing time of the face recognition algorithm and the pedestrian detection algorithm with respect to the terminal type. In the evaluation display interface, there can also be an editing function of the business conclusion to enable the user to edit and input the evaluation conclusion as the business conclusion for display and saving.

[0165] In addition, the displayed business conclusion can be displayed in the form of text or in the form of a table. By displaying the generated business conclusion, the visualization of the business conclusion can also be realized.

[0166] Before displaying the change function, a template selection page can also be displayed, as shown in Figure 8 In the template selection page, the task name, such as algorithm index evaluation of terminal X, the task type, such as terminal algorithm index evaluation, and whether to select the built-in template option, for determining whether to select a built-in template from a built-in template list, can be displayed. The built-in template includes a preset display graph, such as a line chart, a column chart, a curve chart, etc. If no, a custom image template can also be uploaded. The display effect can also be previewed below, such as Figure 8 and Figure 9The three preview images shown below are only for illustration. In addition, the test configuration of the X-axis of the function image to be displayed can also be customized, and the specified indicators corresponding to the Y-axis can also be customized, for example, the X-axis is the number of images in the face library, and the Y-axis is the algorithm accuracy. For the task type of offline type, as shown in Figure 9 whether to select the built-in template option, and then select the specified evaluation indicators in the data source option, for example, which can include: accuracy, average processing time, GPU utilization, etc., and select the line type option to display the above change function in the line chart or pie chart.

[0167] In an implementation manner, after the algorithm is developed and trained on the development platform, the algorithm can also be evaluated by issuing an evaluation task. The evaluation task issued can include necessary parameters required to execute the task, for example, which can include: task name, task type, algorithm name, data source identifier, specified evaluation indicators, identifier of display template, task execution frequency, result notification form, etc.

[0168] In the embodiment, the efficiency of algorithm evaluation can be improved. Further, by displaying a task generation interface; the task generation interface includes a task type selection box; receiving a task selection operation on the task type selection box; in the case that the task type selected by the task selection operation is a test type, a first sub-interface is displayed; in the case that the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; the index data of the specified evaluation indicators can be obtained more intuitively and conveniently through the display interface, and the user experience is improved.

[0169] Based on Figure 4In the illustrated embodiment, the task generation page, the first sub-page and / or the second sub-page displayed by the electronic device according to the embodiments of the present application can also have differences in display modes according to the differences in the deployment scenarios of the electronic device to which the embodiments of the present application are applied. In one implementation, when the algorithm evaluation method provided by the embodiments of the present application is applied to the electronic device deployed at the algorithm provider, the task generation page, the first sub-page and / or the second sub-page can be displayed in a front-end interface, for example, a web interface or an APP (application) interface, and the like. When the algorithm provider or the algorithm demander needs to test the to-be-tested algorithm, the algorithm test task or the offline evaluation task can be generated through the task generation interface displayed in the front-end interface, and the task configuration can be performed through the first sub-page and / or the second sub-page. After the electronic device to which the embodiments of the present application are applied receives the task interface configured, the corresponding index data acquisition operation can be performed on the received task. Further, the evaluation display interface can also be displayed in the front-end interface by the electronic device of the algorithm provider after generating the business conclusion. In another implementation, when the algorithm evaluation method provided by the embodiments of the present application is applied to the electronic device deployed at the algorithm demander, the task generation page, the first sub-page and / or the second sub-page can also be the page displayed by the electronic device deployed at the algorithm demander, so that the algorithm demander can directly configure the task locally and perform the algorithm evaluation process locally. In this case, the evaluation display interface can also be the page displayed by the electronic device deployed at the algorithm demander after generating the business conclusion.

[0170] Optionally, in another embodiment of the present application, the target algorithm evaluation strategy can include an algorithm evaluation template; the algorithm evaluation template includes a condition field and a blank field to be filled with content; the condition field includes a filtering condition for filtering the test configuration and / or the evaluation index;

[0171] In this case, as Figure 10 indicated, the business conclusion generation mode indicated according to the target algorithm evaluation strategy can generate the business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, and can include the following steps:

[0172] S1001, filtering the test configuration when the to-be-tested algorithm is tested and / or the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the filtering condition included in the condition field of the algorithm evaluation template to obtain filtering data;

[0173] The preset text field and the blank field can be included in the above algorithm evaluation template. For example, when the input number of routes is X on the X terminal, the algorithm accuracy is X%, the response time is X seconds, and the system resource utilization is X%. The blank field is represented by X.

[0174] The filtering condition included in the condition field can be a filtering condition for the blank field, that is, the filtering condition is used to filter the text and / or numbers from the index data of the specified evaluation index obtained by the test configuration and / or the test of the algorithm to be tested when the algorithm to be tested is tested, and is used to fill in the blank field.

[0175] In an implementation mode, when the algorithm to be tested is applied to multiple terminals, the index data of the specified evaluation index can include index data obtained when the algorithm to be tested is applied to each terminal, and the filtering condition can be to filter the largest index in each index data as the filtered data. When the algorithm to be tested includes multiple algorithms, the index data of the specified evaluation index can include, for each algorithm, the index data of the specified evaluation index obtained by the test configuration and / or the test of the algorithm to be tested when the algorithm is tested, and the filtering condition can also be to filter the largest index in the specified evaluation index of each algorithm as the filtered data.

[0176] In another implementation mode, the filtering condition can be to filter a specific test configuration, for example, to filter at least one of the terminal type, the terminal version, the number of graphic processing units, the number of task configurations, and / or at least one of the specific evaluation index in the specified evaluation index, for example, the algorithm accuracy, the response data, and the GPU utilization.

[0177] S1002, from the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration, determine the index data and / or the test configuration corresponding to the filtered data;

[0178] In this step, in the case that the screening data includes at least one specific test configuration and at least one specific evaluation index, the specific evaluation index corresponding to each specific test configuration or the specific test configuration corresponding to each specific evaluation index can be determined according to the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration. For example, when the algorithm evaluation template includes the text field: “When the input number is X2 on the X1 terminal, the algorithm accuracy is X3%, the response time is X4 seconds, and the hardware resource utilization is X5%”, the terminal type and / or terminal version in the test configuration need to be screened out for the blank X1, and the algorithm accuracy, response time, and hardware resource utilization in the specified evaluation index corresponding to the test configuration in the blank X1 need to be determined for the blanks X2, X3, X4, and X5. The screening data can include the terminal type and / or terminal version in each test configuration, and the algorithm accuracy, response time, and hardware resource utilization in each specified evaluation index, so as to determine the specified evaluation index corresponding to each test configuration according to the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration.

[0179] When the algorithm evaluation template includes the text field: “In the XX scenario, the algorithm has the highest accuracy”, the screening data can include the specified evaluation index with the maximum algorithm accuracy in the specified evaluation index corresponding to each test configuration, and the data source type in the multiple test configurations, so as to determine the data source type in the test configuration corresponding to the specified evaluation index according to the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration.

[0180] When the algorithm evaluation template includes the text field: “When the input number is 10, the accuracy is more than 98%, and the maximum supported face library size is XX”, the face library size can be screened out from each test configuration, and the specified evaluation index with the algorithm accuracy greater than 98% can be screened out from each specified evaluation index as the screening data, so as to determine the face library size in the test configuration corresponding to the specified evaluation index in the screening data according to the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration. If the determined face library size is multiple, the maximum size in the face library size can be further screened out.

[0181] S1003, filling the determined index data and / or test configuration into the blank field in the algorithm evaluation template to obtain the business conclusion of the algorithm to be tested for the business scenario.

[0182] The algorithm evaluation template can include preset text fields and blank fields for display. After determining the index data and / or test configuration, the determined index data and / or test configuration can be filled into the blank fields in the algorithm evaluation template. Since there are preset text fields in the algorithm evaluation template, when the blank fields are filled with corresponding content, the business conclusion of the to-be-tested algorithm for the business scenario can be obtained.

[0183] In the embodiment, the efficiency of algorithm evaluation can be improved. Further, the target algorithm evaluation strategy includes an algorithm evaluation template. The algorithm evaluation template includes condition fields and blank fields to be filled with content. The condition fields include filtering conditions for filtering test configurations and / or evaluation indexes. The filtering data is obtained by filtering the test configurations when testing the to-be-tested algorithm and / or the index data of the specified evaluation indexes of the to-be-tested algorithm according to the filtering conditions included in the condition fields of the algorithm evaluation template. The index data and / or test configuration corresponding to the filtering data are determined from the correspondence between the index data of the specified evaluation indexes of the to-be-tested algorithm and the test configurations. The determined index data and / or test configuration are filled into the blank fields in the algorithm evaluation template to obtain the business conclusion of the to-be-tested algorithm for the business scenario, thereby providing an implementation basis for generating the business conclusion of the to-be-tested algorithm for the business scenario.

[0184] For the convenience of understanding, the algorithm evaluation method provided by the embodiment of the application is exemplarily introduced below with reference to the accompanying drawings.

[0185] As shown in Figure 11 The algorithm evaluation method provided by the embodiment of the application can be applied to an algorithm test platform. Taking an algorithm as an example, the model needs to be implemented in a development platform to construct, develop and train the model, and then a test task is issued to the algorithm test platform. The test task can be issued in the following manner: configuring necessary parameters required for executing the task and sending the necessary parameters to the algorithm test platform, or the test task can be configured and generated in a task generation interface, a first sub-interface or a second sub-interface. After receiving the issued test task, the algorithm test platform selects a verification set, that is, according to a specified test strategy, the to-be-tested algorithm is tested under at least one test configuration to complete model verification, generate index data and business conclusion, and visualize the index data and feed back to the development platform, for example, in the form of an HTML (Hyper Text Markup Language, Hyper Text Markup Language) link. The algorithm test platform can also be integrated into the development platform, so that the developer or tester can directly complete the algorithm test in the development platform.

[0186] In a multi-terminal and multi-algorithm scenario, as Figure 12As shown, algorithm N1 and algorithm N2 run in terminal M1, and algorithm N1 and algorithm N2 run in terminal M2, a plurality of data sources, i.e. test data source 1, test data source 2 and test data source 3, can be input into algorithm N1 and algorithm N2 running in terminal M1 and algorithm N1 and algorithm N2 running in terminal M2 for algorithm testing, and algorithm result data for each algorithm in each terminal can be obtained, i.e. index data of the specified evaluation index, and then for index X1 in the specified evaluation index, index dimension X1 curve drawing can be performed, for index X2, index dimension X2 pie chart drawing can be performed, and for index X3, index dimension X3 bar chart drawing can be performed. Further, according to the target algorithm evaluation strategy, the index relationship can be found, i.e. the data is determined and filtered, so as to form various business conclusions, and then the business conclusions are pushed according to the accurate demand, for example, in the form of picture-text sharing, one-key customized forwarding or other visual presentation.

[0187] In an implementation manner, for different applicable scenarios, the evaluation scheme can be determined according to the corresponding relationship shown in Table 1 as follows, including: test configuration (including dimension and input source), visualization form, i.e. content to be displayed by the evaluation display interface, algorithm evaluation template.

[0188] Table 1

[0189]

[0190] For example, if the algorithm to be tested is an OCR recognition algorithm, the input is 100 A company electronic business card images, and each image has 5 fields to be recognized. The test task is to select the A company electronic business card as the data source, and the field labels X1, X2, X3, X4 and X5 are recognized. The test result is obtained by executing the test task. The test result can include: number of incorrect recognition, number of missed recognition, number of original characters, number of recognized characters, detection time, recognition time, number of picture frames, etc. The generated business conclusion can be: the OCR recognition algorithm has an accuracy of 98% for the A company electronic business card, and the average recognition time is 0.3s. In addition, the OCR recognition algorithm version iteration test can be performed on the basis of the above. Thus, the comparison results of V1, V2,..., Vn versions are obtained.

[0191] If the algorithm to be tested is pedestrian detection, the index curves and conclusion requirements for the same input source (resolution, frame rate), GPU 1-M and road number 1-N need to be obtained. Then the specified video source, the specified input road number, the algorithm and the deployment environment can be selected as the test input. The test task is executed:

[0192] 1) When the GPU is 1, the input is 1, 2, 3,..., N, the algorithm accuracy, response time and GPU utilization rate are determined;

[0193] 2) GPU is 2, input 1, 2, 3,..., N, determine the accuracy of the algorithm, response time, GPU utilization rate;

[0194] 3) GPU is N, input 1, 2, 3,..., N, determine the accuracy of the algorithm, response time, GPU utilization rate.

[0195] The generated business conclusion can be: when the GPU is configured as N, the input number is Y, the algorithm accuracy is xx%, the response time is xxs, and the GPU utilization rate is xx%.

[0196] If the algorithm to be tested is a face recognition algorithm, in the same deployment environment, the evaluation indexes of the face library with magnitudes of 2w, 10w, 30w, and 100w are given,

[0197] The test task can select the base library number, algorithm, terminal deployment information, base library number, and test dataset as input. Optionally, a specified evaluation index can also be selected. Execute the test task:

[0198] 1) Under 2w base library, determine the accuracy, missed recognition rate, false recognition rate, and average recognition time of face recognition;

[0199] 2) Under 10w base library, determine the accuracy, missed recognition rate, false recognition rate, and average recognition time of face recognition;

[0200] 3) Under 30w base library, determine the accuracy, missed recognition rate, false recognition rate, and average recognition time of face recognition;

[0201] 4) Under 100w base library, determine the accuracy, missed recognition rate, false recognition rate, and average recognition time of face recognition.

[0202] The generated business conclusion can be: the face recognition algorithm, when the base library is XX, the algorithm accuracy is XX, and the average recognition time is XX; or the face recognition algorithm, the maximum supported face library magnitude with an accuracy of XX is XX.

[0203] It can be seen that the embodiment forms a standard and unified evaluation process for different terminal and algorithm deployment forms, and provides visual index data display and accurate business conclusion output, which can be used for algorithm upgrade, algorithm model optimization, algorithm resource evaluation, algorithm strategy selection, and other decision-making and evaluation processes, and improves the production and research efficiency. Moreover, the embodiment can also perform burying points at each stage of algorithm evaluation, obtain historical data such as creation of a test task, execution of a test task, test data, business conclusion customization, and data sharing selection.

[0204] In this embodiment, after the index data of the specified evaluation index of the to-be-tested algorithm obtained by testing according to the specified test strategy is obtained, the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the business conclusion generation manner indicated by the target algorithm evaluation strategy and using the correspondence. It can be seen that the present scheme can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0205] The embodiment of the present application also provides an algorithm evaluation method applied to a client in an algorithm evaluation system, wherein the algorithm evaluation system further comprises a server.

[0206] The client can be any terminal device, for example, a computer, a mobile phone, etc., and can also be a software program in the terminal device, which needs to have the function of interface display. The server can be a computer system in a network that can provide certain services to other machines, and of course, can also be other electronic devices with data processing capability.

[0207] The method comprises:

[0208] displaying a task generation interface;

[0209] receiving a configuration operation for the task generation interface to generate an algorithm evaluation task; wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index of a to-be-tested algorithm obtained by testing according to a specified test strategy;

[0210] sending the algorithm evaluation task to the server, so that the server acquires the index data of the specified evaluation index of the to-be-tested algorithm obtained by testing according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task after receiving the algorithm evaluation task; determines the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the acquired test data of the specified evaluation dimension; determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a business conclusion generation manner based on a test configuration and / or an evaluation index; generates a business conclusion for the to-be-tested algorithm for the business scenario based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to the business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0211] Optionally, the task generation interface comprises a task type selection box; the configuration operation for the task generation interface comprises a task selection operation for the task type selection box; the client receives the configuration operation for the task generation interface, generates an algorithm evaluation task, comprising:

[0212] in the case where the task type selected by the task selection operation is a test type, a first sub-interface is displayed; wherein the first sub-interface comprises a test algorithm selection box, a test strategy selection box and an evaluation index selection box; selection operations for the test algorithm selection box, the test strategy selection box and the evaluation index selection box are respectively received, and selected algorithm identifiers, strategy identifiers and specified evaluation indexes are determined; based on the selected algorithm identifiers, strategy identifiers and specified evaluation indexes, an algorithm test task is generated as an algorithm evaluation task;

[0213] in the case where the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; wherein the second sub-interface comprises a location selection box; a selection operation for the location selection box is received, and selected storage address information is determined; based on the selected storage address information, an offline evaluation task is generated as an algorithm evaluation task; the selected storage address information indicates an acquisition address of offline index data, and the offline index data is index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy.

[0214] Optionally, the server acquires, based on a data acquisition strategy indicated by the algorithm evaluation task, index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy, comprising:

[0215] in the case where the acquired is the algorithm test task, testing the to-be-tested algorithm indicated by the algorithm identifier according to the specified test strategy indicated by the strategy identifier, to obtain index data of the to-be-tested algorithm under the specified evaluation index;

[0216] in the case where the acquired is the offline evaluation task, reading the offline index data from the acquisition address indicated by the selected storage address information.

[0217] Optionally, after the task generation interface is displayed, the method further comprises:

[0218] at least one algorithm evaluation strategy is displayed; wherein any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; a selection operation for the at least one algorithm evaluation strategy is received, and the algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy is sent to the server;

[0219] The server determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, including:

[0220] The algorithm evaluation strategy sent by the client is received as the target algorithm evaluation strategy.

[0221] Optionally, the test configuration includes a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of terminal hardware configuration and data type configuration; the terminal hardware configuration indicates terminal information deployed when the to-be-tested algorithm is tested, and the data type configuration indicates type information of data input when the to-be-tested algorithm is tested;

[0222] The server determines the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the obtained test data of the specified evaluation dimension, including:

[0223] Based on the obtained test data of the specified evaluation dimension, a change function of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0224] Optionally, the method further includes receiving the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration sent by the server, and / or the business conclusion generated by the server;

[0225] A target legend is generated based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, and an evaluation display interface is displayed; wherein the evaluation display interface includes the target legend corresponding to the specified evaluation index, and / or the received business conclusion; the target legend is a legend generated based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0226] Optionally, the terminal hardware configuration includes at least one of terminal configuration and hardware configuration; wherein the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained in the terminal when the to-be-tested algorithm is tested;

[0227] The data type configuration comprises at least one of a data source configuration, a task quantity configuration, and a data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested; the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of the input data when the to-be-tested algorithm is tested.

[0228] Optionally, the terminal configuration comprises at least one of a terminal quantity, a terminal type, and a terminal version; the hardware configuration comprises at least one of a graphics card quantity, a memory size, and a processor computing power; the data source configuration comprises at least one of a data source belonging scene, a data source type, and a data quantity in the data source; and the data format configuration comprises at least one of a resolution, a frame rate, and a type.

[0229] Optionally, the target algorithm evaluation strategy comprises an algorithm evaluation template; the algorithm evaluation template comprises a condition field and a blank field to be filled with content; the condition field comprises a screening condition for screening a test configuration and / or an evaluation index; and the server generates a business conclusion of the to-be-tested algorithm for the business scene according to a business conclusion generation mode indicated by the target algorithm evaluation strategy, based on a corresponding relationship between index data of a specified evaluation index of the to-be-tested algorithm and a test configuration, comprising:

[0230] screening, according to a screening condition comprised in the condition field in the algorithm evaluation template, the test configuration when the to-be-tested algorithm is tested and / or the index data of the specified evaluation index obtained by the to-be-tested algorithm, to obtain screening data;

[0231] determining, from the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, index data and / or a test configuration corresponding to the screening data;

[0232] filling the determined index data and / or test configuration into the blank field in the algorithm evaluation template to obtain the business conclusion of the to-be-tested algorithm for the business scene.

[0233] Optionally, the evaluation indexes comprise at least one of a hardware evaluation index and an algorithm evaluation index.

[0234] The hardware evaluation index comprises at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value.

[0235] The algorithm evaluation index comprises at least one of an algorithm throughput, an algorithm accuracy, and a response time.

[0236] In this embodiment, after the index data of the specified evaluation index of the to-be-tested algorithm obtained by testing according to the specified test strategy is obtained, the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the business conclusion generation manner indicated by the target algorithm evaluation strategy and using the correspondence. It can be seen that the present scheme can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0237] The embodiment of the present application also provides an algorithm evaluation method applied to a server in an algorithm evaluation system, wherein the algorithm evaluation system also comprises a client; and the method comprises the following steps of:

[0238] receiving an algorithm evaluation task generated by the client; wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index of a to-be-tested algorithm obtained by testing according to a specified test strategy;

[0239] acquiring the index data of the specified evaluation index of the to-be-tested algorithm obtained by testing according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task;

[0240] determining a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the acquired test data of the specified evaluation dimension;

[0241] determining a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index;

[0242] generating a business conclusion for the to-be-tested algorithm for the business scenario based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0243] Optionally, the task generation interface comprises a task type selection box; and the configuration operation for the task generation interface comprises a task selection operation for the task type selection box.

[0244] The client receives the configuration operation for the task generation interface, and generates an algorithm evaluation task, which comprises the following steps of:

[0245] In a case where the task type selected by the task selection operation is a test type, a first sub-interface is displayed; the first sub-interface includes a test algorithm selection box, a test strategy selection box, and an evaluation index selection box; selection operations on the test algorithm selection box, the test strategy selection box, and the evaluation index selection box are received respectively, and selected algorithm identification, strategy identification, and specified evaluation indexes are determined; an algorithm test task is generated based on the selected algorithm identification, strategy identification, and specified evaluation indexes, as an algorithm evaluation task;

[0246] In a case where the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; the second sub-interface includes a location selection box; a selection operation on the location selection box is received, and selected storage address information is determined; an offline evaluation task is generated based on the selected storage address information, as an algorithm evaluation task; the selected storage address information indicates an acquisition address of offline index data, and the offline index data is index data of specified evaluation indexes obtained by testing a to-be-tested algorithm according to a specified test strategy.

[0247] Optionally, the acquisition of the index data of the specified evaluation indexes obtained by testing the to-be-tested algorithm according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task includes:

[0248] In a case where the acquired task is the algorithm test task, a to-be-tested algorithm indicated by the algorithm identification is tested according to a specified test strategy indicated by the strategy identification, to obtain index data of the to-be-tested algorithm under the specified evaluation indexes;

[0249] In a case where the acquired task is the offline evaluation task, the offline index data is read from an acquisition address indicated by the selected storage address information.

[0250] Optionally, the client further displays at least one algorithm evaluation strategy after displaying the task generation interface; any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; a selection operation on the at least one algorithm evaluation strategy is received, and an algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy is sent to the server.

[0251] The determination of a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy includes:

[0252] The algorithm evaluation strategy sent by the client is received as the target algorithm evaluation strategy.

[0253] Optionally, the test configuration comprises a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration; the terminal hardware configuration indicates terminal information deployed when the to-be-tested algorithm is tested, and the data type configuration indicates type information of data input when the to-be-tested algorithm is tested.

[0254] The corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration is determined based on the obtained test data of the specified evaluation dimension, comprising:

[0255] Based on the obtained test data of the specified evaluation dimension, a change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0256] Optionally, the terminal hardware configuration comprises at least one of a terminal configuration and a hardware configuration; wherein the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained by the terminal when the to-be-tested algorithm is tested.

[0257] The data type configuration comprises at least one of a data source configuration, a task number configuration, and a data format configuration; wherein the data source configuration indicates a data source to which input data of the to-be-tested algorithm belongs when the to-be-tested algorithm is tested; the task number configuration indicates a maximum number of tasks processed synchronously by the to-be-tested algorithm when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of the input data of the to-be-tested algorithm when the to-be-tested algorithm is tested.

[0258] Optionally, the terminal configuration comprises at least one of a terminal number, a terminal type, and a terminal version; the hardware configuration comprises at least one of a number of graphics cards, a memory size, and a processor computing power; the data source configuration comprises at least one of a scenario to which a data source belongs, a data source type, and a number of data in the data source; and the data format configuration comprises at least one of a resolution, a frame rate, and a type.

[0259] Optionally, after the business conclusion of the to-be-tested algorithm for the business scenario is generated in the business conclusion generation manner indicated by the target algorithm evaluation strategy based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, the method further comprises:

[0260] receiving the business conclusion generated by the server and the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration;

[0261] generating a target legend based on a correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration;

[0262] displaying an evaluation display interface; wherein the evaluation display interface comprises the target legend and / or the received business conclusion.

[0263] Optionally, the target algorithm evaluation strategy comprises an algorithm evaluation template; the algorithm evaluation template comprises a condition field and a blank field to be filled with content; the condition field comprises a screening condition for screening the test configuration and / or the evaluation index; and the business conclusion of the algorithm to be tested for the business scenario is generated based on the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration according to the business conclusion generation manner indicated by the target algorithm evaluation strategy, comprising:

[0264] screening the test configuration when testing the algorithm to be tested and / or the index data of the specified evaluation index obtained by testing the algorithm to be tested according to the screening condition comprised in the condition field in the algorithm evaluation template, to obtain screening data;

[0265] determining the index data and / or the test configuration corresponding to the screening data from the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration;

[0266] filling the determined index data and / or test configuration into the blank field in the algorithm evaluation template to obtain the business conclusion of the algorithm to be tested for the business scenario.

[0267] Optionally, the evaluation indexes comprise at least one of a hardware evaluation index and an algorithm evaluation index.

[0268] The hardware evaluation index comprises at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value.

[0269] The algorithm evaluation index comprises at least one of an algorithm throughput, an algorithm accuracy, and a response time.

[0270] In this embodiment, after the index data of the specified evaluation index of the to-be-tested algorithm obtained by testing according to the specified test strategy is obtained, the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the business conclusion generation manner indicated by the target algorithm evaluation strategy and using the correspondence. It can be seen that the present scheme can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0271] The present application also provides an algorithm evaluation system, as shown in the accompanying drawings, the algorithm evaluation system comprises a client 1301 and a server 1302. Figure 13

[0272] The client 1301 is configured to display a task generation interface, receive a configuration operation on the task generation interface, and generate an algorithm evaluation task. The algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy. The algorithm evaluation task is sent to the server 1302.

[0273] The server 1302 is configured to receive the algorithm evaluation task sent by the client 1301, acquire the index data of the specified evaluation index obtained by testing the to-be-tested algorithm according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task, determine a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration based on the acquired test data of the specified evaluation dimension, determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index, and generate a business conclusion for the to-be-tested algorithm for the business scenario based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to the business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0274] Optionally, the task generation interface comprises a task type selection box, and the configuration operation on the task generation interface comprises a task selection operation on the task type selection box.

[0275] The client receives the configuration operation on the task generation interface to generate the algorithm evaluation task, which comprises:

[0276] ​In a case where the task type selected by the task selection operation is a test type, a first sub-interface is displayed; the first sub-interface includes a test algorithm selection box, a test strategy selection box, and an evaluation index selection box; selection operations on the test algorithm selection box, the test strategy selection box, and the evaluation index selection box are received respectively, and selected algorithm identification, strategy identification, and specified evaluation indexes are determined; based on the selected algorithm identification, strategy identification, and specified evaluation indexes, an algorithm test task is generated as an algorithm evaluation task;

[0277] In a case where the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; the second sub-interface includes a location selection box; a selection operation on the location selection box is received, and selected storage address information is determined; based on the selected storage address information, an offline evaluation task is generated as an algorithm evaluation task; the selected storage address information indicates an acquisition address of offline index data, and the offline index data is index data of specified evaluation indexes obtained by testing a to-be-tested algorithm according to a specified test strategy.

[0278] Optionally, the server acquires, based on a data acquisition strategy indicated by the algorithm evaluation task, index data of the specified evaluation indexes obtained by testing the to-be-tested algorithm according to the specified test strategy, including:

[0279] In a case where the acquired task is the algorithm test task, a to-be-tested algorithm indicated by the algorithm identification is tested according to a specified test strategy indicated by the strategy identification, to obtain index data of the to-be-tested algorithm under the specified evaluation indexes;

[0280] In a case where the acquired task is the offline evaluation task, the offline index data is read from an acquisition address indicated by the selected storage address information.

[0281] Optionally, the client further displays at least one algorithm evaluation strategy after displaying the task generation interface; any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; a selection operation on the at least one algorithm evaluation strategy is received, and an algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy is sent to the server;

[0282] The server determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, including:

[0283] The algorithm evaluation strategy sent by the client is received as the target algorithm evaluation strategy.

[0284] Optionally, the test configuration comprises a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration; the terminal hardware configuration indicates terminal information deployed when the to-be-tested algorithm is tested, and the data type configuration indicates type information of data input when the to-be-tested algorithm is tested.

[0285] The server determines a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the obtained test data of the specified evaluation dimension, comprising:

[0286] Based on the obtained test data of the specified evaluation dimension, a change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0287] Optionally, the terminal hardware configuration comprises at least one of a terminal configuration and a hardware configuration; wherein the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained by the terminal when the to-be-tested algorithm is tested.

[0288] The data type configuration comprises at least one of a data source configuration, a task quantity configuration, and a data format configuration; wherein the data source configuration indicates a data source to which input data of the to-be-tested algorithm belongs when the to-be-tested algorithm is tested; the task quantity configuration indicates a maximum task number processed synchronously by the to-be-tested algorithm when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of the input data of the to-be-tested algorithm when the to-be-tested algorithm is tested.

[0289] Optionally, the terminal configuration comprises at least one of a terminal quantity, a terminal type, and a terminal version; the hardware configuration comprises at least one of a graphics card quantity, a memory size, and a processor computing power; the data source configuration comprises at least one of a scenario to which a data source belongs, a data source type, and a data quantity in the data source; and the data format configuration comprises at least one of a resolution, a frame rate, and a type.

[0290] Optionally, the server is further configured to: after generating the business conclusion of the to-be-tested algorithm for the business scenario in the business conclusion generation manner indicated by the target algorithm evaluation strategy based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, send the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration and / or the generated business conclusion to the client.

[0291] The client is further configured to generate a target legend based on the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration, and display an evaluation display interface; wherein the evaluation display interface includes the target legend corresponding to the specified evaluation index, and / or the received business conclusion.

[0292] Optionally, the target algorithm evaluation strategy includes an algorithm evaluation template; the algorithm evaluation template includes a condition field and a blank field to be filled with content; the condition field includes a filtering condition for filtering the test configuration and / or the evaluation index; and the server generates the business conclusion for the business scenario of the algorithm to be tested based on the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration, according to the business conclusion generation manner indicated by the target algorithm evaluation strategy, including:

[0293] According to the filtering condition included in the condition field in the algorithm evaluation template, the test configuration when the algorithm to be tested is tested and / or the index data of the specified evaluation index obtained by testing the algorithm to be tested are filtered to obtain filtered data;

[0294] From the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration, the index data and / or the test configuration corresponding to the filtered data are determined;

[0295] The determined index data and / or test configuration are filled into the blank field in the algorithm evaluation template to obtain the business conclusion for the business scenario of the algorithm to be tested.

[0296] Optionally, the evaluation indexes include at least one of a hardware evaluation index and an algorithm evaluation index.

[0297] The hardware evaluation index includes at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value.

[0298] The algorithm evaluation index includes at least one of an algorithm throughput, an algorithm accuracy, and a response time.

[0299] In this embodiment, after obtaining the index data of the specified evaluation index of the algorithm to be tested obtained by testing the algorithm to be tested according to the specified test strategy, the correspondence between the index data of the specified evaluation index of the algorithm to be tested and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the algorithm to be tested with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the correspondence by using the business conclusion generation manner indicated by the target algorithm evaluation strategy. It can be seen that the present scheme can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0300] The embodiment of the present application also provides an algorithm evaluation device, as shown in the figure, the device comprises: Figure 14

[0301] an index data acquisition module 1401, configured to acquire index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; wherein the specified test strategy is a test strategy matched with a business scenario of the to-be-tested algorithm, the specified test strategy indicates that the to-be-tested algorithm is tested under at least one test configuration, and the specified evaluation index is at least one of each evaluation index;

[0302] a corresponding relationship determination module 1402, configured to determine a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration based on the acquired test data of the specified evaluation index;

[0303] an evaluation strategy determination module 1403, configured to determine a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index;

[0304] a business conclusion generation module 1404, configured to generate a business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0305] Optionally, the index data acquisition module comprises:

[0306] a task acquisition sub-module, configured to acquire an algorithm test task or an offline evaluation task; wherein the algorithm test task comprises an algorithm identifier, a strategy identifier and a specified evaluation index; the offline evaluation task comprises storage address information; the storage address information indicates an acquisition address of offline index data, and the offline index data is index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy in advance;

[0307] a test sub-module, configured to, in a case where the acquired task is the algorithm test task, test a to-be-tested algorithm indicated by the algorithm identifier according to a specified test strategy indicated by the strategy identifier, to obtain index data of the to-be-tested algorithm under the specified evaluation index;

[0308] an offline index reading sub-module, configured to, in a case where the acquired task is the offline evaluation task, read the offline index data from an acquisition address indicated by the storage address information.

[0309] ​Optionally, the task acquisition submodule comprises:

[0310] The first display unit is configured to display a task generation interface; the task generation interface comprises a task type selection box;

[0311] The task selection operation receiving unit is configured to receive a task selection operation for the task type selection box;

[0312] The second display unit is configured to display a first sub-interface if the task type selected by the task selection operation is a test type; the first sub-interface comprises a test algorithm selection box, a test strategy selection box and an evaluation index selection box; the selection operation for the test algorithm selection box, the test strategy selection box and the evaluation index selection box is received respectively to determine the selected algorithm identifier, strategy identifier and specified evaluation index; and an algorithm test task is generated based on the selected algorithm identifier, strategy identifier and specified evaluation index;

[0313] The third display unit is configured to display a second sub-interface if the task type selected by the task selection operation is an offline type; the second sub-interface comprises a position selection box; the selection operation for the position selection box is received to determine the selected storage address information; and an offline evaluation task is generated based on the selected storage address information.

[0314] Optionally, the test configuration comprises a first dimension configuration and a second dimension configuration; the first dimension configuration and the second dimension are at least one of terminal hardware configuration and data type configuration; the terminal hardware configuration indicates terminal information deployed when the to-be-tested algorithm is tested; and the data type configuration indicates type information of data input when the to-be-tested algorithm is tested.

[0315] The corresponding relationship determination module is specifically configured to:

[0316] Based on the acquired test data of the specified evaluation dimension, a change function of index data of the specified evaluation index of the to-be-tested algorithm with respect to the change of the second dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0317] The terminal hardware configuration comprises at least one of terminal configuration and hardware configuration; the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested; and the hardware configuration indicates hardware information contained in the terminal when the to-be-tested algorithm is tested.

[0318] The data type configuration comprises at least one of a data source configuration, a task quantity configuration, and a data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested; the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of the input data when the to-be-tested algorithm is tested.

[0319] Optionally, the terminal configuration comprises at least one of a terminal quantity, a terminal type, and a terminal version; the hardware configuration comprises at least one of a graphics card quantity, a memory size, and a processor computing power; the data source configuration comprises at least one of a data source belonging scene, a data source type, and a data quantity in the data source; and the data format configuration comprises at least one of a resolution, a frame rate, and a type.

[0320] Optionally, the evaluation strategy determination module comprises:

[0321] The evaluation strategy display submodule is configured to display at least one algorithm evaluation strategy; any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index.

[0322] The selection operation receiving submodule is configured to receive a selection operation for the at least one algorithm evaluation strategy, and select, as an algorithm evaluation strategy associated with the specified test strategy, an algorithm evaluation strategy selected by the selection operation from the at least one algorithm evaluation strategy.

[0323] Optionally, the apparatus further comprises:

[0324] The evaluation display module is configured to display an evaluation display interface after the business conclusion generation module generates a business conclusion of the to-be-tested algorithm for the business scene based on a correspondence between index data of a specified evaluation index of the to-be-tested algorithm and a test configuration in a manner indicated by the target algorithm evaluation strategy; the evaluation display interface comprises a target legend corresponding to the specified evaluation index and / or the generated business conclusion; the target legend is generated based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0325] Optionally, the target algorithm evaluation strategy comprises an algorithm evaluation template; the algorithm evaluation template comprises a condition field and a blank field to be filled with content; the condition field comprises a screening condition for screening a test configuration and / or an evaluation index; and the business conclusion generation module comprises:

[0326] The screening submodule is configured to screen the test configuration when the to-be-tested algorithm is tested and / or the index data of the specified evaluation index of the to-be-tested algorithm, to obtain screening data, according to the screening condition contained in the condition field in the algorithm evaluation template according to the algorithm.

[0327] The data determination submodule is configured to determine the index data and / or the test configuration corresponding to the screening data from the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

[0328] The business conclusion generation submodule is configured to fill the determined index data and / or the test configuration into the blank field in the algorithm evaluation template, to obtain the business conclusion of the to-be-tested algorithm for the business scenario.

[0329] Optionally, the evaluation indexes include at least one of a hardware evaluation index and an algorithm evaluation index; the hardware evaluation index includes at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value; and the algorithm evaluation index includes at least one of an algorithm throughput, an algorithm accuracy, and a response time.

[0330] In this embodiment, after the index data of the specified evaluation index of the to-be-tested algorithm is obtained, the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the business conclusion generation manner indicated by the target algorithm evaluation strategy and using the correspondence. It can be seen that, by using the obtained index data, the business conclusion for the business scenario can be automatically generated, so that the efficiency of algorithm evaluation is improved.

[0331] The embodiment of the application further provides an algorithm evaluation device applied to a client in an algorithm evaluation system, wherein the algorithm evaluation system further includes a server, and the device includes:

[0332] The interface display module is configured to display a task generation interface.

[0333] The algorithm evaluation task generation module is configured to receive a configuration operation for the task generation interface, and generate an algorithm evaluation task, wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index of a to-be-tested algorithm tested according to a specified test strategy.

[0334] The algorithm evaluation task sending module is configured to send the algorithm evaluation task to the server, so that the server, after receiving the algorithm evaluation task, acquires the index data of the specified evaluation indicators obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; determines the correspondence between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configurations based on the acquired test data of the specified evaluation dimensions; determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation indicator; and generates the business conclusion of the to-be-tested algorithm for the business scenario based on the correspondence between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configurations according to the business conclusion generation manner indicated by the target algorithm evaluation strategy.

[0335] In this embodiment, after the index data of the specified evaluation indicators obtained by the to-be-tested algorithm in the test according to the specified test strategy is obtained, the correspondence between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configurations can be determined, which can reflect the change of the index data of the specified evaluation indicators of the to-be-tested algorithm with the change of the test configurations, and then the business conclusion for the business scenario can be generated by using the correspondence according to the business conclusion generation manner indicated by the target algorithm evaluation strategy. It can be seen that, according to the scheme, the business conclusion for the business scenario can be automatically generated by using the acquired index data, thereby improving the efficiency of algorithm evaluation.

[0336] The embodiment of the application further provides an algorithm evaluation device applied to a server in an algorithm evaluation system, the algorithm evaluation system further comprising a client; and the device comprising:

[0337] A task receiving module is configured to receive an algorithm evaluation task generated by the client; wherein the algorithm evaluation task indicates a data acquisition strategy of acquiring index data of specified evaluation indicators obtained by a to-be-tested algorithm in a test according to a specified test strategy;

[0338] An index acquiring module is configured to acquire the index data of the specified evaluation indicators obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task;

[0339] A relationship determining module is configured to determine the correspondence between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configurations based on the acquired test data of the specified evaluation dimensions;

[0340] The policy determining module is configured to determine a preset algorithm evaluation policy associated with the specified test policy as a target algorithm evaluation policy, wherein any algorithm evaluation policy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index;

[0341] The conclusion generating module is configured to generate the business conclusion of the to-be-tested algorithm for the business scenario based on a correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to the business conclusion generation manner indicated by the target algorithm evaluation policy.

[0342] In this embodiment, after obtaining the index data of the specified evaluation index of the to-be-tested algorithm according to the specified test policy, the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration can be determined, which can reflect the change of the index data of the specified evaluation index of the to-be-tested algorithm with the change of the test configuration, and then the business conclusion for the business scenario can be generated by using the correspondence according to the business conclusion generation manner indicated by the target algorithm evaluation policy. It can be seen that the present solution can automatically generate the business conclusion for the business scenario by using the obtained index data, thereby improving the efficiency of algorithm evaluation.

[0343] The present application also provides a client, as shown in Figure 15 The client includes a processor 1501 and a machine readable storage medium 1502, the machine readable storage medium 1502 stores machine executable instructions capable of being executed by the processor 1501, and the processor 1501 is prompted by the machine executable instructions to implement the method steps of the above-mentioned algorithm evaluation method applied to the client.

[0344] The present application also provides a server, as shown in Figure 16 The server includes a processor 1601 and a machine readable storage medium 1602, the machine readable storage medium 1602 stores machine executable instructions capable of being executed by the processor 1601, and the processor 1601 is prompted by the machine executable instructions to implement the method steps of the above-mentioned algorithm evaluation method applied to the client.

[0345] The present application also provides an electronic device, as shown in Figure 17 The electronic device includes a processor 1701, a communication interface 1702, a memory 1703 and a communication bus 1704, wherein the processor 1701, the communication interface 1702 and the memory 1703 complete mutual communication through the communication bus 1704,

[0346] The memory 1703 is configured to store a computer program.

[0347] The processor 1701 is configured to execute the program stored in the memory 1703 to implement the method steps of the algorithm evaluation method.

[0348] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0349] The communication interface is used for communication between the above electronic device and other devices.

[0350] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the above processor.

[0351] The above processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0352] In another embodiment provided by the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the above algorithm evaluation methods.

[0353] In another embodiment provided by the present application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute any of the algorithm evaluation methods in the above embodiments.

[0354] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented in software, all or some of the steps can be implemented in the form of one or more computer programs which are stored in a computer readable medium. The computer readable medium can include one or more of a computer readable storage medium and a computer readable signal medium. The computer readable storage medium can include one or more of a magnetic storage medium (e.g., one or more magnetic tapes), an electronic storage medium (e.g., one or more semiconductor memories or tapes), a mechanical storage medium (e.g., one or more recording chips), or the like. The computer readable signal medium can include a computer readable storage medium which is configured to transmit program code embedded in a modulated data signal. The modulated data signal is one example means for transmitting the program code (e.g., either modulated on a carrier or unmodulated). The program code embodied on the computer readable medium is for executing one or more of the steps in the embodiments described above on one or more of the computers in the system. The computer readable medium can be a computer program product.

[0355] It should be noted that, in the description above, the relative terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Also, the terms "comprises", "comprising", or any other variations thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a..." does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0356] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0357] The above merely describes the preferred embodiments of the present application, but is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An algorithm evaluation method, characterized by, The method comprises: obtaining index data of a specified evaluation index obtained by a to-be-tested algorithm performing testing according to a specified testing strategy; wherein the specified testing strategy is a testing strategy matched with a business scenario of the to-be-tested algorithm, the specified testing strategy indicates that the to-be-tested algorithm performs testing under at least one testing configuration; the specified evaluation index is at least one of each evaluation index; the testing configuration comprises a first dimension configuration and a second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of a terminal hardware configuration, a data type configuration; the terminal hardware configuration comprises at least one of a terminal configuration and a hardware configuration; the terminal configuration indicates terminal deployment information when the to-be-tested algorithm performs testing, and the hardware configuration indicates hardware information contained by a terminal when the to-be-tested algorithm performs testing; the data type configuration comprises at least one of a data source configuration, a task quantity configuration and a data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm performs testing; the task quantity configuration indicates a maximum task number processed synchronously when the to-be-tested algorithm performs testing; and the data format configuration indicates a data format of input data when the to-be-tested algorithm performs testing; determining, based on the obtained testing data of the specified evaluation dimension, a change function of index data of the specified evaluation index of the to-be-tested algorithm with respect to a change of the second dimension configuration under the first dimension configuration as a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the testing configuration; determining a preset algorithm evaluation strategy associated with the specified testing strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a testing configuration and / or an evaluation index; generating, according to a business conclusion generation manner indicated by the target algorithm evaluation strategy, a business conclusion of the to-be-tested algorithm for the business scenario based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the testing configuration.

2. The method of claim 1, wherein, The obtaining of the index data of the specified evaluation index obtained by the to-be-tested algorithm performing testing according to the specified testing strategy comprises: obtaining an algorithm testing task or an offline evaluation task; wherein the algorithm testing task comprises an algorithm identifier, a strategy identifier and a specified evaluation index; the offline evaluation task comprises storage address information; the storage address information indicates an acquisition address of offline index data, and the offline index data is index data of a specified evaluation index obtained by pre-testing a to-be-tested algorithm according to a specified testing strategy; in a case where the obtained task is the algorithm testing task, testing a to-be-tested algorithm indicated by the algorithm identifier according to a specified testing strategy indicated by the strategy identifier to obtain index data of the to-be-tested algorithm under the specified evaluation index; in a case where the obtained task is the offline evaluation task, reading the offline index data from an acquisition address indicated by the storage address information.

3. The method of claim 2, wherein, The obtaining of the algorithm testing task or the offline evaluation task comprises: Display a task generation interface; wherein the task generation interface comprises a task type selection box; Receive a task selection operation for the task type selection box; In a case where the task type selected by the task selection operation is a test type, display a first sub-interface; wherein the first sub-interface comprises a test algorithm selection box, a test strategy selection box, and an evaluation index selection box; receive selection operations for the test algorithm selection box, the test strategy selection box, and the evaluation index selection box, respectively, to determine selected algorithm identification, strategy identification, and specified evaluation index; and generate an algorithm test task based on the selected algorithm identification, strategy identification, and specified evaluation index; In a case where the task type selected by the task selection operation is an offline type, display a second sub-interface; wherein the second sub-interface comprises a location selection box; receive a selection operation for the location selection box to determine selected storage address information; and generate an offline evaluation task based on the selected storage address information.

4. The method of claim 1, wherein, The terminal configuration includes at least one of the number of terminals, the type of terminal, and the version of terminal; the hardware configuration includes at least one of the number of graphics cards, the size of memory, and the computing power of processor; the data source configuration includes at least one of the scenario to which the data source belongs, the type of data source, and the number of data in the data source; and the data format configuration includes at least one of resolution, frame rate, and type.

5. The method according to any one of claims 1 to 3, characterized in that, The determination of the preset algorithm evaluation strategy associated with the specified test strategy includes: Display at least one algorithm evaluation strategy; wherein any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; Receive a selection operation for the at least one algorithm evaluation strategy, and select, as the algorithm evaluation strategy associated with the specified test strategy, the algorithm evaluation strategy selected by the selection operation from the at least one algorithm evaluation strategy.

6. The method according to any one of claims 1 to 3, characterized in that, After the generation of the business conclusion of the to-be-tested algorithm for the business scenario in the manner indicated by the target algorithm evaluation strategy and based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, the method further includes: Display an evaluation display interface; wherein the evaluation display interface comprises a target legend corresponding to the specified evaluation index and / or the generated business conclusion; wherein the target legend is a legend generated based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration.

7. The method according to any one of claims 1 to 3, characterized in that, The target algorithm evaluation strategy includes an algorithm evaluation template; the algorithm evaluation template comprises a condition field and a blank field to be filled with content; and the condition field comprises a filtering condition for filtering a test configuration and / or an evaluation index; The generation of the business conclusion of the to-be-tested algorithm for the business scenario in the manner indicated by the target algorithm evaluation strategy and based on the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration includes: The screening condition contained in the condition field in the algorithm evaluation template is used to screen the test configuration when the to-be-tested algorithm is tested and / or the index data of the specified evaluation index obtained by testing the to-be-tested algorithm, to obtain screening data; Index data and / or test configuration corresponding to the screening data are determined from the correspondence between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration; The determined index data and / or test configuration are filled into the blank field in the algorithm evaluation template, to obtain a business conclusion of the to-be-tested algorithm for the business scenario.

8. The method according to any one of claims 1 to 3, characterized in that, The evaluation indexes include at least one of a hardware evaluation index and an algorithm evaluation index; The hardware evaluation index includes at least one of a hardware resource utilization rate, a hardware power consumption, and a hardware temperature change value; The algorithm evaluation index includes at least one of an algorithm throughput, an algorithm accuracy, and a response time.

9. An algorithm evaluation method, characterized by, The method is applied to a client in an algorithm evaluation system, and the algorithm evaluation system further includes a server. The method includes: displaying a task generation interface; receiving a configuration operation on the task generation interface to generate an algorithm evaluation task; wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy; sending the algorithm evaluation task to the server, so that the server, after receiving the algorithm evaluation task, acquires the index data of the specified evaluation indicators obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; based on the acquired test data of the specified evaluation dimensions, determining a change function of the index data of the specified evaluation indicators of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration as the corresponding relationship between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configuration; the test configuration includes the first dimension configuration and the second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of terminal hardware configuration, data type configuration; the terminal hardware configuration includes at least one of terminal configuration and hardware configuration; the terminal configuration indicates the terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates the hardware information contained in the terminal when the to-be-tested algorithm is tested; the data type configuration includes at least one of data source configuration, task number configuration and data format configuration; the data source configuration indicates the data source to which the input data of the to-be-tested algorithm belongs when the to-be-tested algorithm is tested; the task number configuration indicates the maximum number of tasks processed synchronously by the to-be-tested algorithm when the to-be-tested algorithm is tested; the data format configuration indicates the data format of the input data of the to-be-tested algorithm when the to-be-tested algorithm is tested; determining a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on test configuration and / or evaluation indicators; generating a business conclusion for the to-be-tested algorithm for a business scenario according to the corresponding relationship between the index data of the specified evaluation indicators of the to-be-tested algorithm and the test configuration in the way of generating a business conclusion indicated by the target algorithm evaluation strategy.

10. An algorithm evaluation method, characterized by, The server applied to an algorithm evaluation system, the algorithm evaluation system further includes a client; the method includes: receiving the algorithm evaluation task generated by the client; wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of specified evaluation indicators obtained by a to-be-tested algorithm in a test according to a specified test strategy; acquiring the index data of the specified evaluation indicators obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; Based on the obtained test data of the specified evaluation dimension, a change function of index data of the specified evaluation index of the to-be-tested algorithm with respect to a change of the second dimension configuration under the first dimension configuration is determined as a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration; the test configuration includes the first dimension configuration and the second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of terminal hardware configuration, data type configuration; the terminal hardware configuration includes at least one of terminal configuration and hardware configuration; the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained in the terminal when the to-be-tested algorithm is tested; the data type configuration includes at least one of data source configuration, task quantity configuration and data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested; the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of input data when the to-be-tested algorithm is tested; A preset algorithm evaluation strategy associated with the specified test strategy is determined as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation index; According to the business conclusion generation way indicated by the target algorithm evaluation strategy, the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration is used to generate a business conclusion of the to-be-tested algorithm for a business scenario.

11. An algorithm evaluation system, characterized by The algorithm evaluation system includes a client and a server; The client is configured to display a task generation interface, receive a configuration operation on the task generation interface, generate an algorithm evaluation task, wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation index obtained by testing a to-be-tested algorithm according to a specified test strategy, and send the algorithm evaluation task to the server. The server is configured to receive the algorithm evaluation task sent by the client, acquire the index data of the specified evaluation index obtained by the to-be-tested algorithm according to the specified test strategy based on a data acquisition strategy indicated by the algorithm evaluation task, determine a change function of the index data of the specified evaluation index of the to-be-tested algorithm with respect to a second dimension configuration based on the acquired test data of the specified evaluation dimension, as a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration under a first dimension configuration, the test configuration includes the first dimension configuration and the second dimension configuration, the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration, the terminal hardware configuration includes at least one of a terminal configuration and a hardware configuration, the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, the hardware configuration indicates hardware information contained in a terminal when the to-be-tested algorithm is tested, the data type configuration includes at least one of a data source configuration, a task quantity configuration, and a data format configuration, the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested, the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested, and the data format configuration indicates a data format of input data when the to-be-tested algorithm is tested, a preset algorithm evaluation strategy associated with the specified test strategy is determined as a target algorithm evaluation strategy, any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation index, and a business conclusion of the to-be-tested algorithm for a business scenario is generated based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration according to a business conclusion generation manner indicated by the target algorithm evaluation strategy.

12. The algorithm evaluation system of claim 11, wherein, The task generation interface includes a task type selection box, and the configuration operation for the task generation interface includes a task selection operation for the task type selection box. The client receives the configuration operation for the task generation interface, generates an algorithm evaluation task, and includes: In a case where the task type selected by the task selection operation is a test type, a first sub-interface is displayed, the first sub-interface includes a test algorithm selection box, a test strategy selection box, and an evaluation index selection box, selection operations for the test algorithm selection box, the test strategy selection box, and the evaluation index selection box are received respectively, an algorithm identifier, a strategy identifier, and a specified evaluation index are determined, an algorithm test task is generated based on the selected algorithm identifier, strategy identifier, and specified evaluation index, as an algorithm evaluation task. In a case where the task type selected by the task selection operation is an offline type, a second sub-interface is displayed; the second sub-interface includes a position selection box; a selection operation on the position selection box is received, and selected storage address information is determined; based on the selected storage address information, an offline evaluation task is generated as an algorithm evaluation task; the selected storage address information indicates an acquisition address of offline index data, and the offline index data is index data of a specified evaluation index obtained by performing a test on a to-be-tested algorithm according to a specified test strategy.

13. The algorithm evaluation system of claim 12, wherein, The server acquires, based on a data acquisition strategy indicated by the algorithm evaluation task, index data of the specified evaluation index obtained by performing the test on the to-be-tested algorithm according to the specified test strategy, including: In a case where the acquired task is the algorithm test task, performing a test on the to-be-tested algorithm indicated by the algorithm identifier according to the specified test strategy indicated by the strategy identifier to obtain index data of the to-be-tested algorithm under the specified evaluation index; In a case where the acquired task is the offline evaluation task, reading the offline index data from the acquisition address indicated by the selected storage address information.

14. The algorithm evaluation system of claim 11, wherein, The client further displays at least one algorithm evaluation strategy after displaying the task generation interface; any algorithm evaluation strategy is configured for at least one test configuration and / or at least one evaluation index; receives a selection operation on the at least one algorithm evaluation strategy, and sends, to the server, an algorithm evaluation strategy selected by the selection operation in the at least one algorithm evaluation strategy; The server determines a preset algorithm evaluation strategy associated with the specified test strategy as a target algorithm evaluation strategy, including: Receiving the algorithm evaluation strategy sent by the client as the target algorithm evaluation strategy.

15. An algorithm evaluation apparatus characterized by comprising: The device includes: The index data acquisition module is configured to acquire index data of a specified evaluation index obtained by a to-be-tested algorithm performing testing according to a specified testing strategy; the specified testing strategy is a testing strategy matched with a business scenario of the to-be-tested algorithm, and the specified testing strategy indicates that the to-be-tested algorithm performs testing under at least one testing configuration; the specified evaluation index is at least one of each evaluation index; the testing configuration includes a first dimension configuration and a second dimension configuration; the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration; the terminal hardware configuration includes at least one of a terminal configuration and a hardware configuration; the terminal configuration indicates terminal deployment information when the to-be-tested algorithm performs testing, and the hardware configuration indicates hardware information contained by a terminal when the to-be-tested algorithm performs testing; the data type configuration includes at least one of a data source configuration, a task quantity configuration, and a data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm performs testing; the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm performs testing; and the data format configuration indicates a data format of input data when the to-be-tested algorithm performs testing; The corresponding relationship determination module is configured to determine, based on the acquired testing data of the specified evaluation dimension, a change function of index data of the specified evaluation index of the to-be-tested algorithm with respect to a change of the second dimension configuration as the to-be-tested algorithm is under the first dimension configuration, as a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the testing configuration; The evaluation strategy determination module is configured to determine a preset algorithm evaluation strategy associated with the specified testing strategy as a target algorithm evaluation strategy; any algorithm evaluation strategy indicates a manner of generating a business conclusion based on a testing configuration and / or an evaluation index; The business conclusion generation module is configured to generate, based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the testing configuration, a business conclusion of the to-be-tested algorithm for the business scenario according to a business conclusion generation manner indicated by the target algorithm evaluation strategy.

16. An algorithm evaluation apparatus characterized by comprising: A client applied to an algorithm evaluation system, the algorithm evaluation system further includes a server; the device includes: An interface display module configured to display a task generation interface; An algorithm evaluation task generation module configured to receive a configuration operation for the task generation interface and generate an algorithm evaluation task; the algorithm evaluation task indicates a data acquisition strategy of acquiring index data of a specified evaluation index obtained by a to-be-tested algorithm performing testing according to a specified testing strategy; The algorithm evaluation task sending module is configured to send the algorithm evaluation task to the server, so that the server, after receiving the algorithm evaluation task, acquires the index data of the specified evaluation indicator obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; based on the acquired test data of the specified evaluation dimension, the change function of the index data of the specified evaluation indicator of the to-be-tested algorithm with respect to the change of the second dimension configuration under the first dimension configuration is determined as the corresponding relationship between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration; the test configuration includes the first dimension configuration and the second dimension configuration; wherein the first dimension configuration and the second dimension are at least one of terminal hardware configuration, data type configuration; the terminal hardware configuration includes at least one of terminal configuration and hardware configuration; the terminal configuration indicates the terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates the hardware information contained in the terminal when the to-be-tested algorithm is tested; the data type configuration includes at least one of data source configuration, task number configuration and data format configuration; the data source configuration indicates the data source to which the input data of the to-be-tested algorithm belongs when the to-be-tested algorithm is tested; the task number configuration indicates the maximum number of tasks processed synchronously by the to-be-tested algorithm when the to-be-tested algorithm is tested; the data format configuration indicates the data format of the input data of the to-be-tested algorithm when the to-be-tested algorithm is tested; a preset algorithm evaluation strategy associated with the specified test strategy is determined as a target algorithm evaluation strategy; wherein any algorithm evaluation strategy indicates a way of generating a business conclusion based on a test configuration and / or an evaluation indicator; according to the business conclusion generation manner indicated by the target algorithm evaluation strategy, the business conclusion of the to-be-tested algorithm for a business scenario is generated based on the corresponding relationship between the index data of the specified evaluation indicator of the to-be-tested algorithm and the test configuration.

17. An algorithm evaluation apparatus characterized by comprising: The server applied to an algorithm evaluation system, the algorithm evaluation system further includes: a client; the device includes: A task receiving module is configured to receive an algorithm evaluation task generated by the client; wherein the algorithm evaluation task indicates a data acquisition strategy for acquiring index data of a specified evaluation indicator obtained by a to-be-tested algorithm in a test according to a specified test strategy; An index acquiring module is configured to acquire the index data of the specified evaluation indicator obtained by the to-be-tested algorithm in the test according to the specified test strategy based on the data acquisition strategy indicated by the algorithm evaluation task; The relationship determining module is configured to determine, based on the obtained test data of the specified evaluation dimension, a change function of index data of the specified evaluation index of the to-be-tested algorithm with respect to a change of a second dimension configuration, as a corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and a test configuration, when the to-be-tested algorithm is in a first dimension configuration; the test configuration comprises the first dimension configuration and the second dimension configuration; the first dimension configuration and the second dimension are at least one of a terminal hardware configuration and a data type configuration; the terminal hardware configuration comprises at least one of a terminal configuration and a hardware configuration; the terminal configuration indicates terminal deployment information when the to-be-tested algorithm is tested, and the hardware configuration indicates hardware information contained in a terminal when the to-be-tested algorithm is tested; the data type configuration comprises at least one of a data source configuration, a task quantity configuration, and a data format configuration; the data source configuration indicates a data source to which input data belongs when the to-be-tested algorithm is tested; the task quantity configuration indicates a maximum number of tasks processed synchronously when the to-be-tested algorithm is tested; and the data format configuration indicates a data format of input data when the to-be-tested algorithm is tested. The policy determining module is configured to determine a preset algorithm evaluation policy associated with the specified test policy as a target algorithm evaluation policy; any algorithm evaluation policy indicates a manner of generating a business conclusion based on a test configuration and / or an evaluation index. The conclusion generating module is configured to generate, based on the corresponding relationship between the index data of the specified evaluation index of the to-be-tested algorithm and the test configuration, a business conclusion of the to-be-tested algorithm for a business scenario in a manner indicated by the target algorithm evaluation policy.

18. A client, comprising: The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 9.

19. A server, characterized by The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 10.

20. An electronic device, comprising: The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 9. The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 10. The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 9.

21. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program capable of being executed by the processor, and the processor is prompted by the computer program to implement the method steps of claim 10.

Images