Model testing method, equipment, storage medium and program product

By establishing a distributed architecture between the test node and the model node, automating the model deployment and testing process, the problem of low model testing efficiency in the existing technology is solved, and efficient model performance evaluation and deployment is achieved.

CN113254331BActive Publication Date: 2024-11-08WEBANK (CHINA)
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202110519001.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-12
Publication Date
2024-11-08
Estimated Expiration
2041-05-12

AI Technical Summary

Technical Problem

The existing algorithm model testing methods are inefficient, resulting in the inability to effectively evaluate the model performance and affect the deployment effect.

Method used

By establishing a distributed architecture between the test node and the model node, automating the model deployment and testing process, the test node automatically publishes model information based on the task, the model node automatically acquires model deployment and processing results, and calculates test indicators.

Benefits of technology

It improves the efficiency of model testing, provides an automated test solution, reduces the need for manual writing of test code, and improves the accuracy and efficiency of test results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113254331B_ABST
    Figure CN113254331B_ABST
Patent Text Reader

Abstract

The present invention discloses a model testing method, device, storage medium and program product, the method comprising: when a test task for testing a model to be tested is detected, a target model node is determined from a plurality of model nodes; model information of the model to be tested is sent to the target model node, so that the target model node deploys and starts the model to be tested according to the model information; the target model node is called to process the test data corresponding to the test task using the model to be tested, and the model output result returned by the target model node is obtained; the test result of the indicator to be tested in the test task is calculated according to the model output result. The present invention provides an automated model testing scheme, which improves the efficiency of model testing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model testing method, device, storage medium and program product. Background Art

[0002] With the continuous development of artificial intelligence technology, more and more people from non-computer industries are engaged in algorithm research and data science research, but most algorithm engineers mainly conduct research based on local environments and output algorithm research results. If some algorithm models need to be engineered or continue to provide services after deployment, the performance of the algorithm models needs to be evaluated, otherwise it will be difficult to achieve the expected results. However, if algorithm engineers want to evaluate model performance, they can only write scripts for a single algorithm model to test model performance, which is inefficient. Summary of the invention

[0003] The main purpose of the present invention is to provide a model testing method, device, storage medium and program product, aiming to solve the technical problem of low efficiency of existing algorithm model testing methods.

[0004] To achieve the above object, the present invention provides a model testing method, which is applied to a test node, the test node is communicatively connected with a plurality of model nodes, and the method comprises the following steps:

[0005] When a test task for testing the model to be tested is detected, determining a target model node from the plurality of model nodes;

[0006] Sending the model information of the model to be tested to the target model node, so that the target model node deploys and starts the model to be tested according to the model information;

[0007] Calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node;

[0008] The test results of the indicators to be tested in the test task are calculated according to the output results of the model.

[0009] Optionally, when a test task for testing the model to be tested is detected, the step of determining a target model node from the multiple model nodes includes:

[0010] When a test task for testing the model to be tested is detected, a node whose machine model is the specified machine model in the test task among the multiple model nodes is used as a candidate model node;

[0011] A target model node is determined from candidate model nodes in an idle state.

[0012] Optionally, when a test task for testing the model to be tested is detected, before the step of determining a target model node from the multiple model nodes, the method further includes:

[0013] When a task configuration operation triggered by a user is detected, the test task is created according to the information configured by the task configuration operation, wherein the information configured by the task configuration operation includes one or more of the model information, the test set information corresponding to the test data, the indicators to be tested and the specified model.

[0014] Optionally, when the indicator to be tested is an indicator of a model performance type, the step of calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node includes:

[0015] Acquire a target test set according to the test set information corresponding to the test task;

[0016] Send each test data in the target test set to the target model node respectively, so as to call the target model node to process each test data respectively using the model to be tested, and obtain the model output results corresponding to each test data respectively returned by the target model node;

[0017] The step of calculating the test result of the indicator to be tested in the test task according to the output result of the model includes:

[0018] The test results of the indicators to be tested in the test task are calculated based on the output results of each model and the real labeled data corresponding to each test data in the target test set.

[0019] Optionally, there are multiple target model nodes, and the step of calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node includes:

[0020] According to the preset load strategy, each target model node is called to use the model to be tested to process different test data corresponding to the test task respectively, and the model output result returned by each target model node is obtained.

[0021] To achieve the above object, the present invention further provides a model testing method, which is applied to a model node, wherein a plurality of the model nodes are respectively connected to a test node for communication, and the method comprises the following steps:

[0022] Receiving model information of a model to be tested sent by the test node, wherein when the test node detects a test task for testing the model to be tested, the test node determines a target model node from a plurality of the model nodes and sends the model information to the target model node;

[0023] Deploy and start the model to be tested according to the model information;

[0024] When receiving a call from the test node, the test data corresponding to the test task is processed using the model to be tested to obtain a model output result;

[0025] The model output result is returned to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

[0026] Optionally, the model information includes the model file, dependent environment information and training framework type of the model to be tested.

[0027] The step of deploying and starting the model to be tested according to the model information includes:

[0028] Install the dependent environment required for starting the model to be tested according to the dependent environment information;

[0029] Detecting whether the training framework type is a type supported by the general framework in the model node;

[0030] If so, the model file is loaded in the general framework to start the model to be tested.

[0031] To achieve the above object, the present invention further provides a model testing device, which is deployed on a test node, the test node is communicatively connected with a plurality of model nodes, and the device comprises:

[0032] A determination module, configured to determine a target model node from the plurality of model nodes when a test task for testing a model to be tested is detected;

[0033] A sending module, used for sending the model information of the model to be tested to the target model node, so that the target model node deploys and starts the model to be tested according to the model information;

[0034] A calling module, used for calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node;

[0035] The calculation module is used to calculate the test results of the indicators to be tested in the test task according to the output results of the model.

[0036] To achieve the above object, the present invention further provides a model testing device, which is deployed on a model node, and a plurality of the model nodes are respectively connected to a test node for communication, and the device comprises:

[0037] A receiving module, configured to receive model information of a model to be tested sent by the test node, wherein when the test node detects a test task for testing the model to be tested, the test node determines a target model node from a plurality of the model nodes and sends the model information to the target model node;

[0038] A deployment module, used to deploy and start the model to be tested according to the model information;

[0039] A processing module, configured to, when receiving a call from the test node, use the model to be tested to process the test data corresponding to the test task to obtain a model output result;

[0040] The return module is used to return the model output result to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

[0041] To achieve the above objectives, the present invention also provides a model testing device, which includes: a memory, a processor, and a model testing program stored in the memory and executable on the processor, wherein the model testing program implements the steps of the model testing method described above when executed by the processor.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a model testing program is stored, and when the model testing program is executed by a processor, the steps of the model testing method described above are implemented.

[0043] In addition, to achieve the above object, the present invention also proposes a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the model testing method described above are implemented.

[0044] In the present invention, when the test node detects a test task for testing the model to be tested, the target model node is determined from multiple model nodes, and the model information of the model to be tested is sent to the target model node, so that the target model node can deploy and start the model to be tested according to the model information; the test node calls the target model node to process the test data corresponding to the test task using the model to be tested, and obtains the model output result returned by the target model node, and calculates the test result of the indicator to be tested in the test task according to the model output result. Compared with the existing solution of manually writing test code for a single model for testing, in the present invention, by setting a distributed architecture including test nodes and model nodes, the test node automatically publishes model information based on the test task, the model node automatically deploys and loads the model, and the test node automatically obtains the model processing result to test the indicator, an automated model testing solution is provided, and the efficiency of model testing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the flow chart of the first embodiment of the model testing method of the present invention;

[0047] Figure 3 A schematic diagram of interaction between a test node and a model node involved in an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of an overall flow of a model test involved in an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of an idle machine selection scenario involved in an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a multi-node model deployment scenario involved in an embodiment of the present invention.

[0051] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0053] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0054] It should be noted that the model test device in the embodiment of the present invention can be a smart phone, a personal computer, a server, etc., and no specific limitation is made here. The model test device can be a test node or a model node, and the test node is communicatively connected with multiple model nodes.

[0055] like Figure 1 As shown, the model testing device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the model test device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0057] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module and a model test program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the model test program and other software or programs.

[0058] When the model test device is a test node, Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the model node; and the processor 1001 can be used to call the model test program stored in the memory 1005 and perform the following operations:

[0059] When a test task for testing the model to be tested is detected, determining a target model node from the plurality of model nodes;

[0060] Sending the model information of the model to be tested to the target model node, so that the target model node deploys and starts the model to be tested according to the model information;

[0061] Calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node;

[0062] The test results of the indicators to be tested in the test task are calculated according to the output results of the model.

[0063] Further, when a test task for testing the model to be tested is detected, the step of determining a target model node from the multiple model nodes includes:

[0064] When a test task for testing the model to be tested is detected, a node whose machine model is the specified machine model in the test task among the multiple model nodes is used as a candidate model node;

[0065] A target model node is determined from candidate model nodes in an idle state.

[0066] Further, when a test task for testing the model to be tested is detected, before the step of determining the target model node from the multiple model nodes, the processor 1001 may also be used to call the model test program stored in the memory 1005 to perform the following operations:

[0067] When a task configuration operation triggered by a user is detected, the test task is created according to the information configured by the task configuration operation, wherein the information configured by the task configuration operation includes one or more of the model information, the test set information corresponding to the test data, the indicators to be tested and the specified model.

[0068] Furthermore, when the indicator to be tested is an indicator of a model performance type, the step of calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node includes:

[0069] Acquire a target test set according to the test set information corresponding to the test task;

[0070] Send each test data in the target test set to the target model node respectively, so as to call the target model node to process each test data respectively using the model to be tested, and obtain the model output results corresponding to each test data respectively returned by the target model node;

[0071] The step of calculating the test result of the indicator to be tested in the test task according to the output result of the model includes:

[0072] The test results of the indicators to be tested in the test task are calculated based on the output results of each model and the real labeled data corresponding to each test data in the target test set.

[0073] Furthermore, the number of the target model nodes is multiple, and the step of calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node includes:

[0074] According to the preset load strategy, each target model node is called to use the model to be tested to process different test data corresponding to the test task respectively, and the model output result returned by each target model node is obtained.

[0075] When the model test device is a model node, Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the test node; and the processor 1001 can be used to call the model test program stored in the memory 1005 and perform the following operations:

[0076] Receiving model information of a model to be tested sent by the test node, wherein when the test node detects a test task for testing the model to be tested, the test node determines a target model node from a plurality of the model nodes and sends the model information to the target model node;

[0077] Deploy and start the model to be tested according to the model information;

[0078] When receiving a call from the test node, the test data corresponding to the test task is processed using the model to be tested to obtain a model output result;

[0079] The model output result is returned to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

[0080] Furthermore, the model information includes a model file, dependent environment information, and a training framework type of the model to be tested, and the step of deploying and starting the model to be tested according to the model information includes:

[0081] Install the dependent environment required for starting the model to be tested according to the dependent environment information;

[0082] Detecting whether the training framework type is a type supported by the general framework in the model node;

[0083] If so, the model file is loaded in the general framework to start the model to be tested.

[0084] Based on the above structure, various embodiments of the model testing method are proposed.

[0085] Reference Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the model testing method of the present invention.

[0086] The embodiment of the present invention provides an embodiment of the model testing method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here. In this embodiment, the model testing method is applied to a test node, and the test node is communicatively connected with multiple model nodes. The test node and the model node can be devices such as smart phones, personal computers, and servers, which are not specifically limited in this embodiment. In this embodiment, the model testing method includes:

[0087] Step S10, when a test task for testing the model to be tested is detected, determining a target model node from the multiple model nodes;

[0088] In this embodiment, the test system may include test nodes and model nodes. There are multiple model nodes for deploying and starting running models. The test nodes are used to call model nodes to obtain model output results, thereby completing the test task. Specifically, the test task can be established by the test node based on the task configuration operation triggered by the user or the test task establishment instruction sent by other machines, or it can be directly sent to the test node by other nodes. The test task refers to the task of testing the model to be tested. Specifically, the test task may include model information of the model to be tested, relevant information of the test data set (or test set), relevant information of the model nodes required to be used during the test, and / or provisions of information such as indicators to be tested. Among them, the model to be tested is a model trained by a machine learning algorithm, such as a decision tree model, a logistic regression model, a neural network model, etc. In this embodiment, the type of model is not limited. The model information may include the model name or number of the model to be tested, which can uniquely indicate the model; or the model information may include the model file of the model to be tested, which is a code file that can realize the model task obtained after training. After loading the model file, the data can be input for processing to obtain the model output result; or, the model information may further include the framework type used in model training (hereinafter referred to as the training framework type) on the basis of including the identification information or the model file, or may further include the information of the dependent environment required when starting the model to be tested. The relevant information of the test set may be the index information of the storage location of the test set or the file name of the test set, which can identify the test set. The relevant information of the model node may be the model of the model node or the number of the model node. The indicator to be tested may be a model performance indicator or a stress test performance indicator; wherein the model performance indicator is an indicator related to the prediction performance of the model, such as accuracy, precision, recall, etc.; the stress test performance indicator refers to an indicator related to the service performance of the model, such as TPS (transactions per second), RT (response time), CPU utilization, memory occupancy, etc.

[0089] When the test node detects a test task for testing the model to be tested, the target model node can be determined from multiple model nodes. Specifically, the test node determines one model node from multiple model nodes as the target model node, or determines multiple model nodes from multiple model nodes as the target model nodes, that is, the number of target model nodes can be one or more, which is not limited in this embodiment. There are many methods for determining the target model node from multiple model nodes; for example, the target model node can be randomly selected from multiple model nodes; or, the target model node can be randomly selected from model nodes in an idle state; or, the target model node can be determined according to the relevant information of the model node required to be used in the test specified in the test task. For example, if the relevant information is the number of the model node, the test node can directly use the model node corresponding to the number as the target model node.

[0090] Furthermore, in one embodiment, before step S10, the method further includes:

[0091] Step a, when a task configuration operation triggered by a user is detected, the test task is created according to the information configured by the task configuration operation, wherein the information configured by the task configuration operation includes one or more of the model information, the test set information corresponding to the test data, the indicators to be tested and the specified model.

[0092] The test node can be provided with an interactive function for the user to perform task configuration, and based on the interactive function, the user can perform task configuration. When the test node detects the task configuration operation triggered by the user, the test task can be created according to the information configured by the task configuration operation. The information configured by the task configuration operation may include one or more of the model information, the test set information corresponding to the test data, the index to be tested, and the specified model. For example, in one embodiment, the user can upload the model file and the test set of the model to be tested, and configure the training framework type of the model to be tested, the dependent environment and other model information in the test node, as well as configure the index to be tested (i.e., the index to be tested) and specify which model of model node to use for testing.

[0093] Step S20, sending the model information of the model to be tested to the target model node, so that the target model node deploys and starts the model to be tested according to the model information;

[0094] After determining the target model node, the test node may send the model information of the model to be tested to the target model node. After receiving the model information sent by the test node, the model node may deploy the model to be tested according to the model information. Specifically, if the model information includes the identification information of the model to be tested, such as the model name of the model to be tested, the model node may obtain the model file of the model to be tested from the test node or cloud storage according to the identification information; wherein the model file may be uploaded to the test node or cloud storage by the user in advance. If the model information includes the model file of the model to be tested, the model node may directly obtain the model file from the model information. In one embodiment, the model node saves the model file and records the storage path, starts the model to be tested according to the storage path, or starts the model to be tested after receiving the start instruction sent by the test node. Among them, starting the model to be tested is to load the model file to start the process of the model to be tested. Furthermore, for models trained using different types of frameworks, different frameworks are required to load model files. In some embodiments, in order to expand the types of models that can be tested by the test system, a general framework for loading model files can be deployed in the model node. The general framework supports a variety of mainstream training framework types, that is, loading methods corresponding to different training framework types are set in the general framework; the model information sent by the test node also includes the training framework type of the model to be tested. The model node determines whether the training framework type is a type supported by the general framework. If so, the model file is loaded in the general framework according to the loading method corresponding to the training framework type to start the model to be tested. If not supported, the model node can feedback to the test node. After receiving the feedback, the test node can output prompt information to prompt manual deployment of the framework. Furthermore, in some embodiments, in order to further improve the efficiency of model testing, the model information sent by the test node may also include dependency environment information, and the dependency environment information is information about the dependency environment required to start the model to be tested. Specifically, it may include some libraries needed to start the model to be tested, such as databases or algorithm libraries; the model node installs the dependency environment according to the dependency environment information in the model information. After installation, the model file can be loaded to start the model to be tested.

[0095] Step S30, calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node;

[0096] After the model node starts the model to be tested, the test node calls the target model node to process the test data corresponding to the test task using the model to be tested. Among them, the test data corresponding to the test task can be the test data in the test set specified in the test task, or it can be the test data in the default test set when the test task is not specified, and the test data is the test data in the default test set. In one embodiment, the test node can obtain the test set corresponding to the test task, and send the test data in the test set to the target model node respectively, so as to call the target model node to process each test data using the model to be tested. In another embodiment, the model node can also obtain the test set from the test node or cloud storage, and after receiving the call of the test node, the model to be tested is used to process the data in the test set. The model node returns the model output result obtained by processing the test data using the model to be tested to the test node. Further, in one embodiment, after starting the model to be tested, the model node can return a calling interface to the test node, and the test node calls the model node through the calling interface to process the test data.

[0097] Step S40, calculating the test result of the indicator to be tested in the test task according to the output result of the model.

[0098] After obtaining the model output result returned by the target model node, the test node can calculate the test result of the indicator to be tested in the test task according to the model output result. Specifically, the calculation methods corresponding to different indicators are pre-deployed in the test node. The test node calls the calculation method corresponding to the indicator to be tested to calculate based on the model output result, and the test result of the model to be tested on the indicator to be tested can be obtained. Specifically, when the indicator to be tested is a model performance indicator, the test node can call the corresponding calculation method for calculation based on the model output result and the real labeled data corresponding to the test data; for example, for the accuracy indicator, the test node can compare the model output result corresponding to each test data with the real labeled data to determine whether the model output result is correct, that is, to determine whether the model to be tested accurately predicts the test data, and divides the data volume of the accurately predicted test data by the total data volume of the test data to obtain the accuracy, that is, to obtain the test result of the model to be tested on the accuracy indicator. When the indicator to be tested is a stress test performance indicator, the test node can call the corresponding calculation method for calculation based on the model output result and some other information. For example, the test node can calculate RT based on the time of sending test data and the time of receiving the model output results, calculate TPS based on the number of concurrency and average response time, and calculate memory usage based on the memory usage data when the target model node processes test data, and calculate CPU utilization based on the CPU utilization data when the target model node processes test data.

[0099] In this embodiment, when the test node detects a test task to test the model to be tested, the target model node is determined from multiple model nodes, and the model information of the model to be tested is sent to the target model node, so that the target model node can deploy and start the model to be tested according to the model information; the test node calls the target model node to process the test data corresponding to the test task using the model to be tested, and obtains the model output result returned by the target model node, and calculates the test result of the indicator to be tested in the test task according to the model output result. Compared with the existing solution of manually writing test code for a single model for testing, in this embodiment, by setting a distributed architecture including test nodes and model nodes, the test node automatically publishes model information based on the test task, the model node automatically deploys and loads the model, and the test node automatically obtains the model processing result to test the indicator, an automated model testing solution is provided, and the efficiency of model testing is improved.

[0100] Furthermore, in one embodiment, the test node may output the test results in a visual manner, for example, in the form of a performance report. Figure 3 As shown, the application service in the test node can call the model node to process the test data according to a certain test plan (for example, the order in which each test data is sent) (corresponding to step 1 in the figure), and the model node returns the processing result (corresponding to step 2 in the figure), and the application service records the model output result, and then calculates the test result of the indicator to be tested according to the model output result corresponding to each test data recorded in the end, obtains the performance report and outputs it. If the indicator to be tested is an indicator of the model performance type, the test node can draw a model performance curve according to the test result, for example, a PR curve (Precision Recall curve, precision-recall curve), a ROC curve (Receiver Operating Characteristic Curve, a curve for evaluating classification models) or an AUC curve (Area under the ROC curve, a curve for evaluating model classification performance), etc., and the model performance curve is included in the performance report and output. By visually outputting the test results, users can intuitively obtain the test results of the model to be tested, and then quickly adjust and optimize the model to be tested according to the test results.

[0101] In one embodiment, if Figure 4As shown, users can upload algorithm models and data sets (i.e., test sets) to test nodes or cloud storage. When testing is required, they select the data sets to be used in the data set storage pool, and configure the model performance indicators and test machines (i.e., model nodes) to be tested. The test nodes generate model test tasks based on the user's configuration operations, execute the test tasks to obtain performance reports, and finally output the performance reports.

[0102] Further, based on the above first embodiment, a second embodiment of the model testing method of the present invention is proposed. In this embodiment, step S10 includes:

[0103] Step S101, when a test task for testing a model to be tested is detected, a node whose model is a specified model in the test task among the multiple model nodes is used as a candidate model node;

[0104] When the size of the model file, the type of the model to be tested, etc. are different, the machine model required for the test may be different. Therefore, in this embodiment, the test task can specify the model of the model node used in the test, that is, the test task can include a specified model. The model can refer to the machine type information related to the machine, such as the model, memory size, processor model, etc. The user can configure the required model when configuring the task, and the test node uses the model configured by the task configuration operation as the specified model for the test task. When the test node detects a test task to test the model to be tested, the node among multiple model nodes whose model is the specified model in the test task is used as an alternative model node. For example, Figure 5 As shown, there are four different models: 2C4G, 4C8G, 8C16G and 8C32G (where C represents the number of CPU cores and G represents the memory size). If the specified model is 8C16G, the model node with the model of 8C16G can be used as an alternative model node.

[0105] Step S102: determining a target model node from candidate model nodes in an idle state.

[0106] After the test node determines the candidate model node, it can determine the target model node from the candidate model nodes in the idle state. Specifically, when there are multiple candidate model nodes in the idle state, one or more nodes can be selected as the target model node; when the number of candidate model nodes in the idle state is not enough, for example, a target model node needs to be determined but there is no candidate model node in the idle state at present, then you can wait, for example, add the test task to the waiting queue, until the number of candidate model nodes in the idle state reaches the standard, and then determine it as the target model node. Further, in one embodiment, the test node can send an inquiry message to the model node to determine whether the model node is in the idle state, or the model node can send a report message to the test node at regular intervals to report its current state. Furthermore, the model node being in the idle state can be that the model node has closed the process of the model to be tested corresponding to all previous test tasks.

[0107] In this embodiment, by setting model nodes of different models for selection, the types of models that can be tested by the test system are further expanded, so that the test system can be adapted to test more types of algorithm models. By selecting a model node in an idle state as a target model node, the target model node is not interfered by other processes when running the model to be tested, thereby improving the accuracy of the test results.

[0108] Further, based on the above-mentioned first and / or second embodiments, a third embodiment of the model testing method of the present invention is proposed. In this embodiment, when the indicator to be tested is an indicator of a model performance type, step S30 includes:

[0109] Step S301, obtaining a target test set according to the test set information corresponding to the test task;

[0110] The test node can obtain the target test set according to the test set information corresponding to the test task. The test set information is the relevant information of the test set, which can be the index information of the test set storage location or the file name of the test set and other information that can identify the test set. According to the test information, the test set can be obtained from the local storage of the test node or the cloud storage, and the obtained test set is used as the target test set. The target test set includes multiple test data and the real labeled data corresponding to each test data.

[0111] Step S302, sending each test data in the target test set to the target model node respectively, so as to call the target model node to process each test data respectively using the model to be tested, and obtain the model output results corresponding to each test data respectively returned by the target model node;

[0112] The test node can send each test data in the target test set to the target model node respectively, so as to call the target model node to use the model to be tested to process each test data respectively, obtain the model output result corresponding to each test data respectively, and return it to the test node.

[0113] The step S40 comprises:

[0114] Step S401, calculating the test results of the indicators to be tested in the test task according to the output results of each model and the real labeled data corresponding to each test data in the target test set.

[0115] After receiving the model output result returned by the target model node, the test node obtains the model output result and the real labeled data corresponding to each test data. The test node can call the corresponding calculation method to perform calculations based on the model output result and the real labeled data corresponding to the test data; for example, for the accuracy index, the test node can compare the model output result and the real labeled data corresponding to each test data to determine whether the model output result is correct, that is, to determine whether the model to be tested accurately predicts the test data, and divide the amount of accurately predicted test data by the total amount of test data to obtain the accuracy, that is, to obtain the test result of the model to be tested on the accuracy index.

[0116] In this embodiment, the target test set is obtained according to the test set information corresponding to the test task through the test node, and the target model node is called to use the model to be tested to process each test data in the target test set to obtain the model output result, and then the test result of the indicator to be tested in the test task is calculated according to the output result of each model and the real labeled data corresponding to each test data, thereby realizing the automated testing of model performance type indicators and improving the testing efficiency of model performance type indicators.

[0117] Further, based on the above-mentioned first, second and / or third embodiments, a fourth embodiment of the model testing method of the present invention is proposed. In this embodiment, the number of the target model nodes is multiple, and the step S30 includes:

[0118] Step S303, calling each of the target model nodes according to a preset load strategy to use the model to be tested to process different test data corresponding to the test task respectively, and obtaining the model output results returned by each of the target model nodes.

[0119] In this embodiment, the number of target model nodes can be multiple, that is, the test node sends the model information of the model to be tested to multiple model nodes, and each model node deploys the model to be tested respectively. The test node can call each target model node to process different test data according to the preset load strategy, that is, the test node determines which target model node to call to process those test data according to the preset load strategy. Among them, the preset load strategy can be set as needed. For example, according to the response time of each test node, a faster test data sending rate can be set for the target model node with a shorter response time, and a slower sending rate can be set for the target model node with a longer response time, so that the target model node with a shorter response time can process more test data, thereby improving the efficiency of obtaining the model output results and improving the overall test efficiency.

[0120] Further, in one embodiment, if Figure 6 As shown, the test node may include an application service (not shown in the figure) and a gateway. The application service sends a request to the gateway. After receiving the request, the gateway calls each model node to process the test data according to the customized load strategy, and calculates the test result after obtaining the model output result, and returns the test result to the application service.

[0121] In this embodiment, by selecting multiple model nodes as target model nodes, calling each target model node according to a preset load strategy to use the model to be tested to process different test data, and obtaining the model output results returned by each target model node, the test data processing efficiency is improved, thereby improving the overall test efficiency.

[0122] Further, based on the above-mentioned first, second, third and / or fourth embodiments, a fifth embodiment of the model testing method of the present invention is proposed. In this embodiment, the model testing method is applied to model nodes, and multiple model nodes are respectively connected to the test nodes for communication. The test nodes and model nodes can be devices such as smart phones, personal computers and servers, which are not specifically limited in this embodiment. In this embodiment, the model testing method includes:

[0123] Step A10, receiving the model information of the model to be tested sent by the test node, wherein when the test node detects a test task for testing the model to be tested, the test node determines a target model node from the plurality of model nodes, and sends the model information to the target model node;

[0124] The model node receives the model information of the model to be tested sent by the test node, wherein the specific implementation method of the test node sending the model information to the target model node according to the test task can refer to the above embodiments, which will not be described in detail here.

[0125] Step A20, deploying and loading the model to be tested according to the model information;

[0126] After receiving the model information, the model node deploys and loads the model to be tested according to the model information. The specific implementation method can refer to the above embodiments and will not be described in detail here.

[0127] Step A30, when receiving the call of the test node, using the model to be tested to process the test data corresponding to the test task to obtain a model output result;

[0128] When receiving a call from a test node, the model node uses the model to be tested to process the test data corresponding to the test task to obtain a model output result. The specific implementation method can refer to the above embodiments and will not be described in detail here.

[0129] Step A40, returning the model output result to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

[0130] The model node returns the model output result to the test node. The specific implementation method of the test node calculating the test result of the to-be-tested indicator in the test task according to the model output result can refer to the above embodiments, which will not be described in detail here.

[0131] Further, in one embodiment, the model information includes the model file, dependent environment information and training framework type of the model to be tested, and the step A20 includes:

[0132] Step A201, installing the dependent environment required for loading the model to be tested according to the dependent environment information;

[0133] The specific implementation of the model node installing the dependent environment required for loading the model to be tested according to the dependent environment information can be referred to the above embodiments, which will not be described in detail here.

[0134] Step A202, detecting whether the training framework type is a type supported by the general framework in the model node;

[0135] Step A203: If yes, then load the model file in the general framework to start the model to be tested.

[0136] The model node detects whether the training framework type of the model to be tested is a type supported by the general framework in the model node. If so, the model file is loaded in the general framework to start the model to be tested. The specific implementation method can refer to the above embodiments and will not be described in detail here. Further, if the training framework type of the model to be tested is not a type supported by the general framework in the model node, the model node can output a prompt message to manually deploy the framework.

[0137] In this embodiment, after receiving the model information through the model node, the model to be tested is deployed and started according to the model information, wherein, when the test node detects a test task to test the model to be tested, the target model node is determined from multiple model nodes, and the model information of the model to be tested is sent to the target model node; when the model node receives the call, the model to be tested is used to process the test data corresponding to the test task, and the processed model output result is returned to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result. Compared with the existing solution of manually writing test code for a single model for testing, in this embodiment, by setting a distributed architecture including test nodes and model nodes, the test node automatically publishes model information based on the test task, the model node automatically deploys and loads the model, and the test node automatically obtains the model processing result to test the indicator, providing an automated model testing solution and improving the efficiency of model testing.

[0138] In addition, an embodiment of the present invention further provides a model testing device, which is deployed on a test node, and the test node is communicatively connected with a plurality of model nodes, and the device includes:

[0139] A determination module, configured to determine a target model node from the plurality of model nodes when a test task for testing a model to be tested is detected;

[0140] A sending module, used for sending the model information of the model to be tested to the target model node, so that the target model node deploys and starts the model to be tested according to the model information;

[0141] A calling module, used for calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node;

[0142] The calculation module is used to calculate the test results of the indicators to be tested in the test task according to the output results of the model.

[0143] Furthermore, the determination module includes:

[0144] When a test task for testing the model to be tested is detected, a node whose machine model is the specified machine model in the test task among the multiple model nodes is used as a candidate model node;

[0145] A target model node is determined from candidate model nodes in an idle state.

[0146] Furthermore, the device also includes:

[0147] A generation module is used to create the test task according to the information configured by the task configuration operation when a task configuration operation triggered by a user is detected, wherein the information configured by the task configuration operation includes one or more of the model information, the test set information corresponding to the test data, the indicators to be tested and the specified model.

[0148] Furthermore, when the indicator to be tested is an indicator of a model performance type, the calling module includes:

[0149] An acquisition unit, used for acquiring a target test set according to the test set information corresponding to the test task;

[0150] A first calling unit is used to send each test data in the target test set to the target model node, so as to call the target model node to process each test data respectively using the model to be tested, and obtain the model output results corresponding to each test data returned by the target model node;

[0151] The calculation module comprises:

[0152] A calculation unit is used to calculate the test results of the indicators to be tested in the test task according to the output results of each model and the real labeled data corresponding to each test data in the target test set.

[0153] Furthermore, the number of the target model nodes is multiple, and the calling module includes:

[0154] The second calling unit is used to call each of the target model nodes according to a preset load strategy to use the model to be tested to process different test data corresponding to the test task respectively, and obtain the model output results returned by each of the target model nodes.

[0155] In addition, an embodiment of the present invention further provides a model testing device, which is deployed on a model node, and a plurality of the model nodes are respectively connected to a test node for communication, and the device includes:

[0156] A receiving module, configured to receive model information of a model to be tested sent by the test node, wherein when the test node detects a test task for testing the model to be tested, the test node determines a target model node from a plurality of the model nodes and sends the model information to the target model node;

[0157] A deployment module, used to deploy and start the model to be tested according to the model information;

[0158] A processing module, configured to, when receiving a call from the test node, use the model to be tested to process the test data corresponding to the test task to obtain a model output result;

[0159] The return module is used to return the model output result to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

[0160] Furthermore, the model information includes the model file, dependent environment information and training framework type of the model to be tested, and the deployment module includes:

[0161] An installation unit, used for installing the dependent environment required for starting the model to be tested according to the dependent environment information;

[0162] A detection unit, used to detect whether the training framework type is a type supported by the general framework in the model node;

[0163] A loading unit is used to, if yes, load the model file in the general framework to start the model to be tested.

[0164] The extended contents of the specific implementation manner of the model testing device of the present invention are basically the same as those of the above-mentioned embodiments of the model testing method, and will not be elaborated here.

[0165] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a model testing program is stored. When the model testing program is executed by a processor, the steps of the model testing method described below are implemented.

[0166] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the model testing method described above are implemented.

[0167] The various embodiments of the model testing device, computer-readable storage medium, and computer program product of the present invention may all refer to the various embodiments of the model testing method of the present invention, and will not be described in detail here.

[0168] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0169] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0171] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A model testing method, characterized in that: The method is applied to a test node, the test node is connected to multiple model nodes for communication, the model node is used to deploy and start running models, and the test node is used to call the model node to obtain model output results. The method includes the following steps: When a test task for testing the model to be tested is detected, determining a target model node from the plurality of model nodes; Sending the model information of the model to be tested to the target model node, so that the target model node loads the model file according to the model information, deploys and starts the model to be tested, wherein the model information includes identification information of the model to be tested or the model file of the model to be tested; Calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node; Calculate the test results of the indicators to be tested in the test task according to the output results of the model; Before the step of determining a target model node from the plurality of model nodes when a test task for testing the model to be tested is detected, the method further includes: When a task configuration operation triggered by a user is detected, the test task is created according to the information configured by the task configuration operation, wherein the information configured by the task configuration operation includes one or more of the model information, the test set information corresponding to the test data, the indicators to be tested and the specified model.

2. The model testing method according to claim 1, characterized in that: When a test task for testing a model to be tested is detected, the step of determining a target model node from the plurality of model nodes comprises: When a test task for testing the model to be tested is detected, a node whose machine model is the specified machine model in the test task among the multiple model nodes is used as a candidate model node; A target model node is determined from candidate model nodes in an idle state.

3. The model testing method according to any one of claims 1 to 2, characterized in that: When the indicator to be tested is an indicator of a model performance type, the step of calling the target model node to process the test data corresponding to the test task using the model to be tested and obtaining the model output result returned by the target model node includes: Acquire a target test set according to the test set information corresponding to the test task; Send each test data in the target test set to the target model node respectively, so as to call the target model node to process each test data respectively using the model to be tested, and obtain the model output results corresponding to each test data respectively returned by the target model node; The step of calculating the test result of the indicator to be tested in the test task according to the output result of the model includes: The test results of the indicators to be tested in the test task are calculated based on the output results of each model and the real labeled data corresponding to each test data in the target test set.

4. The model testing method according to any one of claims 1 to 2, characterized in that: The number of the target model nodes is multiple, and the step of calling the target model node to process the test data corresponding to the test task using the model to be tested, and obtaining the model output result returned by the target model node includes: According to the preset load strategy, each target model node is called to use the model to be tested to process different test data corresponding to the test task respectively, and the model output result returned by each target model node is obtained.

5. A model testing method, characterized in that: The method is applied to a model node, wherein a plurality of the model nodes are respectively connected to a test node for communication, the model node is used to deploy and start running the model, and the test node is used to call the model node to obtain the model output result, and the method comprises the following steps: Receive model information of the model to be tested sent by the test node, wherein when the test node detects a task configuration operation triggered by a user, the test task is created according to the information configured by the task configuration operation, the information configured by the task configuration operation includes the model information, the test set information corresponding to the test data, one or more of the indicators to be tested and the specified model, and when a test task for testing the model to be tested is detected, a target model node is determined from a plurality of the model nodes, and the model information is sent to the target model node; the model information includes identification information of the model to be tested or a model file of the model to be tested; Load the model file according to the model information, deploy and start the model to be tested; When receiving a call from the test node, the test data corresponding to the test task is processed using the model to be tested to obtain a model output result; The model output result is returned to the test node, so that the test node can calculate the test result of the indicator to be tested in the test task according to the model output result.

6. The model testing method according to claim 5, characterized in that: The model information includes the model file, dependent environment information and training framework type of the model to be tested, The step of deploying and starting the model to be tested according to the model information includes: Install the dependent environment required for starting the model to be tested according to the dependent environment information; Detecting whether the training framework type is a type supported by the general framework in the model node; If so, the model file is loaded in the general framework to start the model to be tested.

7. A model testing device, characterized in that: The model testing device comprises: a memory, a processor, and a model testing program stored in the memory and executable on the processor, wherein the model testing program implements the steps of the model testing method according to any one of claims 1 to 6 when executed by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a model testing program, and when the model testing program is executed by a processor, the steps of the model testing method according to any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model testing method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • TTCN-3-based distributed testing framework applicable to software system

    CN102035896A

  • Apparatus for testing network performance parameter, system thereof, and method thereof

    CN102487345A

  • Software testing method and software testing system based on distributed testing node cluster

    CN110297766A

  • MQTT server performance test method and system, computer equipment and storage medium

    CN111124850A

  • Model test method and server

    CN111476349A