Model performance prediction method, performance prediction model construction method and device

Abstract

The invention relates to the technical field of financial science and technology, in particular to a model performance prediction method and a performance prediction model construction method and device. The method comprises the following steps: determining influence factors of a to-be-predicted model; inputting the influence factors into a performance prediction model to obtain a resource consumption key prediction value and an average time consumption key prediction value required by calling a to-be-predicted model, the performance prediction model constructing a training sample set by a sample resource consumption key prediction value of a sample model, a sample average time consumption key prediction value and a plurality of influence factors, training a plurality of initial prediction models; and according to the resource consumption key prediction value and the average time consumption key prediction value, determining a resource consumption overall prediction value and an average time consumption overall prediction value of the to-be-predicted model. According to the scheme, before the model is developed, the resource overhead and performance after the model is online can be predicted, the labor burden is reduced, reworking is avoided, and the model delivery efficiency is ensured.

Description

Technical Field

[0001] This specification relates to the field of fintech, and in particular to a method for predicting model performance, a method for constructing a performance prediction model, and an apparatus therefor. Background Art

[0002] In recent years, the fields of big data and machine learning have experienced rapid development, driving innovation and transformation in multiple industries. In machine learning, the balance between model effectiveness and model performance (such as, the computational efficiency of the model, the resource consumption of the model, etc.) is a key issue, especially when deploying and running models in a production environment. Among them, model effectiveness mainly refers to the prediction ability of the model, which is usually measured by evaluation metrics such as accuracy, recall, precision, F1 value, AUC, etc. The better the model effectiveness, the higher the degree of fitting of the model to the data in the task. Model performance refers to the efficiency of the model in the real-time calculation process, involving inference time, memory occupancy, computational resource consumption, etc. A model with good performance should be able to complete inference with reasonable time and resource consumption. Finding the right balance requires comprehensive consideration of various factors such as business requirements, hardware limitations, response time requirements, etc.

[0003] Before a model is launched, stress testing and evaluation are carried out, and only when the stress test performance reaches a certain standard is it allowed to be launched. If the model fails the stress test, analysts often need to readjust the model, including modifying model parameters and retraining the model, which greatly increases the workload of all links in model production and significantly affects the delivery efficiency. Summary of the Invention

[0004] To solve the problems of the prior art, this specification provides a method for predicting model performance, a method for constructing a performance prediction model, and an apparatus therefor.

[0005] An embodiment of this specification provides a method for predicting model performance, the method comprising: determining influencing factors of a model to be predicted; inputting the influencing factors into a performance prediction model to obtain a key predicted value of resource consumption required for invoking the model to be predicted and a key predicted value of average elapsed time, wherein the performance prediction model is obtained by constructing a training sample set from sample key values of resource consumption, sample key values of average elapsed time of sample models, and multiple influencing factors, and training multiple initial prediction models; and determining an overall predicted value of resource consumption and an overall predicted value of average elapsed time of the model to be predicted according to the key value of resource consumption and the key value of average elapsed time.

[0006] According to one aspect of the embodiments of the present specification, the method further includes: determining a theoretical value of resource consumption prediction and a theoretical value of average time consumption prediction for the to-be-predicted model according to the total resource consumption when a single feature set in at least one original feature set on which the to-be-predicted model depends is called, the total resource consumption when a single derived feature set in at least one derived feature set on which the to-be-predicted model depends is called, the maximum average time consumption when a single feature set in at least one original feature set on which the to-be-predicted model depends is called, and the maximum average time consumption when a single derived feature set in at least one derived feature set on which the to-be-predicted model depends is called; determining whether the overall predicted value of resource consumption of the to-be-predicted model is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the to-be-predicted model is greater than the theoretical value of average time consumption prediction; determining the final predicted amount of resource consumption and the final predicted amount of average time consumption of the to-be-predicted model according to the comparison result between the overall predicted value of resource consumption and the theoretical value of resource consumption prediction, and the comparison result between the overall predicted value of average time consumption and the theoretical value of average time consumption prediction; if the final predicted amount of resource consumption meets the first preset condition and the final predicted amount of average time consumption meets the second preset condition, ending the performance prediction of the to-be-predicted model.

[0007] According to one aspect of the embodiments of the present specification, the determining the final predicted amount of resource consumption and the final predicted amount of average time consumption according to the comparison result between the overall predicted value of resource consumption and the theoretical value of resource consumption prediction, and the comparison result between the overall predicted value of average time consumption and the theoretical value of average time consumption prediction includes: determining whether the overall predicted value of resource consumption of the to-be-predicted model is greater than the theoretical value of resource consumption prediction, and whether the overall predicted value of average time consumption of the to-be-predicted model is greater than the theoretical value of average time consumption prediction; if not, determining the theoretical value of resource consumption prediction as the final predicted amount of resource consumption and determining the theoretical value of average time consumption prediction as the final predicted amount of average time consumption; if so, determining the overall predicted value of resource consumption as the final predicted amount of resource consumption and determining the overall predicted value of average time consumption of the to-be-predicted model as the final predicted amount of average time consumption.

[0008] According to one aspect of the embodiments of the present specification, the first preset condition includes that the final predicted amount of resource consumption does not exceed a preset resource consumption threshold; the second preset condition includes that the final predicted amount of average time consumption does not exceed a preset average time consumption threshold.

[0009] An embodiment of this specification discloses a method for constructing a performance prediction model. The performance prediction model is constructed in the following manner: Analyze the sample feature set dependency relationship of the sample model, and initiate a call to the sample model according to the sample feature set dependency relationship, where the sample feature set dependency relationship at least includes an original sample feature set and a derived sample feature set; By calling the original sample feature set and the derived sample feature set, determine the key value of sample resource consumption and the key value of sample average time consumption generated during the sample model call; Use the key value of sample resource consumption, the key value of sample average time consumption, and multiple influencing factors to construct a training sample set, and use the training sample set to train multiple initial prediction models to obtain a performance prediction model; Among them, the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0010] According to one aspect of the embodiment of this specification, analyzing the sample feature set dependency relationship of the sample model and initiating a call to the sample model according to the sample feature set dependency relationship includes: Initiate a call to the original sample feature set to obtain a first call result; According to the first call result, initiate a call to the derived sample feature set to obtain a second call result.

[0011] According to one aspect of the embodiment of this specification, by calling the original sample feature set and the derived sample feature set, determining the key value of sample resource consumption and the key value of sample average time consumption generated during the sample model call includes: Initiate a call to the original sample feature set to obtain the maximum value of the first resource consumption and the first average time consumption of the first call result; Initiate a call to the derived sample feature set to obtain the maximum value of the second resource consumption and the second average time consumption of the second call result; According to the first resource consumption and the second resource consumption, determine the key value of sample resource consumption, and according to the maximum value of the first average time consumption and the maximum value of the second average time consumption, determine the key value of sample average time consumption.

[0012] According to one aspect of the embodiment of this specification, using the key value of sample resource consumption, the key value of sample average time consumption, and multiple influencing factors to construct a training sample set and using the training sample set to train multiple initial prediction models to obtain a performance prediction model includes: Calculate multiple influencing factors according to the original sample feature set and the derived sample feature set; Input the multiple influencing factors into the multiple initial prediction models respectively to obtain the output results of each initial prediction model; According to the labels of each initial prediction model and the output results of each initial prediction model, calculate the loss value, and iteratively update the model parameters of each initial prediction model through the loss value until the performance prediction model is obtained; Among them, the first resource consumption, the second resource consumption, the maximum value of the first average time consumption, and the maximum value of the second average time consumption are respectively used as the labels of each initial prediction model.

[0013] An embodiment of this specification provides a model performance prediction device, which includes: an influencing factor determination unit for determining the influencing factors of the model to be predicted; and for inputting the influencing factors into a performance prediction model to obtain a key prediction value of resource consumption required for invoking the model to be predicted and a key prediction value of average time consumption, where the performance prediction model is obtained by constructing a training sample set from the key prediction value of sample resource consumption, the key prediction value of sample average time consumption of a sample model, and multiple influencing factors, and training multiple initial prediction models; a prediction value determination unit for determining an overall prediction value of resource consumption and an overall prediction value of average time consumption of the model to be predicted according to the key prediction value of resource consumption and the key prediction value of average time consumption.

[0014] An embodiment of this specification provides a performance prediction model construction device, which includes: a calling unit for parsing the dependency relationship of the sample feature set of the sample model and initiating a call to the sample model according to the dependency relationship of the sample feature set, where the dependency relationship of the sample feature set at least includes an original sample feature set and a derived sample feature set; a key value determination unit for determining a key value of sample resource consumption and a key value of sample average time consumption generated during the call of the sample model by calling the original sample feature set and the derived sample feature set; a training unit for constructing a training sample set by using the key value of sample resource consumption, the key value of sample average time consumption, and multiple influencing factors, and training multiple initial prediction models by using the training sample set to obtain a performance prediction model; where the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0015] An embodiment of this specification further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, it implements the model performance prediction and performance prediction model construction methods.

[0016] An embodiment of this specification further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the model performance prediction and performance prediction model construction methods.

[0017] This solution can predict the resource overhead and performance after the model goes online before model development, reduce the manual burden, avoid rework, and ensure the model delivery efficiency. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of this specification. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0019] Figure 1 The following shows a flowchart of a method for predicting model performance according to an embodiment of this specification;

[0020] Figure 2a The following shows a flowchart of a method for determining the final predicted amount of resource consumption and the final predicted average time consumption according to an embodiment of this specification;

[0021] Figure 2b The following shows a flowchart of a method for determining the final predicted amount of resource consumption and the final predicted average time consumption according to an embodiment of this specification;

[0022] Figure 3 The following shows a flowchart of a method for constructing a performance prediction model according to an embodiment of this specification;

[0023] Figure 4 The following shows a flowchart of a specific process for calling a model according to an embodiment of this specification;

[0024] Figure 5 The following shows a flowchart of a method for determining the key values of sample resource consumption and the key values of sample average time consumption generated during the process of calling a sample model according to an embodiment of this specification;

[0025] Figure 6 The following shows a flowchart of a method for constructing a performance prediction model according to an embodiment of this specification;

[0026] Figure 7 The following shows a schematic structural diagram of a model performance prediction device according to an embodiment of this specification;

[0027] Figure 8 The following shows a schematic structural diagram of a performance prediction model construction device according to an embodiment of this specification;

[0028] Figure 9 The following shows a schematic diagram of a model deployment architecture according to an embodiment of this specification;

[0029] Figure 10 The following shows a schematic diagram of the call sequence of the dependent feature set of a model to be predicted according to an embodiment of this specification;

[0030] Figure 11 The following shows a schematic structural diagram of a computer device according to an embodiment of this specification.

[0031] Description of the Attached Drawing Signs:

[0032] 701, influencing factor determination unit; 702, prediction unit; 703, predicted value determination unit; 801, calling unit; 802, key value determination unit; 803, training unit; 1102, computer device; 1104, processor; 1106, memory; 1108, drive mechanism; 1110, input / output module; 1112, input device; 1114, output device; 1116, presentation device; 1118, graphical user interface; 1120, network interface; 1122, communication link; 1124, communication bus. Detailed implementation manners

[0033] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of this specification and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0035] This specification provides method operation steps as described in the embodiments or flowcharts, but may include more or fewer operation steps based on routine or non-creative labor. The order of steps listed in the embodiments is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual system or device product is executed, it can be executed in the order shown in the embodiments or the drawings or executed in parallel.

[0036] In the related art, when using a trained model as an API service for encapsulation, services are usually provided externally in the form of API interfaces, so that users can obtain the model prediction results by calling the API. During the service provision process, the service provider processes the input data in real time and returns the prediction results, paying more attention to the model training process and the model effect, but there is no direct and effective control method for the resource consumption generated by the features on which the model depends and the overall performance of the model in the process of calling the model, and it is impossible to achieve the balance between the model effect and the model performance of the model. For this reason, this specification provides a model performance prediction method, a performance prediction model construction method and a device to solve the above technical problems.

[0037] It should be noted that the model performance prediction method, the performance prediction model construction method and the device in this specification can be used in the field of fintech technology, and can also be used in any field other than the fintech technology field, including data analysis, software development, information technology and other fields. This specification does not limit the application fields of the model performance prediction method, the performance prediction model construction method and the device.

[0038] The information, data and signals involved in this disclosure are all authorized by users or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions.

[0039] Figure 1 The figure shows a flowchart of a method for determining the overall predicted value of the resource consumption and the overall predicted value of the average time consumption of a model to be predicted in an embodiment of this specification, which at least includes the following steps 101-103, and steps 101 to 103 are executed by a model performance prediction device.

[0040] Step 101, determine the influencing factors of the model to be predicted. In this step, in order to predict the resource consumption and average time consumption of the model to be predicted, it is necessary to obtain the influencing factors of the model to be predicted.

[0041] In the embodiments of this specification, in the actual production environment, the influencing factors are the factors that cause resource competition when the feature sets that affect multiple models to be predicted are called simultaneously. In some examples, the influencing factors include, but are not limited to, resource consumption and average time consumption.

[0042] In the embodiments of this specification, resource competition refers to the situation where multiple processes or threads compete for computing resources during execution. For example, central processing unit (CPU) resource competition specifically refers to the computing resources that the CPU competes for when multiple processes or threads are executing. Since the processing power of the CPU is limited, when multiple processes simultaneously request the CPU for computing, a competition situation will occur, which may lead to a decline in system performance and an increase in response time. In a multitasking operating system, the CPU can allocate CPU time (i.e., the time spent by the computer processor when executing a specific program) to different processes or threads through time slicing or other scheduling algorithms to minimize the impact of resource competition and improve the overall efficiency and response ability of the system.

[0043] In this step, the influencing factors of the model to be predicted are calculated by monitoring parameters such as the CPU consumption and average time consumption generated when a single feature set in the original feature set and the derivative feature set (if any) on which the model to be predicted depends is called. Among them, according to the data similar to those in Table 1 and Table 2 in this specification, the parameters such as the CPU consumption and average time consumption when a single feature set in the original feature set and the derivative feature set on which the model to be predicted depends is called are determined. In the embodiments of this specification, the feature set performance tables corresponding to Table 1 and Table 2 record the parameters such as the CPU consumption and average time consumption when a single feature set of all the candidate original feature sets and derivative feature sets is called. Therefore, the parameters can be directly obtained from the tables to calculate the influencing factors.

[0044] In some embodiments, the influencing factors include, but are not limited to: the number of original feature sets on which the model to be predicted depends (denoted as num_fc), the total CPU consumption when a single feature set in the original feature sets on which the model to be predicted depends is called (denoted as sum_cpu_fc), the maximum value of the average time consumption when a single feature set in the original feature sets on which the model to be predicted depends is called (denoted as max_avg_cost_fc), the number of secondary derivative feature sets on which the model to be predicted depends (denoted as num_sfc), the total CPU consumption when a single feature set in the secondary derivative feature sets on which the model to be predicted depends is called (denoted as sum_cpu_sfc), the maximum value of the average time consumption when a single feature set in the secondary derivative feature sets on which the model to be predicted depends is called (denoted as max_avg_cost_sfc), the coefficient of variation of the CPU consumption when a single feature set in the original feature sets on which the model to be predicted depends is called (denoted as cv_cpu_fc), the range of the CPU consumption when a single feature set in the feature sets on which the model depends is called (denoted as range_cpu_fc), the coefficient of variation of the average time consumption when a single feature set in the feature sets on which the model depends is called (denoted as cv_avg_cost_fc), the range of the average time consumption when a single feature set in the feature sets on which the model depends is called (denoted as range_avg_cost_fc), the range of the CPU consumption when a single feature set in the feature sets on which the model depends is called (denoted as range_cpu_fc), the coefficient of variation of the CPU consumption when a single feature set in the secondary derivative feature sets on which the model depends is called (denoted as cv_cpu_sfc), the range of the CPU consumption when a single feature set in the secondary derivative feature sets on which the model depends is called (denoted as range_cpu_sfc), the coefficient of variation of the average time consumption when a single feature set in the secondary derivative feature sets on which the model depends is called (denoted as cv_avg_cost_sfc), the range of the average time consumption when a single feature set in the secondary derivative feature sets on which the model depends is called (denoted as range_avg_cost_sfc), the total CPU resources of the relevant production server (denoted as sum_cpu_total), the concurrency number corresponding to the performance index (the number of customers supported to initiate requests simultaneously) (denoted as num_concurrency), etc. The method for determining the influencing factors of the model to be predicted in this step is similar to the solution of the embodiments shown in Figure 6 which will not be elaborated here as it is similar to the solution of the embodiments shown later.

[0045] Step 102: Input the influencing factors into the performance prediction model to obtain the key predicted value of resource consumption and the key predicted value of average time consumption required for calling the model to be predicted. The performance prediction model is constructed by using the sample key predicted value of resource consumption, the sample key predicted value of average time consumption of the sample model, and multiple influencing factors to form a training sample set, and training multiple initial prediction models.

[0046] In this specification, the performance prediction model is used to predict the resource consumption and average time consumption required to call the model to be predicted before the model goes online, so as to determine the model performance. For the specific description of building the performance prediction model, see the following embodiments Figures 3 - 6 for the described performance prediction model building method.

[0047] In this step, the influencing factors of the model to be predicted are used as the feature data of the performance prediction model. After inputting into the performance prediction model, the performance prediction model outputs the prediction results related to the resource consumption and time consumption to be occupied when calling the model to be predicted, that is, the prediction results are the resource consumption prediction value and average time consumption prediction value for the model to be predicted. Specifically, the performance prediction model includes 4 prediction models, namely the first prediction model, the second prediction model, the third prediction model and the fourth prediction model. In this step, the influencing factors of the model to be predicted are respectively input into these 4 prediction models, and the results output by the 4 prediction models are: the output of the first prediction model (denoted as sum_cpu_fc_Y), the output of the second prediction model (denoted as sum_cpu_sfc_Y), the output of the third prediction model (denoted as max_avg_cost_fc_Y) and the output of the fourth prediction model (denoted as max_avg_cost_sfc_Y). Among them, sum_cpu_fc_Y and sum_cpu_sfc_Y are the key prediction values of resource consumption, and max_avg_cost_fc_Y and max_avg_cost_sfc_Y are the key prediction values of average time consumption.

[0048] Step 103: Determine the overall resource consumption prediction value and overall average time consumption prediction value of the model to be predicted according to the key prediction values of resource consumption and key prediction values of average time consumption.

[0049] In the embodiments of this specification, according to the sequential process of the invocation model, the overall predicted value of the resource consumption (cpu_predict) of the model to be predicted includes: the cpu consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model to be predicted, the cpu consumption generated by the simultaneous invocation of at least one derived feature set, and the total cpu consumption generated by its own algorithm and the API entry; the overall predicted value of the average time consumption of the model to be predicted is the sum of the maximum value of the average time consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model to be predicted, the maximum value of the average time consumption generated by the simultaneous invocation of at least one derived feature set, the average time consumption of its own algorithm, and the average time consumption of the API entry. Among them, the key predicted value of resource consumption includes the cpu consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model to be predicted, and the cpu consumption generated by the simultaneous invocation of at least one derived feature set relied on by the model to be predicted. The key predicted value of average time consumption includes the maximum value of the average time consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model to be predicted and the maximum value of the average time consumption generated by the simultaneous invocation of the derived feature set relied on by the model to be predicted.

[0050] In this step, determining the overall predicted value of the resource consumption and the overall predicted value of the average time consumption of the model to be predicted according to the key value of resource consumption and the key value of average time consumption specifically includes: inputting the key value of resource consumption and the key value of average time consumption into the overall predicted model of resource consumption and the overall predicted model of average time consumption respectively, so as to determine the overall predicted value of resource consumption and the overall predicted value of average time consumption.

[0051] Among them, the overall predicted model of the resource consumption of the model to be predicted can be expressed as follows:

[0052]

[0053] Among them, SUM() represents summation, cpu() represents listing the resource consumption amounts of multiple objects in the parentheses. cpu(original feature set relied on by the model to be predicted) is the cpu consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model to be predicted. SUM(cpu(original feature set relied on by the model to be predicted)) is the sum of the cpu consumption generated by the simultaneous invocation of at least one original feature set finally relied on by the model. cpu(derived feature set relied on by the model to be predicted) is the cpu consumption generated by the simultaneous invocation of at least one derived feature set relied on by the model. SUM(cpu(derived feature set relied on by the model to be predicted)) is the sum of the cpu consumption generated by the simultaneous invocation of at least one derived feature set relied on by the model.

[0054] The key predicted values of resource consumption in this step include SUM(cpu(original feature set on which the model to be predicted depends)) and SUM(cpu(derived feature set on which the model to be predicted depends)); OneOf() represents the cpu consumption generated by the algorithm of the model to be predicted itself, and specific constant values are selected according to the different algorithms selected for the model to be predicted; api_cpu represents the cpu consumption generated when the api interface invokes the model to be predicted with the original feature set and the derived feature set as parameters.

[0055] In this step, the overall prediction model of the average time consumption of the model to be predicted can be expressed as follows:

[0056]

[0057] Among them, MAX() represents finding the maximum value, avg_cost() represents listing the average time consumption of multiple objects in the parentheses. avg_cost(original feature set on which the model to be predicted depends) is the average time consumption generated when at least one original feature set on which the model to be predicted finally depends is called simultaneously. MAX(avg_cost(original feature set on which the model to be predicted depends)) is the average time consumption generated when at least one original feature set on which the model to be predicted finally depends is called simultaneously. avg_cost(derived feature set on which the model to be predicted depends) is the maximum value of the average time consumption generated when at least one derived feature set on which the model to be predicted depends is called simultaneously. MAX(avg_cost(derived feature set on which the model to be predicted depends)) is the maximum value of the average time consumption generated when at least one derived feature set on which the model to be predicted depends is called simultaneously.

[0058] The key predicted values of the average time consumption in this step include MAX(avg_cost(original feature set on which the model to be predicted depends)) and MAX(avg_cost(derived feature set on which the model to be predicted depends)); OneOf() represents the average time consumption generated by the algorithm of the model to be predicted itself, and specific constant values are selected according to the different algorithms selected for the model to be predicted; api_cost represents the average time consumption generated when the api interface invokes the model to be predicted with the original feature set and the derived feature set as parameters.

[0059] In this step, the key predicted values of resource consumption are substituted into the overall prediction formula of the resource consumption of the model to be predicted to determine the overall predicted value of the resource consumption of the model to be predicted. Among them, the overall prediction process of the resource consumption of the model to be predicted is expressed as follows:

[0060]

[0061] Among them, cpu_predict_model represents the overall predicted value of the resource consumption of the model to be predicted, sum_cpu_fc_Y represents the output of the first prediction model, sum_cpu_sfc_Y represents the output of the second prediction model, and the outputs of the first prediction model and the second prediction model constitute the key predicted value of resource consumption. OneOf(lr_cpu, rf_cpu, dt_cpu) represents the cpu consumption generated by itself according to the algorithm selected for the model to be predicted, and api_cpu represents the cpu consumption generated by the api interface when initiating a call to the model to be predicted with the original feature set and the derived feature set as parameters.

[0062] In this step, the key predicted value of the average time consumption is substituted into the overall prediction formula of the average time consumption of the model to be predicted to determine the key predicted value of the average time consumption of the model to be predicted. Among them, the overall prediction process of the average time consumption of the model to be predicted is shown as follows:

[0063]

[0064] Among them, avg_cost_predict_model represents the overall predicted value of the average time consumption of the model to be predicted, max_avg_cost_fc_Y represents the output of the third prediction model, max_avg_cost_sfc_Y represents the output of the fourth prediction model, and max_avg_cost_fc_Y and max_avg_cost_sfc_Y constitute the key predicted value of the average time consumption. api_cost represents the average time consumption generated by the api interface when initiating a call to the model to be predicted with the original feature set and the derived feature set as parameters.

[0065] Therefore, in this step, the key predicted value of resource consumption, the resource consumption determined according to its own algorithm, and the resource consumption at the API entry are added up to determine the overall predicted value of the resource consumption of the model to be predicted; the key predicted value of the average time consumption, the average time consumption determined according to its own algorithm, and the average time consumption at the API entry are added up to determine the overall predicted value of the average time consumption of the model to be predicted.

[0066] This specification determines how to better allocate server resources to each model to be predicted by predicting the resource consumption and average time consumption after each model to be predicted goes online in a resource competition situation, so as to optimize the model performance and model effect of each model to be predicted, make each model to be predicted meet the resource budget and service throughput capacity expectations, balance the model effect and model performance of the model to be predicted, greatly reduce the workload of each link in model production, and significantly improve the delivery efficiency.

[0067] Figure 2aThe following is a flowchart of a method for determining the predicted value of the final resource consumption and the predicted value of the final average time consumption according to an embodiment of this specification. In this specification, after calculating the overall predicted value of the resource consumption and the overall predicted value of the average time consumption of the model to be predicted, it is necessary to further determine their accuracy. Therefore, it is necessary to judge by combining the theoretical values of the resource consumption prediction and the theoretical values of the average time consumption prediction when there is no resource competition. Figure 2a The method at least includes the following steps 201-207.

[0068] Step 201: Determine the theoretical predicted value of the resource consumption and the theoretical predicted value of the average time consumption of the model to be predicted according to the sum of the resource consumption when a single feature set in at least one original feature set on which the model to be predicted depends is called, the sum of the resource consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is called, the maximum average time consumption when a single feature set in at least one original feature set on which the model to be predicted depends is called, and the maximum average time consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is called.

[0069] In the embodiment of this specification, when resource competition is not considered, the monitoring tool is used in advance to separately count the cpu resource consumption and the average time consumption generated when each feature set or derivative sample feature set in all the original sample feature sets and derivative sample feature sets available for constructing the model to be predicted is separately called, and pre-store them in the system. See Example Tables 1 and 2. Among them, Table 1 is the performance table of the original sample feature set, and Table 2 is the performance table of the secondary derivative sample feature set. Subsequently, the analyst selects some original sample feature sets and derivative sample feature sets from all the original sample feature sets and derivative sample feature sets recorded in the table to construct the model to be predicted. After determining the dependent features of the model to be predicted, the system calculates the sum of the resource consumption generated when each dependent original feature set of the model to be predicted is called individually (denoted as sum_cpu_fc in this specification) and the maximum average time consumption, the sum of the resource consumption and the maximum average time consumption when each derivative feature set finally depended on by the model to be predicted is called individually according to the cpu resource consumption and the average time consumption data monitored in advance corresponding to each feature set recorded in the table, and inputs these data into the overall resource consumption prediction model and the overall average time consumption prediction model to obtain the theoretical predicted value of the resource consumption and the theoretical predicted value of the average time consumption, denoted as cpu_predict_fallback and avg_cost_predict_fallback respectively. In this specification, different models to be predicted correspond to different theoretical predicted values of the resource consumption and the theoretical predicted values of the average time consumption. Therefore, the theoretical predicted values of the resource consumption and the theoretical predicted values of the average time consumption in this specification can be understood as dynamically changing values, rather than constant values.

[0070] Table 1

[0071]

[0072]

[0073] Table 2

[0074]

[0075] In this specification, the data monitored by the monitoring tool is the data when each feature set is separately called (for example, when each feature set is deployed on different servers). There is no resource competition among the individually called feature sets. However, in the actual process of calling the model, it is inevitable that multiple original feature sets and derived feature sets on which the model depends are stored on the same server. Therefore, there must be resource competition in the actual process of calling the model. That is to say, when actually considering resource competition, the resource consumption generated by simultaneously calling each feature set will be higher than the resource consumption generated by individually calling a feature set due to resource competition. For example, when multiple feature sets deployed on the same group of servers are simultaneously called, multiple feature sets compete for the CPU resources of the same group of servers, and the call time of each feature set will increase, and the generated CPU resource consumption will also increase. Therefore, the CPU resource consumption and average time consumption generated when each feature set or derived sample feature set in all the original sample feature sets and derived sample feature sets used to construct the model to be predicted is separately called, which are statistically obtained by using the monitoring tool without considering resource competition, are used as the theoretical values of resource consumption prediction and the theoretical values of average time consumption prediction. The theoretical values of resource consumption prediction and the theoretical values of average time consumption prediction can be used as the theoretical values of resource consumption prediction and the theoretical values of average time consumption prediction (the theoretical values can also be lower limit values) to determine whether the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted are within the normal range, so as to determine whether the performance prediction result of the model to be predicted is within the normal range.

[0076] As shown below, among them, the parameters on the right side of the equal sign in the four equations are all the data monitored by the monitoring tool, and the right side is the parameters input into the theoretical value calculation formula.

[0077] SUM(cpu(original feature set on which the model to be predicted depends)) = sum_cpu_fc

[0078] SUM(cpu(derived feature set on which the model to be predicted depends)) = sum_cpu_sfc

[0079] MAX(avg_cost(original feature set on which the model to be predicted depends)) = max_avg_cost_fc

[0080] MAX(average cost (derived feature set on which the model to be predicted depends)) = max_avg_cost_sfc

[0081]

[0082] Among them, cpu_predict_fallback represents the theoretical value of the resource consumption prediction of the model to be predicted, sum_cpu_fc represents the total cpu consumption when a single original feature set in the original feature set on which the model to be predicted depends is called without resource competition, sum_cpu_sfc represents the total cpu consumption when a single derived feature set in the derived feature set on which the model to be predicted depends is called without resource competition, OneOf(lr_cpu, rf_cpu, dt_cpu) represents the total cpu consumption of the algorithm of the model to be predicted itself without resource competition, and api_cpu represents the cpu consumption of the api interface without resource competition.

[0083]

[0084] Among them, avg_cost_predict_fallback represents the theoretical value of the average time consumption prediction of the model to be predicted. The theoretical value of the resource consumption prediction and the theoretical value of the average time consumption prediction are used to evaluate whether the performance prediction result of the model to be predicted belongs to the normal range. max_avg_cost_fc represents the maximum average time consumption when a single original feature set in the original feature set on which the model to be predicted depends is called without resource competition, max_avg_cost_sfc represents the maximum average time consumption when a single derived feature set in the derived feature set on which the model to be predicted depends is called individually without resource competition, OneOf(lr_cost, rf_cost, dt_cost) represents the average time consumption of the algorithm of the model to be predicted itself without resource competition, and api_cost represents the average time consumption of the api interface without resource competition.

[0085] Step 202: Determine the final predicted resource consumption and the final predicted average time consumption of the model to be predicted according to the comparison result between the overall predicted resource consumption value and the theoretical value of the resource consumption prediction, and the comparison result between the overall predicted average time consumption value and the theoretical value of the average time consumption prediction.

[0086] The specific process of determining the final predicted resource consumption and the final predicted average time consumption of the model to be predicted is as Figure 2b shown, Figure 2b This is a flowchart of a method for determining the final predicted resource consumption and the final predicted average time consumption according to an embodiment of this specification, including at least Step 2021 to Step 2023.

[0087] In step 2021, determine whether the overall predicted value of the resource consumption of the model to be predicted is greater than the theoretical predicted value of the resource consumption, and whether the overall predicted value of the average time consumption of the model to be predicted is greater than the theoretical predicted value of the average time consumption.

[0088] As described above, in the actual consideration of resource competition, the resource consumption of each feature set will be higher due to resource competition than when a single feature set is called. Therefore, in this step, compare the overall predicted value of the resource consumption of the model to be predicted calculated in step 103 with the theoretical predicted value of the resource consumption, that is, judge the magnitudes of cpu_predict_model and cpu_predict_fallback. Compare the overall predicted value of the average time consumption of the model to be predicted calculated in step 103 with the theoretical predicted value of the average time consumption, that is, judge the magnitudes of avg_cost_predict_model and avg_cost_predict_fallback.

[0089] In step 2022, if the answer is no, determine the theoretical predicted value of the resource consumption as the final predicted amount of the resource consumption, and determine the theoretical predicted value of the average time consumption as the final predicted amount of the average time consumption.

[0090] In step 2023, if the answer is yes, determine the overall predicted value of the resource consumption as the final predicted amount of the resource consumption, and determine the overall predicted value of the average time consumption of the model to be predicted as the final predicted amount of the average time consumption.

[0091] In this specification, if cpu_predict_model is greater than cpu_predict_fallback, then cpu_predict_model is the finally predicted cpu resource consumption amount, that is, the final predicted amount of the resource consumption, indicating that the prediction result of the performance prediction model conforms to the common sense cognition under resource competition. Otherwise, use the theoretical value cpu_predict_fallback as the final predicted amount of the resource consumption.

[0092] In this specification, if avg_cost_predict_model is greater than avg_cost_predict_fallback, then avg_cost_predict_model is the final predicted amount of the average time consumption.

[0093] In step 203, if the final predicted amount of the resource consumption meets the first preset condition and the final predicted amount of the average time consumption meets the second preset condition, then end the performance prediction of the model to be predicted.

[0094] Among them, the first preset condition includes that the predicted final resource consumption does not exceed the preset resource consumption threshold; the second preset condition includes that the predicted final average time consumption does not exceed the preset average time consumption threshold.

[0095] In this step, the predicted final resource consumption is continuously compared with the preset resource consumption threshold set by the system to determine whether the predicted final resource consumption meets the first preset condition; the predicted final average time consumption is compared with the preset resource consumption threshold set by the system to determine whether the predicted final average time consumption meets the second preset condition. If both are met, it indicates that the performance of the model to be predicted is good in terms of resource consumption and average time consumption, and there is no need to adjust the feature set of the model to be predicted, and the performance prediction of the model to be predicted is completed. Analysts can use the current model to be predicted to carry out subsequent work. If not met, it indicates that the performance of the model to be predicted in terms of resource consumption and average time consumption does not meet the expectation, and it is necessary to further adjust the original feature set or derivative feature set on which the model to be predicted depends for re-prediction until the prediction result meets the performance standard.

[0096] In this specification, the preset resource consumption threshold can be set to 1000 millicores, and the preset average time consumption threshold can be set to 200 milliseconds. These two thresholds can be constant standards. Of course, for different systems or scenarios, the thresholds can also be adaptively set, which is not limited in this specification.

[0097] In this specification, if the predicted final resource consumption and the predicted final average time consumption of the model to be predicted meet the standards set by the system, it indicates that the model performance and efficiency of the model to be predicted meet the service throughput capacity expectation, and the performance prediction of the model to be predicted can be completed and subsequent work can be carried out normally. Otherwise, it is necessary to adjust the list of the original feature set and the derivative feature set of the model to be predicted, re-select the appropriate original feature set or derivative feature set, re-execute the steps described above, and re-predict the model to be predicted until the prediction result meets the performance standard. Before model development, this specification predicts the resource overhead and performance after the prediction model is launched, which is beneficial to making good resource budgets and service throughput capacity expectations, reducing the burden on analysts, and improving the model optimization efficiency.

[0098] In this specification, the system can also, after the model to be predicted is launched, use the model to be predicted to regularly predict and determine the newly added models that finally meet the standards for going online (the newly added models can be called incremental models), regularly capture the actual data of the performance test of the incremental models, evaluate the effects of the four prediction models, and add the actual data of the incremental models to the training sample set of the prediction models to continuously optimize the prediction models.

[0099] Figure 3 The figure shows a flowchart of a method for constructing a performance prediction model according to an embodiment of this specification, and the method includes at least 301-303. Figure 3The method is executed by a performance prediction model construction device in the system.

[0100] Step 301: Analyze the sample feature set dependency relationship of the sample model, and initiate a call to the sample model according to the sample feature set dependency relationship, where the sample feature set dependency relationship at least includes an original sample feature set and a derived sample feature set.

[0101] In the embodiments of this specification, the sample model is a pre-trained calculation model. By analyzing the configuration of the sample model, the sample feature set dependency relationship of the sample model can be determined. Both the original sample feature set and the derived sample feature set in this step are sample feature data sets. Among them, the sample feature set dependency relationship includes: the original sample feature set and the derived sample feature set. The original sample feature set and the derived sample feature set belong to the feature data used by the sample model during training. The feature parameters of the sample model come from the original sample feature set and the derived sample feature set. The sample feature set dependency relationship records the dependency relationship between the sample model and the original sample feature set, the dependency relationship between the sample model and the derived sample feature set, and the dependency relationship between the original sample feature set and the derived sample feature set.

[0102] Among them, for a series of original data that changes over time, a pre-developed derivative algorithm is used for real-time operation to obtain a series of original features. These original features are used to describe certain attributes or characteristics of the data, and the set of these features is called the original feature set. For several original feature sets, a pre-developed derivative algorithm is used for real-time operation to obtain a series of new features, and this new feature set is called the secondary derivative feature set. Among them, the derived sample feature set can include multiple derived sample feature sets. For example, the secondary derived sample feature set, the tertiary derived sample feature set, etc. In this specification, the secondary derived sample feature set is used as an example to represent the derived sample feature set.

[0103] In a machine learning model, the selection and invocation of feature data are crucial, directly affecting the performance and accuracy of the model. In the embodiments of this specification, it is assumed that there are 3 existing sample models, namely "Model 1", "Model 2", and "Model 3". Among them, if the customer's requirement is to invoke "Model 2", then first, by analyzing the sample feature set dependency relationship of Model 2, it can be determined that "Model 2" depends on the original sample feature set 1, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N. Further, according to the sample feature set dependency relationship, it can be determined that the secondary derivative sample feature set 2 depends on the original sample feature set 1 and the original sample feature set N, and the secondary derivative sample feature set N depends on the original sample feature set 1 and the original sample feature set 2. Therefore, "Model 2" ultimately depends on the original sample feature set 1, the original sample feature set 2, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N. When the customer invokes the API to obtain the model prediction result, the invocation order of the dependent feature set of Model 2 of the service provider is shown as Figure 10 shown.

[0104] Step 302: By invoking the original sample feature set and the derivative sample feature set, determine the key value of sample resource consumption and the key value of sample average time consumption generated during the sample model invocation process.

[0105] In the embodiments of this specification, invoking a sample model is essentially invoking the feature data of the sample model, specifically: invoking the original sample feature set and the derivative sample feature set of the sample model. Based on the embodiments in Step 101, the process of invoking "Model 2" actually includes the process of invoking the original sample feature set 1, the original sample feature set 2, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N.

[0106] In this specification, invoking a sample model includes multiple steps, at least including: invoking the original sample feature set; invoking the secondary derivative sample feature set according to the result of invoking the original sample feature set; invoking the model according to the invocation result of the secondary derivative sample feature set and the invocation result of the original sample feature set, and other steps.

[0107] Correspondingly, the resource consumption of "Model 2" is the sum of the CPU consumption of the original sample feature set that it ultimately relies on, the secondary derivative sample feature set that it ultimately relies on, its own algorithm, and the API entry; the average time consumption of "Model 2" is the sum of the maximum value of the average time consumption of the original sample feature set that it ultimately relies on, the maximum value of the average time consumption of the secondary derivative sample feature set that it ultimately relies on, the average time consumption of its own algorithm, and the average time consumption of the API entry. Among them, the maximum value of the resource consumption and average time consumption of the original sample feature set that "Model 2" ultimately relies on and the secondary derivative sample feature set that it ultimately relies on are difficult to predict. Therefore, it is necessary to count or determine the key values ​​of sample resource consumption and average time consumption corresponding to these two key steps for the subsequent construction of a performance prediction model.

[0108] Step 303: construct a training sample set using the sample resource consumption key value, the sample average time consumption key value and multiple influencing factors, and use the training sample set to train multiple initial prediction models to obtain a performance prediction model.

[0109] Among them, the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption. In this specification, the purpose of building a performance prediction model is to predict the resource consumption and average time consumption required for the model to be predicted to be called before the model to be predicted goes online, so as to determine the performance of the model to be predicted, so that analysts can preliminarily screen the model features that can make the model performance meet the standards, and use the screened model features to focus on improving the model effect at work. While further reducing the burden on analysts, it avoids rework and re-forecasting of the model to be predicted due to non-compliance with performance standards, improves model delivery efficiency, and achieves a balance between model performance and model effect.

[0110] After determining the sample resource consumption key value and the sample average time consumption key value of the key steps in the sample model calling process in step 102, the sample resource consumption key value and the sample average time consumption key value together with multiple influencing factors of the sample model constitute a training sample set, and the training sample set is used to train multiple initial prediction models to construct a performance prediction model. When the model to be predicted is put into production, the feature set and performance data on which the model to be predicted depends can be further used to train the performance prediction model.

[0111] Combination Figure 10 The following is a specific process of calling the dependent feature set of a model to be predicted according to an embodiment of this specification. Taking the embodiment of "Model 2" in step 301 as an example, after the client initiates a request to call Model 2, it includes multiple calling sub-steps, as follows:

[0112] Sub-step 1: The customer initiates a call to "Model 2";

[0113] Sub-step 2: After the service provider's API entry parses the dependencies of "Model 2", it simultaneously initiates calls to the original sample feature set 1, the original sample feature set 2, and the original sample feature set N.

[0114] Sub-step 3: After the service provider's API entry obtains the call results of the original sample feature set 1, the original sample feature set 2, and the original sample feature set N, it simultaneously initiates calls to the secondary derivative sample feature set 2 and the secondary derivative sample feature set N;

[0115] Sub-step 4: The service provider's API entry uses the obtained results of the original sample feature set 1, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N as parameters to initiate a call to Model 2.

[0116] Sub-step 5: The service provider's entry returns the obtained results of Model 2 to the customer.

[0117] According to the foregoing description, for various different types of models, it is difficult to predict the maximum values of the resource consumption and average time consumption of sub-steps 2 and 3 involved in calling the model. Therefore, this specification needs to count the resource consumption and average time consumption of sub-step 2 and the resource consumption and average time consumption of sub-step 3 during the sample model call process, for constructing a prediction model to provide prediction services for more types of models to be predicted.

[0118] In another embodiment, the process of calling the model can be implemented through Figure 4 the embodiment shown in Figure 4 The embodiment shown at least includes the following steps 401-402.

[0119] Step 401: Initiate a call to the original sample feature set to obtain a first call result. This step corresponds to sub-step 2. After the service provider's API entry parses the dependencies of "Model 2", it simultaneously initiates calls to the original sample feature set 1, the original sample feature set 2, and the original sample feature set N. In this step, part of the input parameters can be used to obtain the original data, and the original data is analyzed and processed to obtain the original sample feature set. For example, if "Model 2" is a model related to credit investigation, by inputting user information (such as account, contact information, etc.), the corresponding original transaction data, operation records, and other data of the user can be queried. Further analyzing and calculating the original transaction data, operation records, and other original data, the features of the user's information (such as operation frequency, consumption habits, etc.) can be obtained, and these features belong to the original sample feature set in this specification. Initiating a call to the original sample feature set, the obtained call result is called the first call result.

[0120] Step 402: Initiate a call to the derived sample feature set based on the first call result, and obtain a second call result. In this step, corresponding to sub-step 3, initiate a simultaneous call to the secondary derived sample feature set based on the first call result, and the second call result obtained is the result after analyzing the secondary derived feature set. For example, the first call result includes the number of transactions the user makes every 7 days and the number of transactions the user makes every 1 month, so that the change trend, coefficient of variation, etc. of the number of transactions of the user every 7 days and every month within the time window can be derived and calculated.

[0121] In the embodiments of this specification, the process of model call further includes: initiating a call to the sample model based on the first call result and the second call result, and returning the final call result. In this step, based on the results obtained by separately calling the original sample feature set and the derived sample feature set in the foregoing steps, initiate a call to the model. For example, the results obtained by calling the original sample feature set and the derived sample feature set can be used as parameters to initiate a call to the model to obtain the final call result. In the embodiments of this specification, the original sample feature set and the secondary derived sample feature set have a great impact on the prediction accuracy of the to-be-predicted model to be constructed subsequently. Therefore, it is necessary to record the resource consumption and average time consumption occupied during the call process through a monitoring tool.

[0122] Figure 5 The figure shows a flowchart of a method for determining the key values of sample resource consumption and sample average time consumption generated during the sample model call process in the embodiments of this specification, which specifically includes the following steps 501-503.

[0123] According to Figure 4In the embodiments herein, the resource consumption of sample model 2 is the sum of the CPU consumption of the original sample feature set on which sample model 2 depends being called simultaneously, the secondary derivative sample feature set on which sample model 2 depends being called simultaneously, the own algorithm of sample model 2, and the API entry for calling sample model 2; the average time consumption is the sum of the maximum value of the average time consumption of the original sample feature set on which sample model 2 depends being called simultaneously, the maximum value of the average time consumption of the secondary derivative sample feature set on which sample model 2 depends being called simultaneously, the average time consumption of the own algorithm of sample model 2, and the average time consumption of the API entry for calling sample model 2. For the resource consumption and average time consumption generated by calling sub-step 4 in sample model 2, in common machine learning algorithms, they can be classified by algorithm and summarized as constants. For the resource consumption and average time consumption generated by calling sub-step 5 in sample model 2, this step can also be summarized as constant resource consumption and time consumption in the actual production process. And through the inventor's research, it is found that the proportion of sub-steps 4 and 5 in the resource consumption and average time consumption of the prediction target is much smaller than that of the original sample feature set and the secondary derivative sample feature set part, and has a very small impact on the prediction accuracy. Therefore, the embodiments of this specification only need to count the resource consumption and average time consumption corresponding to sub-step 2 and sub-step 3 during the call of the sample model, and call the statistical results the key values of sample resource consumption and the key values of sample average time consumption.

[0124] Step 501, initiate a call to the original sample feature set to obtain the maximum value of the first resource consumption and the first average time consumption of the first call result.

[0125] In this specification, by using a monitoring tool, the sum of the resources consumed when simultaneously initiating calls to each original sample feature set on which the sample model depends is counted as the first resource consumption. On the other hand, the average time consumption of each original sample feature set when simultaneously calling each original sample feature set is counted. In some embodiments, the maximum average time consumption is selected from multiple average time consumptions as the maximum value of the first average time consumption.

[0126] Step 502, initiate a call to the derivative sample feature set to obtain the maximum value of the second resource consumption and the second average time consumption of the second call result. In this specification, by using a monitoring tool, the sum of the resources consumed when simultaneously initiating calls to each derivative sample feature set on which the sample model depends is counted as the second resource consumption. On the other hand, the average time consumption of each derivative sample feature set when simultaneously calling each derivative sample feature set is counted. In some embodiments, the maximum average time consumption is selected from multiple average time consumptions as the maximum value of the second average time consumption. In this specification, the first resource consumption and the second resource consumption include CPU consumption resources, such as CPU time.

[0127] Step 503: Determine the sample resource consumption key value according to the first resource consumption and the second resource consumption, and determine the sample average time consumption key value according to the maximum value of the first average time consumption and the maximum value of the second average time consumption. The sample resource consumption key value in this step is Figure 4 the resource consumption corresponding to invocation sub-step 2 and invocation sub-step 3 in the embodiment. The sample average time consumption key value in this step is Figure 4 the average time consumption corresponding to invocation sub-step 2 and invocation sub-step 3 in the embodiment. These two types of data are obtained through statistics as known parameters and are also used as the training sample set for training the initial prediction model to guide the iterative update of the initial prediction model. Finally, a performance prediction model is constructed, thereby forming an effective tool for predicting the resource overhead of the model to be predicted before it goes online, and achieving a balance between the model effect and the model performance of the model to be predicted.

[0128] Figure 6 The figure shows a flowchart of a method for constructing a performance prediction model according to an embodiment of the present specification, which at least includes the following steps 601-604.

[0129] Step 601: Calculate multiple influencing factors according to the original sample feature set and the derived sample feature set.

[0130] In the embodiment of the present specification, in an actual production environment, since the calculation models for invoking the original sample feature set and the secondary derived sample feature set are deployed in the same group of servers, the four indicators corresponding to the sample resource consumption key value and the sample average time consumption key value will be affected by resource competition. Therefore, it is necessary to determine the factors that affect resource competition when multiple feature sets are invoked simultaneously, that is, the influencing factors.

[0131] In this specification, the influencing factors determining the resource competition of each indicator include, but are not limited to: the number of original sample feature sets on which the sample model depends (denoted as num_fc), the total CPU consumption when a single call is made to the original sample feature set on which the model depends (denoted as sum_cpu_fc), the coefficient of variation of the CPU consumption when a single call is made to the original sample feature set on which the sample model depends (denoted as cv_cpu_fc), the range of the CPU consumption when a single call is made to the original sample feature set on which the sample model depends (denoted as range_cpu_fc), the maximum value of the average time consumption when a single call is made to the original sample feature set on which the sample model depends (denoted as max_avg_cost_fc), the coefficient of variation of the average time consumption when a single call is made to the original sample feature set on which the sample model depends (denoted as cv_avg_cost_fc), the range of the average time consumption when a single call is made to the original sample feature set on which the sample model depends (denoted as range_avg_cost_fc), the number of secondary derivative sample feature sets on which the sample model depends (denoted as num_sfc), the total CPU consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as sum_cpu_sfc), the coefficient of variation of the CPU consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as cv_cpu_sfc), the range of the CPU consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as range_cpu_sfc), the maximum value of the average time consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as max_avg_cost_sfc), the coefficient of variation of the average time consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as cv_avg_cost_sfc), the range of the average time consumption when a single call is made to the secondary derivative sample feature set on which the sample model depends (denoted as range_avg_cost_sfc), the total amount of CPU resources of the relevant production server (denoted as sum_cpu_total), and the concurrency number corresponding to the performance indicator (the number of requests that support customers to initiate simultaneously) (denoted as num_concurrency).

[0132] In the embodiments of this specification, in order to determine the sample resource consumption and sample average time consumed during the invocation of the sample model Score1, when the model is launched, the data in Table 1 and Table 2 below (Table 1 and Table 2 have already appeared in step 201) are obtained through performance testing. Among them, Table 1 is the performance table of the original sample feature set, and Table 2 is the performance table of the secondary derivative sample feature set. Among them, the cpu corresponding to the feature sets in Table 1 and Table 2 refers to the cpu resource consumption, and avg cost refers to the average time consumed. These parameters are monitored by a monitoring tool, indicating the cpu consumption during a single invocation of the original sample feature set, the average time consumed during a single invocation of the original sample feature set, the cpu consumption during a single invocation of the secondary derivative sample feature set, and the average time consumed.

[0133] Table 1

[0134]

[0135] Table 2

[0136]

[0137] Taking the sample model Score1 as an example, by analyzing the dependency relationship of the sample feature set of the sample model Score1, it can be determined that the sample feature sets on which the sample model Score1 depends include: the original sample feature set 1, the original sample feature set 5, the original sample feature set 13, the original sample feature set 22, the secondary derivative sample feature set 2, and the secondary derivative sample feature set 12. Further, a list of the secondary derivative sample feature sets on which the sample model Score1 depends is determined, as shown in Table 3 below, the dependency feature set table of the secondary derivative sample feature set.

[0138] Table 3

[0139]

[0140] It can be seen that for the two secondary derivative sample feature sets (the secondary derivative sample feature set 2 and the secondary derivative sample feature set 12), the union of the original sample feature set ids on which they depend is {5, 6, 7, 9, 22}, and then taking the union with the list of the original sample feature sets on which the sample model Score1 depends {1, 5, 13, 22} gives {1, 5, 6, 7, 9, 13, 22}. That is, the list of the original sample feature sets finally depended on by the sample model Score1 is as shown in Table 4 below. Table 4 is the list of the finally depended-on original sample feature sets.

[0141] Table 4

[0142]

[0143]

[0144] In this step, according to the data in Table 3 and Table 4, calculate the specific values of the 16 factors that affect resource competition in the sample model Score1, and use them as the feature data for training the prediction model. As follows:

[0145] num_fc: 7, indicating that the number of original sample feature sets on which the sample model Score1 depends is 7;

[0146] sum_cpu_fc: 22 + 16 + 32 + 28 + 17 + 38 + 25 = 178, representing the total cpu consumption when each of the original sample feature sets on which the sample model Score1 depends is called individually;

[0147] cv_cpu_fc: standard deviation / average (178 / 7 ≈ 25.43), representing the coefficient of variation of the cpu consumption when each of the original sample feature sets on which the sample model Score1 depends is called individually;

[0148]

[0149] Denoted as cv(22, 16, 32, 28, 17, 38, 25) ≈ 0.312845,

[0150] range_cpu_fc: 38 - 16 = 22, representing the range of cpu consumption when each of the original sample feature sets on which the sample model Score1 depends is called individually;

[0151] max_avg_cost_fc: 45, representing the maximum value of the average time consumption when each of the original sample feature sets on which the sample model Score1 depends is called individually;

[0152] cv_avg_cost_fc: cv(26, 18, 35, 34, 20, 45, 28) ≈ 0.318679, representing the coefficient of variation of the average time consumption when each of the secondary derivative sample feature sets on which the sample model Score1 depends is called individually;

[0153] range_avg_cost_fc: 45 - 18 = 27, representing the range of the average time consumption when each of the original sample feature sets on which the sample model depends is called individually;

[0154] num_sfc: 2, indicating that the number of secondary derivative sample feature sets on which the sample model depends is 2;

[0155] sum_cpu_sfc: 7 + 23 = 30, representing the total cpu consumption when each of the secondary derivative sample feature sets on which the sample model depends is called individually;

[0156] cv_cpu_sfc: cv(7, 23) ≈ 0.754247, representing the coefficient of variation of the CPU consumption when a single call is made to the quadratic derivative sample feature set on which the sample model depends;

[0157] range_cpu_sfc: 23 - 7 = 16, representing the range of the CPU consumption when a single call is made to the quadratic derivative sample feature set on which the sample model depends;

[0158] max_avg_cost_sfc: 25, representing the maximum value of the average time-consuming when a single call is made to the quadratic derivative sample feature set on which the sample model depends;

[0159] cv_avg_cost_sfc: cv(7, 25) ≈ 0.795495, representing the coefficient of variation of the average time-consuming when a single call is made to the quadratic derivative sample feature set on which the sample model depends;

[0160] range_avg_cost_sfc: 25 - 7 = 18, representing the range of the average time-consuming when a single call is made to the quadratic derivative sample feature set on which the sample model depends.

[0161] When the sample model Score1 is under performance testing, record the remaining two influencing factors:

[0162] sum_cpu_total: 1280000, the total CPU resources of the production server

[0163] and num_concurrency: 50, the number of concurrency corresponding to the performance metrics.

[0164] Step 602, input multiple influencing factors into multiple initial prediction models respectively to obtain the output results of each initial prediction model. In the embodiments of this specification, the initial prediction models can be selected from linear regression models and polynomial regression models. Each initial prediction model uses the influencing factors as sample feature data, and Y1, Y2, Y3, Y4 are used as labels respectively, and a certain regression algorithm (for example, polynomial regression algorithm) is selected for training.

[0165] In this step, the 16 influencing factors of the sample model Score1 are input into four initial prediction models respectively, and these four initial prediction models are trained using the polynomial regression algorithm respectively. Further substituting the influencing factors can obtain the output results of the four initial prediction models.

[0166] The output result of the first initial prediction model is as follows:

[0167] sum_cpu_fc_Y = model1(num_fc, sum_cpu_fc, cv_cpu_fc, range_cpu_fc,

[0168] max_avg_cost_fc, cv_avg_cost_fc, range_avg_cost_fc, num_sfc,

[0169] sum_cpu_sfc, cv_cpu_sfc, range_cpu_sfc, max_avg_cost_sfc,

[0170] cv_avg_cost_sfc, range_avg_cost_sfc, sum_cpu_total,

[0171] num_concurrency )

[0173] The output result of the second initial prediction model is as follows:

[0174] sum_cpu_sfc_Y = model2(num_fc, sum_cpu_fc, cv_cpu_fc, range_cpu_fc,

[0175] max_avg_cost_fc, cv_avg_cost_fc, range_avg_cost_fc, num_sfc,

[0176] sum_cpu_sfc, cv_cpu_sfc, range_cpu_sfc, max_avg_cost_sfc,

[0177] cv_avg_cost_sfc, range_avg_cost_sfc, sum_cpu_total,

[0178] num_concurrency )

[0180] The output result of the third initial prediction model is as follows:

[0181] max_avg_cost_fc_Y = model3(num_fc, sum_cpu_fc, cv_cpu_fc, range_cpu_fc,

[0182] max_avg_cost_fc, cv_avg_cost_fc, range_avg_cost_fc, num_sfc,

[0183] sum_cpu_sfc, cv_cpu_sfc, range_cpu_sfc, max_avg_cost_sfc,

[0184] cv_avg_cost_sfc, range_avg_cost_sfc, sum_cpu_total,

[0185] num_concurrency )

[0187] The output results of the fourth initial prediction model are as follows:

[0188] max_avg_cost_sfc_Y = model4(num_fc, sum_cpu_fc, cv_cpu_fc, range_cpu_fc,

[0189] max_avg_cost_fc, cv_avg_cost_fc, range_avg_cost_fc, num_sfc,

[0190] sum_cpu_sfc, cv_cpu_sfc, range_cpu_sfc, max_avg_cost_sfc,

[0191] cv_avg_cost_sfc, range_avg_cost_sfc, sum_cpu_total,

[0192] num_concurrency )

[0194] Step 603: Calculate the loss value according to the labels of each initial prediction model and the output results of each initial prediction model, and iteratively update the model parameters of each initial prediction model through the loss value until the preset conditions are met to obtain the performance prediction model, where the maximum values of the first resource consumption, the second resource consumption, the maximum of the first average time-consuming, and the maximum of the second average time-consuming are used as the labels of each initial prediction model respectively.

[0195] In this step, the process of constructing each prediction model includes: training the first prediction model based on the output result of the first initial prediction model and the first resource consumption; training the second prediction model based on the output result of the second initial prediction model and the second resource consumption; training the third prediction model based on the output result of the third initial prediction model and the maximum value of the first average time consumption; training the fourth prediction model based on the output result of the fourth initial prediction model and the maximum value of the second average time consumption. Specifically, the output result of the first initial prediction model is sum_cpu_fc_Y. Calculate the loss function based on the output result of the first initial prediction model and the first resource consumption label Y1, and use the value of the loss function corresponding to the current training to iterate the model parameters of the first initial prediction model until the value of the loss function converges to a preset threshold to construct the first prediction model. The output result of the second initial prediction model is sum_cpu_sfc_Y. Calculate the loss function based on the output result of the second initial prediction model and the second resource consumption label Y2, and use the value of the loss function corresponding to the current training to iterate the model parameters of the second initial prediction model until the value of the loss function converges to a preset threshold to construct the second prediction model. Similarly, the output result of the third initial prediction model is max_avg_cost_fc_Y. Construct the third prediction model based on the output result of the third initial prediction model and the maximum value label Y3 of the first average time consumption; the output result of the fourth initial prediction model is max_avg_cost_sfc_Y. Construct the fourth prediction model based on the output result of the fourth initial prediction model and the maximum value label Y3 of the second average time consumption.

[0196] In this specification, the first resource consumption is the index monitored during the performance test of the sample model in step 501: SUM(cpu(original sample feature set on which the model depends)), the second resource consumption is the index: SUM(cpu(derived sample feature set on which the model depends)), the maximum value index of the first average time consumption is: MAX(avg_cost(original sample feature set on which the model depends)) and the maximum value index of the second average time consumption is: MAX(avg_cost(derived sample feature set on which the model depends)). The above 4 indexes are respectively used as the model prediction labels Y1, Y2, Y3 and Y4.

[0197] Specifically, use the first resource consumption index as the label Y1 of the first initial prediction model, use the second resource consumption index as the label Y2 of the second initial prediction model, use the maximum value index of the first average time consumption as the label Y3 of the third initial prediction model, and use the maximum value index of the second average time consumption as the label Y4 of the fourth initial prediction model.

[0198] In summary, after the first prediction model, the second prediction model, the third prediction model, and the fourth prediction model are constructed, the performance prediction model of this specification is constructed. The performance prediction model constructed in this specification becomes an effective tool for predicting the performance and resource overhead of the production environment before the model to be predicted goes online, helping analysts predict the final resource overhead of the model to be predicted in advance before training the model to be predicted, and further achieving a balance between the model effect and the model performance before the model goes online.

[0199] As Figure 7 shown is a schematic structural diagram of a model performance prediction device according to an embodiment of this specification. The basic structure of the model performance prediction device is described in this figure. The functional units and modules therein can be implemented in software, or a general-purpose chip or a specific chip can be used to implement model performance prediction. The device specifically includes:

[0200] An influencing factor determination unit 701, configured to determine the influencing factors of the model to be predicted;

[0201] A prediction unit 702, configured to input the influencing factors into the performance prediction model to obtain a key prediction value of resource consumption required for invoking the model to be predicted and a key prediction value of average time consumption, where the performance prediction model is obtained by constructing a training sample set from the key prediction value of sample resource consumption, the key prediction value of sample average time consumption of the sample model, and multiple influencing factors, and training multiple initial prediction models;

[0202] A prediction value determination unit 703, configured to determine an overall prediction value of resource consumption and an overall prediction value of average time consumption of the model to be predicted according to the key prediction value of resource consumption and the key prediction value of average time consumption.

[0203] The embodiment of this specification also discloses the structure of another model performance prediction device, which specifically includes:

[0204] A theoretical value determination unit, configured to determine a predicted theoretical value of resource consumption and a predicted theoretical value of average time consumption of the model to be predicted according to the sum of the resource consumption when a single feature set in at least one original feature set on which the model to be predicted depends is invoked, the sum of the resource consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is invoked, the maximum value of the average time consumption when a single feature set in at least one original feature set on which the model to be predicted depends is invoked, and the maximum value of the average time consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is invoked;

[0205] A final value determination unit, configured to determine a final predicted amount of resource consumption and a final predicted amount of average time consumption of the model to be predicted according to the comparison result between the overall prediction value of resource consumption and the predicted theoretical value of resource consumption, and the comparison result between the overall prediction value of average time consumption and the predicted theoretical value of average time consumption;

[0206] A condition judgment unit, configured to end the performance prediction of the to-be-predicted model if the predicted final resource consumption quantity satisfies a first preset condition and the predicted final average time consumption quantity satisfies a second preset condition.

[0207] An embodiment of this specification discloses a theoretical value determination unit, which specifically includes:

[0208] A judgment module, configured to judge whether the overall predicted value of the resource consumption of the to-be-predicted model is greater than the theoretical value of the predicted resource consumption, and whether the overall predicted value of the average time consumption of the to-be-predicted model is greater than the theoretical value of the predicted average time consumption;

[0209] A first determination module, configured to, if the answer is no, determine the theoretical value of the predicted resource consumption as the predicted final resource consumption quantity, and determine the theoretical value of the predicted average time consumption as the predicted final average time consumption quantity;

[0210] A second determination module, configured to, if the answer is yes, determine the overall predicted value of the resource consumption as the predicted final resource consumption quantity, and determine the overall predicted value of the average time consumption of the to-be-predicted model as the predicted final average time consumption quantity.

[0211] As Figure 8 shown is a schematic structural diagram of a performance prediction model construction device according to an embodiment of this specification. In this figure, the basic structure of the performance prediction model construction device is described. The functional units and modules therein can be implemented in software, or a general-purpose chip or a specific chip can be used to implement the performance prediction model construction. The device specifically includes:

[0212] A calling unit 801, configured to analyze the sample feature set dependency relationship of the sample model, and initiate a call to the sample model according to the sample feature set dependency relationship, where the sample feature set dependency relationship at least includes an original sample feature set and a derived sample feature set;

[0213] A key value determination unit 802, configured to determine the key value of the sample resource consumption and the key value of the sample average time consumption generated during the call of the sample model by calling the original sample feature set and the derived sample feature set;

[0214] A training unit 803, configured to construct a training sample set by using the key value of the sample resource consumption, the key value of the sample average time consumption, and multiple influencing factors, and use the training sample set to train multiple initial prediction models to obtain a performance prediction model; where the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

[0215] An embodiment of this specification also discloses the structure of a calling unit, which specifically includes:

[0216] The first call module is used to initiate a call to the original sample feature set to obtain a first call result;

[0217] The second call module is used to initiate a call to the derived sample feature set according to the first call result to obtain a second call result.

[0218] The embodiments of this specification also disclose a structure of a key value determination unit, which specifically includes:

[0219] The first acquisition module is used to initiate a call to the original sample feature set simultaneously to obtain the maximum value of the first resource consumption and the first average time consumption of the first call result;

[0220] The second acquisition module is used to initiate a call to the derived sample feature set simultaneously to obtain the maximum value of the second resource consumption and the second average time consumption of the second call result;

[0221] The key value determination module is used to determine the sample resource consumption key value according to the first resource consumption and the second resource consumption, and to determine the sample average time consumption key value according to the maximum value of the first average time consumption and the maximum value of the second average time consumption.

[0222] The embodiments of this specification also disclose a structure of a training unit, which specifically includes:

[0223] The influencing factor determination module is used to calculate a plurality of influencing factors according to the original sample feature set and the derived sample feature set;

[0224] The prediction result acquisition module is used to input the plurality of influencing factors into a plurality of initial prediction models respectively to obtain the output results of the respective initial prediction models;

[0225] The iterative update module is used to calculate a loss value according to the labels of the respective initial prediction models and the output results of the respective initial prediction models, and iteratively update the model parameters of the respective initial prediction models through the loss value until a preset condition is met to obtain the performance prediction model; wherein, the first resource consumption, the second resource consumption, the maximum value of the first average time consumption, and the maximum value of the second average time consumption are respectively used as the labels of the respective initial prediction models.

[0226] Figure 9 The figure shows a schematic diagram of a model deployment architecture according to an embodiment of this specification. Among them, the model, the feature set, and the secondary derived feature set are all the sources of resource overhead and operation time consumption when the model cloud service provides services. A series of original data changing over time are subjected to real-time operation of a pre-developed derivative algorithm to obtain a series of features, and this feature set is called the feature set. A series of pre-developed derivative algorithms are used to perform real-time operations on several feature sets to obtain a series of new features, and this new feature set is called the secondary derived feature set.

[0227] Figure 10 The figure shows a schematic diagram of the call order of the dependent feature set of a to-be-predicted model in an embodiment of this specification. As Figure 10 shown. By analyzing the dependency relationship of the sample feature set of Model 2, it can be determined that "Model 2" depends on the original sample feature set 1, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N. Further, it can be determined that the secondary derivative sample feature set 2 depends on the original sample feature set 1 and the original sample feature set N, and the secondary derivative sample feature set N depends on the original sample feature set 1 and the original sample feature set 2. Therefore, "Model 2" ultimately depends on the original sample feature set 1, the original sample feature set 2, the original sample feature set N, the secondary derivative sample feature set 2, and the secondary derivative sample feature set N. Regarding Figure 10 the introduction, see specifically Figure 4 the description.

[0228] As Figure 11 shown, a computer device provided in an embodiment of this specification is shown. The model performance prediction method described in this application can be applied to the computer device. The computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 1102 may also include any memory 1106 for storing any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash devices, hard disks, optical discs, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 1102. In one case, when the processor 1104 executes the associated instructions stored in any memory or combination of memories, the computer device 1102 may perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disc drive mechanism, etc.

[0229] The computer device 1102 may also include an input / output module 1110 (I / O) for receiving various inputs (via the input device 1112) and for providing various outputs (via the output device 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface (GUI) 1118. In other embodiments, the input / output module 1110 (I / O), the input device 1112, and the output device 1114 may not be included, and it may only be a computer device in the network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.

[0230] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0231] Corresponding to Figures 1 - 6 In the embodiments of this specification, a computer-readable storage medium is also provided. A computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the above method are executed.

[0232] The embodiments of this specification also provide a computer-readable instruction. When the processor executes the instruction, the program therein causes the processor to execute the method as Figures 1 - 6 shown.

[0233] It should be understood that in the various embodiments of this specification, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this specification.

[0234] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this specification generally represents an "or" relationship between the associated objects before and after.

[0235] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this specification can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this specification.

[0236] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0237] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the couplings, direct couplings, or communication connections shown or discussed among each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be in electrical, mechanical, or other forms of connection.

[0238] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this specification.

[0239] In addition, the functional units in each embodiment of this specification can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0240] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this specification. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0241] Specific embodiments are used in this specification to elaborate on the principles and implementation manners of this specification. The description of the above embodiments is only used to help understand the method and its core idea of this specification; at the same time, for those of ordinary skill in the art, according to the idea of this specification, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this specification.

Claims

1. A method for predicting model performance, characterized in that, The method includes: Determine the influencing factors of the model to be predicted; Input the influencing factors into the performance prediction model to obtain the key prediction values of resource consumption and the key prediction values of average time consumption required for calling the model to be predicted. Among them, the performance prediction model is obtained by constructing a training sample set with the sample key prediction values of resource consumption, sample key prediction values of average time consumption of the sample model, and multiple influencing factors, and training multiple initial prediction models; Determine the overall prediction value of resource consumption and the overall prediction value of average time consumption of the model to be predicted according to the key prediction values of resource consumption and the key prediction values of average time consumption.

2. The method according to claim 1, characterized in that The method further includes: Determine the theoretical prediction value of resource consumption and the theoretical prediction value of average time consumption of the model to be predicted according to the sum of resource consumption when a single feature set in at least one original feature set on which the model to be predicted depends is called, the sum of resource consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is called, the maximum value of average time consumption when a single feature set in at least one original feature set on which the model to be predicted depends is called, and the maximum value of average time consumption when a single derivative feature set in at least one derivative feature set on which the model to be predicted depends is called; Determine the final predicted amount of resource consumption and the final predicted amount of average time consumption of the model to be predicted according to the comparison result between the overall prediction value of resource consumption and the theoretical prediction value of resource consumption, and the comparison result between the overall prediction value of average time consumption and the theoretical prediction value of average time consumption; If the final predicted amount of resource consumption meets the first preset condition and the final predicted amount of average time consumption meets the second preset condition, end the performance prediction of the model to be predicted.

3. The method according to claim 2, characterized in that, The step of determining the final predicted amount of resource consumption and the final predicted amount of average time consumption according to the comparison result between the overall prediction value of resource consumption and the theoretical prediction value of resource consumption, and the comparison result between the overall prediction value of average time consumption and the theoretical prediction value of average time consumption includes: Judge whether the overall prediction value of resource consumption of the model to be predicted is greater than the theoretical prediction value of resource consumption, and whether the overall prediction value of average time consumption of the model to be predicted is greater than the theoretical prediction value of average time consumption; If not, determine the theoretical prediction value of resource consumption as the final predicted amount of resource consumption, and determine the theoretical prediction value of average time consumption as the final predicted amount of average time consumption; If so, determine the overall prediction value of resource consumption as the final predicted amount of resource consumption, and determine the overall prediction value of average time consumption of the model to be predicted as the final predicted amount of average time consumption.

4. The method according to claim 2 or 3, wherein The first preset condition includes that the final predicted amount of resource consumption does not exceed the preset resource consumption threshold; The second preset condition includes that the final predicted amount of average time consumption does not exceed the preset average time consumption threshold.

5. A method for constructing a performance prediction model, characterized in that, The performance prediction model is constructed in the following manner: Analyze the dependency relationship of the sample feature sets of the sample model, and initiate a call to the sample model according to the dependency relationship of the sample feature sets. Among them, the dependency relationship of the sample feature sets includes at least the original sample feature set and the derivative sample feature set; By invoking the original sample feature set and the derived sample feature set, determine the key values of sample resource consumption and the key values of sample average time consumption generated during the sample model invocation process; Construct a training sample set by using the key values of sample resource consumption, the key values of sample average time consumption, and multiple influencing factors, and use the training sample set to train multiple initial prediction models to obtain a performance prediction model; wherein, the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

6. The method according to claim 5, wherein Analyze the sample feature set dependency of the sample model, and initiate the invocation of the sample model according to the sample feature set dependency, including: Initiate an invocation of the original sample feature set to obtain a first invocation result; According to the first invocation result, initiate an invocation of the derived sample feature set to obtain a second invocation result.

7. The method according to claim 6, characterized in that, By invoking the original sample feature set and the derived sample feature set, determining the key values of sample resource consumption and the key values of sample average time consumption generated during the sample model invocation process includes: Initiate an invocation of the original sample feature set to obtain the maximum values of the first resource consumption and the first average time consumption of the first invocation result; Initiate an invocation of the derived sample feature set to obtain the maximum values of the second resource consumption and the second average time consumption of the second invocation result; According to the first resource consumption and the second resource consumption, determine the key values of sample resource consumption, and according to the maximum value of the first average time consumption and the maximum value of the second average time consumption, determine the key values of sample average time consumption.

8. The method according to claim 7, wherein Constructing a training sample set by using the key values of sample resource consumption, the key values of sample average time consumption, and multiple influencing factors, and using the training sample set to train multiple initial prediction models to obtain a performance prediction model includes: Calculate multiple influencing factors according to the original sample feature set and the derived sample feature set; Input the multiple influencing factors into the multiple initial prediction models respectively to obtain the output results of the respective initial prediction models; Calculate the loss value according to the labels of the respective initial prediction models and the output results of the respective initial prediction models, and iteratively update the model parameters of the respective initial prediction models through the loss value until a preset condition is met to obtain the performance prediction model; wherein, the first resource consumption, the second resource consumption, the maximum value of the first average time consumption, and the maximum value of the second average time consumption are respectively used as the labels of the respective initial prediction models.

9. A model performance prediction device, characterized in that, The device includes: An influencing factor determination unit for determining the influencing factors of the model to be predicted; A prediction unit for inputting the influencing factors into the performance prediction model to obtain the key predicted values of resource consumption and the key predicted values of average time consumption required for invoking the model to be predicted, wherein the performance prediction model is constructed by using the key predicted values of sample resource consumption, the key predicted values of sample average time consumption of the sample model, and multiple influencing factors to construct a training sample set and training multiple initial prediction models; A predicted value determination unit for determining the overall predicted value of resource consumption and the overall predicted value of average time consumption of the model to be predicted according to the key predicted values of resource consumption and the key predicted values of average time consumption.

10. An apparatus for constructing a performance prediction model, characterized in that, The device includes: A calling unit, configured to parse the sample feature set dependency relationship of the sample model, and initiate a call to the sample model according to the sample feature set dependency relationship, where the sample feature set dependency relationship at least includes an original sample feature set and a derived sample feature set; A key value determination unit, configured to determine a sample resource consumption key value and a sample average time consumption key value generated during the sample model call by calling the original sample feature set and the derived sample feature set; A training unit, configured to construct a training sample set by using the sample resource consumption key value, the sample average time consumption key value, and multiple influencing factors, and use the training sample set to train multiple initial prediction models to obtain a performance prediction model; where the multiple initial prediction models include: an initial prediction model for predicting resource consumption and an initial prediction model for predicting average time consumption.

11. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images