Model quality assessment method and device
The allocation of test data to multiple prediction terminals in clustering is solved, and the problems of low efficiency in the quality evaluation of prediction models and excessive computing resources in the prior art are solved, and more efficient model quality evaluation is achieved.
Patent Information
- Application Number
- CN201910636167.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-07-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-07-15
AI Technical Summary
In the prior art, the quality evaluation of prediction models is inefficient, resulting in excessive computing resources occupied and prone to lag or stuck problems.
Multiple test data are allocated to multiple prediction terminals for prediction value calculation through clustering, reducing the calculation amount and operation time of the evaluation terminal, and improving the efficiency of model quality evaluation.
It effectively reduces the calculation time required to calculate the quality parameters of the predictive model, improves the efficiency of the quality evaluation of the predictive model, and avoids the problems of lag and stuck caused by insufficient computing resources.
Smart Images

Figure CN110363427B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to a model quality assessment method and device. Background Art
[0002] Supervised learning refers to using a set of training data to learn the mapping relationship between the data input and output, and then applying this mapping relationship to unknown data to achieve the purpose of classifying or regressing the location data. In the field of prediction, the mapping relationship between the training data and the true value of the training data that can be learned by the machine through supervised learning can be called a prediction model. The quality of a prediction model can affect the accuracy of the prediction results output by the model. Therefore, before a prediction model is put into use, it is necessary to pre-test the quality of the model, that is, how the model performs on new test data to determine whether the model can be put into use. Obviously, the existing technology has the following problems: the method of evaluating the quality of the prediction model is inefficient. Summary of the invention
[0003] The embodiments of the present application provide a model quality assessment method and device, which can improve the efficiency of prediction model quality assessment.
[0004] The present application embodiment provides a model quality assessment method, which is applicable to an assessment terminal, including:
[0005] Obtain a prediction model and at least two test data subsets, wherein the test data subsets include at least one test data marked with a true value;
[0006] Sending a prediction request carrying the prediction model and the test data subset to a corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset;
[0007] When the prediction terminal in the terminal cluster completes the prediction, the prediction value of the test data is obtained from the prediction terminal to obtain the prediction value of the test data in each test data subset;
[0008] Classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs;
[0009] The model quality parameter of the prediction model is calculated according to the category to which the test data in the test data subset belongs.
[0010] In some embodiments, calculating the model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs includes:
[0011] Count the proportion of each category in all categories;
[0012] The model quality parameter of the prediction model is calculated based on the proportion of each category in all categories.
[0013] In some embodiments, classifying the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs includes:
[0014] Determine a target prediction value from the prediction values of the test data sent by all prediction terminals in the terminal cluster;
[0015] The test data is classified based on the target predicted value, the predicted value of the test data, and the true value.
[0016] In some embodiments, obtaining a prediction model and at least two test data subsets includes:
[0017] Obtaining a prediction model and a test data set, wherein the test data set includes at least one test data marked with a true value;
[0018] A partitioning operation is performed on the test data in the test data set to obtain at least two test data subsets.
[0019] In some embodiments, obtaining a prediction model and a test data set includes:
[0020] Obtain the prediction model and the test data set to be preprocessed;
[0021] Determining the data type of the test data to be preprocessed;
[0022] The data format of the test data to be preprocessed in the test data set to be preprocessed is converted based on the data type to obtain a test data set.
[0023] In some embodiments, the obtaining the predicted value of the test data from the prediction terminal further includes:
[0024] Obtaining a preset value range and a preset threshold value corresponding to the value range;
[0025] Determining a numerical range within which the predicted value of the test data lies;
[0026] The predicted value is assigned a value according to a preset threshold value corresponding to the numerical range of the predicted value to obtain the predicted value of the test data after the assignment processing.
[0027] Classifying the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs includes:
[0028] The test data is classified according to the predicted value after the assignment processing of the test data to obtain the category to which the test data in the test data subset belongs.
[0029] In some embodiments, sending a prediction request carrying the prediction model and the test data subset to a corresponding prediction terminal in the terminal cluster includes:
[0030] Obtaining the startup status information and data acquisition status information of the prediction terminal;
[0031] Determine the startup state and data acquisition state of the prediction terminal according to the startup state information and the data acquisition state information;
[0032] When the startup state of the prediction terminal is the shutdown state, turning on the prediction terminal;
[0033] When the startup state of the prediction terminal is the on state, and the data acquisition state is the waiting state, a prediction request carrying the prediction model and the test data subset is sent to a corresponding prediction terminal in the terminal cluster.
[0034] The present application embodiment provides a model quality assessment method, which is applicable to a prediction terminal, including:
[0035] Receiving a test data subset and a prediction model sent by an evaluation terminal, wherein the test data subset includes at least one test data;
[0036] Using the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data;
[0037] When the prediction is completed, the predicted value of the test data is sent to the evaluation terminal.
[0038] In some embodiments, using the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data includes:
[0039] Obtaining a preset value range and a preset threshold value corresponding to the value range;
[0040] Using the prediction model to predict the test data in the test data subset to obtain the to-be-processed prediction value of the test data;
[0041] Determine the numerical range of the predicted value to be processed;
[0042] The predicted value to be processed is assigned a value according to a preset threshold value corresponding to the numerical range of the predicted value to be processed to obtain the predicted value of the test data.
[0043] The present application also provides a model quality assessment device, including:
[0044] An acquisition unit, configured to acquire a prediction model and at least two test data subsets, wherein the test data subsets include at least one test data labeled with a true value;
[0045] A sending unit, configured to send a prediction request carrying the prediction model and the test data subset to a corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset;
[0046] A prediction value unit, configured to obtain the prediction value of the test data from the prediction terminal when the prediction terminal in the terminal cluster completes the prediction, so as to obtain the prediction value of the test data in each test data subset;
[0047] A classification unit, used to classify the test data according to the predicted value and the true value of the test data, so as to obtain the category to which the test data in the test data subset belongs;
[0048] A calculation unit is used to calculate the model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs.
[0049] In some embodiments, the embodiments of the present application also provide a storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the steps in any model quality assessment method provided in the embodiments of the present application.
[0050] In some embodiments, the embodiments of the present application also provide a server, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any model quality assessment method provided in the embodiments of the present application.
[0051] The embodiments of the present application can obtain a prediction model and multiple test data subsets, wherein the test data subset includes at least one test data marked with a true value; send a prediction request carrying the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset; when the prediction terminal in the terminal cluster completes the prediction, obtain the predicted value of the test data from the prediction terminal to obtain the predicted value of the test data in each test data subset; classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in multiple test data subsets belongs; and calculate the model quality parameters of the prediction model according to the category to which the test data in multiple test data subsets belongs.
[0052] In the present application, in order to solve the problem that a large amount of computing resources are required to calculate the model quality parameters of the prediction model, as well as the problem of computing jams and freezes caused by insufficient computing resources, a cluster approach can be used to predict object behavior indicators, and multiple test data are assigned to the prediction terminal in the prediction cluster to calculate the predicted value. This can greatly reduce the computing time consumed in calculating the predicted values of all test data, and at the same time reduce the calculation amount of the evaluation terminal. Therefore, this solution can improve the efficiency of the prediction model quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0054] Figure 1a is a scenario diagram of a model quality assessment system provided in an embodiment of the present application;
[0055] Figure 1b This is a schematic diagram of the first flow chart of the model quality assessment method provided in the embodiment of the present application;
[0056] Figure 2a This is a second flow chart of the model quality assessment method provided in the embodiment of the present application;
[0057] Figure 2b It is a schematic diagram of the Deep&Wide model structure provided in the embodiment of the present application;
[0058] Figure 3a This is a third flow chart of the model quality assessment method provided in the embodiment of the present application;
[0059] Figure 3b It is a schematic diagram of the preprocessing process provided in the embodiment of the present application;
[0060] Figure 3c This is a schematic diagram of the Spark architecture provided by the embodiment of the present application;
[0061] Figure 3d It is a schematic diagram of the operation process of Spark provided in the embodiment of the present application;
[0062] Figure 3e It is a schematic diagram of the assignment processing flow provided by an embodiment of the present application;
[0063] Figure 4 This is a schematic diagram of the first structure of the model quality assessment device provided in the embodiment of the present application;
[0064] Figure 5 This is a second structural diagram of the model quality assessment device provided in an embodiment of the present application;
[0065] Figure 6 It is a schematic diagram of the structure of the evaluation terminal provided in the embodiment of the present application. DETAILED DESCRIPTION
[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0067] The embodiments of the present application provide a model quality assessment method and device.
[0068] An embodiment of the present application provides a model quality assessment system, including: a model quality assessment device provided in an embodiment of the present application, for example, a model quality assessment device suitable for an evaluation terminal, a model quality assessment device suitable for a prediction terminal, etc.
[0069] In some embodiments, the model quality assessment system may be composed of a plurality of model quality assessment devices applicable to the assessment terminal and a plurality of model quality assessment devices applicable to the prediction terminal. For example, the model quality assessment system may be composed of a model quality assessment device applicable to the assessment terminal and a plurality of model quality assessment devices applicable to the prediction terminal. For example, the model quality assessment system may be composed of a plurality of model quality assessment devices applicable to the assessment terminal and a plurality of model quality assessment devices applicable to the prediction terminal.
[0070] The evaluation terminal and the prediction terminal may be devices such as terminals and servers. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, or a personal computer (PC); the server may be a single server or a server cluster consisting of multiple servers.
[0071] In some embodiments, the evaluation terminal and the prediction terminal may be a server cluster composed of multiple servers. For example, the prediction terminal may be composed of multiple servers to implement the model quality evaluation method of the present application. For example, the evaluation terminal may also be composed of multiple servers to implement.
[0072] For example, reference Figure 1aA schematic diagram of a model quality assessment system scenario is provided. The model quality assessment system may include a terminal cluster consisting of multiple prediction terminals (respectively prediction terminal A, prediction terminal B, and prediction terminal C) and an evaluation terminal. Each prediction terminal may be a single server, and each evaluation terminal may also be a single server.
[0073] exist Figure 1a In the process, the evaluation terminal can obtain the prediction model and multiple test data subsets marked with true values, and send a prediction request carrying the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster. For example, the prediction request carrying the prediction model and a part of the test data subset is sent to the prediction terminal A, the prediction request carrying the prediction model and another part of the test data subset is sent to the prediction terminal B, and the prediction request carrying the prediction model and the remaining part of the test data subset is sent to the prediction terminal C; then, the prediction completion information of the prediction terminal in the terminal cluster is monitored. When the prediction terminal in the terminal cluster completes the prediction, the prediction value of the test data is obtained from the prediction terminal to obtain the prediction value of the test data in each test data subset; finally, the model quality parameters of the prediction model are calculated according to the predicted values and true values of the test data in multiple test data subsets.
[0074] exist Figure 1a In the process, the prediction terminal in the terminal cluster can receive the test data subset and the prediction model sent by the evaluation terminal, wherein the test data subset includes at least one test data; the prediction model is used to predict the test data in the test data subset to obtain the predicted value of the test data; when the prediction is completed, the predicted value of the test data is sent to the evaluation terminal.
[0075] Specifically, the evaluation terminal can send a prediction request carrying a prediction model and a test data subset to the corresponding prediction terminal in the terminal cluster; when the prediction terminal receives the request, the prediction terminal can receive the test data subset and the prediction model sent by the evaluation terminal, and then use the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data. When the prediction is completed, the prediction terminal can send the predicted value of the test data to the evaluation terminal, and the evaluation terminal can obtain the predicted value of the test data from the prediction terminal, obtain the predicted value of the test data in each test data subset, and calculate the model quality parameters of the prediction model based on the predicted values and true values of the test data in multiple test data subsets.
[0076] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0077] In this embodiment, a model quality assessment method is provided, which is applicable to an assessment terminal, such as Figure 1bAs shown, the specific process of the model quality assessment method can be as follows:
[0078] 101. Obtain a prediction model and at least two test data subsets, where the test data subset includes at least one test data labeled with a true value.
[0079] The prediction model is a neural network model used for prediction, for example, the prediction model can be used to predict the object behavior index, and the index can be used to represent the object network behavior. For example, the prediction model can be used to predict the number of times the object clicks on a specific advertisement on a website.
[0080] Among them, the object behavior index is an index used to reflect the behavior of the object on the network for specific network information. For example, according to the type classification of the object behavior index, the object behavior index can be the number, probability, frequency and other indicators of the behavior generated by the object for specific network information. For example, the object behavior index can include the object's click-through rate, click volume, number of visits, visit time, object payment amount and other indicators. For another example, according to the type classification of the network information targeted by the object behavior index, the object behavior index can reflect the network information such as advertisements, recommended videos, news headlines, etc. placed on the web page by the object.
[0081] In some embodiments, depending on the type of the object behavior indicator, the prediction model may include an advertisement click-through rate prediction model, a web page visit number prediction model, an object payment amount prediction model, and the like.
[0082] Specifically, in some embodiments, the prediction model may include an advertisement click-through rate prediction model, which may predict the probability of an object clicking on a specific advertisement on a web page.
[0083] Among them, the test data subset may include multiple test data, which can be used to test the model quality parameters of the prediction model. The test data in the test data subset may be similar to the training data used by the prediction model in the training stage. For example, the training data used by the advertising click-through rate prediction model in the training stage may include the attribute information of the advertisement and the attribute information of the object, so the test data in the test data subset may also include the attribute information of the advertisement and the attribute information of the object.
[0084] The attribute information of the advertisement may include information such as the image size of the advertisement, the time when the advertisement appears on the web page, the field to which the advertisement belongs, etc.; the attribute information of the object may include information such as the age, gender, education level, etc. of the object.
[0085] In some embodiments, the test data in the test data subset may be collected and sorted in advance by humans or machines and annotated with their true values, which may be annotated as true or false.
[0086] There are many ways to obtain the prediction model and test data subset, such as obtaining the user behavior indicator prediction model and test data subset from the database through the network, or reading the user behavior indicator prediction model and test data subset stored in the local memory, etc.
[0087] In some embodiments, a test data subset may be obtained through a network and a prediction model in a local memory may be read.
[0088] In some embodiments, the test data subset may be stored in the test data set, and obtaining the prediction model and the test data subset may include the following steps:
[0089] (1) Obtain a prediction model and a test data set, where the test data set includes at least one test data with a true value marked.
[0090] There are many ways to obtain the test data set, such as obtaining the test data set from a database through a network, or reading the test data set stored in a local memory, and so on.
[0091] For example, in some embodiments, obtaining the prediction model and the test data set may specifically include the following steps:
[0092] a. Obtain the prediction model and the test data set to be preprocessed;
[0093] b. Determine the data type of the test data to be preprocessed;
[0094] c. Convert the data format of the test data to be preprocessed in the test data set to be preprocessed based on the data type to obtain a test data set.
[0095] The test data set to be preprocessed may include one or more test data to be preprocessed, and the test data to be preprocessed may include multiple data types, for example, the test data to be preprocessed may be a string type, a byte array type, a vector type, and the like.
[0096] In some embodiments, the data type of the test data to be preprocessed is determined, and a format conversion method corresponding to the data type is selected to perform format conversion on the data type to obtain a test data set.
[0097] For example, all information in the preprocessed test data may be serialized and encoded, and the obtained sequences may be encapsulated into a data frame format to normalize the test data, thereby further improving the accuracy of the obtained model quality parameters.
[0098] (2) Perform a partition operation on the test data in the test data set to obtain multiple test data subsets.
[0099] The partitioning operation refers to selecting part of the test data from the test data set and using the part of the test data as the test data subset.
[0100] For example, in some embodiments, the partition operation may be to divide the test data into
[0101] The test data in the data set are equally divided. For example, the test data set includes 1000 test data, and these test data are equally divided into 10 to obtain 100 test data subsets.
[0102] 102. Send a prediction request carrying a prediction model and a test data subset to a corresponding prediction terminal in the terminal cluster.
[0103] The prediction request may carry the prediction model and the test data subset, and may also carry the address of a database storing the prediction model and the test data subset. For example, in some embodiments, the prediction request may carry the test data subset and the address of a model database storing the prediction model.
[0104] Among them, the terminal cluster may include one or more prediction terminals, and there are various ways to send prediction models and test data subsets to the corresponding prediction terminals in the terminal cluster, such as reading the address information of all prediction terminals in the terminal cluster in the local memory, and sending the prediction model and test data subset to the prediction terminal according to the address information through the wireless network; for example, directly sending the prediction model and test data subset to the corresponding prediction terminal in the terminal cluster through wired transmission, and so on.
[0105] In some embodiments, the prediction model and the test data subset may be sent to the corresponding prediction terminal in the terminal cluster through the network, for example, the following steps may be specifically included:
[0106] a. Obtain the startup status information and data acquisition status information of the prediction terminal;
[0107] b. Determine the startup status and data acquisition status of the prediction terminal according to the startup status information and data acquisition status information;
[0108] c. When the prediction terminal is in the closed state, open the prediction terminal;
[0109] d. When the startup state of the prediction terminal is the on state and the data acquisition state is the waiting state, the test data subset and the prediction model are sent to the prediction terminal.
[0110] Among them, the startup status information may include the current startup status of the predicted terminal, which may include an on state and a off state, etc.; the data acquisition status information may include the current data acquisition status of the predicted terminal, which may include a waiting acquisition state and a stopped acquisition state, etc.
[0111] In this way, it is possible to prevent mistransmission caused by forcibly sending the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster when the prediction terminal is not yet ready to obtain the prediction model and the test data subset.
[0112] 103. When the prediction terminal in the terminal cluster completes the prediction, the prediction value of the test data is obtained from the prediction terminal to obtain the prediction value of the test data in each test data subset.
[0113] In some embodiments, it is possible to determine whether a prediction terminal in a terminal cluster has completed prediction by monitoring prediction completion information of the prediction terminal, and the specific steps may include:
[0114] a. Regularly receive messages sent by the predicted terminal corresponding address;
[0115] b. Determine the message type in the message;
[0116] c. If the message type of the message is a prediction completion information type, the message is regarded as prediction completion information.
[0117] For example, the format of the prediction completion information can refer to Table 1:
[0118] Message Type Source address Message release time Message acquisition time 0x00 21:C4:66:12 2017050421273 2017050421324
[0119] Table 1
[0120] Among them, the message type 0x00 means that the message type of the message is the prediction completion information type, the message source address 21:C4:66:12 can be the network address of the receiving prediction terminal, the message release time 2017050421273 can refer to the time when the message is released by the prediction terminal, and the message acquisition time 2017050421324 can refer to the time when the message is received.
[0121] In some embodiments, the amount of calculation can be reduced and the value range of the prediction value can be standardized by reducing the value accuracy of the prediction value. Therefore, after obtaining the prediction value of the test data from the prediction terminal, the following steps can also be included:
[0122] a. Obtain a preset value range and a preset threshold value corresponding to the value range;
[0123] b. Determine the numerical range of the predicted value of the test data;
[0124] c. Assign the predicted value according to the preset threshold value corresponding to the numerical range of the predicted value to obtain the predicted value of the test data after assignment processing.
[0125] There are many methods for obtaining the preset numerical range and the preset threshold value corresponding to the numerical range. For example, the preset numerical range and the preset threshold value corresponding to the numerical range can be set by a person skilled in the art, or can be obtained from a database through a network, or can be obtained by reading a local memory, etc.
[0126] For example, in some embodiments, the preset numerical range and the preset threshold value corresponding to the numerical range can be saved in the local memory in the form of a table. By reading the table in the local memory, the preset numerical range and the preset threshold value corresponding to the numerical range can be obtained.
[0127] The format of the table can refer to Table 2:
[0128] Numerical range [0.00,0.100] (0.100,0.255] (0.255,0.355] (0.355,0.500] Preset Threshold 0.1 0.2 0.3 0.4
[0129] Table 2
[0130] For example, based on the preset numerical range provided in Table 2 and the preset threshold value corresponding to the numerical range, the numerical range in which the predicted value of the test data is located can be determined, and the predicted value can be assigned according to the preset threshold value corresponding to the numerical range in which the predicted value is located to obtain the predicted value of the test data after the assignment processing.
[0131] 104. Classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data belongs in the test data subset.
[0132] For example, in some embodiments, the predicted value of the test data may be 0 or 1, and the true value of the test data may also be 0 or 1, and the test data may be classified according to the predicted value and the true value of the test data. The classification results may include multiple types, for example, the test data may be classified into true sample data, false positive sample data, true negative sample data, false negative sample data, and the like.
[0133] Among them, true sample data refers to test data whose predicted value and true value are both true; false positive sample data refers to test data whose predicted value is true and true value is false; true negative sample data refers to test data whose predicted value and true value are both false; false negative sample data refers to test data whose predicted value is false and true value is true.
[0134] In some embodiments, classifying the test data according to the predicted values and true values of the test data sent by all the prediction terminals in the terminal cluster to obtain the category to which the test data belongs may specifically include the following steps:
[0135] A. Determine the target prediction value from the prediction values of the test data sent by all prediction terminals in the terminal cluster;
[0136] B. Classify the test data based on the target predicted value, the predicted value of the test data, and the true value.
[0137] A target prediction value is selected from the prediction values of the test data sent by all prediction terminals in the terminal cluster, and whether the prediction value of the test data is true or false is determined according to the size relationship between the prediction value and the target prediction value.
[0138] Specifically, classifying the test data based on the target predicted value, the predicted value of the test data, and the true value may include the following steps:
[0139] a. Classify the test data whose predicted value is greater than or equal to the target predicted value as positive sample data, and classify the test data whose predicted value is less than the target predicted value as negative sample data;
[0140] b. Classify the positive samples whose true values are the preset threshold as true sample data;
[0141] c. Classify negative samples whose true values are the preset threshold as false positive sample data.
[0142] The preset threshold value can be set by a technician and saved in a local memory, or obtained from a database via a network.
[0143] For example, the predicted values of the test data sent by all prediction terminals in the terminal cluster include 0.1, 0.2, 0.3, 0.4, and 0.5. Assuming that the target prediction value 0.3 is selected from these prediction values, all prediction values less than the target prediction value are judged to be false, and all prediction values greater than the target prediction value are judged to be true, that is, the prediction values of 0.1 and 0.2 are both false, and the prediction values of 0.3, 0.4, and 0.5 are all true.
[0144] When both the predicted value and the true value are true, the test data is classified as true sample data; when the predicted value is true and the true value is false, the test data is classified as false positive sample data; when both the predicted value and the true value are false, the test data is classified as true negative sample data; when the predicted value is false and the true value is true, the test data is classified as false negative sample data.
[0145] 105. Calculate the model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs.
[0146] The specific steps of calculating the model quality parameter of the prediction model according to the categories to which the test data in the plurality of test data subsets belong may include:
[0147] a. Count the proportion of each category in all categories;
[0148] b. Calculate the model quality parameter of the prediction model based on the proportion of each category in all categories.
[0149] For example, in some embodiments, the categories of test data may include positive sample data, negative sample data, true sample data, and false positive sample data, and the specific steps of counting the proportion of each category in all categories may include:
[0150] a. Count the number of true negative sample data, the number of false negative sample data, the number of true sample data, and the number of false positive sample data in the test data set.
[0151] b. Calculate the true positive rate based on the number of true sample data and the number of positive sample data in the test data set.
[0152] c. Calculate the false positive rate based on the number of false positive sample data and the number of negative sample data in the test data set.
[0153] The formula for the true rate is as follows:
[0154]
[0155] The formula for the false positive rate is as follows:
[0156]
[0157] In some embodiments, the model quality parameter of the prediction model may be calculated based on the proportion of each category in all categories.
[0158] Among them, model quality parameters may include receiver operating characteristic (ROC), area under curve (AUC), and the like.
[0159] In some embodiments, the model quality parameter of the prediction model can be calculated by the true positive rate and the false positive rate. For example, when the model quality parameter is AUC, the AUC value can be calculated by the true positive rate and the false positive rate, and the formula is as follows:
[0160]
[0161] Among them, P is the true positive rate; N is the false positive rate; i is the target prediction value; n is a positive integer, representing all prediction data sent by the nth prediction terminal.
[0162] The model quality evaluation scheme provided in the embodiment of the present application can be applied in various model quality evaluation scenarios. For example, taking the model quality parameter evaluation of the advertisement click-through rate model as an example, the advertisement click-through rate model and the test data subset can be first obtained from the database through the network. The test data subset includes 1,000 test data marked with real values. The test data may include object attributes and advertisement attributes. For example, the object attributes may include the object's age, gender and other information. The advertisement attributes may control the time and location of the advertisement on a specific web page, as well as the type of advertisement, advertisement content and other information;
[0163] Then, the advertisement click rate model is sent to each prediction terminal in the terminal cluster, and 100 test data out of 1000 test data are sent to each prediction terminal in the terminal cluster; the prediction completion information of the prediction terminal is monitored, and when the prediction completion information sent by the prediction terminal is monitored, the prediction value of the test data is obtained from the prediction terminal;
[0164] Finally, the model quality parameters of the prediction model are calculated based on the predicted values and true values of the test data sent by all prediction terminals in the terminal cluster.
[0165] The solution provided in the embodiment of the present application can predict object behavior indicators in a clustering manner, and allocate multiple test data to the prediction terminal in the prediction cluster to calculate the prediction value, thereby greatly reducing the computing time consumed by calculating the prediction value of all test data, and at the same time reducing the calculation amount of the evaluation terminal. Therefore, the present application can solve the problem that the current method of calculating the model quality parameters of the prediction model requires a large amount of computing resources and a long calculation time.
[0166] As can be seen from the above, the embodiment of the present application can obtain a prediction model and multiple test data subsets, wherein the test data subset includes at least one test data marked with a true value;
[0167] Send a prediction request carrying a prediction model and a test data subset to the corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset; when the prediction terminal in the terminal cluster completes the prediction, obtain the prediction value of the test data from the prediction terminal to obtain the prediction value of the test data in each test data subset; classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in multiple test data subsets belongs; calculate the model quality parameters of the prediction model according to the category to which the test data in multiple test data subsets belongs. Therefore, this solution can predict the behavior indicators of the object in a clustering manner, reducing the operation time of calculating the prediction value based on the test data, and thus, this solution can improve the efficiency of the prediction model quality assessment.
[0168] In this embodiment, a model quality assessment method is provided, which is applicable to a prediction terminal, such as Figure 2a As shown, the specific process of the model quality assessment method can be as follows:
[0169] 201. Receive a test data subset and a prediction model sent by an evaluation terminal.
[0170] There are various methods for receiving the test data subset and prediction model sent by the evaluation terminal, for example, receiving the test data subset and prediction model sent by the evaluation terminal through the network, for example, receiving the test data subset and prediction model sent by the evaluation terminal through wired communication, and so on.
[0171] In some embodiments, a prediction request carrying a prediction model and a test data subset sent by an evaluation terminal may be received, thereby acquiring the test data subset and the prediction model.
[0172] The prediction request may carry a prediction model and a test data subset, and may also carry an address of a database storing the prediction model and the test data subset, and the prediction terminal may obtain the prediction model and the test data subset by accessing the address. For example, in some embodiments, the prediction request may carry a test data subset and an address of a model database storing the prediction model, and the prediction terminal may obtain the prediction model by accessing the address of the model database.
[0173] In some embodiments, in order to prevent the test data subset and the prediction model sent by the evaluation terminal from being not yet ready to be received, the test data subset includes at least one test data. Therefore, before the test data subset and the prediction model sent by the evaluation terminal are received, the following steps may be specifically included:
[0174] Determine the current startup status and data acquisition status;
[0175] The startup status information and the data acquisition status information are sent to the evaluation terminal according to the current startup status and the data acquisition status.
[0176] Among them, the startup status information may include the current startup status of the predicted terminal, which may include an on state and a off state, etc.; the data acquisition status information may include the current data acquisition status of the predicted terminal, which may include a waiting acquisition state and a stopped acquisition state, etc.
[0177] 202. Use the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data.
[0178] There are many prediction methods, such as logistic regression, factorization machine, deep neural network (DNN), etc.
[0179] Among them, deep neural networks can include factorization machine-based neural networks (Factorization Machine Neural Network, FNN), sampling-based neural networks (Sampling Neural Network, SNN), and so on.
[0180] The prediction object may be an object behavior indicator, such as click volume, click rate, etc. In the following prediction process, the click volume will be described as the prediction object.
[0181] For example, in some embodiments, the prediction model is a deep neural network based on a factorization machine (Deep Factorization Machine, DeepFM), and the prediction model is used to predict the click volume of the test data in the test data subset, and obtaining the predicted value of the test data may specifically include the following steps:
[0182] a. Obtain a preset value range and a preset threshold value corresponding to the value range;
[0183] b. Use the prediction model to predict the click volume of the test data in the test data subset to obtain the prediction value to be processed for the test data.
[0184] c. Determine the numerical range of the predicted value to be processed.
[0185] d. Assign a value to the predicted value to be processed according to the preset threshold value corresponding to the numerical range of the predicted value to be processed to obtain the predicted value of the test data.
[0186] There are many methods for obtaining the preset numerical range and the preset threshold value corresponding to the numerical range. For example, the preset numerical range and the preset threshold value corresponding to the numerical range can be set by a person skilled in the art, or can be obtained from a database through a network, or can be obtained by reading a local memory, etc.
[0187] It should be noted that the preset numerical range obtained by the prediction terminal and the preset threshold corresponding to the numerical range can be the same as the preset numerical range obtained by the evaluation terminal and the preset threshold corresponding to the numerical range, or they can be different, and are not limited here. The prediction terminal assigns the predicted value to be processed according to the preset threshold corresponding to the numerical range of the predicted value to be processed, and the process of obtaining the predicted value of the test data can refer to the evaluation terminal, and will not be repeated here.
[0188] In some embodiments, the Deep&Wide model (a kind of DeepFM) can be used as a prediction model to predict the click volume of the test data in the test data subset to obtain the to-be-processed prediction value of the test data. The schematic diagram of the model structure can be referred to Figure 2b .
[0189] Among them, Figure 2b It can be seen that the deep&wide model is divided into the Deep part and the Wide part. After the test data in the test data subset is input into the FM part, and the test data in the test data subset is input into the DNN part, low-dimensional features can be extracted through the FM part and high-dimensional features can be extracted through the DNN part, so that the model can consider both low-dimensional features and high-dimensional features in the test data at the same time.
[0190] The features of the test data used in the Deep&Wide model include two categories: one is continuous features, which are mainly used for the training of the Deep part; the other is discrete features, which are mainly used for the training of the Wide part. Among them, continuous features can include real value features, embedding features, etc., and discrete features can include sparse features, cross features, etc.
[0191] When using the Deep&Wide model to predict click volume, since the weight update in the model will be affected by the joint influence of the Wide part and the Deep part on the model training error, the model can take into account both the low-dimensional features and high-dimensional features in the test data when predicting click volume, and its size and complexity can be controlled, thereby improving the overall model performance and prediction accuracy.
[0192] 203. When the prediction is completed, the predicted value of the test data is sent to the evaluation terminal.
[0193] For example, in some embodiments, after the predicted values of the test data in all test data subsets are calculated, prediction completion information can be sent to the evaluation terminal, and when the acquisition instruction returned by the evaluation terminal is received, the predicted values of the test data are sent to the evaluation terminal.
[0194] The model quality evaluation scheme provided in the embodiment of the present application can be applied in various model quality evaluation scenarios, such as: taking the prediction of advertisement click volume as an example, receiving the test data subset and the advertisement click volume prediction model sent by the evaluation terminal; using the Deep&Wide model to predict the click volume of the test data in the test data subset to obtain the predicted value of the test data; sending the prediction completion information to the evaluation terminal; and when receiving the acquisition instruction of the evaluation terminal, sending the predicted value of the test data to the evaluation terminal.
[0195] The solution provided in the embodiment of the present application can more efficiently obtain accurate advertising click prediction values, further reducing the computing time for calculating prediction values based on test data. Therefore, the solution can improve the efficiency of prediction model quality assessment.
[0196] As can be seen from the above, the embodiment of the present application can receive the test data subset and prediction model sent by the evaluation terminal; use the prediction model to predict the click volume of the test data in the test data subset to obtain the predicted value of the test data; send the prediction completion information to the evaluation terminal; when receiving the acquisition instruction of the evaluation terminal, send the predicted value of the test data to the evaluation terminal. Therefore, this solution can calculate the predicted value of the test data, reduce the amount of calculation of the evaluation terminal, and prevent the machine from freezing or crashing due to excessive calculation when the evaluation terminal performs model quality evaluation. Therefore, this solution can improve the efficiency of the prediction model quality evaluation.
[0197] The method described in the above embodiment will be further described in detail below.
[0198] In this embodiment, the method of the embodiment of the present invention is described in detail by taking an example of specific application in an advertisement click volume model evaluation scenario, using a cluster of multiple prediction terminals to calculate the click volume prediction value of test data, and the evaluation terminal performing click volume model quality evaluation based on these prediction values.
[0199] refer to Figure 3a ,The specific process of ad click model evaluation is as follows:
[0200] (1) The evaluation terminal obtains an advertisement click volume model and a test data subset, wherein the test data subset includes at least one test data marked with a true value.
[0201] The ad click volume model may be a Deep & Wide model, and the test data may include ad data and object data. Specifically, the ad data may include ad attribute data and ad location data. For example, ad attribute data may include ad content information, ad duration, number of ads, etc.; ad location data may include ad location information on a web page, ad time information on a web page, etc.; object data may include the age information, income information, gender information, education information, etc. of the object.
[0202] In some embodiments, the evaluation terminal may read the advertisement click volume model stored in the local memory, and obtain the test data set to be preprocessed from the database server through the network, wherein the test data set to be preprocessed may include at least one test data to be preprocessed.
[0203] Since the test data includes information in a variety of different formats, it is necessary to perform data preprocessing on all the test data to be preprocessed in the test data set to obtain preprocessed test data sets of uniform specifications, and finally perform partitioning operations on the preprocessed test data sets in these preprocessed test data sets to obtain test data subsets.
[0204] There are many preprocessing methods, for example, by determining the data type of the test data to be preprocessed, and then converting the data format of the test data to be preprocessed in the test data set to be preprocessed based on the data type, and finally obtaining the processed test data set.
[0205] Specifically, refer to Figure 3b The preprocessing flow diagram shown can read the test data to be preprocessed, analyze the data type of the test data to be preprocessed, and when the test data to be preprocessed is a multi-dimensional vector, convert the test data to be preprocessed into an integer (INT) type; when the test data to be preprocessed is a one-dimensional vector, convert the test data to be preprocessed into a bit string (ByteString) type, thereby converting the test data to be preprocessed into serialized codes through Protocol Buffers (a data representation language used to serialize data), and finally encapsulating the obtained sequences into a data frame (Data Frame) format to obtain a test data set.
[0206] Among them, the format diagram of the data frame can refer to Table 3, which can include three pre-processed test data, and the pre-processed test data can include advertising attribute data such as advertising content information, advertising location data such as the time and location of the advertisement on the web page, object data such as the age and income of the object, and the true value marked on each pre-processed test data, which can indicate whether an object of a specific age and income clicks on an advertisement with specific content, appearance time, and appearance location. For example, taking the test data numbered 00x0 as an example, it can be seen that an object with an age of x and an income of X clicked on an advertisement with content A, appearance time a, and appearance location a'; for another example, taking the test data numbered 00x1 as an example, it can be seen that an object with an age of y and an income of Y did not click on an advertisement with content B, appearance time b, and appearance location b'.
[0207]
[0208] Table 3
[0209] In some embodiments, after the evaluation terminal performs the above-mentioned serialization preprocessing, a processed test data set is obtained, and then a partitioning operation may be performed on the processed test data in the processed test data set.
[0210] For example, taking the processed test data set including 100 processed test data as an example, the evaluation terminal can divide the 100 processed test data into 10 parts according to the numbers of the 100 processed test data, each part including 10 processed test data, and taking every 10 processed test data as the test data subset.
[0211] For example, the evaluation terminal divides the test data numbered 0x00 to 0x09 into test data subset 1, divides the test data numbered 0x10 to 0x19 into test data subset 2, divides the test data numbered 0x20 to 0x29 into test data subset 3, ... divides the test data numbered 0x90 to 0x99 into test data subset 10, and so on.
[0212] (ii) The evaluation terminal sends the advertisement click volume model and the test data subset to the corresponding prediction terminal in the terminal cluster, as well as the prediction completion information of all prediction terminals in the listening terminal cluster.
[0213] The evaluation terminal may send the same advertisement click volume model and different test data subsets to each prediction terminal in the terminal cluster. After sending, the evaluation terminal may monitor the prediction completion information of all prediction terminals in the terminal cluster.
[0214] For example, in this embodiment, the evaluation terminal may send the advertisement click volume model and the test data subset to the prediction terminals according to the serial number sequence of the prediction terminals in the terminal cluster.
[0215] For example, taking the terminal cluster including 10 prediction terminals and the prediction terminal numbers as shown in Table 4, the test data subsets 1, 2, 3..., 10 obtained above can be sent to the 10 prediction terminals respectively, and the advertising click volume model can be sent to the 10 prediction terminals, that is, the test data subset 1 is sent to the prediction terminal numbered 1, the test data subset 2 is sent to the prediction terminal numbered 2, ... the test data subset 10 is sent to the prediction terminal numbered 10, and the advertising click volume model is sent to the 10 prediction terminals.
[0216] Prediction Terminal Prediction Terminal A Prediction Terminal B .. Prediction Terminal C serial number 1 2 .. 10
[0217] Table 4
[0218] Specifically, in some embodiments, in order to maintain efficient, secure and flexible communication between the prediction terminal and the evaluation terminal, the evaluation terminal and the prediction terminal in the terminal cluster can be arranged with the Spark architecture (a data processing framework), so that the evaluation terminal sends the advertising click model and the test data subset to the corresponding prediction terminal in the terminal cluster, as well as the prediction completion information of all prediction terminals in the listening terminal cluster.
[0219] The composition diagram of the Spark architecture can be found in Figure 3c In the Spark architecture, each prediction terminal in the terminal cluster acts as a worker node (Worker Node) and is used as an executor (Executor) to perform multiple prediction tasks (Task). The prediction task may include Task 1: the prediction terminal receives the test data subset and the advertisement click volume model sent by the evaluation terminal; Task 2: the prediction terminal uses the advertisement click volume model to predict the click volume of the test data in the test data subset to obtain the predicted click volume value of the test data; Task 3: the prediction terminal sends the prediction completion information to the evaluation terminal, and so on.
[0220] In some embodiments, the evaluation terminal can serve as a cluster manager to manage and monitor data reception, task reception, etc. of each prediction terminal.
[0221] In other embodiments, the relay server may also serve as a cluster management terminal to manage and monitor the data reception and task reception of each prediction terminal, while the evaluation terminal serves as a driver program to manage and monitor the data reception and task reception of each prediction terminal by communicating with the relay server.
[0222] In another embodiment, the evaluation terminal may be composed of a cluster management terminal and a driving terminal.
[0223] Specifically, the Spark operation process diagram can be referred to Figure 3d ,like Figure 3d As shown, the specific steps include:
[0224] a. Build the Spark operating environment and start the driver;
[0225] b. The driver applies for executor resources of the working nodes in the running terminal cluster;
[0226] c. The executor applies to the driver for the task to be executed;
[0227] d. The driver distributes the prediction task to the executor;
[0228] e. The driver side constructs a directed acyclic graph (DAG), decomposes the DAG into multiple stages, and sends tasks to the executors of the working nodes according to the task scheduler in the driver side;
[0229] f. The executor runs the task, and when the execution is completed, the executor releases all computing resources.
[0230] (3) The prediction terminal receives the test data subset and the advertisement click volume model sent by the evaluation terminal.
[0231] The specific manner in which the prediction terminal receives the test data subset and the advertisement click volume model sent by the evaluation terminal can refer to the above-mentioned Spark operation process, which will not be described in detail here.
[0232] (iv) The prediction terminal uses the advertisement click volume model to predict the click volume of the test data in the test data subset to obtain the predicted click volume value of the test data.
[0233] For example, the prediction terminal may use the advertisement click volume model (a Deep & Wide model) to predict the click volume of the test data in the test data subset to obtain the predicted click volume value of the test data.
[0234] The specific prediction process of the prediction terminal may refer to step 202, which will not be described in detail here.
[0235] (V) The prediction terminal sends prediction completion information to the evaluation terminal.
[0236] The specific process of the prediction terminal sending the prediction completion information to the evaluation terminal may refer to step 203, which will not be described in detail here.
[0237] (VI) When the evaluation terminal monitors the prediction completion information sent by the prediction terminal, the evaluation terminal obtains the click volume prediction value of the test data from the prediction terminal, and calculates the test data of the sub-prediction value based on the prediction value and the true value of the test data sent by all the prediction terminals in the terminal cluster.
[0238] For example, in some embodiments, in order to standardize the accuracy of the test data in the test data subset, so that the obtained model quality parameters are more accurate, and at the same time reduce the amount of calculation when the evaluation terminal calculates the model quality parameters, reference Figure 3e As shown in the assignment processing flow chart, the evaluation terminal can adjust the accuracy of the click volume prediction value obtained from the prediction terminal, and then calculate the AUC based on the true value and the click volume prediction value after the accuracy adjustment to obtain the model quality sub-parameter, and finally calculate the model quality parameter based on multiple model quality sub-parameters.
[0239] Specifically, the steps for precision adjustment are as follows:
[0240] a. Obtain a preset value range and a preset threshold value corresponding to the value range;
[0241] b. Determine the numerical range of the predicted value of the test data;
[0242] c. Assign the predicted value according to the preset threshold value corresponding to the numerical range of the predicted value to obtain the predicted value of the test data after assignment processing.
[0243] For example, according to the preset numerical range provided in Table 2 and the preset threshold value corresponding to the numerical range, the numerical range of the predicted value of the test data can be determined. For example, assuming that the predicted value is 0.211, the predicted value is compared with the numerical range in Table 2 to find that the numerical range of the predicted value is (0.100, 0.255], then the preset threshold value 0.2 corresponding to the range is assigned to the predicted value, and the predicted value after the assignment is 0.2.
[0244] After completing the above-mentioned accuracy adjustment, the model quality parameters of the prediction model can be calculated based on the true values of all test data and the predicted values after accuracy adjustment.
[0245] For example, taking the model quality parameter AUC as an example, the evaluation terminal calculates the AUC of the prediction model according to the true value of all test data and the prediction value after precision adjustment. Please refer to step 105, so it will not be repeated here.
[0246] This embodiment can improve the efficiency of prediction model quality assessment. The specific effect can be referred to Table 5, where Table 5 shows the time consumed by the model quality assessment method provided by the present invention and the traditional model quality assessment method when processing test data subsets with different numbers of dimensions and types, and the improvement of time consumption reduction.
[0247] The number of dimensions refers to the number of dimensions of the multidimensional vector data in the test data subset, and the number of types refers to the number of types of data in the test data subset.
[0248] Number of dimensions Number of types Traditional methods are time-consuming The present invention is time consuming Lift 12 0 4805s 123s 3907% 13 1 6134s 141s 4350% 15 3 6134s 152s 1036%
[0249] Table 5
[0250] In addition, the present invention can also handle large sample model quality assessment that the original model quality assessment method cannot handle, where large samples refer to sample data with a number of millions. For example, when the number of test data subsets reaches 1.9 billion, the number of dimensions reaches 12, and the number of types is 1, the present invention can take about 17 minutes and 47 seconds.
[0251] As can be seen from the above, in this embodiment, the evaluation terminal can obtain the advertising click volume model and the test data subset, and send the advertising click volume model and the test data subset to the corresponding prediction terminal in the terminal cluster, and then monitor the prediction completion information of all prediction terminals in the terminal cluster, wherein the test data subset includes at least one test data marked with a true value; the prediction terminal can receive the test data subset and the advertising click volume model sent by the evaluation terminal, and use the advertising click volume model to predict the click volume of the test data in the test data subset to obtain the click volume prediction value of the test data, and finally send the prediction completion information to the evaluation terminal; when the evaluation terminal monitors the prediction completion information sent by the prediction terminal, the evaluation terminal can maliciously obtain the click volume prediction value of the test data from the prediction terminal, and calculate the model quality parameters of the prediction model based on the predicted value and true value of the test data sent by all prediction terminals in the terminal cluster.
[0252] Therefore, this solution can predict object behavior indicators in a clustering manner. The evaluation terminal does not need to calculate the corresponding click prediction value of the test data based on the advertisement click model, thereby reducing the calculation amount of the evaluation terminal. At the same time, it can also ensure that the weights, biases, etc. in the advertisement click model will not change, so that it can be saved for future use; in addition, multiple prediction terminals can perform click prediction at the same time, greatly reducing the time required for click prediction. Therefore, this solution can improve the efficiency of prediction model quality evaluation.
[0253] In order to better implement the above method, the embodiment of the present application also provides a model quality assessment device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer; the server can be a single server or a server cluster composed of multiple servers.
[0254] In this embodiment, the method of the embodiment of the present invention is described in detail by taking the model quality assessment device being specifically integrated into a single server as an example.
[0255] For example, Figure 4 As shown, the model quality assessment device may include an acquisition unit 401, a sending unit 402, a prediction value unit 403, a classification unit 404 and a calculation unit 405 as follows:
[0256] (I) Acquisition unit 401:
[0257] The acquisition unit is used to acquire the prediction model and multiple test data subsets, wherein the test data subset includes at least one test data marked with a true value.
[0258] In some embodiments, the acquisition unit 401 includes an acquisition subunit and a division subunit:
[0259] An acquisition subunit, used to acquire a prediction model and a test data set, wherein the test data set includes at least one test data marked with a true value;
[0260] The partitioning subunit is used to perform a partitioning operation on the test data in the test data set to obtain multiple test data subsets.
[0261] In some embodiments, the acquisition subunit may be specifically used for:
[0262] Obtain the prediction model and the test data set to be preprocessed;
[0263] Determine the data type of the test data to be preprocessed;
[0264] The data format of the test data to be preprocessed in the test data set to be preprocessed is converted based on the data type to obtain a test data set.
[0265] (ii) Sending unit 402:
[0266] The sending unit is used to send a prediction request carrying a prediction model and a test data subset to a corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset.
[0267] In some embodiments, the sending unit 402 may be specifically configured to:
[0268] Obtaining the startup status information and data acquisition status information of the prediction terminal;
[0269] Determine the startup state and data acquisition state of the prediction terminal according to the startup state information and the data acquisition state information;
[0270] When the prediction terminal is in the closed state, the prediction terminal is turned on;
[0271] When the startup state of the prediction terminal is the on state and the data acquisition state is the waiting state, the test data subset and the prediction model are sent to the prediction terminal.
[0272] (III) Prediction value unit 403:
[0273] The prediction value unit 403 is used to obtain the prediction value of the test data from the prediction terminal when the prediction terminal in the terminal cluster completes the prediction, so as to obtain the prediction value of the test data in each test data subset.
[0274] (IV) Classification unit 404:
[0275] The classification unit 404 is used to classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data belongs in multiple test data subsets.
[0276] In some embodiments, the classification unit 404 may be specifically configured to:
[0277] Determine a target prediction value from the prediction values of the test data sent by all prediction terminals in the terminal cluster;
[0278] Classify the test data based on the target predicted value, the predicted value of the test data, and the true value.
[0279] (V) Calculation unit 405:
[0280] The calculation unit is used to calculate the model quality parameter of the prediction model according to the category to which the test data in the multiple test data subsets belong.
[0281] In some embodiments, the calculation unit 405 may be specifically configured to:
[0282] Count the proportion of each category in all categories;
[0283] The model quality parameter of the prediction model is calculated based on the proportion of each category in all categories.
[0284] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0285] As can be seen from the above, the model quality assessment device of this embodiment can obtain the prediction model and multiple test data subsets by the acquisition unit, and the test data subset includes at least one test data marked with a true value; the sending unit sends a prediction request carrying the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset; when the prediction terminal in the terminal cluster completes the prediction, the prediction value unit obtains the prediction value of the test data from the prediction terminal to obtain the prediction value of the test data in each test data subset; the classification unit classifies the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in multiple test data subsets belongs; the calculation unit calculates the model quality parameters of the prediction model according to the category to which the test data in multiple test data subsets belongs.
[0286] Since this scheme can use a clustering approach to predict object behavior indicators and distribute multiple test data to the prediction terminals in the prediction cluster so that they can calculate the predicted values, it can greatly reduce the computing time consumed by the evaluation terminal to obtain the predicted values of all test data, while reducing the calculation amount of the evaluation terminal. Therefore, this scheme can improve the efficiency of the quality evaluation of the prediction model.
[0287] In order to better implement the above method, the embodiment of the present application also provides a model quality assessment device, which can be integrated in an electronic device, wherein the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, or a personal computer; the server can be a single server or a server cluster composed of multiple servers. In some embodiments, the model quality assessment device can also be integrated in a terminal cluster composed of multiple assessment terminals.
[0288] In this embodiment, the method of the embodiment of the present invention is described in detail by taking each server being integrated with the model quality assessment device as an example.
[0289] For example, Figure 5 As shown, the model quality assessment device may include a receiving unit 501, a prediction unit 502, and a sending unit 503, as follows:
[0290] (I) Receiving unit 501:
[0291] The receiving unit 501 is configured to receive a test data subset and a prediction model sent by an evaluation terminal, wherein the test data subset includes at least one test data.
[0292] (ii) Prediction unit 502:
[0293] The prediction unit 502 is used to predict the test data in the test data subset using the prediction model to obtain the predicted value of the test data.
[0294] In order to reduce the distribution range of the predicted values, thereby reducing the computing resources required for storing and sending the predicted values, in some embodiments, the numerical range of the predicted values may be standardized, and the prediction unit 502 may be specifically used to:
[0295] Obtain a preset value range and a preset threshold value corresponding to the value range;
[0296] The prediction model is used to predict the click volume of the test data in the test data subset to obtain the prediction value to be processed of the test data;
[0297] Determine the numerical range of the predicted value to be processed;
[0298] The predicted value to be processed is assigned a value according to a preset threshold value corresponding to the numerical range of the predicted value to be processed, so as to obtain the predicted value of the test data.
[0299] (III) Sending unit 503:
[0300] The sending unit 503 is used to send the predicted value of the test data to the evaluation terminal when the prediction is completed.
[0301] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0302] As can be seen from the above, the model quality assessment device of this embodiment obtains a preset numerical range and a preset threshold value corresponding to the numerical range by the receiving unit; the prediction unit predicts the test data in the test data subset using the prediction model to obtain the predicted value of the test data; when the prediction is completed, the sending unit sends the predicted value of the test data to the evaluation terminal. Since this solution can reduce the distribution range of the predicted value, thereby reducing the computing resources required to store and send the predicted value, this solution can improve the efficiency of the prediction model quality assessment.
[0303] The embodiment of the present application also provides an evaluation terminal, which can be a server, a personal computer, a mobile phone, a tablet computer, a micro processing box, a drone, or a data processing device, etc. Figure 6As shown, it shows a schematic diagram of the structure of the evaluation terminal involved in the embodiment of the present application, specifically:
[0304] The evaluation terminal may include one or more processors 601 of processing cores, one or more computer-readable storage media memories 602, a power supply 603, an input module 604, and a communication module 605. Those skilled in the art will appreciate that Figure 4 The evaluation terminal structure shown in the figure does not constitute a limitation on the evaluation terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0305] The processor 601 is the control center of the evaluation terminal. It uses various interfaces and lines to connect various parts of the entire evaluation terminal. By running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, it executes various functions of the evaluation terminal and processes data, thereby monitoring the evaluation terminal as a whole. In some embodiments, the processor 601 may include one or more processing cores; in some embodiments, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, object interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.
[0306] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the evaluation terminal, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.
[0307] The evaluation terminal also includes a power supply 603 for supplying power to various components. In some embodiments, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 603 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0308] The evaluation terminal may further include an input module 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to object settings and function controls.
[0309] The evaluation terminal may also include a communication module 605. In some embodiments, the communication module 605 may include a wireless module. The evaluation terminal may perform short-range wireless transmission through the wireless module of the communication module 605, thereby providing the subject with wireless broadband Internet access. For example, the communication module 605 may be used to help the subject send and receive e-mail , browse the web, access streaming media, etc.
[0310] Although not shown, the evaluation terminal may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 601 in the evaluation terminal will load the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application programs stored in the memory 602, thereby realizing various functions, as follows:
[0311] Obtain a prediction model and multiple test data subsets, wherein the test data subset includes at least one test data marked with a true value; send a prediction request carrying the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster through the communication module 605, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset; when the prediction terminal in the terminal cluster completes the prediction, obtain the predicted value of the test data from the prediction terminal, and obtain the predicted value of the test data in each test data subset; classify the test data according to the predicted value and the true value of the test data, and obtain the category to which the test data in multiple test data subsets belongs; calculate the model quality parameter of the prediction model according to the category to which the test data in multiple test data subsets belongs.
[0312] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0313] From the above, it can be seen that in the present application, the evaluation terminal adopts a clustering approach to predict the object behavior indicators, and distributes multiple test data to the prediction terminals in the prediction cluster through the communication module, so that they can calculate the predicted values. This can greatly reduce the computing time consumed by the evaluation terminal to obtain the predicted values of all test data, and at the same time reduce the calculation amount of the evaluation terminal. Therefore, this solution can improve the efficiency of the quality evaluation of the prediction model.
[0314] The embodiment of the present application also provides a prediction terminal, which can be a server, a personal computer, a mobile phone, a tablet computer, a micro processing box, a drone, or an image acquisition device, etc.
[0315] Specifically, the structure of the prediction terminal may be similar to that of the evaluation terminal, for example, including a processor with one or more processing cores, a memory with one or more computer-readable storage media, a power supply, an input module, a communication module and other components.
[0316] In some embodiments, in order to improve the efficiency of calculating the prediction value, the prediction value of all prediction data may be calculated by a terminal cluster, wherein the terminal cluster may include multiple prediction terminals, and the processor in the prediction terminal may load the executable file corresponding to the process of one or more application programs into the memory according to the following instructions, and the processor may run the application program stored in the memory, thereby realizing various functions, as follows:
[0317] The prediction terminal receives the test data subset and the prediction model sent by the evaluation terminal through the communication module; uses the prediction model to predict the click volume of the test data in the test data subset to obtain the predicted value of the test data; sends the prediction completion information to the evaluation terminal; and when receiving the acquisition instruction of the evaluation terminal, sends the predicted value of the test data to the evaluation terminal.
[0318] The specific structure of the prediction terminal can be found in the above-mentioned embodiment of the evaluation terminal, which will not be described in detail here.
[0319] As can be seen from the above, this embodiment can receive the test data subset and prediction model sent by the evaluation terminal; use the prediction model to predict the click volume of the test data in the test data subset to obtain the predicted value of the test data; send the prediction completion information to the evaluation terminal; when receiving the acquisition instruction of the evaluation terminal, send the predicted value of the test data to the evaluation terminal. Therefore, in this embodiment, each prediction terminal in the terminal cluster can predict the object behavior index, thereby reducing the computing time consumed by calculating the predicted value of all test data, and thus, this solution can improve the efficiency of the prediction model quality evaluation.
[0320] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0321] To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the model quality assessment methods provided in the embodiment of the present application. For example, the instructions can execute the following steps:
[0322] Obtain a prediction model and multiple test data subsets, wherein the test data subset includes at least one test data marked with a true value; send a prediction request carrying the prediction model and the test data subset to the corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct the prediction terminal to use the prediction model to predict the test data in the test data subset; when the prediction terminal in the terminal cluster completes the prediction, obtain the predicted value of the test data from the prediction terminal to obtain the predicted value of the test data in each test data subset; classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in multiple test data subsets belongs; calculate the model quality parameters of the prediction model according to the category to which the test data in multiple test data subsets belongs.
[0323] In some embodiments, the instructions stored in the storage medium may further perform the following steps:
[0324] A test data subset and a prediction model are received from an evaluation terminal, wherein the test data subset includes at least one test data; the prediction model is used to predict the test data in the test data subset to obtain a predicted value of the test data; and when the prediction is completed, the predicted value of the test data is sent to the evaluation terminal.
[0325] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0326] Since the instructions stored in the storage medium can execute the steps in any model quality assessment method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model quality assessment method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0327] The above is a detailed introduction to a model quality assessment method and device provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A model quality assessment method, characterized in that: Suitable for evaluation terminals including: Obtain a prediction model and at least two test data subsets, wherein the test data subsets include at least one test data annotated with a true value, wherein the prediction model is used to predict an object behavior indicator, wherein the object behavior refers to a model quality parameter of the prediction model calculated according to the category to which the test data in the test data subsets belong, and is used to reflect the behavior of the object on the network for specific network information, wherein the specific network information includes advertisements, recommended videos, and news headlines; Sending a prediction request carrying the prediction model and the test data subset to a corresponding prediction terminal in the terminal cluster, wherein the prediction request is used to instruct each of the prediction terminals to use the prediction model to predict the test data in the test data subset carried by the prediction request, the terminal cluster includes multiple prediction terminals, and each of the prediction terminals carries the prediction model carried by the prediction request after obtaining the prediction request; When the prediction terminal in the terminal cluster completes the prediction, the prediction value of the test data is obtained from the prediction terminal to obtain the prediction value of the test data in each test data subset predicted by each prediction terminal in the terminal cluster; Classify the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs; Calculating a model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs; The step of classifying the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs includes: Selecting a target prediction value from the prediction values of the test data sent by all prediction terminals in the terminal cluster; Based on the target prediction value, the prediction value and the true value, the test data is classified to obtain the category to which the test data in the test data subset belongs, and the category includes true sample data, false positive sample data, true negative sample data, and false negative sample data.
2. The model quality assessment method according to claim 1, characterized in that: Calculating the model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs includes: Count the proportion of each category in all categories; The model quality parameter of the prediction model is calculated based on the proportion of each category in all categories.
3. The model quality assessment method according to claim 1, characterized in that: Obtain a predictive model and at least two test data subsets, including: Obtaining a prediction model and a test data set, wherein the test data set includes at least one test data marked with a true value; A partitioning operation is performed on the test data in the test data set to obtain at least two test data subsets.
4. The model quality assessment method according to claim 3, characterized in that: Get the prediction model and test dataset, including: Obtain the prediction model and the test data set to be preprocessed; Determining the data type of the test data to be preprocessed; The data format of the test data to be preprocessed in the test data set to be preprocessed is converted based on the data type to obtain a test data set.
5. The model quality assessment method according to claim 1, characterized in that: The step of obtaining the predicted value of the test data from the prediction terminal further includes: Obtaining a preset value range and a preset threshold value corresponding to the value range; Determining a numerical range within which the predicted value of the test data lies; Performing value assignment processing on the predicted value according to a preset threshold value corresponding to the numerical range of the predicted value, to obtain a predicted value of the test data after value assignment processing; Classifying the test data according to the predicted value and the true value of the test data to obtain the category to which the test data in the test data subset belongs includes: The test data is classified according to the predicted value after the assignment processing of the test data to obtain the category to which the test data in the test data subset belongs.
6. The model quality assessment method according to any one of claims 1 to 5, characterized in that: Sending a prediction request carrying the prediction model and the test data subset to a corresponding prediction terminal in the terminal cluster includes: Obtaining the startup status information and data acquisition status information of the prediction terminal; Determine the startup state and data acquisition state of the prediction terminal according to the startup state information and the data acquisition state information; When the startup state of the prediction terminal is the shutdown state, turning on the prediction terminal; When the startup state of the prediction terminal is the on state, and the data acquisition state is the waiting state, a prediction request carrying the prediction model and the test data subset is sent to a corresponding prediction terminal in the terminal cluster.
7. A model quality assessment method, characterized in that: Applicable to forecast terminals, including: Receiving a test data subset and a prediction model sent by an evaluation terminal, wherein the test data subset includes at least one test data marked with a true value, and the prediction model is used to predict an object behavior indicator, wherein the object behavior indicator is used to reflect the behavior of the object on the network for specific network information, wherein the specific network information includes advertisements, recommended videos, and news headlines; Using the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data; When the prediction is completed, the predicted value of the test data is sent to the evaluation terminal so that the evaluation terminal can select a target predicted value from the predicted values of the test data sent by all prediction terminals in the terminal cluster; based on the target predicted value, the predicted value and the true value, the test data is classified to obtain the category to which the test data in the test data subset belongs, the category including true sample data, false positive sample data, true negative sample data, and false negative sample data; the model quality parameters of the prediction model are calculated according to the category to which the test data in the test data subset belongs.
8. The model quality assessment method according to claim 7, characterized in that: Using the prediction model to predict the test data in the test data subset to obtain the predicted value of the test data includes: Obtaining a preset value range and a preset threshold value corresponding to the value range; Using the prediction model to predict the test data in the test data subset to obtain the to-be-processed prediction value of the test data; Determine the numerical range of the predicted value to be processed; The predicted value to be processed is assigned a value according to a preset threshold value corresponding to the numerical range of the predicted value to be processed to obtain the predicted value of the test data.
9. A model quality assessment device, characterized in that: include: an acquisition unit, configured to acquire a prediction model and at least two test data subsets, wherein the test data subsets include at least one test data annotated with a true value, wherein the prediction model is used to predict an object behavior indicator, wherein the object behavior indicator is used to reflect the behavior of the object on the network for specific network information, wherein the specific network information includes advertisements, recommended videos, and news headlines; A sending unit, configured to send a prediction request carrying the prediction model and a test data subset to a corresponding prediction terminal in a terminal cluster, wherein the prediction request is used to instruct each of the prediction terminals to use the prediction model to predict the test data in the test data subset carried by the prediction request, and the terminal cluster includes a plurality of prediction terminals, and each of the prediction terminals carries the prediction model carried by the prediction request after obtaining the prediction request; A prediction value unit, configured to obtain the prediction value of the test data from the prediction terminal when the prediction terminal in the terminal cluster completes the prediction, and obtain the prediction value of the test data in each test data subset predicted by each prediction terminal in the terminal cluster; A classification unit, used to classify the test data according to the predicted value and the true value of the test data, so as to obtain the category to which the test data in the test data subset belongs; A calculation unit, configured to calculate a model quality parameter of the prediction model according to the category to which the test data in the test data subset belongs; The classification unit is used to: select a target prediction value from the prediction values of the test data sent by all prediction terminals in the terminal cluster; classify the test data based on the target prediction value, the prediction value and the true value to obtain the category to which the test data in the test data subset belongs, and the category includes true sample data, false positive sample data, true negative sample data, and false negative sample data.
10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the model quality assessment method as described in any one of claims 1 to 6 or 7 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the model quality assessment method according to any one of claims 1 to 6 or 7 to 8.
