Model quantitative evaluation method, device and equipment

Abstract

The invention provides a model quantitative evaluation method, device and equipment, and relates to the technical field of artificial intelligence recommendation, and the method comprises the steps: taking an evaluation sample set which comprises a plurality of evaluation samples and the label information of each evaluation sample; evaluating each evaluation sample by adopting a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on a training sample and a label information sample; target quantitative information corresponding to the recommendation model is determined based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, and the target quantitative information is used for evaluating the online condition of the recommendation model. Through the evaluation of the evaluation sample by the recommendation model, the label information of the evaluation sample and the real recommendation information, the evaluation of the condition after the recommendation model is online is quantitatively obtained under the condition that the recommendation model is not directly online, and the accuracy of model evaluation is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence recommendation technology, and in particular to a model quantitative evaluation method, device and equipment. Background Art

[0002] With the development of the Internet, recommendation systems, as the main system of the Internet, have been affecting people's lives all the time. In various application scenarios, recommendation models are the core components of recommendation systems. Recommendation systems recommend different items to different users through recommendation models, which directly affects the user experience of recommendation services. In recent years, with the rapid development of recommendation model technology, the research on recommendation models has become an important branch in the field of artificial intelligence.

[0003] In the industry, the recommendation model, which is the core of the recommendation system, is also frequently iterated and updated in pursuit of better services and user experience. The iterative optimization of the recommendation model is divided into offline and online. Considering the frequency of recommendation model iteration and the stability of the online system, the common practice is to first evaluate offline whether the iterative optimization of the recommendation model has achieved an improvement in offline indicators. If there is an improvement in offline indicators, the next step of online verification will be continued, otherwise the optimization will continue offline. Based on the above iterative path, the evaluation method of the offline model is particularly important, because the offline evaluation determines whether the subsequent online verification effect and the direction of subsequent iterative optimization.

[0004] At present, the commonly used offline evaluation indicators for recommendation models include the area under the ROC curve (AUC), group AUC (GAUC) and top-k accuracy, etc. However, these indicators are all ranking metrics, and can only qualitatively evaluate that the better the ranking, the better the online effect, but cannot quantitatively and directly reflect the positive online effect, resulting in low accuracy of model evaluation. Summary of the invention

[0005] The present invention provides a model quantitative evaluation method, device and equipment to solve the problem of low accuracy of model evaluation.

[0006] The present invention provides a model quantitative evaluation method, comprising: Acquire an evaluation sample set, the evaluation sample set including a plurality of evaluation samples and label information of each of the evaluation samples; Using a recommendation model to evaluate each of the evaluation samples respectively to obtain recommendation information corresponding to each of the evaluation samples; the recommendation model is obtained by training based on training samples and label information samples; Based on the recommendation information corresponding to each of the evaluation samples, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used to evaluate the situation of the recommendation model after it is put into operation.

[0007] According to a model quantitative evaluation method provided by the present invention, the target quantitative information corresponding to the recommendation model is determined based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, including: Respectively comparing the recommendation information corresponding to each of the evaluation samples with the real recommendation information of each of the evaluation samples; Based on the comparison result, each of the evaluation samples is divided into a first set or a second set; Based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set, and a preset target label, target quantitative information corresponding to the recommendation model is determined.

[0008] According to a model quantitative evaluation method provided by the present invention, the method of dividing each evaluation sample into a first set or a second set based on the comparison result includes: When the comparison result shows that the recommendation information and the real recommendation information are the same, dividing the evaluation samples corresponding to the recommendation information into the first set; When the comparison result is that the recommendation information and the real recommendation information are different, the evaluation samples corresponding to the recommendation information are divided into the second set.

[0009] According to a model quantitative evaluation method provided by the present invention, the target quantitative information corresponding to the recommendation model is determined based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and a preset target label, including: Respectively determining a first quantity of the label information of each evaluation sample in the first set being the preset target label information and a second quantity of the label information of each evaluation sample in the second set being the preset target label information; Target quantitative information corresponding to the recommendation model is determined based on the first number, the second number, a third number of each of the samples in the first set, and a fourth number of each of the samples in the second set.

[0010] According to a model quantitative evaluation method provided by the present invention, determining the target quantitative information corresponding to the recommendation model based on the first number, the second number, the third number of each of the samples in the first set, and the fourth number of each of the samples in the second set includes: Determine a ratio of the first number to the third number as an average a posteriori accuracy rate of the first set; Determine the ratio of the sum of the first number and the second number to the sum of the third number and the fourth number as the posterior accuracy of the overall set; the overall set includes the first set and the second set; The difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

[0011] According to a model quantitative evaluation method provided by the present invention, the method further includes: The ratio of the second number to the fourth number is determined as the average posterior accuracy of the second set.

[0012] The present invention also provides a model quantitative evaluation device, comprising the following modules: An acquisition module, used to acquire an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; A first evaluation module is used to evaluate each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on training samples and label information samples; The second evaluation module is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, and the target quantitative information is used to evaluate the situation of the recommendation model after it is put into operation.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned model quantitative evaluation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for quantitatively evaluating a model as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned model quantitative evaluation methods.

[0016] The model quantitative evaluation method, device and equipment provided by the present invention obtain an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; each evaluation sample is evaluated by a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used to evaluate the situation after the recommendation model is put online. Through the evaluation of the evaluation samples by the recommendation model, the label information of the evaluation samples and the real recommendation information, the evaluation of the situation after the recommendation model is put online can be quantitatively obtained without directly going online, thereby improving the accuracy of the model evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the model quantitative evaluation method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the model quantitative evaluation method provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the model quantitative evaluation device provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 2 The quantitative evaluation method of the model of the present invention is described.

[0024] Figure 1 is one of the flow charts of the model quantitative evaluation method provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 103.

[0025] Step 101: Acquire an evaluation sample set, where the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample.

[0026] It should be noted that the model quantitative evaluation method provided by the present invention can be applied to the quantitative evaluation scenario of the recommendation model in the recommendation system in the field of artificial intelligence recommendation or recommendation model driven to improve the quality of products or services. The field of artificial intelligence recommendation or recommendation model driven to improve the quality of products or services is sensitive to the changes in the effect of the recommendation model. Therefore, it is often necessary to iterate and update the recommendation model to improve the effect of the recommendation model, thereby improving the quality of recommended products or services and improving user experience. The execution subject of the method can be a model quantitative evaluation device, such as an electronic device, or a control module in the model quantitative evaluation device for executing the model quantitative evaluation method.

[0027] Specifically, an evaluation sample set is obtained from the data layer. The evaluation sample set includes multiple evaluation samples and label information of each evaluation sample. The evaluation sample may be business evaluation data, and the label information may be a recommendation or other label.

[0028] The data layer is a module that automatically maintains data. It supports data set upload or database table specification, model file upload or model file storage location specification, model estimation result direct upload or database table specification. In addition, the data layer supports data cleaning and automatic storage in the underlying data table of the business scenario, as well as further deep processing of data to extract important indicators for presentation and analysis of business concerns. The data layer provides functions such as automatic data collection, cleaning, processing and query based on big data tools, providing basic support for query and analysis of business data.

[0029] Step 102: Use a recommendation model to evaluate each evaluation sample to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples.

[0030] Specifically, the recommendation model is trained based on the training samples and the label information samples, and the recommendation model is used to evaluate each of the evaluation samples. After the recommendation model is trained, each evaluation sample is input into the recommendation model to obtain the recommendation information (i.e., the recommended topic P_item) corresponding to each evaluation sample output by the recommendation model.

[0031] Step 103: Based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, determine the target quantitative information corresponding to the recommendation model, wherein the target quantitative information is used to evaluate the situation of the recommendation model after it is put into operation.

[0032] Specifically, the real recommendation information is the actual recommendation information of each evaluation sample (i.e., the actual recommendation topic R_item). For example, different insurance products are recommended online in the insurance recommendation scenario. Since different recommended products have different conversion rate data, the offline recommendation model is used to estimate the recommendation results of each recommended insurance product and the actual online recommendation effect. The subsequent online effect of this offline recommendation model can be quantitatively evaluated.

[0033] Based on the recommendation information, label information and real recommendation information corresponding to each evaluation sample, the target quantitative information corresponding to the recommendation model can be determined, wherein the target quantitative information is used to evaluate the situation after the recommendation model is launched, that is, the online recommendation effect or the benefits obtained after the recommendation model is launched.

[0034] The model quantitative evaluation method provided by the present invention obtains an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; each evaluation sample is evaluated by a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used to evaluate the situation after the recommendation model is put online. Through the evaluation of the evaluation samples by the recommendation model, the label information of the evaluation samples and the real recommendation information, the evaluation of the situation after the recommendation model is put online can be quantitatively obtained without directly going online, thereby improving the accuracy of the model evaluation.

[0035] Optionally, the specific implementation of step 103 includes: The recommendation information corresponding to each of the evaluation samples is compared with the actual recommendation information of each of the evaluation samples; based on the comparison result, each of the evaluation samples is divided into a first set or a second set; based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set and the preset target label, the target quantitative information corresponding to the recommendation model is determined.

[0036] Specifically, the recommendation information corresponding to each evaluation sample is compared with the real recommendation information of each evaluation sample, and based on the comparison result, each evaluation sample is divided into a first set or a second set, wherein the first set includes evaluation samples corresponding to the same recommendation information as the real recommendation information, i.e., the set JE of P_item=R_item, and the second set includes evaluation samples corresponding to the recommendation information different from the real recommendation information, i.e., the set JN of P_item!=R_item. Then, based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set and the preset target label, the target quantitative information corresponding to the recommendation model can be determined.

[0037] The model quantitative evaluation method provided by the present invention compares the recommendation information corresponding to each evaluation sample estimated offline with the actual recommendation information recommended online, quantitatively obtains the approximate benefits that can be obtained after the recommendation model is put online, better points out the direction for the iteration of the model, and greatly accelerates the optimization process of the model and the system.

[0038] Optionally, the dividing each of the evaluation samples into the first set or the second set based on the comparison result includes: When the comparison result is that the recommendation information and the true recommendation information are the same, the evaluation samples corresponding to the recommendation information are divided into the first set; when the comparison result is that the recommendation information and the true recommendation information are different, the evaluation samples corresponding to the recommendation information are divided into the second set.

[0039] Specifically, when the comparison result is that the recommended information and the actual recommended information are the same, it means that the information recommended by the recommendation model is consistent with the actual information, the performance of the recommendation model is relatively good, and the evaluation samples corresponding to the recommended information are divided into the first set; when the comparison result is that the recommended information and the actual recommended information are different, it means that the information recommended by the recommendation model is inconsistent with the actual information, the performance of the recommendation model is relatively poor, and the evaluation samples corresponding to the recommended information are divided into the second set.

[0040] Optionally, determining the target quantitative information corresponding to the recommendation model based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set, and a preset target label includes: Respectively determine a first quantity of the label information of each evaluation sample in the first set that is the preset target label information and a second quantity of the label information of each evaluation sample in the second set that is the preset target label information; based on the first quantity, the second quantity, the third quantity of each sample in the first set and the fourth quantity of each sample in the second set, determine the target quantitative information corresponding to the recommendation model.

[0041] Specifically, by comparing the label information of each evaluation sample in the first set with the preset target label information, it can be determined that the label information of each evaluation sample in the first set is a first quantity of the preset target label information, and by comparing the label information of each evaluation sample in the second set with the preset target label information, it can be determined that the label information of each evaluation sample in the second set is a second quantity of the preset target label information. For example, assuming that there are JEM evaluation samples in the first set (set JE), JNM evaluation samples in JN, and the target label information is whether to click (i.e., the posterior label is whether to click), JEC samples in set JE are clicked samples, and JNC samples in set JN are clicked samples.

[0042] Based on the first number, the second number, the third number of each sample in the first set, and the fourth number of each sample in the second set, target quantitative information corresponding to the recommendation model may be determined.

[0043] Optionally, determining the target quantitative information corresponding to the recommendation model based on the first number, the second number, a third number of each of the samples in the first set, and a fourth number of each of the samples in the second set includes: The ratio of the first quantity to the third quantity is determined as the average posterior accuracy of the first set; the ratio of the sum of the first quantity and the second quantity to the sum of the third quantity and the fourth quantity is determined as the posterior accuracy of the overall set; the overall set includes the first set and the second set; the difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

[0044] For example, the average posterior accuracy of the first set JES = JEC / JEM, the posterior accuracy of the entire set JTS = (JEC+JNC) / (JEM+JNM), and the target quantitative information Target corresponding to the recommendation model = JES-JTS.

[0045] Optionally, the method further comprises: The ratio of the second number to the fourth number is determined as the average posterior accuracy of the second set.

[0046] Specifically, the average a posteriori accuracy of the second set JNS=JNC / JNM, and the average a posteriori accuracy of the second set is used to determine the accuracy of the evaluation samples in the second set being accurately recommended.

[0047] Figure 2 This is the second flow chart of the model quantitative evaluation method provided by the present invention, such as Figure 2 As shown, the method includes steps 201 to 208.

[0048] Step 201: Obtain an evaluation sample set, where the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample.

[0049] Step 202: Use the recommendation model to evaluate each evaluation sample respectively to obtain recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on the training samples and the label information samples.

[0050] Step 203: Compare the recommendation information corresponding to each evaluation sample with the actual recommendation information of each evaluation sample.

[0051] Step 204: When the comparison result shows that the recommended information and the actual recommended information are the same, the evaluation samples corresponding to the recommended information are divided into the first set; when the comparison result shows that the recommended information and the actual recommended information are different, the evaluation samples corresponding to the recommended information are divided into the second set.

[0052] Step 205: Determine respectively a first quantity of label information of each evaluation sample in the first set that is preset target label information and a second quantity of label information of each evaluation sample in the second set that is preset target label information.

[0053] Step 206: Determine the ratio of the first number to the third number of samples in the first set as the average posterior accuracy of the first set.

[0054] Step 207: determine the ratio of the sum of the first quantity and the second quantity to the sum of the fourth quantity of each sample in the third quantity and the second set as the posterior accuracy of the overall set; the overall set includes the first set and the second set.

[0055] Step 208: The difference between the average a posteriori accuracy of the first set and the a posteriori accuracy of the entire set is determined as the target quantitative information corresponding to the recommendation model; the target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0056] Next, the quantitative evaluation system architecture provided by the present invention is described.

[0057] The quantitative evaluation system mainly includes three basic layers: data layer, analysis layer and display layer. The effective hierarchical structure division makes the system platform easier to manage and maintain.

[0058] The data layer is the guarantee of data resources for the entire system platform. The system platform is a general analysis platform for various businesses in various business scenarios. Therefore, the construction of the data layer plays a very important role in the design of the system platform.

[0059] The data layer is a module that automatically maintains data. It supports data set upload or database table specification, model file upload or model file storage location specification, model estimation result direct upload or database table specification. In addition, the data layer supports data cleaning and automatic storage in the underlying data table of the business scenario, as well as further deep processing of data to extract important indicators for presentation and analysis of business concerns. The data layer provides functions such as automatic data collection, cleaning, processing and query based on big data tools, providing basic support for query and analysis of business data.

[0060] The analysis layer is the core of the entire system platform, including the real-time reception of evaluation data processed and cleaned by the data layer, the process design and iterative update of the analysis algorithm, and the output of the analysis results. The analysis layer is the most complex layer, that is, the relevant algorithms, analysis templates, and process formulation designed are all completed at this layer, which is the embodiment of the automation of the entire system platform.

[0061] In the design of the analysis layer of the quantitative evaluation system, the model quantitative evaluation method provided by the present invention is adopted. This method adopts an algorithm that compares the estimated results of the recommendation model with the actual posterior results and divides the sets. The algorithm can quantitatively derive the approximate benefits that can be obtained after the model is launched. The specific implementation process can refer to the above description of the model quantitative evaluation method. In order to avoid repetition, it will not be repeated here.

[0062] The display layer mainly includes some page optimization and layout, including the selection of evaluation data, the display of evaluation conclusions, and the personalized display of tables. The selection of evaluation data includes the selection of evaluation data of certain users in a city and corresponding business scenarios within the target time period. The display layer is actually the interaction process between the user and the system platform, that is, the user passes the data to the system platform for analysis, and the platform formats the evaluation conclusions and presents them to the user.

[0063] The present invention designs a complete quantitative evaluation system framework, including the acquisition and maintenance of evaluation data, the process template of evaluation, and the display of evaluation results, etc., which greatly improves the efficiency of model evaluation iteration. It can establish a standard evaluation process in a procedural manner, standardize the evaluation process, and is simple to operate. Business personnel without background can also use it, which greatly shortens the analysis time, and can find out the reasons for data changes in the first time. In addition, a complete offline evaluation model is designed to quantitatively evaluate the online effect. It can simulate the online situation without directly going online, and obtain a reasonable quantitative evaluation scheme, quantitatively evaluate the quantitative benefits that can be obtained after the model goes online, better point out the direction for the iteration of the model, speed up the optimization process of the model and the system, automatically save more than 90% of the complicated trivialities, and improve the efficiency of the model quantitative evaluation. And visually display the analysis results in the form of tables and text.

[0064] The model quantitative evaluation device provided by the present invention is described below. The model quantitative evaluation device described below and the model quantitative evaluation method described above can be referred to each other.

[0065] Figure 3 is a schematic diagram of the structure of the model quantitative evaluation device provided by the present invention, such as Figure 3 As shown, the model quantitative evaluation device 300 includes: an acquisition module 301, a first evaluation module 302 and a second evaluation module 303; wherein,

[0066] An acquisition module 301 is used to acquire an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; A first evaluation module 302 is used to evaluate each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on training samples and label information samples; The second evaluation module 303 is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, and the target quantitative information is used to evaluate the situation of the recommendation model after it is put into use.

[0067] The model quantitative evaluation device provided by the present invention obtains an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; each evaluation sample is evaluated by a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on training samples and label information samples; based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used to evaluate the situation after the recommendation model is put online. Through the evaluation of the evaluation samples by the recommendation model, the label information of the evaluation samples and the real recommendation information, the evaluation of the situation after the recommendation model is put online can be quantitatively obtained without directly going online, thereby improving the accuracy of the model evaluation.

[0068] Optionally, the second evaluation module 303 is specifically configured to: Respectively comparing the recommendation information corresponding to each of the evaluation samples with the real recommendation information of each of the evaluation samples; Based on the comparison result, each of the evaluation samples is divided into a first set or a second set; Based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set, and a preset target label, target quantitative information corresponding to the recommendation model is determined.

[0069] Optionally, the second evaluation module 303 is further configured to: When the comparison result shows that the recommendation information and the real recommendation information are the same, dividing the evaluation samples corresponding to the recommendation information into the first set; When the comparison result is that the recommendation information and the real recommendation information are different, the evaluation samples corresponding to the recommendation information are divided into the second set.

[0070] Optionally, the second evaluation module 303 is further configured to: Respectively determining a first quantity of the label information of each evaluation sample in the first set being the preset target label information and a second quantity of the label information of each evaluation sample in the second set being the preset target label information; Target quantitative information corresponding to the recommendation model is determined based on the first number, the second number, a third number of each of the samples in the first set, and a fourth number of each of the samples in the second set.

[0071] Optionally, the second evaluation module 303 is further configured to: Determine a ratio of the first number to the third number as an average a posteriori accuracy rate of the first set; Determine the ratio of the sum of the first number and the second number to the sum of the third number and the fourth number as the posterior accuracy of the overall set; the overall set includes the first set and the second set; The difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

[0072] Optionally, the model quantitative evaluation device 300 further includes: A determination module is used to determine the ratio of the second number to the fourth number as the average posterior accuracy of the second set.

[0073] Figure 4 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 4 As shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the model quantitative evaluation method, which includes: taking an evaluation sample set, the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; using the recommendation model to evaluate each evaluation sample respectively, and obtaining the recommendation information corresponding to each evaluation sample; the recommendation model is obtained by training based on the training sample and the label information sample; based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, determining the target quantitative information corresponding to the recommendation model, and the target quantitative information is used to evaluate the situation after the recommendation model is put online.

[0074] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model quantitative evaluation method provided by the above methods, and the method includes: taking an evaluation sample set, the evaluation sample set including multiple evaluation samples and label information of each evaluation sample; using a recommendation model to evaluate each of the evaluation samples separately to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; based on the recommendation information corresponding to each of the evaluation samples, the label information and the actual recommendation information, determine the target quantitative information corresponding to the recommendation model, and the target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the model quantitative evaluation method provided by the above-mentioned methods, the method comprising: taking an evaluation sample set, the evaluation sample set comprising multiple evaluation samples and label information of each evaluation sample; using a recommendation model to evaluate each of the evaluation samples respectively to obtain recommendation information corresponding to each of the evaluation samples; the recommendation model is trained based on training samples and label information samples; based on the recommendation information corresponding to each of the evaluation samples, the label information and the actual recommendation information, determining the target quantitative information corresponding to the recommendation model, the target quantitative information being used to evaluate the situation after the recommendation model is launched.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model quantitative evaluation method, characterized in that: include: Acquire an evaluation sample set, the evaluation sample set including a plurality of evaluation samples and label information of each of the evaluation samples; Using the recommendation model to evaluate each of the evaluation samples respectively, and obtain recommendation information corresponding to each of the evaluation samples; The recommendation model is obtained by training based on training samples and label information samples; Based on the recommendation information corresponding to each of the evaluation samples, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used to evaluate the situation of the recommendation model after it is put into operation.

2. The model quantitative evaluation method according to claim 1, characterized in that: The determining, based on the recommendation information corresponding to each of the evaluation samples, the label information and the real recommendation information, the target quantitative information corresponding to the recommendation model includes: Respectively comparing the recommendation information corresponding to each of the evaluation samples with the real recommendation information of each of the evaluation samples; Based on the comparison result, each of the evaluation samples is divided into a first set or a second set; Based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set, and a preset target label, target quantitative information corresponding to the recommendation model is determined.

3. The model quantitative evaluation method according to claim 2, characterized in that: The dividing each of the evaluation samples into the first set or the second set based on the comparison result includes: When the comparison result shows that the recommendation information and the real recommendation information are the same, dividing the evaluation samples corresponding to the recommendation information into the first set; When the comparison result is that the recommendation information and the real recommendation information are different, the evaluation samples corresponding to the recommendation information are divided into the second set.

4. The model quantitative evaluation method according to claim 2, characterized in that: The determining, based on the label information of each of the evaluation samples in the first set, the label information of each of the evaluation samples in the second set, and a preset target label, target quantitative information corresponding to the recommendation model includes: Respectively determining a first quantity of the label information of each evaluation sample in the first set being the preset target label information and a second quantity of the label information of each evaluation sample in the second set being the preset target label information; Target quantitative information corresponding to the recommendation model is determined based on the first number, the second number, a third number of each of the samples in the first set, and a fourth number of each of the samples in the second set.

5. The model quantitative evaluation method according to claim 4, characterized in that: The determining, based on the first number, the second number, the third number of each of the samples in the first set, and the fourth number of each of the samples in the second set, target quantitative information corresponding to the recommendation model includes: Determine a ratio of the first number to the third number as an average a posteriori accuracy rate of the first set; Determine the ratio of the sum of the first number and the second number to the sum of the third number and the fourth number as the posterior accuracy of the overall set; the overall set includes the first set and the second set; The difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

6. The model quantitative evaluation method according to claim 5, characterized in that: The method further comprises: The ratio of the second number to the fourth number is determined as the average posterior accuracy of the second set.

7. A model quantitative evaluation device, characterized in that: include: An acquisition module, used to acquire an evaluation sample set, wherein the evaluation sample set includes multiple evaluation samples and label information of each evaluation sample; A first evaluation module is used to evaluate each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; The recommendation model is obtained by training based on training samples and label information samples; The second evaluation module is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information and the real recommendation information, and the target quantitative information is used to evaluate the situation of the recommendation model after it is put into operation.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the model quantitative evaluation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model quantitative evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model quantitative evaluation method according to any one of claims 1 to 6 is implemented.

Images