Model quantitative evaluation method, device and equipment

Abstract

The application provides a model quantitative evaluation method, device and equipment, and relates to the technical field of artificial intelligence recommendation, and the method comprises the following steps: obtaining an evaluation sample set, the evaluation sample set comprising a plurality of evaluation samples and label information of each evaluation sample; evaluating each evaluation sample by using a recommendation model to obtain corresponding recommendation information of each evaluation sample; the recommendation model is obtained by training based on training samples and label information samples; based on the corresponding recommendation information of each evaluation sample, the label information and real recommendation information, target quantitative information corresponding to the recommendation model is determined, and the target quantitative information is used for evaluating the situation after the recommendation model is put into operation. Through the evaluation of the evaluation sample by the recommendation model, the label information of the evaluation sample and the real recommendation information, the evaluation of the situation after the recommendation model is put into operation is quantitatively obtained without being directly put into operation, and the accuracy of model evaluation is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence recommendation technology, and in particular to a method, apparatus and equipment for quantitative evaluation of models. Background Technology

[0002] With the development of the internet, recommender systems, as a major internet system, have been constantly influencing people's lives. In various application scenarios, recommender models are a core component of recommender systems, directly impacting the user experience of the recommendation service by recommending different items to different users. In recent years, with the rapid development of recommender model technology, research on recommender models has become an important branch of the field of artificial intelligence.

[0003] In industry, recommendation models, the core of recommendation systems, are frequently iterated and updated to pursue better service and user experience. Iterative optimization of recommendation models is divided into offline and online processes. Due to the frequency of model iteration and considerations for the stability of online systems, a common practice is to first evaluate the iterative optimization of the recommendation model offline to see if it has improved offline metrics. If there is an improvement in offline metrics, then the next step of online validation will proceed; otherwise, offline optimization continues. Based on this iterative path, the offline model evaluation method is particularly important because offline evaluation determines whether to conduct subsequent online validation and the direction of subsequent iterative optimization.

[0004] Currently, commonly used offline evaluation metrics for recommendation models include the area under the ROC curve (AUC), group AUC (GAUC), and top-k accuracy. However, these metrics are all ranking metrics, which can only qualitatively indicate that a better ranking means better online performance. They cannot quantitatively reflect the positive online performance, resulting in low model evaluation accuracy. Summary of the Invention

[0005] This invention provides a method, apparatus, and equipment for quantitative model evaluation, which addresses the problem of low accuracy in model evaluation.

[0006] This invention provides a method for quantitative evaluation of a model, comprising: Obtain an evaluation sample set, which includes multiple evaluation samples and label information for each evaluation sample; A recommendation model is used to evaluate each of the evaluation samples to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; Based on the recommendation information, the tag information, and the real recommendation information corresponding to each evaluation sample, the target quantitative information corresponding to the recommendation model is determined. The target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0007] According to a model quantitative evaluation method provided by the present invention, determining the target quantitative information corresponding to the recommendation model based on the recommendation information, the label information, and the real recommendation information corresponding to each evaluation sample includes: The recommendation information corresponding to each evaluation sample is compared with the actual recommendation information of each evaluation sample. Based on the comparison results, each of the evaluation samples is divided into the first set or the second set; Based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and the preset target label, the target quantitative information corresponding to the recommendation model is determined.

[0008] According to a model quantitative evaluation method provided by the present invention, the step of dividing each evaluation sample into a first set or a second set based on comparison results includes: If the comparison result shows that the recommended information and the real recommended information are the same, the evaluation sample corresponding to the recommended information is assigned to the first set; If the comparison result shows that the recommended information and the actual recommended information are different, the evaluation sample corresponding to the recommended information is assigned to the second set.

[0009] According to a model quantitative evaluation method provided by the present invention, determining the target quantitative information corresponding to the recommendation model based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and a preset target label includes: The number of times the label information of each evaluation sample in the first set is the preset target label information and the number of times the label information of each evaluation sample in the second set is the preset target label information are determined respectively; Based on the first quantity, the second quantity, the third quantity of each sample in the first set, and the fourth quantity of each sample in the second set, the target quantitative information corresponding to the recommendation model is determined.

[0010] According to a model quantitative evaluation method provided by the present invention, determining the target quantitative information corresponding to the recommendation model based on the first quantity, the second quantity, the third quantity of each sample in the first set, and the fourth quantity of each sample in the second set includes: The ratio of the first quantity to the third quantity is determined as the average posterior accuracy of the first set; The ratio of the sum of the first and second quantities to the sum of the third and fourth quantities is determined as the posterior accuracy of the entire set; the entire set includes the first set and the second set. The difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

[0011] According to a model quantitative evaluation method provided by the present invention, the method further includes: The ratio of the second quantity to the fourth quantity is determined as the average posterior accuracy of the second set.

[0012] The present invention also provides a model quantitative evaluation device, comprising the following modules: The acquisition module is used to acquire an evaluation sample set, which includes multiple evaluation samples and label information for each evaluation sample; The first evaluation module is used to evaluate each of the evaluation samples using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; The second evaluation module is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information, the tag information, and the real recommendation information corresponding to each evaluation sample. The target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the model quantitative evaluation method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model quantitative evaluation method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model quantitative evaluation method as described above.

[0016] The present invention provides a model quantitative evaluation method, apparatus, and device. This method involves acquiring an evaluation sample set, which includes multiple evaluation samples and label information for each sample; evaluating each evaluation sample using a recommendation model to obtain recommendation information corresponding to each sample; the recommendation model being trained based on training samples and label information samples; and determining target quantitative information corresponding to the recommendation model based on the recommendation information, label information, and real recommendation information for each evaluation sample. This target quantitative information is used to evaluate the performance of the recommendation model after its deployment. By using the evaluation of evaluation samples by the recommendation model, the label information of the evaluation samples, and the real recommendation information, a quantitative assessment of the performance of the recommendation model after deployment can be obtained without direct deployment, thus improving the accuracy of model evaluation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the quantitative evaluation method for models provided by this invention.

[0019] Figure 2 This is the second flowchart of the quantitative evaluation method for models provided by this invention.

[0020] Figure 3 This is a schematic diagram of the structure of the model quantitative evaluation device provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] The following is combined with Figures 1-2 The present invention describes a quantitative evaluation method for the model.

[0024] Figure 1 This is one of the flowcharts illustrating the quantitative evaluation method for models provided by this invention, such as... Figure 1 As shown, the method includes the following steps 101-103.

[0025] Step 101: Obtain the evaluation sample set, which includes multiple evaluation samples and the label information of each evaluation sample.

[0026] It should be noted that the model quantitative evaluation method provided by this invention can be applied to the quantitative evaluation of recommendation models in recommendation systems used in the fields of artificial intelligence recommendation or recommendation models-driven improvements to product quality or services. These fields are highly sensitive to changes in the effectiveness of recommendation models, thus frequently requiring iteration and updates to improve their performance and consequently enhance the quality of recommended products or services and improve user experience. The execution entity of this method can be a model quantitative evaluation device, such as an electronic device or a control module within that device for executing the model quantitative evaluation method.

[0027] Specifically, an evaluation sample set is obtained from the data layer. The evaluation sample set includes multiple evaluation samples and the label information of each evaluation sample. The evaluation samples can be business evaluation data, and the label information can be recommendations or other labels.

[0028] The data layer is a module that automatically maintains data, supporting dataset uploads or specified database tables, model file uploads or specified model file storage locations, and direct uploads of model prediction results or specified database tables. Furthermore, the data layer supports data cleaning and automatic storage in the underlying data tables for business scenarios, as well as further data processing to extract key metrics relevant to business presentation and analysis. Based on big data tools, the data layer provides automatic data collection, cleaning, processing, and querying functions, providing a fundamental guarantee for business data querying and analysis.

[0029] Step 102: Use a recommendation model to evaluate each of the evaluation samples to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples.

[0030] Specifically, the recommendation model is trained based on training samples and labeled information samples, and is used to evaluate each of the evaluation samples. After the recommendation model is trained, each evaluation sample is input into the recommendation model to obtain the recommendation information (i.e., recommendation topic P_item) corresponding to each evaluation sample output by the recommendation model.

[0031] Step 103: Based on the recommendation information, the label information, and the real recommendation information corresponding to each evaluation sample, determine the target quantitative information corresponding to the recommendation model. The target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0032] Specifically, the real recommendation information refers to the actual recommendation information of each evaluation sample (i.e., the actual recommendation topic R_item). For example, in the insurance recommendation scenario, different insurance products are recommended online. Since different recommended products have different conversion rate data, the recommendation results of each recommended insurance product and the actual online recommendation effect can be estimated by using an offline recommendation model. This allows for a quantitative assessment of the subsequent online effect of the offline recommendation model.

[0033] Based on the recommendation information, tag information, and real recommendation information corresponding to each evaluation sample, the target quantitative information corresponding to the recommendation model can be determined. The target quantitative information is used to evaluate the situation after the recommendation model is launched, that is, the online recommendation effect or the benefits obtained after the recommendation model is launched.

[0034] The quantitative evaluation method for models provided by this invention involves acquiring an evaluation sample set, which includes multiple evaluation samples and label information for each evaluation sample; evaluating each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; and determining target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information, and the actual recommendation information. This target quantitative information is used to evaluate the performance of the recommendation model after it is deployed online. By using the evaluation of evaluation samples by the recommendation model, the label information of the evaluation samples, and the actual recommendation information, a quantitative assessment of the performance of the recommendation model after deployment can be obtained without direct deployment, thus improving the accuracy of model evaluation.

[0035] Optionally, the specific implementation of step 103 above includes: The recommendation information corresponding to each evaluation sample and the actual recommendation information of each evaluation sample are compared respectively; based on the comparison results, each evaluation sample is divided into a first set or a second set; based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set and the preset target label, the target quantitative information corresponding to the recommendation model is determined.

[0036] Specifically, the recommendation information corresponding to each evaluation sample is compared with the actual recommendation information of each evaluation sample. Based on the comparison results, each evaluation sample is divided into a first set or a second set. The first set includes evaluation samples corresponding to recommendation information that is the same as the actual recommendation information, i.e., the set JE where P_item = R_item. The second set includes evaluation samples corresponding to recommendation information that is different from the actual recommendation information, i.e., the set JN where P_item != R_item. Then, based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and the preset target label, the target quantitative information corresponding to the recommendation model can be determined.

[0037] The quantitative evaluation method for models provided by this invention, by comparing the recommendation information corresponding to each evaluation sample estimated offline with the actual recommendation information of online recommendations, can quantitatively determine the approximate benefits that the recommendation model can obtain after going online, thus better pointing out the direction for model iteration and greatly accelerating the optimization process of the model and system.

[0038] Optionally, the step of dividing each of the evaluation samples into a first set or a second set based on the comparison results includes: If the comparison result shows that the recommended information and the real recommended information are the same, the evaluation sample corresponding to the recommended information is assigned to the first set; if the comparison result shows that the recommended information and the real recommended information are not the same, the evaluation sample corresponding to the recommended information is assigned to the second set.

[0039] Specifically, if the comparison result shows that the recommended information is the same as the actual recommended information, it means that the information recommended by the recommendation model matches the actual information and the performance of the recommendation model is relatively good. The evaluation samples corresponding to the recommended information are then assigned to the first set. If the comparison result shows that the recommended information is different from the actual recommended information, it means that the information recommended by the recommendation model does not match the actual information and the performance of the recommendation model is relatively poor. The evaluation samples corresponding to the recommended information are then assigned to the second set.

[0040] Optionally, determining the target quantitative information corresponding to the recommendation model based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and a preset target label includes: The first number of times the label information of each evaluation sample in the first set is the preset target label information and the second number of times the label information of each evaluation sample in the second set is the preset target label information are determined respectively; based on the first number, the second number, the third number of each sample in the first set and the fourth number of each sample in the second set, the target quantitative information corresponding to the recommendation model is determined.

[0041] Specifically, by comparing the label information of each evaluation sample in the first set with the preset target label information, the label information of each evaluation sample in the first set can be determined to be the first quantity of the preset target label information. Similarly, by comparing the label information of each evaluation sample in the second set with the preset target label information, the label information of each evaluation sample in the second set can be determined to be the second quantity of the preset target label information. For example, suppose there are JEM evaluation samples in the first set (set JE), JNM evaluation samples in JN, and the target label information is whether it was clicked (i.e., the posterior label is whether it was clicked). Set JE has JEC samples that were clicked, and set JN has JNC samples that were clicked.

[0042] Based on the first quantity, the second quantity, the third quantity of each sample in the first set, and the fourth quantity of each sample in the second set, the target quantitative information corresponding to the recommendation model can be determined.

[0043] Optionally, determining the target quantitative information corresponding to the recommendation model based on the first quantity, the second quantity, the third quantity of each sample in the first set, and the fourth quantity of each sample in the second set includes: The ratio of the first quantity to the third quantity is determined as the average posterior accuracy of the first set; the ratio of the sum of the first and second quantities to the sum of the third and fourth quantities is determined as the posterior accuracy of the entire set; the entire set includes the first set and the second set; the difference between the average posterior accuracy of the first set and the posterior accuracy of the entire set is determined as the target quantitative information corresponding to the recommendation model.

[0044] For example, the average posterior accuracy of the first set is JES = JEC / JEM, the posterior accuracy of the entire set is JTS = (JEC + JNC) / (JEM + JNM), and the quantitative information of the target corresponding to the recommendation model is Target = JES - JTS.

[0045] Optionally, the method further includes: The ratio of the second quantity to the fourth quantity is determined as the average posterior accuracy of the second set.

[0046] Specifically, the mean posterior accuracy of the second set, JNS = JNC / JNM, is used to determine the accuracy at which the evaluation samples in the second set are accurately recommended.

[0047] Figure 2 This is the second flowchart illustrating the quantitative evaluation method for models provided by this invention, as shown below. Figure 2 As shown, the method includes steps 201-208.

[0048] Step 201: Obtain the evaluation sample set, which includes multiple evaluation samples and the label information of each evaluation sample.

[0049] Step 202: Use the recommendation model to evaluate each evaluation sample and obtain the recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples.

[0050] Step 203: Compare the recommendation information corresponding to each evaluation sample with the actual recommendation information of each evaluation sample.

[0051] Step 204: If the comparison result shows that the recommended information and the true recommended information are the same, the evaluation samples corresponding to the recommended information are assigned to the first set; if the comparison result shows that the recommended information and the true recommended information are different, the evaluation samples corresponding to the recommended information are assigned to the second set.

[0052] Step 205: Determine the first number of preset target label information for each evaluation sample in the first set and the second number of preset target label information for each evaluation sample in the second set.

[0053] Step 206: The ratio of the first quantity to the third quantity of each sample in the first set is determined as the average posterior accuracy of the first set.

[0054] Step 207: The ratio of the sum of the first and second quantities to the sum of the fourth quantities of each sample in the third and second sets is determined as the posterior accuracy of the overall set; the overall set includes the first and second sets.

[0055] Step 208: The difference between the average posterior accuracy of the first set and the posterior accuracy of the entire set is determined as the target quantitative information corresponding to the recommendation model; the target quantitative information is used to evaluate the performance of the recommendation model after it is launched.

[0056] The architecture of the quantitative evaluation system provided by this invention will be described next.

[0057] The quantitative assessment system mainly consists of three basic layers: data layer, analysis layer, and presentation layer. The effective hierarchical structure makes the system platform easier to manage and maintain.

[0058] The data layer is responsible for ensuring the data resources of the entire system platform. The system platform is a general analysis platform for various business scenarios, so the construction of the data layer plays a very important role in the design of the system platform.

[0059] The data layer is a module that automatically maintains data, supporting dataset uploads or specified database tables, model file uploads or specified model file storage locations, and direct uploads of model prediction results or specified database tables. Furthermore, the data layer supports data cleaning and automatic storage in the underlying data tables for business scenarios, as well as further data processing to extract key metrics relevant to business presentation and analysis. Based on big data tools, the data layer provides automatic data collection, cleaning, processing, and querying functions, providing a fundamental guarantee for business data querying and analysis.

[0060] The analysis layer is the core of the entire system platform. It includes receiving the processed and cleaned evaluation data from the data layer in real time, designing and iteratively updating the analysis algorithms, and outputting the analysis results. The analysis layer is the most complex layer, as all the designed algorithms, analysis templates, and process specifications are completed at this layer. It embodies the automation of the entire system platform.

[0061] In the design of the analysis layer of the quantitative evaluation system, the model quantitative evaluation method provided by this invention is adopted. This method uses an algorithm that compares the predicted results of the recommended model with the actual posterior results and divides the set. This algorithm can quantitatively determine the approximate benefits that the model can obtain after it goes online. The specific implementation process can be referred to the above description of the model quantitative evaluation method. To avoid repetition, it will not be repeated here.

[0062] The presentation layer mainly includes page optimization and layout, such as the selection methods for evaluation data, the display of evaluation conclusions, and the personalized display of tables. The selection methods for evaluation data include the selection of evaluation data from specific users in a city within a target time period and the corresponding business scenario. The presentation layer is essentially the interaction process between the user and the system platform; that is, the user provides the system platform with the data to be analyzed, and the platform presents the evaluation conclusions to the user in a formatted manner.

[0063] This invention designs a complete quantitative evaluation system framework, including the acquisition and maintenance of evaluation data, evaluation process templates, and the display of evaluation results, greatly improving the efficiency of model evaluation iteration. It can establish a standardized evaluation process in a streamlined manner, standardizing the evaluation process. The system is simple to operate, even for business personnel without background experience, significantly shortening analysis time and enabling immediate identification of the causes of data changes. Furthermore, it designs a complete quantitative evaluation method for the online impact of offline evaluation models, simulating online conditions without direct deployment and obtaining a reasonable quantitative evaluation scheme. This quantitatively evaluates the quantitative benefits that the model can obtain after deployment, better guiding model iteration, accelerating the optimization process of the model and system, and automating over 90% of tedious tasks, thus improving the efficiency of quantitative model evaluation. Moreover, the analysis results are visually displayed in a table and text format, providing an intuitive understanding.

[0064] The model quantitative evaluation device provided by the present invention is described below. The model quantitative evaluation device described below can be referred to in correspondence with the model quantitative evaluation method described above.

[0065] Figure 3 This is a schematic diagram of the structure of the quantitative evaluation device for models provided by the present invention, as shown below. Figure 3 As shown, the model quantitative evaluation device 300 includes: an acquisition module 301, a first evaluation module 302, and a second evaluation module 303; wherein,

[0066] The acquisition module 301 is used to acquire an evaluation sample set, which includes multiple evaluation samples and label information of each evaluation sample; The first evaluation module 302 is used to evaluate each of the evaluation samples using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples. The second evaluation module 303 is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information, the label information and the real recommendation information corresponding to each evaluation sample. The target quantitative information is used to evaluate the situation after the recommendation model is launched.

[0067] The present invention provides a quantitative evaluation device for models. This device acquires an evaluation sample set, which includes multiple evaluation samples and their label information. A recommendation model is used to evaluate each evaluation sample to obtain corresponding recommendation information. The recommendation model is trained based on training samples and label information samples. Based on the recommendation information, label information, and real recommendation information for each evaluation sample, target quantitative information is determined for the recommendation model. This target quantitative information is used to evaluate the performance of the recommendation model after it is deployed online. By using the evaluation of evaluation samples, the label information of the evaluation samples, and the real recommendation information, the device can quantitatively assess the performance of the recommendation model after deployment without direct deployment, thus improving the accuracy of model evaluation.

[0068] Optionally, the second evaluation module 303 is specifically used for: The recommendation information corresponding to each evaluation sample is compared with the actual recommendation information of each evaluation sample. Based on the comparison results, each of the evaluation samples is divided into the first set or the second set; Based on the label information of each evaluation sample in the first set, the label information of each evaluation sample in the second set, and the preset target label, the target quantitative information corresponding to the recommendation model is determined.

[0069] Optionally, the second evaluation module 303 is further configured to: If the comparison result shows that the recommended information and the real recommended information are the same, the evaluation sample corresponding to the recommended information is assigned to the first set; If the comparison result shows that the recommended information and the actual recommended information are different, the evaluation sample corresponding to the recommended information is assigned to the second set.

[0070] Optionally, the second evaluation module 303 is further configured to: The number of times the label information of each evaluation sample in the first set is the preset target label information and the number of times the label information of each evaluation sample in the second set is the preset target label information are determined respectively; Based on the first quantity, the second quantity, the third quantity of each sample in the first set, and the fourth quantity of each sample in the second set, the target quantitative information corresponding to the recommendation model is determined.

[0071] Optionally, the second evaluation module 303 is further configured to: The ratio of the first quantity to the third quantity is determined as the average posterior accuracy of the first set; The ratio of the sum of the first and second quantities to the sum of the third and fourth quantities is determined as the posterior accuracy of the entire set; the entire set includes the first set and the second set. The difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

[0072] Optionally, the model quantitative evaluation device 300 further includes: The determination module is used to determine the ratio of the second quantity to the fourth quantity as the average posterior accuracy of the second set.

[0073] Figure 4 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device 400 may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a model quantitative evaluation method. This method includes: retrieving an evaluation sample set, the evaluation sample set including multiple evaluation samples and label information of each evaluation sample; evaluating each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; and determining target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information, and the real recommendation information, the target quantitative information being used to evaluate the performance of the recommendation model after it is deployed online.

[0074] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model quantitative evaluation method provided by the above methods. The method includes: taking an evaluation sample set, the evaluation sample set including multiple evaluation samples and label information of each evaluation sample; evaluating each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model is trained based on training samples and label information samples; and determining target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information, and the real recommendation information, the target quantitative information being used to evaluate the situation after the recommendation model is launched.

[0076] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the model quantitative evaluation method provided by the above methods. The method includes: taking an evaluation sample set, the evaluation sample set including multiple evaluation samples and label information for each evaluation sample; evaluating each evaluation sample using a recommendation model to obtain recommendation information corresponding to each evaluation sample; the recommendation model being trained based on training samples and label information samples; and determining target quantitative information corresponding to the recommendation model based on the recommendation information corresponding to each evaluation sample, the label information, and the actual recommendation information, the target quantitative information being used to evaluate the situation after the recommendation model is deployed online.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quantitative evaluation of a model, characterized in that, Applications include insurance recommendation scenarios, including: Obtain an evaluation sample set, which includes multiple evaluation samples and tag information for each evaluation sample. The evaluation samples are business evaluation data, and the tag information is recommendation tags, which indicate whether the data was clicked. A recommendation model is used to evaluate each of the evaluation samples to obtain recommendation information corresponding to each evaluation sample. The recommendation information is an online recommendation of different insurance products in the insurance recommendation scenario. The recommendation model is trained based on training samples and label information samples. Based on the recommendation information, the tag information, and the real recommendation information corresponding to each evaluation sample, the target quantitative information corresponding to the recommendation model is determined. The target quantitative information is used to evaluate the online recommendation effect or the benefits obtained after the recommendation model is launched. The step of determining the target quantitative information corresponding to the recommendation model based on the recommendation information, the label information, and the real recommendation information corresponding to each evaluation sample includes: The recommendation information corresponding to each evaluation sample is compared with the actual recommendation information of each evaluation sample to obtain the comparison results; If the comparison result shows that the recommended information and the real recommended information are the same, the evaluation sample corresponding to the recommended information is assigned to the first set; if the comparison result shows that the recommended information and the real recommended information are not the same, the evaluation sample corresponding to the recommended information is assigned to the second set. The first number of evaluation samples in the first set whose label information is a preset target label information and the second number of evaluation samples in the second set whose label information is a preset target label information are respectively determined; The ratio of the first quantity to the third quantity of each sample in the first set is determined as the average posterior accuracy of the first set; the ratio of the sum of the first quantity and the second quantity to the sum of the third quantity and the fourth quantity of each sample in the second set is determined as the posterior accuracy of the overall set; the overall set includes the first set and the second set; the difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

2. The model quantitative evaluation method according to claim 1, characterized in that, The method further includes: The ratio of the second quantity to the fourth quantity is determined as the average posterior accuracy of the second set.

3. A model quantitative evaluation device, characterized in that, Applications include insurance recommendation scenarios, including: The acquisition module is used to acquire an evaluation sample set, which includes multiple evaluation samples and tag information for each evaluation sample. The evaluation samples are business evaluation data, and the tag information is recommendation tags, which indicate whether the data is clickable. The first evaluation module is used to evaluate each of the evaluation samples using a recommendation model to obtain recommendation information corresponding to each evaluation sample. The recommendation information is online recommendation of different insurance products in the insurance recommendation scenario. The recommendation model is trained based on training samples and label information samples. The second evaluation module is used to determine the target quantitative information corresponding to the recommendation model based on the recommendation information, the tag information and the real recommendation information corresponding to each evaluation sample. The target quantitative information is used to evaluate the online recommendation effect or the benefits obtained after the recommendation model is launched. The recommendation information is the online recommendation of different insurance products in the insurance recommendation scenario. The step of determining the target quantitative information corresponding to the recommendation model based on the recommendation information, the label information, and the real recommendation information corresponding to each evaluation sample includes: The recommendation information corresponding to each evaluation sample is compared with the actual recommendation information of each evaluation sample to obtain the comparison results; If the comparison result shows that the recommended information and the real recommended information are the same, the evaluation sample corresponding to the recommended information is assigned to the first set; if the comparison result shows that the recommended information and the real recommended information are not the same, the evaluation sample corresponding to the recommended information is assigned to the second set. The first number of evaluation samples in the first set whose label information is a preset target label information and the second number of evaluation samples in the second set whose label information is a preset target label information are respectively determined; The ratio of the first quantity to the third quantity of each sample in the first set is determined as the average posterior accuracy of the first set; the ratio of the sum of the first quantity and the second quantity to the sum of the third quantity and the fourth quantity of each sample in the second set is determined as the posterior accuracy of the overall set; the overall set includes the first set and the second set; the difference between the average posterior accuracy of the first set and the posterior accuracy of the overall set is determined as the target quantitative information corresponding to the recommendation model.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the model quantitative evaluation method as described in claim 1 or 2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model quantitative evaluation method as described in claim 1 or 2.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model quantitative evaluation method as described in claim 1 or 2.

Images