Resource estimation model evaluation method and device, storage medium and electronic equipment

Abstract

The invention discloses a resource estimation model evaluation method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring at least two media resource samples; dividing the at least two media resources into at least two sample sets according to the sample number of the at least two media resource samples; evaluating the first resource estimation result through a first preset deviation threshold value corresponding to the first sample set to obtain a first evaluation result, and evaluating the second resource estimation result through a second preset deviation threshold value corresponding to the second sample set to obtain a second evaluation result; and integrating the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model. The embodiment of the invention can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving and the like. The technical problem that the evaluation accuracy of the resource estimation model is low is solved.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a method, apparatus, storage medium, and electronic device for evaluating a resource prediction model. Background Art

[0002] Currently, before relevant resources (such as advertisements) are put on the market, it is often necessary to first make a prediction and judgment on the relevant resources to obtain prediction information for measuring subsequent specific placement information (such as placement order, placement volume).

[0003] In order to obtain more accurate prediction information, the prior art often uses a pre-trained resource prediction model to perform the above prediction and judgment operations. However, the quality evaluation effect of the existing resource prediction model is poor, resulting in low evaluation accuracy of the resource prediction model. As a result, the accuracy of the prediction information obtained by the resource prediction model for predicting and judging relevant resources is low, thereby causing the technical problem of low accuracy of resource prediction. Therefore, there is a technical problem of low accuracy of resource prediction in the related art. Summary of the Invention

[0004] Embodiments of this application provide a method, apparatus, storage medium, and electronic device for evaluating a resource prediction model, so as to at least solve the technical problem of low evaluation accuracy of the resource prediction model in the related art.

[0005] According to one aspect of the embodiments of this application, a method for evaluating a resource prediction model is provided, including: obtaining at least two media resource samples, where the at least two media resource samples are used to evaluate the prediction deviation degree of the resource prediction model, and the resource prediction model is used to predict media resources; dividing the at least two media resources into at least two sample sets according to the number of the at least two media resource samples, where the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set; when obtaining a first resource prediction result obtained by the resource prediction model predicting the first sample set and a second resource prediction result obtained by predicting the second sample set, evaluating the first resource prediction result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluating the second resource prediction result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result; integrating the first evaluation result and the second evaluation result to obtain a model quality score of the resource prediction model, where the model quality score is used to evaluate the prediction deviation degree of the resource prediction model.

[0006] According to another aspect of the embodiments of the present application, there is also provided an evaluation device for a resource estimation model, including: an acquisition unit, configured to acquire at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of the resource estimation model, and the resource estimation model is used to estimate media resources; a division unit, configured to divide the at least two media resources into at least two sample sets according to the number of the at least two media resource samples, where the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set; an evaluation unit, configured to, in the case of obtaining a first resource estimation result obtained by the resource estimation model estimating the first sample set and a second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result; an integration unit, configured to integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, where the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

[0007] As an alternative solution, the evaluation unit includes: a first acquisition module, configured to acquire a first non-deviation consumption ratio of the first sample set, where the first non-deviation consumption ratio is a first ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, indicated by the first resource estimation result, in the first sample set; a second acquisition module, configured to use the first sample number of the first sample set and the first preset deviation threshold to acquire a first non-deviation parameter corresponding to the first sample set, where the first non-deviation parameter is used to indicate a second ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, in the case that the resource estimation model has no estimation deviation; a first determination module, configured to determine a quotient value of the first non-deviation consumption ratio and the first non-deviation parameter as the first evaluation result.

[0008] As an alternative solution, the above-mentioned second acquisition module includes: a first calculation sub-module, configured to take the square root of the above-mentioned first sample quantity and take the reciprocal of the result of the square root operation to obtain the first standard deviation corresponding to the above-mentioned first sample set; a second calculation sub-module, configured to divide the above-mentioned first preset deviation threshold by the above-mentioned first standard deviation to obtain the first standard deviation multiple corresponding to the above-mentioned first sample set; a third calculation sub-module, configured to use the above-mentioned first standard deviation multiple as the input of a preset density function to obtain the first coverage ratio output by the above-mentioned preset density function, where the above-mentioned preset density function is a cumulative density function representing a normal distribution; a first determination sub-module, configured to determine the above-mentioned first coverage ratio as the above-mentioned first non-deviation parameter.

[0009] As an alternative solution, the above-mentioned first acquisition module includes: a second determination sub-module, configured to determine, according to the above-mentioned first resource estimation result, a third media resource sample in the above-mentioned first sample set whose deviation degree from the true value is less than or equal to the above-mentioned first preset deviation threshold; a third determination sub-module, configured to, when the first resource consumption corresponding to the above-mentioned first sample set is obtained, determine the ratio of the second resource consumption corresponding to the above-mentioned third media resource sample to the above-mentioned first resource consumption as the first non-deviation consumption ratio.

[0010] As an alternative solution, the above-mentioned integration unit includes: a third acquisition module, configured to acquire the first consumption ratio of the first resource consumption corresponding to the above-mentioned first sample set in the third resource consumption corresponding to the at least two sample sets, and acquire the second consumption ratio of the fourth resource consumption corresponding to the above-mentioned second sample set in the third resource consumption; a second determination module, configured to, when the first product result of the above-mentioned first evaluation result and the above-mentioned first consumption ratio and the second product result of the above-mentioned second evaluation result and the above-mentioned second consumption ratio are obtained, use the sum value of the above-mentioned first product result and the above-mentioned second product result to determine the above-mentioned model quality score.

[0011] As an alternative solution, the above device further includes: a third determination module, configured to, after evaluating the above prediction result by the above preset deviation threshold to obtain the model quality score of the above resource prediction model, determine that the above prediction deviation degree of the above resource prediction model satisfies the overall non-convergence condition when the above model quality score is less than the first preset score threshold; a fourth determination module, configured to, after evaluating the above prediction result by the above preset deviation threshold to obtain the model quality score of the above resource prediction model, determine at least one media resource sample with a product result less than the second preset score threshold from the above at least two media resource samples, where the above product result includes the above first product result and the above second product result, and the above second preset score threshold is less than the above first preset score threshold; a fifth determination module, configured to, after evaluating the above prediction result by the above preset deviation threshold to obtain the model quality score of the above resource prediction model, obtain the target sample quantity corresponding to the above at least one media resource sample, and determine that the above prediction deviation degree of the above resource prediction model satisfies the local non-convergence condition at the above target sample quantity.

[0012] As an alternative solution, the above device further includes: a prediction module, configured to, after evaluating the above prediction result by the above preset deviation threshold to obtain the model quality score of the above resource prediction model, in response to a resource prediction request triggered for at least two media resources, input the above at least two media resources into the above resource prediction model for prediction processing to obtain a prediction result, where the above prediction result includes first prediction information for indicating the recommendation order of each media resource in the above at least two media resources and second prediction information for indicating the click prediction value of each media resource.

[0013] According to another aspect of the embodiments of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the evaluation method of the above resource prediction model.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the above processor executes the evaluation method of the above resource prediction model through the computer program.

[0015] In the embodiments of the present application, first, according to the number of samples of at least two media resource samples, the at least two media resource samples are divided into at least two sample sets with different numbers of samples and corresponding to different preset deviation thresholds. Then, based on the preset deviation thresholds corresponding to each sample set in the at least two sample sets, the resource estimation results obtained by estimating the corresponding sample sets through a resource estimation model are evaluated to obtain the evaluation results corresponding to each sample set. Furthermore, the evaluation results corresponding to each sample set are integrated to obtain a model quality score for evaluating the estimation deviation degree of the resource estimation model. In the overall evaluation process of the resource estimation model, the index of the number of samples of the media resource samples is considered in detail, and different numbers of samples correspond to the confidence situations of different preset deviation thresholds. Thus, it is possible to first obtain the estimation deviation situations of the resource estimation model under the confidence situations of different numbers of samples and corresponding different preset deviation thresholds, and then perform integrated detection, comprehensively considering the influencing factors of the sample information of the resource samples themselves on the estimation deviation of the resource estimation model at a finer-grained dimension, achieving the purpose of improving the evaluation accuracy of the resource estimation model. Furthermore, the accuracy of the estimation information obtained by the resource estimation model for estimating and judging relevant resources is improved, thereby achieving the technical effect of improving the accuracy of resource estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an application environment of an optional resource estimation model evaluation method according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a process of an optional resource estimation model evaluation method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of an optional resource estimation model evaluation method according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an optional resource estimation model evaluation method according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of an optional resource estimation model evaluation method according to an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of an optional resource estimation model evaluation device according to an embodiment of the present application;

[0023] Figure 7 Structural schematic diagram of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0024] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to one aspect of the embodiments of the present application, an evaluation method for a resource prediction model is provided. Optionally, as an optional implementation manner, the above-mentioned evaluation method for the resource prediction model can be but is not limited to being applied to an environment such as Figure 1 shown. Among them, it may include but is not limited to the client 102 and the server 112. The client 102 may include but is not limited to a display 104, a processor 106, and a memory 108. The server 112 includes a database 114 and a processing engine 116.

[0027] The specific process may be as follows:

[0028] Step S102, the client 102 obtains at least two media resource samples, where the at least two media resource samples are used to evaluate the prediction deviation degree of the resource prediction model;

[0029] Steps S104 - S106, the client 102 sends a resource prediction model evaluation request to the server 112, where the resource prediction model evaluation request is used to request an evaluation of the prediction deviation degree of the resource prediction model;

[0030] Step S108, the server 112 responds to the resource prediction model evaluation request, divides at least two media resources into at least two sample sets according to the number of samples of at least two media resource samples, wherein the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set;

[0031] Step S110, the server 112 obtains a first resource prediction result obtained by predicting the first sample set by the resource prediction model and a second resource prediction result obtained by predicting the second sample set;

[0032] Step S112, the server 112 evaluates the first resource prediction result through the first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluates the second resource prediction result through the second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result;

[0033] Step S114, the server 112 integrates the first evaluation result and the second evaluation result to obtain a model quality score of the resource prediction model, wherein the model quality score is used to evaluate the prediction deviation degree of the resource prediction model;

[0034] Steps S116 - S118, send the model quality score to the client 102 through the network 110, wherein the processor 106 in the client 102 is used to receive the model quality score, process relevant data, display the model quality score on the display 104, and store the relevant data in the memory 108.

[0035] Except Figure 1 For the examples shown, the above steps can be completed independently by the client or the server, or jointly by the client and the server. For example, the client 102 executes the above steps S108 to S114, etc., so as to reduce the processing pressure on the server 112. The client 102 includes but is not limited to laptop computers, tablet computers, desktop computers, smart TVs, etc. The present application does not limit the specific implementation manner of the client 102. The server 112 can be a single server or a server cluster composed of multiple servers, or a cloud server.

[0036] Optionally, as an alternative implementation, as Figure 2 shown, the evaluation method of the resource prediction model can be executed by an electronic device, such as Figure 1 the client or the server shown, and the specific steps include:

[0037] S202. Obtain at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of a resource estimation model, and the resource estimation model is used to estimate media resources.

[0038] S204. Divide the at least two media resources into at least two sample sets according to the number of the at least two media resource samples, where the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set.

[0039] S206. When obtaining a first resource estimation result obtained by the resource estimation model for estimating the first sample set and a second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result.

[0040] S208. Integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, where the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

[0041] Optionally, in this embodiment, the above evaluation method of the resource estimation model may be, but is not limited to, applied to the evaluation scenario of the refined ranking model in the field of advertising recommendation. Taking this scenario as an example, in the existing industry practice, for a resource estimation model used for subsequent click or view or other operation behaviors of media resources such as advertisements, it often directly processes the estimated value and the actual true value output by the resource estimation model for resource samples. Since this method has relatively coarser calculation dimensions, it does not consider the influence of the sample information of the resource samples themselves on the estimation deviation of the resource estimation model in a more fine-grained dimension.

[0042] For example, as the main indicator for evaluating the prediction deviation degree of the resource prediction model, the bias index value is the predicted click-through rate pctr of the media resource divided by the overall click-through rate ctr, and then subtracted by 1. The larger the absolute value of bias, the greater the prediction deviation range of the model. Further, on the one hand, the existing bias index is calculated on a relatively coarse dimension. For example, the bias index of the ad slot makes it difficult to observe fine-grained prediction anomalies. On the other hand, even when calculating to the dimension of the ad category granularity, there will still be a phenomenon of mutual cancellation of the bias index. For example, the bias of ad category A is +50%, that is, the prediction is 50% higher, and the bias of ad category B is -50%, that is, the prediction is 50% lower. At this time, the average value of the ad category bias is (+50% + -50%) / 2 = 0, that is, there is no deviation, which obviously masks the prediction deviation situation. And if calculating to the ad granularity, there may be some ads with very large fluctuations in their bias index due to few positive samples. For example, if an ad is only exposed 1 time and only clicked 1 time, then the click-through rate of this ad = 100%. Since the predicted value is generally less than 100%, the bias must be less than 0, and this bias value is obtained with only 1 positive sample, which is obviously not reliable enough.

[0043] For the above problems, using the evaluation method of the above resource prediction model, first divide at least two media resource samples into at least two sample sets with different sample numbers and corresponding to different preset deviation thresholds according to the sample numbers of at least two media resource samples, and then respectively based on the preset deviation thresholds corresponding to each sample set in the at least two sample sets, evaluate the resource prediction results obtained after predicting the corresponding sample set through the resource prediction model, so as to obtain the evaluation results corresponding to each sample set. Furthermore, integrate the evaluation results corresponding to each sample set to obtain a model quality score for evaluating the prediction deviation degree of the resource prediction model. In the whole evaluation process, the index of the sample number of the media resource sample is considered in detail, and different sample numbers correspond to the confidence situations of different preset deviation thresholds. Furthermore, it is possible to first obtain the prediction deviation situations of the resource prediction model under different sample numbers, and then perform integrated detection, achieving the purpose of comprehensively considering the influencing factors of the sample information of the resource sample itself on the prediction deviation of the resource prediction model in a more fine-grained dimension, thereby realizing the technical effect of improving the evaluation accuracy of the resource prediction model, and further improving the accuracy of the subsequent media resource prediction based on the resource prediction model.

[0044] Optionally, in this embodiment, the media resource sample may be, but is not limited to, a sample used to evaluate the prediction deviation degree of the resource prediction model. Among them, the resource prediction model may be, but is not limited to, outputting a corresponding predicted value for the input media resource sample, and the gap between the predicted value and the true value corresponding to the media resource sample is the prediction deviation degree of the resource prediction model.

[0045] Optionally, in this embodiment, when the media resource sample is an advertisement sample, the resource prediction model may be, but is not limited to, performing prediction processing on the input advertisement sample to obtain a click predicted value corresponding to the advertisement sample. When the media resource sample is a stock sample, the resource prediction model may be, but is not limited to, performing prediction processing on the input stock sample to obtain a change predicted value corresponding to the stock sample.

[0046] It can be understood that the above resource prediction model may be, but is not limited to, used for predicting / forecasting information in a certain industry or dimension. Correspondingly, the media resource sample is a sample used to train and evaluate the resource prediction model in a certain industry or dimension. This embodiment does not limit the specific implementation manners of the media resource sample and the resource prediction model.

[0047] Optionally, in this embodiment, each media resource sample may be, but is not limited to, including multiple positive samples, and the sample quantity of each media resource sample in at least two media resource samples is the above-mentioned multiple positive samples.

[0048] Illustratively, taking the media resource sample as an advertisement sample as an example, an advertisement sample may be, but is not limited to, a media video with a duration of A, including multiple video segments with a duration of B. Among them, the multiple video segments with a duration of B are the multiple positive samples included in an advertisement sample, and A and B are positive numbers and B is less than A.

[0049] Optionally, in this embodiment, the first media resource sample is divided into the first sample set, and the second media resource sample is divided into the second sample set. Among them, at least two media resource samples include the first media resource sample and the second media resource sample, and at least two sample sets include the first sample set and the second sample set. The sample quantities in different sample sets are different, and different sample sets correspond to different preset deviation thresholds.

[0050] Optionally, in this embodiment, the preset deviation threshold may be, but is not limited to, a preset bias index. The bias index value is the click-through rate predicted value pctr of the media resource divided by the overall click-through rate ctr, and then minus 1. The larger the absolute value of the bias, the greater the prediction deviation amplitude of the model.

[0051] It should be noted that the larger the number of samples of the media resource samples, the smaller the preset deviation threshold of the corresponding sample set. That is, the larger the number of samples, the stricter the bias standard, that is, the smaller the allowable deviation range.

[0052] Optionally, in this embodiment, a resource estimation result obtained by the resource estimation model after estimating at least two sample sets is obtained, where the resource estimation result includes a first resource estimation result obtained after estimating the first sample set, and a second resource estimation result obtained after estimating the second sample set.

[0053] Optionally, in this embodiment, the resource estimation result can be used but not limited to indicating the proportion of samples in the sample set whose estimated value output by the resource estimation model exceeds the preset deviation threshold corresponding to the sample set from the true value, that is, the proportion of abnormal cases, and can also be used but not limited to indicating the proportion of samples in the sample set whose estimated value output by the resource estimation model does not exceed the preset deviation threshold corresponding to the sample set from the true value, that is, the proportion of normal cases, where the proportion of normal cases is equal to 1 minus the proportion of abnormal cases.

[0054] It should be noted that even if the resource estimation model is completely accurate, for example, in a 95% confidence interval, due to random fluctuations, a certain proportion of advertisements will be judged as abnormal cases exceeding the bias threshold. Therefore, the proportion of normal cases (1 - the proportion of abnormal cases of the model) theoretically cannot reach 1. In order to make the value range of the model evaluation index fall between 0 and 1, after calculating the proportion of normal cases (1 - the proportion of abnormal cases of the model), it is necessary to divide by a non-bias parameter of the "theoretical upper limit of the unbiased proportion" for normalization, and finally calculate the evaluation set of the sample set.

[0055] Optionally, in this embodiment, the evaluation result of the sample set can be used but not limited to indicating the estimation deviation degree of the resource estimation model under the sample set of this sample quantity segment. In order to further obtain the estimation deviation degree of the resource estimation model under the overall sample quantity segment, the evaluation combinations of each sample set are integrated to obtain the model quality score of the resource estimation model, where the higher the model quality score, the lower the estimation deviation degree of the resource estimation model, and the higher the accuracy of subsequent resource estimation.

[0056] For further illustration with an example, taking the acquisition of two media resource samples as an example, the evaluation method of the above resource estimation model can be but not limited to as Figure 3 shown, and the specific steps include:

[0057] Step S302: Obtain a first media resource sample and a second media resource sample. Here, the number of samples in the first media resource sample is 15, and the number of samples in the second media resource sample is 30;

[0058] Step S304: Divide the first media resource sample into a first sample set, and divide the second media resource sample into a second sample set. Here, the first preset deviation threshold corresponding to the first sample set is 50%, and the second preset deviation threshold corresponding to the second sample set is 40%;

[0059] Step S306: Obtain a first resource estimation result obtained by the resource estimation model after estimating the first sample set, and a second resource estimation result obtained by estimating the second sample set. Here, the first resource estimation result is used to indicate, among the 15 samples in the first sample set, the proportion of resource consumption occupied by M samples whose deviation degree (i.e., the estimated value divided by the true value, then subtracted by 1, and then taking the absolute value) of the estimated value output by the resource estimation model from the true value of the sample is less than the first preset deviation threshold of 50% (i.e., the resource consumption of M samples divided by the resource consumption of 15 samples). The second resource estimation result is used to indicate, among the 30 samples in the second sample set, the proportion of resource consumption occupied by M samples whose deviation degree of the estimated value output by the resource estimation model from the true value of the sample is less than the first preset deviation threshold of 40%;

[0060] Step S308: Evaluate the first resource estimation result with the first preset deviation threshold to obtain a first evaluation result of the first sample set, and evaluate the second resource estimation result with the second preset deviation threshold to obtain a second evaluation result of the second sample set. Here, the first evaluation result is used to indicate the estimation deviation degree of the resource estimation model in the first sample set, and the second evaluation result is used to indicate the estimation deviation degree of the resource estimation model in the second sample set;

[0061] Step S310: Perform weighted integration on the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model. Here, the model quality score can be, but is not limited to, the first evaluation result multiplied by the proportion of resource consumption of the first sample set in the total sample set, plus the second evaluation result multiplied by the proportion of resource consumption of the second sample set in the total sample set.

[0062] Through the embodiments provided in this application, first, at least two media resource samples are divided into at least two sample sets with different sample numbers and corresponding to different preset deviation thresholds according to the sample numbers of at least two media resource samples. Then, based on the preset deviation thresholds corresponding to each sample set in the at least two sample sets, the resource estimation results obtained by estimating the corresponding sample sets through a resource estimation model are evaluated to obtain the evaluation results corresponding to each sample set. Furthermore, the evaluation results corresponding to each sample set are integrated to obtain a model quality score for evaluating the estimation deviation degree of the resource estimation model. In the overall evaluation process of the resource estimation model, the index of the sample number of the media resource samples is considered in detail, and different sample numbers correspond to the confidence situations of different preset deviation thresholds. Therefore, it is possible to first obtain the estimation deviation situations of the resource estimation model under the confidence situations of different sample numbers and corresponding different preset deviation thresholds, and then perform integrated detection, comprehensively considering the influencing factors of the sample information of the resource samples themselves on the estimation deviation of the resource estimation model at a finer-grained dimension, achieving the purpose of improving the evaluation accuracy of the resource estimation model. Furthermore, the accuracy of the estimation information obtained by the resource estimation model for estimating and judging relevant resources is improved, thus achieving the technical effect of improving the accuracy of resource estimation.

[0063] As an alternative solution, evaluating the first resource estimation result through the first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result includes:

[0064] S1. Obtain the first non-deviation consumption ratio of the first sample set, where the first non-deviation consumption ratio is the first ratio of the media resource samples in the first sample set with an estimation deviation degree within the first preset deviation threshold indicated by the first resource estimation result in the first sample set;

[0065] S2. Use the first sample number and the first preset deviation threshold of the first sample set to obtain a first non-deviation parameter corresponding to the first sample set, where the first non-deviation parameter is used to indicate the second ratio of the media resource samples with an estimation deviation degree within the first preset deviation threshold in the first sample set when there is no estimation deviation in the resource estimation model;

[0066] S3. Determine the quotient of the first non-deviation consumption ratio and the first non-deviation parameter as the first evaluation result.

[0067] Optionally, in this embodiment, obtain the first non-bias consumption ratio of the first sample set, where the first non-bias consumption ratio is used to indicate the resource consumption ratio occupied by M samples in N samples of the first sample set, where the deviation degree between the predicted value output by the resource prediction model and the true value of the sample is less than the first preset deviation threshold (that is, the resource consumption of M samples divided by the resource consumption of N samples).

[0068] It should be noted that even if the resource prediction model is completely accurate, for example, in a 95% confidence interval, due to random fluctuations, a certain proportion of advertisements will be judged as abnormal cases exceeding the bias threshold. Therefore, the proportion of normal cases (1 - the proportion of model abnormal cases) cannot theoretically reach 1. In order to make the value range of the model evaluation index fall between 0 and 1, after calculating the proportion of normal cases (1 - the proportion of model abnormal cases), it is necessary to divide by a non-bias parameter of the "theoretical upper limit of unbiased proportion" (such as the above first non-bias parameter) for normalization, and finally calculate the evaluation set of the sample set.

[0069] It can be understood that the first non-bias parameter is used to indicate how many proportions of the predicted deviations of the samples in the prediction results of the resource prediction model should be within the first preset deviation threshold if there is no prediction deviation in the resource prediction model at all.

[0070] It should be noted that the quotient of the first non-bias consumption ratio and the first non-bias parameter is determined as the first evaluation result.

[0071] Through the embodiment provided by the present application, by using the non-bias parameter as the theoretical upper limit of the unbiased proportion, normalizing the calculated proportion of normal cases of the model, and finally calculating the evaluation result corresponding to the sample set, and making the value range of the evaluation index of the resource prediction model always fall within 0-1, thereby achieving the technical effect of improving the accuracy of the evaluation of the resource prediction model.

[0072] As an optional solution, using the first sample quantity of the first sample set and the first preset deviation threshold, obtaining the first non-bias parameter corresponding to the first sample set includes:

[0073] S1, perform a square root operation on the first sample quantity, and then perform a reciprocal operation on the result of the square root operation to obtain the first standard deviation corresponding to the first sample set;

[0074] S2, divide the first preset deviation threshold by the first standard deviation to obtain the first standard deviation multiple corresponding to the first sample set;

[0075] S3. Use the first standard deviation multiple as the input of the preset density function to obtain the first coverage ratio output by the preset density function, where the preset density function is the cumulative density function representing the normal distribution.

[0076] S4. Determine the first coverage ratio as the first non-bias parameter.

[0077] Optionally, in this embodiment, obtain the first sample quantity of the first sample set, perform a square root operation on the first sample quantity, and determine the reciprocal of the result of the square root operation as the first standard deviation corresponding to the first sample set.

[0078] Optionally, in this embodiment, divide the first preset deviation threshold by the first standard deviation to obtain the first standard deviation multiple corresponding to the first sample set.

[0079] Optionally, in this embodiment, use the first standard deviation multiple as the input of the preset density function to obtain the first coverage ratio output by the preset density function, and determine the first coverage ratio as the first non-bias parameter.

[0080] For further illustration, as shown in Table 1, where column A represents that the first sample quantity of the first sample set is 10, column A1 represents that the first preset deviation parameter of the first sample set is 50%, column A2 represents that the first standard deviation of the first sample set is 1 / sqrt(10) which is equal to 0.316, column A3 is obtained by dividing column A1 by column A2, representing that the first standard deviation multiple of the first sample set is 0.5 / 0.316 which is equal to 1.581, and column A4 is obtained by the following formula: A4 = 2·Φ(A3) - 1, where Φ is the cumulative density function representing the normal distribution (preset density function).

[0081] It can be understood that column A4 represents that if there is no deviation at all, then on average, what proportion of the samples should have a deviation degree within the standard of 50%.

[0082] Through the embodiments provided in this application, the influence of the sample quantity of media resource samples on model evaluation is considered in detail. By calculating the non-bias parameter obtained, the purpose of evaluating the prediction deviation of the resource prediction model at a finer-grained dimension is achieved, thereby improving the accuracy of the evaluation of the resource prediction model, and thus realizing the technical effect of improving the accuracy of resource prediction.

[0083] Table 1

[0084] A A1 A2 A3 A4 n Standard Standard Deviation Multiple of Standard Deviation Coverage Ratio 10 0.5 0.316 1.581 88.6％

[0085] As an alternative solution, obtaining the first non-bias consumption ratio corresponding to the first sample set includes:

[0086] S1. Determine, from the first sample set, third media resource samples whose estimated results deviate from the true values by less than or equal to a first preset deviation threshold according to the first resource estimation result.

[0087] S2. When the first resource consumption corresponding to the first sample set is obtained, determine the ratio of the second resource consumption corresponding to the third media resource samples to the first resource consumption as the first non-deviation consumption ratio.

[0088] Optionally, in this embodiment, from the N samples included in the first sample set, determine M samples whose estimated results deviate from the true values by less than or equal to the first preset deviation threshold, and determine the M samples as the third media resource samples.

[0089] Optionally, in this embodiment, determine the ratio of the second resource consumption corresponding to the third media resource samples to the first resource consumption as the first non-deviation consumption ratio (i.e., the resource consumption of the M samples divided by the resource consumption of the N samples).

[0090] For further illustration, taking the acquisition of two media resource samples as an example, an optional method for evaluating a resource estimation model is as Figure 4 shown, and the specific steps include:

[0091] Step S402. Obtain a first media resource sample and a second media resource sample.

[0092] Step S404. Divide the first media resource sample into a first sample set, and divide the second media resource sample into a second sample set.

[0093] Step S406. Obtain a first resource estimation result obtained by the resource estimation model after estimating the first sample set, and a second resource estimation result obtained by estimating the second sample set.

[0094] Step S408. Obtain the first non-deviation consumption ratio of the first sample set, and use the first sample quantity of the first sample set and the first preset deviation threshold to obtain a first non-deviation parameter corresponding to the first sample set. Determine the quotient of the first non-deviation consumption ratio and the first non-deviation ratio as the first evaluation result. Also, obtain the second non-deviation consumption ratio of the second sample set, and use the second sample quantity of the second sample set and the second preset deviation threshold to obtain a second non-deviation parameter corresponding to the second sample set. Determine the quotient of the second non-deviation consumption ratio and the second non-deviation ratio as the second evaluation result. Among them, step S408 includes step S4081 and step S4082.

[0095] Step S4081: Take the square root of the first sample quantity, and then take the reciprocal of the result of the square root operation to obtain the first standard deviation corresponding to the first sample set; divide the first preset deviation threshold by the first standard deviation to obtain the first standard deviation multiple corresponding to the first sample set; use the first standard deviation multiple as the input of the preset density function to obtain the first coverage ratio output by the preset density function, and determine the first non-deviation parameter of the first sample quantity as the first coverage ratio.

[0096] Step S4082: Take the square root of the second sample quantity, and then take the reciprocal of the result of the square root operation to obtain the second standard deviation corresponding to the second sample set; divide the second preset deviation threshold by the second standard deviation to obtain the second standard deviation multiple corresponding to the second sample set; use the second standard deviation multiple as the input of the preset density function to obtain the second coverage ratio output by the preset density function, and determine the second non-deviation parameter of the second sample quantity as the second coverage ratio.

[0097] Step S410: Perform weighted integration on the first evaluation result and the second evaluation result to obtain the model quality score of the resource estimation model, where the model quality score can be, but is not limited to, the first evaluation result multiplied by the resource consumption proportion of the first sample set in the total sample set, plus the second evaluation result multiplied by the resource consumption proportion of the second sample set in the total sample set.

[0098] As an alternative solution, integrating the first evaluation result and the second evaluation result to obtain the model quality score of the resource estimation model includes:

[0099] S1: Obtain the first consumption proportion of the first resource consumption corresponding to the first sample set in the third resource consumption corresponding to at least two sample sets, and obtain the second consumption proportion of the fourth resource consumption corresponding to the second sample set in the third resource consumption;

[0100] S2: When obtaining the first product result of the first evaluation result and the first consumption proportion, and the second product result of the second evaluation result and the second consumption proportion, use the sum value of the first product result and the second product result to determine the model quality score.

[0101] Optionally, in this embodiment, the first resource consumption corresponding to the first sample set can be, but is not limited to, the resource consumption corresponding to the resource samples included in the first sample set, and the third resource consumption corresponding to at least two sample sets can be, but is not limited to, the resource consumption corresponding to the resource samples included in at least two sample sets.

[0102] It can be understood that when the media resource sample is an advertisement sample, the corresponding resource consumption can be, but is not limited to, the advertising investment cost; when the media resource sample is a stock sample, the corresponding resource consumption can be, but is not limited to, the stock investment cost, and the present embodiment does not impose additional restrictions on the specific resource consumption.

[0103] It should be noted that, in the case of obtaining the evaluation results and resource consumption ratios corresponding to each sample set, the evaluation results corresponding to each sample set are multiplied by the resource consumption ratios to obtain the product results corresponding to each sample set, and then the sum value obtained by adding up each product result is determined as the model quality score.

[0104] Through the embodiments provided by the present application, considering indicators such as the data volume and resource consumption of media resource samples, integrating and detecting bias indicators under different confidence levels as much as possible, so as to provide a model quality score under a certain confidence level, thereby improving the accuracy of the evaluation of the resource prediction model, and thus achieving the technical effect of improving the accuracy of resource prediction.

[0105] As an optional solution, after obtaining the model quality score of the resource prediction model by evaluating the prediction result through a preset deviation threshold, the method further includes:

[0106] S1. When the model quality score is less than the first preset score threshold, it is determined that the prediction deviation degree of the resource prediction model satisfies the overall non-convergence condition;

[0107] S2. Determine at least one media resource sample whose product result is less than the second preset score threshold from at least two media resource samples, where the product result includes a first product result and a second product result, and the second preset score threshold is less than the first preset score threshold;

[0108] S3. Obtain the target sample quantity corresponding to at least one media resource sample, and determine that the prediction deviation degree of the resource prediction model satisfies the local non-convergence condition at the target sample quantity.

[0109] Optionally, in this embodiment, when the model quality score is less than the first preset score threshold, it is determined that the prediction deviation degree of the resource prediction model does not meet the requirements as a whole, that is, it satisfies the overall non-convergence condition.

[0110] It should be noted that when the final model quality score is lower than the first preset score threshold, it can be, but is not limited to, giving an alarm, sending a prompt message, and further analyzing whether the prediction deviation of the resource prediction model is larger on sparse samples or dense samples.

[0111] Optionally, in this embodiment, at least one media resource sample whose product result is less than the second preset score threshold is determined from at least two media resource samples, and it is determined that the estimation deviation degree of the resource estimation model is relatively large within the interval of the target sample quantity corresponding to at least one media resource sample. It is possible but not limited to further perform corresponding parameter adjustment and optimization for the interval of the target sample quantity.

[0112] Through the embodiments provided in this application, when an abnormal situation occurs in the estimation resource model (such as the model quality score is lower than the first preset score threshold), problems can be investigated based on data volume, consumption, etc. in the sub - dimensions, and disaster tolerance priorities can be formulated, so as to more efficiently detect the quality of the resource estimation model and conduct problem diagnosis and investigation, thereby achieving the technical effect of improving the efficiency of evaluating and using the resource estimation model.

[0113] As an alternative solution, after evaluating the estimation result through a preset deviation threshold to obtain the model quality score of the resource estimation model, the method further includes:

[0114] S1, in response to a resource estimation request triggered for at least two media resources, input the at least two media resources into the resource estimation model for estimation processing to obtain an estimation result, where the estimation result includes first estimation information for indicating the recommended order of each media resource among the at least two media resources and second estimation information for indicating the click prediction value of each media resource.

[0115] Optionally, in this embodiment, after evaluating the estimation result through a preset deviation threshold to obtain the model quality score of the resource estimation model, a verification process is performed on the model quality score. Among them, when the model quality score is less than the first preset score threshold, it is determined that the estimation deviation degree of the resource estimation model satisfies the overall non - convergence condition.

[0116] When the above - mentioned model quality score is not less than the first preset score threshold, it is determined that the estimation deviation degree of the resource estimation model satisfies the overall convergence condition, obtain a resource estimation request triggered for at least two media resources, and respond, perform estimation processing on the at least two media resources to obtain an estimation result, where the estimation result includes the above - mentioned first estimation information and the above - mentioned second estimation information.

[0117] For example, when the media resource is an advertising resource, input at least two advertising resources into the resource estimation model for estimation processing, and the obtained estimation result includes first estimation information for indicating the push (recommendation) order of at least two advertising resources, and second estimation information for indicating the click prediction value of each advertising resource after being pushed according to the above - mentioned push information.

[0118] For further illustration, in the case where the media resource is a stock resource, at least two stock resources are input into the resource prediction model for prediction processing. The obtained prediction results include first prediction information for indicating the recommended order of at least two stock resources, and second prediction information for indicating the view prediction value of each stock resource after being recommended according to the above-mentioned recommendation information.

[0119] Through the embodiments provided in this application, the index of the sample quantity of the media resource sample is considered in detail, and different sample quantities correspond to different confidence levels of the preset deviation threshold. Furthermore, it is possible to first obtain the prediction deviation situation of the resource prediction model under different sample quantities, and then perform integrated detection, achieving the purpose of comprehensively considering the influencing factors of the sample information of the resource sample itself on the prediction deviation of the resource prediction model in a more fine-grained dimension. Thus, the technical effect of improving the evaluation accuracy of the resource prediction model is achieved, and furthermore, the accuracy of the subsequent media resource prediction based on the resource prediction model is improved.

[0120] As an optional solution, the above-mentioned resource prediction model is applied to an evaluation scenario of the prediction deviation of a refined ranking model. Starting from the advertisement dimension, indicators such as the prediction deviation of the advertisement recommendation algorithm and the sample confidence are comprehensively considered to form a quality score calculation method for evaluating the prediction deviation situation of the refined ranking model.

[0121] It should be noted that in the existing industry practice, generally, the ratio of the predicted value deviating from the true value, that is, the bias index, is used as the main index for evaluating the prediction deviation. For example, if the predicted click-through rate of an overall advertisement traffic is pctr and the overall click-through rate is ctr, then bias = pctr / ctr - 1. The larger the absolute value of bias, the greater the amplitude of the prediction deviation.

[0122] Furthermore, on the one hand, the existing bias index is calculated on a relatively coarse dimension. For example, the bias index of an advertisement position makes it difficult to observe fine-grained prediction anomalies. On the other hand, even when calculating to the dimension of the advertisement category, there will still be a phenomenon of mutual cancellation of the bias index. For example, the bias of advertisement category A is +50%, that is, the prediction is 50% higher, and the bias of advertisement category B is -50%, that is, the prediction is 50% lower. At this time, the average value of the advertisement category bias is (+50% + -50%) / 2 = 0, that is, there is no deviation, which obviously masks the prediction deviation situation. And if calculating to the advertisement granularity, there may also be some advertisements with large fluctuations in their bias index due to few positive samples. For example, if an advertisement is only exposed once and only clicked once, then the click-through rate of this advertisement = 100%. Since the predicted value is generally less than 100%, the bias must be less than 0, and this bias value is obtained with only one positive sample, which is obviously not confident enough.

[0123] For the above problems, an evaluation method for the prediction deviation of the refined ranking model, which uses the evaluation method based on the above resource prediction model, specifically includes the following steps:

[0124] Step 1: Segment the advertisements according to the number of positive samples, as shown in Column A of Table 2;

[0125] Step 2: Calculate the resource consumption of each segment obtained in Step 1, as shown in Column B of Table 2;

[0126] Step 3: According to the segments obtained in Step 1, formulate a bias standard for each segment, as shown in Column D of Table 2. Among them, the more positive samples there are, the stricter the bias standard is, that is, the smaller the allowable deviation range is;

[0127] Step 4: Calculate the consumption that does not meet the bias standard under each segment in Step 1, that is, the unbiased consumption, as shown in Column E of Table 2;

[0128] Table 2

[0129]

[0130] Step 5: Calculate the theoretical upper limit of the unbiased proportion of each segment in Step 1. Specifically, when the predicted number of positive samples is n, according to the bias standard in Step 3, it can be based on the standard deviation of Calculate how many times the standard in Column D corresponds to the standard deviation, and then according to this multiple, calculate the quantile corresponding to the normal distribution. Finally, for n within the range of Column A, perform an exhaustive search and take the average of the results.

[0131] Taking the positive sample segment of 10 - 20 as an example, the calculation process of the theoretical upper limit of the unbiased proportion is as follows in Table 3:

[0132] Table 3

[0133]

[0134]

[0135] Among them, Column A is all values from 10 to 20. Column A1 is the bias standard of 50%. Column A2 is obtained from Column A according to 1 / √n. For example, 0.316 is equal to 1 / √10. Then Column A3 is obtained by dividing Column A1 by Column A2. Column A4 is obtained from the following formula:

[0136] A4 = 2·Φ(A3) - 1

[0137] Where Φ represents the cumulative density function of the normal distribution. Column A4 represents that if there is no deviation at all, then on average, what proportion of the advertisements should have a deviation degree within the standard of 50%.

[0138] Finally, take the average of all the values in column A4, and the final 0.941 is obtained, which represents the result of column G in the calculation example.

[0139] Repeating the above results, the results of other positive sample number segments can be obtained, as shown in Table 4 (where the 200+ segment is calculated by substituting 200-300):

[0140] Table 4

[0141] Column A Column G 10-20 94.1％ 20-50 97.5％ 50-100 96.5％ 100-200 98.2％ 200+ 98.1％

[0142] The schematic diagram of the cumulative density function curve of the normal distribution is as Figure 5 shown.

[0143] Step 6: Normalize the proportion of abnormal cases in bias estimation. Even if the model is completely accurate, within the 95% confidence interval, due to random fluctuations, a certain proportion of advertisements will exceed the bias threshold and be judged as abnormal cases. Therefore, (1 - the proportion of abnormal cases in the model) theoretically cannot reach 1. In order to make the value range of the model evaluation index fall between 0 and 1, after calculating (1 - the proportion of abnormal cases in the model), it is necessary to divide by an "upper limit of the unbiased proportion theory" (column G) for normalization, and finally calculate the segment score, as shown in column H of Table 2.

[0144] Step 7: Finally, based on the segment score and segment consumption, calculate the weighted score of the model in all rules score = segment score * consumption proportion (column C of Table 2), and obtain the final model quality score, as shown in column I of Table 2.

[0145] Step 8: Configure the detection to give an alarm when the final model quality score is lower than a certain threshold, and then it is possible to further analyze whether the prediction deviation is larger in sparse samples or dense samples.

[0146] It should be noted that in the above step 7, instead of performing weighted aggregation, directly detecting the segment scores in step 6 can also have a similar effect. The above bias may not necessarily be the overall bias of the ad slots, but can also be the prediction deviation of a certain industry or a certain dimension. In this case, regard this industry or this dimension as a whole, perform segmentation and bias standard division, and substitute them into the above steps for calculation.

[0147] It should be noted that the above method can be used, but is not limited to, the scenario of using a fine-rank model for prediction before advertising placement. The general behavior chain of users on advertisements in the APP can be summarized as: display (exposure) -> click -> conversion (registering an account, downloading the APP, following the official account, form submission, etc.). There is generally an estimation algorithm at each arrow to predict the probability of generating the behavior on the right side of the arrow given the left side, such as for display -> click, it is the click-through rate estimation algorithm. By improving the accuracy of these estimation algorithms, we can recommend advertisement content that users are interested in and that advertisers are willing to pay for. Since the advertisement recommendation system also involves charging advertisers, according to the eCPM deduction model, it is required that the advertisement recommendation algorithm, especially the advertisement recommendation algorithm in the fine-rank stage, should not only ensure excellent ranking ability but also ensure that the predicted value does not deviate too much, otherwise it is easy to cause the advertisement deduction to be too high or too low.

[0148] Through the embodiments provided in this application, this solution comprehensively considers indicators such as the data volume and consumption of samples, and integrates and detects bias indicators under different confidence levels as much as possible, so as to provide a model quality score under a certain confidence level. At the same time, when the model has abnormal conditions, it is possible to check problems and formulate disaster tolerance priorities according to the data volume, consumption, etc. in the sub-dimensions, which helps the advertisement recommendation system to more efficiently detect the model quality and conduct problem diagnosis and troubleshooting.

[0149] It can be understood that in the specific implementation manner of this application, data related to user information, etc. is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0150] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0151] According to another aspect of the embodiments of this application, there is also provided an evaluation device for a resource estimation model for implementing the above-mentioned evaluation method of the resource estimation model. As Figure 6 shown, the device includes:

[0152] An acquisition unit 602, configured to acquire at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of the resource estimation model, and the resource estimation model is used to estimate media resources;

[0153] A dividing unit 604, configured to divide at least two media resources into at least two sample sets according to the number of samples of at least two media resource samples, wherein the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set;

[0154] An evaluation unit 606, configured to, when obtaining a first resource estimation result obtained by the resource estimation model after estimating the first sample set and a second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result;

[0155] An integration unit 608, configured to integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, where the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

[0156] As an optional solution, the evaluation unit 606 includes:

[0157] A first obtaining module, configured to obtain a first non-deviation consumption ratio of the first sample set, where the first non-deviation consumption ratio is a first ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, indicated by the first resource estimation result, in the first sample set;

[0158] A second obtaining module, configured to use the first sample number of the first sample set and the first preset deviation threshold to obtain a first non-deviation parameter corresponding to the first sample set, where the first non-deviation parameter is used to indicate a second ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, in the case that there is no estimation deviation in the resource estimation model;

[0159] A first determining module, configured to determine the quotient of the first non-deviation consumption ratio and the first non-deviation parameter as the first evaluation result.

[0160] As an optional solution, the second obtaining module includes:

[0161] A first calculating sub-module, configured to perform a square root operation on the first sample number and then perform a reciprocal operation on the result of the square root operation to obtain a first standard deviation corresponding to the first sample set;

[0162] A second calculation sub-module, configured to divide a first preset deviation threshold by a first standard deviation to obtain a first standard deviation multiple corresponding to a first sample set;

[0163] A third calculation sub-module, configured to use the first standard deviation multiple as an input of a preset density function to obtain a first coverage ratio output by the preset density function, where the preset density function is a cumulative density function representing a normal distribution;

[0164] A first determination sub-module, configured to determine the first coverage ratio as a first non-deviation parameter.

[0165] As an alternative solution, the first acquisition module includes:

[0166] A second determination sub-module, configured to determine, according to a first resource estimation result, third media resource samples in the first sample set whose deviation degree of the estimation result from the true value is less than or equal to a first preset deviation threshold;

[0167] A third determination sub-module, configured to, when obtaining a first resource consumption corresponding to the first sample set, determine a ratio of a second resource consumption corresponding to the third media resource sample to the first resource consumption as a first non-deviation consumption ratio.

[0168] As an alternative solution, the integration unit 608 includes:

[0169] A third acquisition module, configured to obtain a first consumption ratio of a first resource consumption corresponding to the first sample set in third resource consumptions corresponding to at least two sample sets, and obtain a second consumption ratio of a fourth resource consumption corresponding to the second sample set in the third resource consumption;

[0170] A second determination module, configured to, when obtaining a first product result of a first evaluation result and the first consumption ratio and a second product result of a second evaluation result and the second consumption ratio, determine a model quality score by using a sum value of the first product result and the second product result.

[0171] As an alternative solution, the apparatus further includes:

[0172] A third determination module, configured to, after evaluating an estimation result by using a preset deviation threshold to obtain a model quality score of a resource estimation model, and when the model quality score is less than a first preset score threshold, determine that an estimation deviation degree of the resource estimation model satisfies an overall non-convergence condition;

[0173] A fourth determination module, configured to, after evaluating an estimation result through a preset deviation threshold to obtain a model quality score of a resource estimation model, determine at least one media resource sample with a product result less than a second preset score threshold from at least two media resource samples, where the product result includes a first product result and a second product result, and the second preset score threshold is less than the first preset score threshold;

[0174] A fifth determination module, configured to, after evaluating an estimation result through a preset deviation threshold to obtain a model quality score of a resource estimation model, obtain a target sample quantity corresponding to at least one media resource sample, and determine that the estimation deviation degree of the resource estimation model satisfies a local non-convergence condition at the target sample quantity.

[0175] As an alternative solution, the apparatus further includes:

[0176] An estimation module, configured to, after evaluating an estimation result through a preset deviation threshold to obtain a model quality score of a resource estimation model, in response to a resource estimation request triggered for at least two media resources, input the at least two media resources into the resource estimation model for estimation processing to obtain an estimation result, where the estimation result includes first estimation information for indicating a recommended order of each media resource among the at least two media resources and second estimation information for indicating a click prediction value of each media resource.

[0177] According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the above-mentioned evaluation method of a resource estimation model. The electronic device may be, but is not limited to, Figure 1 the client 102 or the server 112 shown in Figure 7 . Taking the electronic device as the client 102 as an example in this embodiment, further as shown in

[0178] , the electronic device includes a memory 702 and a processor 704. A computer program is stored in the memory 702, and the processor 704 is configured to execute the steps in any one of the above method embodiments through the computer program.

[0179] Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through the computer program:

[0180] S1, obtain at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of a resource estimation model, and the resource estimation model is used to estimate media resources;

[0181] S2. Divide at least two media resources into at least two sample sets according to the number of samples of at least two media resource samples, where the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set;

[0182] S3. After obtaining the first resource estimation result obtained by the resource estimation model for estimating the first sample set and the second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through the first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through the second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result;

[0183] S4. Integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, where the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

[0184] Optionally, those of ordinary skill in the art can understand that Figure 7 The structure shown is only schematic Figure 7 and does not limit the structure of the above electronic device. For example, the electronic device may further include more or fewer components (such as a network interface, etc.) than those shown Figure 7 or have a different configuration from that shown Figure 7 .

[0185] Among them, the memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the evaluation method and device of the resource estimation model in the embodiments of the present application. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, implements the above-mentioned evaluation method of the resource estimation model. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 702 may further include a memory remotely disposed relative to the processor 704, and these remote memories may be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network and their combinations. Among them, the memory 702 can specifically but not limitedly be used to store information such as the model quality score. As an example, such as Figure 7As shown, the above-mentioned memory 702 may but is not limited to include the acquisition unit 602, the division unit 604, the evaluation unit 606, and the integration unit 608 in the evaluation device of the above-mentioned resource prediction model. In addition, it may also include but is not limited to other module units in the evaluation device of the above-mentioned resource prediction model, which will not be elaborated in this example.

[0186] Optionally, the above-mentioned transmission device 706 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 706 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable, so as to communicate with the Internet or a local area network. In one example, the transmission device 706 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0187] In addition, the above-mentioned electronic device further includes: a display 708, which is used to display information such as the model quality score; and a connection bus 710, which is used to connect each module component in the above-mentioned electronic device.

[0188] In other embodiments, the above-mentioned client or server may be a node in a distributed system. Among them, the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes can form a peer-to-peer network, and any form of computing device, such as an electronic device such as a server or a client, can become a node in the blockchain system by joining the peer-to-peer network.

[0189] According to one aspect of the present application, a computer program product is provided. The computer program product includes computer programs / instructions, and the computer programs / instructions include program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit, various functions provided by the embodiments of the present application are executed.

[0190] The serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0191] It should be noted that the computer system of the electronic device is only an example and should not bring any limitations to the functions and usage scopes of the embodiments of the present application.

[0192] A computer system includes a Central Processing Unit (CPU), which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) or a program loaded from a storage section into a Random Access Memory (RAM). In the random access memory, various programs and data required for system operation are also stored. The central processing unit, the read-only memory, and the random access memory are connected to each other via a bus. An Input / Output interface (I / O interface) is also connected to the bus.

[0193] The following components are connected to the input / output interface: an input section including a keyboard, a mouse, etc.; an output section including, for example, a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a Local Area Network card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. A removable medium, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive as needed so that a computer program read from it can be installed into the storage section as needed.

[0194] In particular, according to an embodiment of the present application, the processes described in each method flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section, and / or installed from a removable medium. When the computer program is executed by the central processing unit, various functions defined in the system of the present application are executed.

[0195] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above various optional implementation manners.

[0196] Optionally, in this embodiment, the above computer-readable storage medium may be set to store a computer program for executing the following steps:

[0197] S1. Obtain at least two media resource samples, where the at least two media resource samples are used to evaluate the prediction deviation degree of a resource prediction model, and the resource prediction model is used to predict media resources;

[0198] S2. Divide the at least two media resources into at least two sample sets according to the number of the at least two media resource samples, where the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set;

[0199] S3. In the case of obtaining a first resource prediction result obtained by the resource prediction model predicting the first sample set and a second resource prediction result obtained by predicting the second sample set, evaluate the first resource prediction result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource prediction result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result;

[0200] S4. Integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource prediction model, where the model quality score is used to evaluate the prediction deviation degree of the resource prediction model.

[0201] Optionally, in this embodiment, those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing relevant hardware of an electronic device, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc, etc.

[0202] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages and disadvantages of the embodiments.

[0203] If the integrated unit in the above embodiments is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0204] In the above embodiments of the present application, the descriptions of the respective embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0205] In several embodiments provided by the present application, it should be understood that the recorded client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0206] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] In addition, the functional units in the respective embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0208] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An evaluation method for a resource estimation model, characterized in that Including: Obtain at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of a resource estimation model, and the resource estimation model is used to estimate media resources; Divide the at least two media resources into at least two sample sets according to the number of samples of the at least two media resource samples, where the number of samples in different sample sets is different, and different sample sets correspond to different preset deviation thresholds. The at least two sample sets include a first sample set and a second sample set; When obtaining a first resource estimation result obtained by the resource estimation model estimating the first sample set and a second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through the first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through the second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result; Integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, where the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

2. The method according to claim 1, wherein The evaluating the first resource estimation result through the first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result includes: Obtain a first non-deviation consumption ratio of the first sample set, where the first non-deviation consumption ratio is a first ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, indicated by the first resource estimation result, in the first sample set; Use the first sample number of the first sample set and the first preset deviation threshold to obtain a first non-deviation parameter corresponding to the first sample set, where the first non-deviation parameter is used to indicate a second ratio of media resource samples in the first sample set, whose estimation deviation degree is within the first preset deviation threshold, in the case that there is no estimation deviation in the resource estimation model; Determine the quotient of the first non-deviation consumption ratio and the first non-deviation parameter as the first evaluation result.

3. The method according to claim 2, wherein The using the first sample number of the first sample set and the first preset deviation threshold to obtain a first non-deviation parameter corresponding to the first sample set includes: Perform a square root operation on the first sample number, and perform a reciprocal operation on the result of the square root operation to obtain a first standard deviation corresponding to the first sample set; Divide the first preset deviation threshold by the first standard deviation to obtain a first standard deviation multiple corresponding to the first sample set; Use the first standard deviation multiple as an input of a preset density function to obtain a first coverage ratio output by the preset density function, where the preset density function is a cumulative density function representing a normal distribution; Determine the first coverage ratio as the first non-deviation parameter.

4. The method according to claim 2, wherein Obtaining the first non-deviation consumption ratio corresponding to the first sample set includes: According to the first resource estimation result, determining third media resource samples from the first sample set, where the degree of deviation of the estimation result from the true value is less than or equal to the first preset deviation threshold; When the first resource consumption corresponding to the first sample set is obtained, determining the ratio of the second resource consumption corresponding to the third media resource samples to the first resource consumption as the first non-deviation consumption ratio.

5. The method according to claim 1, wherein Integrating the first evaluation result and the second evaluation result to obtain the model quality score of the resource estimation model includes: Obtaining the first consumption ratio of the first resource consumption corresponding to the first sample set in the third resource consumption corresponding to the at least two sample sets, and obtaining the second consumption ratio of the fourth resource consumption corresponding to the second sample set in the third resource consumption; When the first product result of the first evaluation result and the first consumption ratio, and the second product result of the second evaluation result and the second consumption ratio are obtained, determining the model quality score using the sum value of the first product result and the second product result.

6. The method according to claim 5, wherein After evaluating the estimation result through the preset deviation threshold to obtain the model quality score of the resource estimation model, the method further includes: When the model quality score is less than the first preset score threshold, determining that the estimation deviation degree of the resource estimation model satisfies the overall non-convergence condition; Determining at least one media resource sample whose product result is less than the second preset score threshold from the at least two media resource samples, where the product result includes the first product result and the second product result, and the second preset score threshold is less than the first preset score threshold; Obtaining the target sample quantity corresponding to the at least one media resource sample, and determining that the estimation deviation degree of the resource estimation model satisfies the local non-convergence condition at the target sample quantity.

7. The method according to any one of claims 1 to 6, characterized in that, After evaluating the estimation result through the preset deviation threshold to obtain the model quality score of the resource estimation model, the method further includes: In response to a resource estimation request triggered for at least two media resources, inputting the at least two media resources into the resource estimation model for estimation processing to obtain an estimation result, where the estimation result includes first estimation information for indicating the recommended order of each media resource in the at least two media resources and second estimation information for indicating the click prediction value of each media resource.

8. An evaluation device for a resource estimation model, characterized in that, Including: An obtaining unit, configured to obtain at least two media resource samples, where the at least two media resource samples are used to evaluate the estimation deviation degree of a resource estimation model, and the resource estimation model is used to estimate media resources; A dividing unit, configured to divide the at least two media resources into at least two sample sets according to the number of samples of the at least two media resource samples, wherein the number of samples in different sample sets is different, different sample sets correspond to different preset deviation thresholds, and the at least two sample sets include a first sample set and a second sample set; An evaluation unit, configured to, when obtaining a first resource estimation result obtained by the resource estimation model for estimating the first sample set and a second resource estimation result obtained by estimating the second sample set, evaluate the first resource estimation result through a first preset deviation threshold corresponding to the first sample set to obtain a first evaluation result, and evaluate the second resource estimation result through a second preset deviation threshold corresponding to the second sample set to obtain a second evaluation result; An integration unit, configured to integrate the first evaluation result and the second evaluation result to obtain a model quality score of the resource estimation model, wherein the model quality score is used to evaluate the estimation deviation degree of the resource estimation model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when run by an electronic device, executes the method described in any one of claims 1 to 7.

10. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images