Model quality evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a model quality evaluation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring estimated value distribution data and true value distribution data; performing aggregation processing on the estimated value distribution data and the true value distribution data by adopting a first aggregation period with a first time window and a second aggregation period with a second time window respectively to correspondingly obtain estimated value aggregation results and true value aggregation results in a plurality of first aggregation periods and a plurality of second aggregation periods; determining a first pre-estimated deviation value in each first aggregation period based on the pre-estimated value and the real value aggregation result in the first aggregation period; determining a second pre-estimated deviation value in each second aggregation period based on the pre-estimated value and the real value aggregation result in the second aggregation period; and performing model quality evaluation on the information recommendation model based on the first estimated deviation value and the second estimated deviation value. Through the method, the model quality can be effectively evaluated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model quality assessment method, device, electronic device and storage medium. Background Art

[0002] In the process of advertising, the behavior chain of advertising can generally be summarized as: display (exposure) → click → conversion. There is a prediction algorithm (model) at the arrow, which predicts the probability of the behavior on the right side of the arrow given the left side of the arrow. For example, there is a click-through rate prediction algorithm at the arrow of display → click. By improving the accuracy of the prediction algorithm, the advertising content of interest can be recommended to users.

[0003] However, the above-mentioned estimation models for ad recommendations are affected by factors such as features, samples, traffic, delayed sample return, model training, and updates, which may cause fluctuations in the estimated values ​​output by the estimation models. Currently, there is no unified and widely agreed-upon model volatility indicator to evaluate the quality of the estimation model. Summary of the invention

[0004] The embodiments of the present application provide a model quality assessment method, device, electronic device and storage medium, which can effectively assess the quality of the model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a model quality assessment method, the method comprising: obtaining estimated value distribution data and true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object for the recommended target in the time dimension that is pre-labeled; respectively using a first aggregation period with a first time window and a second aggregation period with a second time window to aggregate the estimated value distribution data and the true value distribution data, and correspondingly obtaining the estimated value aggregation results within multiple first aggregation periods, multiple first aggregation periods, The real value aggregation results within the aggregation cycle, the estimated value aggregation results within multiple second aggregation cycles, and the real value aggregation results within multiple second aggregation cycles; based on the estimated value aggregation results within the first aggregation cycle and the real value aggregation results within the first aggregation cycle, determine the first estimated deviation value of the information recommendation model in each of the first aggregation cycle; based on the estimated value aggregation results within the second aggregation cycle and the real value aggregation results within the second aggregation cycle, determine the second estimated deviation value of the information recommendation model in each of the second aggregation cycle; based on the first estimated deviation value and the second estimated deviation value, perform model quality evaluation on the information recommendation model.

[0007] The embodiment of the present application provides a model quality assessment device, including: a data acquisition module, used to obtain estimated value distribution data and true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object for the recommended target in the time dimension that is pre-labeled; an aggregation processing module, used to respectively use a first aggregation cycle with a first time window and a second aggregation cycle with a second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, and obtain corresponding estimated value aggregation results within multiple first aggregation cycles and true value aggregation results within multiple first aggregation cycles. value aggregation results, estimated value aggregation results within multiple second aggregation cycles, and real value aggregation results within multiple second aggregation cycles; a first deviation determination module, used to determine the first estimated deviation value of the information recommendation model in each of the first aggregation cycle based on the estimated value aggregation results within the first aggregation cycle and the real value aggregation results within the first aggregation cycle; a second deviation determination module, used to determine the second estimated deviation value of the information recommendation model in each of the second aggregation cycle based on the estimated value aggregation results within the second aggregation cycle and the real value aggregation results within the second aggregation cycle; a quality assessment module, used to perform model quality assessment on the information recommendation model based on the first estimated deviation value and the second estimated deviation value.

[0008] In the above scheme, the aggregation processing module is also used to use a first aggregation period with the first time window to perform aggregation processing on the estimated value distribution data and the true value distribution data respectively, and obtain estimated value aggregation results within multiple first aggregation periods and true value aggregation results within multiple first aggregation periods; use a second aggregation period with the second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data respectively, and obtain estimated value aggregation results within multiple second aggregation periods and true value aggregation results within multiple second aggregation periods; the first time window is smaller than the second time window.

[0009] In the above scheme, the aggregation processing module is also used to divide the estimated value distribution data and the real value distribution data respectively on the time dimension according to the first aggregation period, and obtain estimated value aggregation results within multiple first aggregation periods and real value aggregation results within multiple first aggregation periods; wherein, each estimated value aggregation result within the first aggregation period includes multiple estimated values ​​within the first aggregation period, and each real value aggregation result within the first aggregation period includes multiple real values ​​within the first aggregation period.

[0010] In the above scheme, the aggregation processing module is also used to divide the estimated value distribution data and the real value distribution data in the time dimension according to the second aggregation period, respectively, to obtain estimated value aggregation results within multiple second aggregation periods and real value aggregation results within multiple second aggregation periods; wherein, each estimated value aggregation result within the second aggregation period includes multiple estimated values ​​within the second aggregation period, and each real value aggregation result within the second aggregation period includes multiple real values ​​within the second aggregation period.

[0011] In the above scheme, the first deviation determination module is also used to obtain multiple estimated values ​​in the estimated value aggregation results within any first aggregation period and multiple real values ​​in the real value aggregation results within the first aggregation period, for the estimated value aggregation results and the real value aggregation results within the first aggregation period; wherein each estimated value corresponds to a real value; determine the ratio of the sum of each of the estimated values ​​to the sum of the corresponding real values, and determine the difference between the ratio and 1 as the ratio difference between the estimated value and the real value; average the ratio difference between the estimated value and the real value to obtain the first estimated deviation value of the information recommendation model within the first aggregation period.

[0012] In the above scheme, the second deviation determination module is also used to obtain multiple estimated values ​​in the estimated value aggregation result within any second aggregation period and multiple real values ​​in the real value aggregation result within the second aggregation period for the estimated value aggregation result and the real value aggregation result within the second aggregation period; wherein each estimated value corresponds to a real value; determine the ratio of the sum of each of the estimated values ​​to the sum of the corresponding real values, and determine the difference between the ratio and 1 as the ratio difference between the estimated value and the real value; average the ratio difference between the estimated value and the real value to obtain the second estimated deviation value of the information recommendation model within the second aggregation period.

[0013] In the above scheme, the data acquisition module is also used to predict the estimated value of the target object for the recommended target at each moment under the time dimension through the information recommendation model; perform data distribution statistics on the estimated value at each moment according to the time increasing law to obtain the estimated value distribution data; obtain the pre-marked true value of the target object for the recommended target at each moment; perform data distribution statistics on the true value at each moment according to the time increasing law to obtain the true value distribution data.

[0014] In the above scheme, the quality assessment module is also used to determine the volatility of the information recommendation model in the first aggregation period and the second aggregation period based on the first estimated deviation value and the second estimated deviation value; the volatility is used to characterize the algorithm stability of the information recommendation model in the first aggregation period and the second aggregation period; the volatility is used to perform model quality assessment on the information recommendation model.

[0015] In the above scheme, the quality assessment module is also used to determine multiple first estimated deviation values ​​corresponding to the second estimated deviation value within the second aggregation period; determine the difference between each first estimated deviation value in the multiple first estimated deviation values ​​and the second estimated deviation value; calculate the square value of each of the differences, and perform square root operation on the square value to obtain a square root operation result corresponding to each of the first estimated deviation values; and average all the square root operation results to obtain the volatility of the information recommendation model in the first aggregation period and the second aggregation period.

[0016] In the above scheme, the model quality assessment device also includes an alarm module, which is used to generate an alarm message when the volatility of the information recommendation model in the first aggregation period and the second aggregation period is greater than a volatility threshold; the alarm message.

[0017] In the above scheme, the model quality assessment device also includes a model upgrade module, which is used to respond to a model upgrade request for the information recommendation model, obtain an upgrade module corresponding to the information recommendation model; determine the upgrade volatility of the upgraded model under the first aggregation period and the second aggregation period; if the volatility of the information recommendation model is less than the upgrade volatility of the upgraded model, prohibit upgrading the information recommendation model.

[0018] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the model quality assessment method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the model quality assessment method provided in the embodiment of the present application when executed by a processor.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the model quality assessment method provided in the embodiment of the present application is implemented.

[0021] The embodiments of the present application have the following beneficial effects: the embodiments of the present application use a first aggregation cycle with a first time window to aggregate the estimated value distribution data and the true value distribution data, obtain the estimated value aggregation results and the true value aggregation results within multiple first aggregation cycles, and then determine the first estimated deviation value within each first aggregation cycle, and use a second aggregation cycle with a second time window to aggregate the estimated value distribution data and the true value distribution data, obtain the estimated value aggregation results and the true value aggregation results within multiple second aggregation cycles, and then determine the second estimated deviation value within each second aggregation cycle. Since the time windows of the first aggregation cycle and the second aggregation cycle are different in length, the volatility of the deviation value can be determined by the first estimated deviation value and the second estimated deviation value within different time windows, and then the volatility of the information recommendation model within a certain period of time can be determined, and the evaluation of the model quality can be effectively realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural diagram of the model quality assessment system architecture provided in the embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 3A It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0025] Figure 3B It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0026] Figure 3C It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0027] Figure 3D It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0028] Figure 3E It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0029] Figure 3F It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0030] Figure 3G It is a flowchart of the model quality assessment method provided in the embodiment of the present application;

[0031] Figure 4 is a schematic diagram of the deviation value distribution provided in the embodiment of the present application;

[0032] Figure 5This is a schematic diagram of an advertisement recommendation display interface provided by an embodiment of the present application;

[0033] Figure 6 It is a schematic diagram of the estimated deviation values ​​of the two algorithms provided in the embodiments of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0035] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) Bias is one of the main indicators for evaluating the algorithm fitting ability in the advertising field. It indicates the deviation between the estimated distribution and the actual distribution. For example, in the click-through rate prediction algorithm, the PCTR bias is calculated by dividing the sum of the estimated values ​​of each sample in the data set, Y', by the sum of the samples with click behavior in the data set, Y, and then subtracting 1, that is, Y' / Y-1.

[0040] The following problems exist in the related art: 1) Bias has positive and negative properties. When the bias is counted for a longer period of time, bias = 0 may occur, but the actual estimated deviation is large. For example, when counting the PCTR bias within one day, half of the samples of algorithm A have a PCTR bias = -10%, and the other half have a PCTR bias = +10%. The final result is that the PCTR bias of algorithm A on this day is 0%. Half of the samples of algorithm B have a PCTR bias = -1%, and the other half have a PCTR bias = +2%. The final result is that the PCTR bias of algorithm B on this day is +0.5%. From the perspective of the volatility of the overall algorithm, algorithm B has less volatility and higher stability than algorithm A. However, if the PCTR bias of algorithm A is smaller according to the statistical results within one day, the wrong conclusion that algorithm A is better will be obtained. 2) Bias is displayed according to the segmented samples, that is, after being presented according to a shorter time window, it is a series of irregular deviation values, and there is a lack of effective means to judge which algorithm has a more stable overall fitting ability.

[0041] Based on at least one of the above problems existing in the related art, the embodiments of the present application provide a model quality assessment method, apparatus, device, computer-readable storage medium and computer program product, which can effectively evaluate the model quality. The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The device provided in the embodiments of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), a smart phone, a smart speaker, a smart watch, a smart TV, a vehicle-mounted terminal and other types of user terminals, and can also be implemented as a server. Below, an exemplary application when the device is implemented as a terminal or a server will be described.

[0042] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the model quality assessment system 100 provided in an embodiment of the present application. To support an exemplary application, the terminal 400 is connected to the server 200 via the network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0043] During the model quality assessment process, the user inputs an assessment operation for performing model quality assessment on the information recommendation model into the terminal 400 . The terminal 400 generates a request for model quality assessment in response to the assessment operation and sends the request for model quality assessment to the server 200 via the network 300 . After receiving the request for model quality assessment, the server 200 responds to the request for model quality assessment and obtains the estimated value distribution data and the true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be assessed for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object marked in advance for the recommended target in the time dimension; the estimated value distribution data and the true value distribution data are respectively aggregated using a first aggregation period with a first time window and a second aggregation period with a second time window, and correspondingly obtain the estimated value aggregation results within a plurality of first aggregation periods, the true value aggregation results within a plurality of first aggregation periods, the estimated value aggregation results within a plurality of second aggregation periods, and the true value aggregation results within a plurality of second aggregation periods; based on the estimated value aggregation results within the first aggregation period and the true value aggregation results within the first aggregation period, determine the first estimated deviation value of the information recommendation model within each first aggregation period; based on the estimated value aggregation results within the second aggregation period and the true value aggregation results within the second aggregation period, determine the second estimated deviation value of the information recommendation model within each second aggregation period; and perform model quality assessment on the information recommendation model based on the first estimated deviation value and the second estimated deviation value.

[0044] In some embodiments, the model quality assessment method of the embodiment of the present application can also be executed by the terminal, that is, the user can input an assessment operation for performing model quality assessment on the information recommendation model through the terminal 400, and the terminal 400 performs an assessment operation for model quality assessment on the information recommendation model to obtain estimated value distribution data and true value distribution data; respectively use a first aggregation period with a first time window and a second aggregation period with a second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, and correspondingly obtain estimated value aggregation results within multiple first aggregation periods, true value aggregation results within multiple first aggregation periods, estimated value aggregation results within multiple second aggregation periods, and true value aggregation results within multiple second aggregation periods; determine a first estimated deviation value of the information recommendation model in each first aggregation period based on the estimated value aggregation results within the first aggregation period and the true value aggregation results within the first aggregation period; determine a second estimated deviation value of the information recommendation model in each second aggregation period based on the estimated value aggregation results within the second aggregation period and the true value aggregation results within the second aggregation period; and perform model quality assessment on the information recommendation model based on the first estimated deviation value and the second estimated deviation value.

[0045] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0046] See also Figure 2 , 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device is used to implement the model quality assessment method, for example, the electronic device can be implemented as a terminal. Among them, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0047] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0048] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0049] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0050] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0051] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0052] An operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic businesses and processing hardware-based tasks; a network communication module 452, for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; a presentation module 453, for enabling information to be presented (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., display screens, speakers, etc.) associated with a user interface 430; an input processing module 454, for detecting one or more user inputs or interactions from one of the one or more input devices 432 and translating the detected inputs or interactions.

[0053] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The model quality assessment device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 4551, an aggregation processing module 4552, a first deviation determination module 4553, a second deviation determination module 4554 and a quality assessment module 4555. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0054] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model quality assessment method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Progr ammable Gate Array, FPGA) or other electronic components.

[0055] The model quality assessment method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the model quality assessment method in each embodiment of the present application can be executed by a server, or by a terminal, or can be executed through interaction between a server and a terminal.

[0056] See also Figure 3A , Figure 3A is a flow chart of the model quality assessment method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained, Figure 3A The body of the step is the server.

[0057] In step 101, estimated value distribution data and true value distribution data are obtained. The estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object marked in advance for the recommended target in the time dimension.

[0058] In some embodiments, the recommended target is a resource recommended and displayed on a page, screen, or other media, such as an advertisement. The target object is an object that can interact with the recommended target, such as multiple users on a media platform. The interactive behavior between the target object and the recommended target may include, but is not limited to, clicks, conversions (registering an account, downloading an application, following a public account, submitting a form), etc. The information recommendation model is a pre-trained neural network model, a deep learning model, etc. The information recommendation model is used to estimate multiple targets to be recommended, obtain an estimated value of each target to be recommended, and determine the recommended target recommended and displayed on the media from multiple targets to be recommended based on the estimated value of each target to be recommended. The number of recommended targets can be one or more. The estimated value of the target object for the recommended target can be the probability that the target object interacts with the recommended target. The true value of the target object for the recommended target represents whether the target object has interacted with the recommended target. If the target object interacts with the recommended target at a certain historical time scale, the true value of the target object for the recommended target at the historical time scale is 1. Otherwise, if there is no interaction between the target object and the recommended target at a certain historical time scale, the true value of the target object for the recommended target at that historical time scale is 0.

[0059] Obtain the estimated value of the target object output by the information recommendation model for the recommended target at each historical time scale within the preset historical time period, and the real value of the target object pre-marked for the recommended target at each historical time scale. Perform distribution statistics based on each estimated value and the corresponding historical time scale to obtain the estimated value distribution data within the preset historical time period. Perform distribution statistics based on each real value and the corresponding historical time scale to obtain the real value distribution data within the preset historical time period.

[0060] It should be noted that the embodiments of the present application do not limit the preset historical time period, nor do they limit the selection rules of each historical time scale. The intervals between the historical time scales can be equal or unequal. For example, within the preset historical time period, a historical time scale can be taken every 1 minute, or the historical time scale can be taken at increasing time intervals of 1 minute, 2 minutes, 3 minutes, etc.

[0061] In some embodiments, see Figure 3B , Figure 3A The illustrated step 101 may be implemented by following steps 1011 to 1014, which are described in detail below.

[0062] In step 1011, in the time dimension, the information recommendation model is used to predict the estimated value of the target object for the recommendation target at each moment.

[0063] In some embodiments, the time dimension is a preset historical time period. In the preset historical time period, the information of the target object and the recommended target is used as the input of the information recommendation model, and the probability of the target object interacting with the recommended target at each historical time scale is predicted by the information recommendation model as the estimated value of the target object for the recommended target at each moment.

[0064] In step 1012, data distribution statistics are performed on the estimated value at each moment according to the time increasing rule to obtain estimated value distribution data.

[0065] In some embodiments, after obtaining the estimated value at each moment, each historical time scale can be arranged on the horizontal axis in an increasing order from small to large, and the estimated value corresponding to each historical time scale can be counted in the same coordinate system to complete the data distribution statistics and obtain the estimated value distribution data.

[0066] In step 1013, the real value of the pre-labeled target object for the recommended target at each moment is obtained.

[0067] In some embodiments, the real value of the target object for the recommended target at each moment will be automatically recorded in the database. If the target object has an interactive behavior with the recommended target, the real value of the target object for the recommended target at that moment is pre-marked as 1. Otherwise, if there is no interactive behavior between the target object and the recommended target, the real value of the target object for the recommended target at that moment is pre-marked as 0. The real value of the target object for the recommended target at each pre-marked historical time scale within the preset historical time period can be obtained from the database.

[0068] In step 1014, data distribution statistics are performed on the real value at each moment according to the time increasing law to obtain real value distribution data.

[0069] In some embodiments, after obtaining the true value at each moment, each historical time scale can be arranged on the horizontal axis in an increasing order from small to large, and the true value corresponding to each historical time scale can be counted in the same coordinate system to complete the data distribution statistics and obtain the true value distribution data.

[0070] The embodiment of the present application obtains the real value and the estimated value at each moment, and performs data distribution statistics according to the time increasing law to obtain the estimated value distribution data and the real value distribution data, thereby improving the standardization and acquisition speed of the data.

[0071] In step 102, the estimated value distribution data and the true value distribution data are aggregated using a first aggregation period with a first time window and a second aggregation period with a second time window, respectively, to obtain corresponding estimated value aggregation results within multiple first aggregation periods, true value aggregation results within multiple first aggregation periods, estimated value aggregation results within multiple second aggregation periods, and true value aggregation results within multiple second aggregation periods.

[0072] Continue to see Figure 3A In some embodiments, the first time window and the second time window are preset time lengths. The first time window and the second time window are not equal. The first polymerization period takes the first time window as a period, and the second polymerization period takes the second time window as a period. Exemplarily, the first polymerization period with the first time window is 10 minutes, and the second polymerization period with the second time window is 1 day.

[0073] In some embodiments, the estimated value distribution data and the real value distribution data are aggregated based on the first aggregation period, and correspondingly, the estimated value aggregation results within multiple first aggregation periods and the real value aggregation results within multiple first aggregation periods are obtained. The estimated value distribution data and the real value distribution data are aggregated based on the second aggregation period, and correspondingly, the estimated value aggregation results within multiple second aggregation periods and the real value aggregation results within multiple second aggregation periods are obtained.

[0074] In some embodiments, Figure 3A The illustrated step 102 may be implemented by following the steps 1021 to 1022, which are described in detail below.

[0075] In step 1021, the estimated value distribution data and the true value distribution data are aggregated respectively using a first aggregation period with a first time window to obtain estimated value aggregation results within multiple first aggregation periods and true value aggregation results within multiple first aggregation periods.

[0076] In some embodiments, based on the estimated value distribution data, the estimated value aggregation result can be calculated once in each first aggregation period to obtain the estimated value aggregation results in multiple first aggregation periods. Based on the real value distribution data, the real value aggregation result can be calculated once in each first aggregation period to obtain the real value aggregation results in multiple first aggregation periods.

[0077] In some embodiments, step 1021 can be implemented by the following steps, which are described in detail below.

[0078] According to the first aggregation period, the estimated value distribution data and the true value distribution data are divided in the time dimension respectively, and the estimated value aggregation results within multiple first aggregation periods and the true value aggregation results within multiple first aggregation periods are obtained.

[0079] The estimated value aggregation result within each first aggregation period includes multiple estimated values ​​within the first aggregation period, and the real value aggregation result within each first aggregation period includes multiple real values ​​within the first aggregation period.

[0080] In some embodiments, the preset historical time period can be divided into multiple first aggregation periods, and each first aggregation period includes multiple historical time scales. According to the first aggregation period, the estimated value distribution data and the real value distribution data are divided in the preset historical time period, and each historical time scale in each first aggregation period has an estimated value and a real value. The estimated values ​​corresponding to each historical time scale in each first aggregation period are aggregated to obtain the estimated value aggregation result of each first aggregation period, and the real values ​​corresponding to each historical time scale in each first aggregation period are aggregated to obtain the real value aggregation result of each first aggregation period.

[0081] In step 1022, a second aggregation period with a second time window is used to aggregate the estimated value distribution data and the true value distribution data, respectively, to obtain estimated value aggregation results within multiple second aggregation periods and true value aggregation results within multiple second aggregation periods.

[0082] The first time window is smaller than the second time window.

[0083] In some embodiments, based on the estimated value distribution data, the estimated value aggregation result can be calculated once in each second aggregation period to obtain the estimated value aggregation results in multiple second aggregation periods. Based on the real value distribution data, the real value aggregation result can be calculated once in each second aggregation period to obtain the real value aggregation results in multiple second aggregation periods.

[0084] In some embodiments, step 1022 may be implemented by the following steps, which are described in detail below.

[0085] According to the second aggregation period, the estimated value distribution data and the true value distribution data are divided in the time dimension respectively, and the estimated value aggregation results within multiple second aggregation periods and the true value aggregation results within multiple second aggregation periods are obtained.

[0086] The estimated value aggregation result within each second aggregation period includes multiple estimated values ​​within the second aggregation period, and the real value aggregation result within each second aggregation period includes multiple real values ​​within the second aggregation period.

[0087] In some embodiments, the preset historical time period can be divided into multiple second aggregation periods, and each second aggregation period includes multiple historical time scales. According to the second aggregation period, the estimated value distribution data and the real value distribution data are divided in the preset historical time period, and each historical time scale in each second aggregation period obtained corresponds to an estimated value and a real value. The estimated values ​​corresponding to each historical time scale in each second aggregation period are aggregated to obtain the estimated value aggregation result of each second aggregation period, and the real values ​​corresponding to each historical time scale in each second aggregation period are aggregated to obtain the real value aggregation result of each second aggregation period.

[0088] The embodiment of the present application adopts a first aggregation period of a first time window and a second aggregation period of a second time window to aggregate the estimated value distribution data and the true value distribution data respectively, and obtains corresponding aggregation results, which facilitates the subsequent calculation of volatility that characterizes the stability of the model.

[0089] In step 103, based on the estimated value aggregation result in the first aggregation period and the real value aggregation result in the first aggregation period, a first estimated deviation value of the information recommendation model in each first aggregation period is determined.

[0090] In some embodiments, for each first aggregation period, a first estimated deviation value within the first aggregation period may be calculated based on a plurality of real values ​​in the real value aggregation result within the first aggregation period and a plurality of estimated values ​​in the corresponding estimated value aggregation result.

[0091] In some embodiments, see Figure 3C , Figure 3A The illustrated step 103 may be implemented by following steps 1031 to 1033, which are described in detail below.

[0092] In step 1031, for the estimated value aggregation result and the true value aggregation result within any first aggregation period, multiple estimated values ​​in the estimated value aggregation result within the first aggregation period and multiple true values ​​in the true value aggregation result within the first aggregation period are obtained; wherein each estimated value corresponds to a true value.

[0093] In some embodiments, for each first aggregation period, estimated values ​​corresponding to multiple historical time scales can be obtained from the estimated value aggregation results within the first aggregation period, and real values ​​corresponding to multiple historical time scales can be obtained from the real value aggregation results within the first aggregation period. The estimated value of each historical time scale corresponds to the real value of the same historical time scale.

[0094] In step 1032, the ratio of the sum of the estimated values ​​to the corresponding sum of the true values ​​is determined, and the difference between the ratio and 1 is determined as the ratio difference between the estimated value and the true value.

[0095] In some embodiments, for each first aggregation cycle, multiple estimated values ​​in the estimated value aggregation result within the first aggregation cycle are added together to obtain the sum of the estimated values. Multiple real values ​​in the real value aggregation result within the first aggregation cycle are added together to obtain the sum of the real values. The ratio of the sum of the estimated values ​​to the sum of the real values ​​is subtracted from 1 to obtain the ratio difference between the estimated value and the real value of the first aggregation cycle.

[0096] In step 1033, the ratio difference between the estimated value and the true value is averaged to obtain a first estimated deviation value of the information recommendation model in the first aggregation period.

[0097] In some embodiments, the first estimated deviation value is used to characterize the difference between the estimated value and the true value. For each first aggregation period, based on the number of estimated values ​​or true values ​​in the first aggregation period, the ratio difference between the estimated value and the true value is averaged to obtain the first estimated deviation value of the information recommendation model in the first aggregation period. The first estimated deviation value of the information recommendation model in the first aggregation period can satisfy the following formula (I).

[0098] average(bias,T1)=avg(y1' / y1-1) Formula (I)

[0099] Among them, average(bias,T1) is the first estimated deviation value of the information recommendation model in the first aggregation period, T1 is the first aggregation period, y1' is the sum of multiple estimated values ​​in the first aggregation period, y1 is the sum of multiple true values ​​in the first aggregation period, avg(y1' / y1-1) is the ratio difference between the estimated value and the true value in the first aggregation period, avg(y1' / y1-1)=(y1' / y1-1) / n, n is the number of estimated values ​​in the first aggregation period.

[0100] like Figure 4 As shown, Figure 4 It is a distribution diagram of the first estimated deviation value in the first aggregation period, or a distribution diagram of the second estimated deviation value in the second aggregation period.

[0101] The embodiment of the present application determines the estimated deviation value through the bias calculation method, which can be used to characterize the difference between the estimated value output by the information recommendation model and the true value, thereby facilitating the subsequent calculation of volatility.

[0102] In step 104, based on the estimated value aggregation result in the second aggregation period and the real value aggregation result in the second aggregation period, a second estimated deviation value of the information recommendation model in each second aggregation period is determined.

[0103] In some embodiments, for each second aggregation period, a second estimated deviation value within the second aggregation period may be calculated based on a plurality of real values ​​in the real value aggregation result within the second aggregation period and a plurality of estimated values ​​in the corresponding estimated value aggregation result.

[0104] In some embodiments, see Figure 3D , Figure 3A The illustrated step 104 may be performed through the following steps 1041 to 1043, which are described in detail below.

[0105] In step 1041, for the estimated value aggregation result and the true value aggregation result within any second aggregation period, multiple estimated values ​​in the estimated value aggregation result within the second aggregation period and multiple true values ​​in the true value aggregation result within the second aggregation period are obtained; wherein each estimated value corresponds to a true value.

[0106] In some embodiments, for each second aggregation period, estimated values ​​corresponding to multiple historical time scales can be obtained from the estimated value aggregation results within the second aggregation period, and real values ​​corresponding to multiple historical time scales can be obtained from the real value aggregation results within the second aggregation period. The estimated value of each historical time scale corresponds to the real value of the same historical time scale.

[0107] In step 1042, the ratio of the sum of the estimated values ​​to the corresponding sum of the true values ​​is determined, and the difference between the ratio and 1 is determined as the ratio difference between the estimated value and the true value.

[0108] In some embodiments, for each second aggregation cycle, multiple estimated values ​​in the estimated value aggregation result in the second aggregation cycle are added together to obtain the sum of the estimated values. Multiple real values ​​in the real value aggregation result in the second aggregation cycle are added together to obtain the sum of the real values. The ratio of the sum of the estimated values ​​to the sum of the real values ​​is subtracted from 1 to obtain the ratio difference between the estimated value and the real value of the second aggregation cycle.

[0109] In step 1043, the ratio difference between the estimated value and the true value is averaged to obtain a second estimated deviation value of the information recommendation model in the second aggregation period.

[0110] In some embodiments, the second estimated deviation value is used to characterize the difference between the estimated value and the true value. For each second aggregation period, based on the number of estimated values ​​or true values ​​in the second aggregation period, the ratio difference between the estimated value and the true value is averaged to obtain the second estimated deviation value of the information recommendation model in the second aggregation period. The second estimated deviation value of the information recommendation model in the second aggregation period can satisfy the following formula (II).

[0111] average(bias,T2)=avg(y2' / y2-1) formula (2)

[0112] Among them, average(bias,T2) is the second estimated deviation value of the information recommendation model in the second aggregation period, T2 is the second aggregation period, y2' is the sum of multiple estimated values ​​in the second aggregation period, y2 is the sum of multiple true values ​​in the second aggregation period, avg(y2' / y2-1) is the ratio difference between the estimated value and the true value in the second aggregation period, avg(y2' / y2-1)=(y2' / y2-1) / m, m is the number of estimated values ​​in the second aggregation period.

[0113] The embodiment of the present application determines the estimated deviation value through the bias calculation method, which can be used to characterize the difference between the estimated value output by the information recommendation model and the true value, thereby facilitating the subsequent calculation of volatility.

[0114] In step 105, based on the first estimated deviation value and the second estimated deviation value, a model quality assessment is performed on the information recommendation model.

[0115] In some embodiments, the volatility may be calculated based on the first estimated deviation value of each first aggregation period and the second estimated deviation value of each second aggregation period, and the volatility may be used to evaluate the model quality of the information recommendation model.

[0116] The embodiment of the present application uses a first aggregation cycle with a first time window to aggregate the estimated value distribution data and the true value distribution data, obtains the estimated value aggregation results and the true value aggregation results within multiple first aggregation cycles, and then determines the first estimated deviation value within each first aggregation cycle, and uses a second aggregation cycle with a second time window to aggregate the estimated value distribution data and the true value distribution data, obtains the estimated value aggregation results and the true value aggregation results within multiple second aggregation cycles, and then determines the second estimated deviation value within each second aggregation cycle. Since the time windows of the first aggregation cycle and the second aggregation cycle are different in length, the volatility of the deviation value can be determined by the first estimated deviation value and the second estimated deviation value within different time windows, and then the volatility of the information recommendation model within a certain period of time can be determined, and the evaluation of the model quality can be effectively realized.

[0117] In some embodiments, see Figure 3E , Figure 3A The illustrated step 105 may be performed through the following steps 1051 to 1052, which are described in detail below.

[0118] In step 1051, based on the first estimated deviation value and the second estimated deviation value, the volatility of the information recommendation model in the first aggregation period and the second aggregation period is determined.

[0119] The volatility is used to characterize the algorithm stability of the information recommendation model in the first aggregation period and the second aggregation period.

[0120] In some embodiments, if the first time window of the first aggregation period is smaller than the second time window of the second aggregation period, the second aggregation period may include multiple first aggregation periods. For a second aggregation period, the volatility of the information recommendation model in the first aggregation period and the second aggregation period may be determined based on the second estimated deviation value of the second aggregation period and the first estimated deviation value of each first aggregation period in the second aggregation period. The volatility is used to evaluate the model quality of the information recommendation model. A smaller volatility (or closer to 0) indicates that the algorithm stability of the information recommendation model in the first aggregation period and the second aggregation period is higher.

[0121] In some embodiments, see Figure 3F , Figure 3E The illustrated step 1051 can be performed through the following steps 10511 to 10514, which are described in detail below.

[0122] In step 10511, a plurality of first estimated deviation values ​​corresponding to the second estimated deviation values ​​within the second aggregation period are determined.

[0123] In some embodiments, if the first time window of the first aggregation cycle is smaller than the second time window of the second aggregation cycle, the second aggregation cycle may include multiple first aggregation cycles. For one second aggregation cycle, the first estimated deviation value in each first aggregation cycle in the second aggregation cycle may be obtained as multiple first estimated deviation values ​​corresponding to the second estimated deviation value in the second aggregation cycle.

[0124] In step 10512, the difference between each first estimated deviation value and the second estimated deviation value in the plurality of first estimated deviation values ​​is determined.

[0125] In some embodiments, for any second aggregation period, each first estimated deviation value corresponding to the second estimated deviation value in the second aggregation period is subtracted from the second estimated deviation value of the second aggregation period to obtain corresponding multiple difference values.

[0126] In step 10513, the square value of each difference is calculated, and a square root operation is performed on the square value to obtain a square root operation result corresponding to each first estimated deviation value.

[0127] In some embodiments, a square operation is performed on each difference to obtain the square value of each difference, and a square root is taken on the square value of each difference to obtain a square root operation result corresponding to each first estimated deviation value.

[0128] In step 10514, the average of all square root operation results is calculated to obtain the volatility of the information recommendation model in the first aggregation period and the second aggregation period.

[0129] In some embodiments, all the square root operation results are added and averaged based on the number of the first aggregation period in the second aggregation period to obtain the volatility of the information recommendation model in the first aggregation period and the second aggregation period. The volatility satisfies the following formula (III).

[0130]

[0131] Among them, the volatility (bias, T1, T2) is the volatility of the information recommendation model in the first aggregation period and the second aggregation period, and N is the number of the first aggregation period in the second aggregation period.

[0132] The embodiment of the present application ensures that the volatility is a positive number by taking the square of the difference as mentioned above, and takes the average after adding the square root to obtain the volatility used to characterize the stability of the model, which can effectively realize the evaluation of the model quality.

[0133] Optionally, in some embodiments, for any second aggregation period, the ratio of each first estimated deviation value corresponding to the second estimated deviation value in the second aggregation period to the second estimated deviation value of the second aggregation period can replace the difference in formula (III). The subsequent calculation steps are the same and will not be repeated. The ratio can also characterize the stability of the model. It should be noted that after the difference in formula (III) is changed to a ratio, the closer the volatility is to 1, the more stable the model is.

[0134] In step 10513, the square value of each difference is calculated and the square root is taken.

[0135] In some embodiments, see Figure 3G ,exist Figure 3E After step 1051 shown, steps 1051A to 1051B may also be performed, which are described in detail below.

[0136] In step 1051A, when determining the volatility under the third aggregation period, multiple second aggregation periods corresponding to the third aggregation period are determined, and the volatility under each second aggregation period in the multiple second aggregation periods is obtained; the third time window corresponding to the third aggregation period is larger than the second time window.

[0137] In some embodiments, if the third time window corresponding to the third aggregation period is larger than the second time window, the third aggregation period may include multiple second aggregation periods, and the volatility of the information recommendation model corresponding to each second aggregation period in the third aggregation period under the first aggregation period and the second aggregation period is obtained.

[0138] In step 1051B, the volatility in multiple second aggregation periods is averaged to obtain the volatility in the third aggregation period.

[0139] In some embodiments, the number of second aggregation cycles within the third aggregation cycle is determined, and based on the number of second aggregation cycles within the third aggregation cycle, the sum of the volatility of the information recommendation model corresponding to each second aggregation cycle in the third aggregation cycle in the first aggregation cycle and the second aggregation cycle is averaged to obtain the volatility in the third aggregation cycle.

[0140] In step 1052, the volatility is used to evaluate the model quality of the information recommendation model.

[0141] In some embodiments, the model quality of the information recommendation model is evaluated based on the volatility in the third aggregation period. The greater the volatility in the third aggregation period, the more unstable the estimation of the information recommendation model is, and the worse the model quality is.

[0142] The embodiment of the present application can observe the estimated volatility of a longer period by taking the volatility mean of the second aggregation period, and better evaluate the model quality.

[0143] In some embodiments, Figure 3E After step 1052 shown, the following steps may be performed, which are described in detail below: When the volatility of the information recommendation model in the first aggregation period and the second aggregation period is greater than the volatility threshold, generate alarm information. Output alarm information.

[0144] In some embodiments, a volatility threshold can be set in advance. When the volatility of the information recommendation model in the first aggregation period and the second aggregation period is greater than the volatility threshold, an alarm message is automatically generated and output. The alarm message indicates that the volatility of the information recommendation model has suddenly changed at this time, and the information recommendation model may be abnormal. It should be noted that the embodiment of the present application does not specifically limit the volatility threshold, and the volatility threshold can be set according to the stability requirements or volatility tolerance of the information recommendation model application scenario. In the embodiment of the present application, when the volatility exceeds the threshold, the system can issue an alarm in time to quickly access and troubleshoot the algorithm abnormality.

[0145] In some embodiments, Figure 3E After step 1052 shown, the following steps may be performed, which are described in detail below: In response to a model upgrade request for the information recommendation model, an upgrade model corresponding to the information recommendation model is obtained. An upgrade volatility of the upgraded model in the first aggregation period and the second aggregation period is determined. If the volatility of the information recommendation model is less than the upgrade volatility of the upgraded model, upgrading the information recommendation model is prohibited.

[0146] In some embodiments, when performing iterative upgrade of the information recommendation model, in response to a model upgrade request for the information recommendation model, an upgraded model corresponding to the information recommendation model is obtained. The upgraded model is the model after the iterative upgrade of the information recommendation model. The upgrade volatility of the upgraded model in the first aggregation period and the second aggregation period is determined by the above volatility calculation method. If the volatility of the information recommendation model is less than the upgrade volatility of the upgraded model, the quality of the upgraded model after the upgrade deteriorates, and the information recommendation model is prohibited from being upgraded.

[0147] The embodiment of the present application uses volatility as an evaluation indicator for iterative upgrading, thereby ensuring the long-term stability of the information recommendation model fitting.

[0148] Optionally, in the model quality assessment provided in the embodiments of the present application, in addition to using the estimated deviation value to participate in the calculation of volatility, the accuracy evaluation indicators of the corresponding algorithm itself, such as Recall@N (recall rate of topN), can also be used. For example, for the LearningToRank (LTR) algorithm, Recall@N is used. At this time, the calculation of bias in formulas (1) to (3) can be replaced by the calculation formula of Recall@N.

[0149] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0150] The embodiment of the present application is implemented in the scenario of evaluating the volatility of the fitting ability of the advertising recommendation algorithm. For example, the information recommendation model is the advertising recommendation algorithm, the recommendation target is to place advertisements, and the target object is the user. Figure 5 As shown, Figure 5It is the interface displayed by the application from which the advertisement is placed. Common application APPs will have advertisements placed. The user behavior chain on advertisements in these APPs can generally be summarized as: display (exposure) -> click -> conversion (register account, download APP, follow public account, submit form, etc.). There is basically an estimation algorithm at the arrow, which predicts the probability of generating the behavior on the right side of the arrow given the left side of the arrow, such as display -> click, which is the click-through rate estimation algorithm. By improving the accuracy of these estimation algorithms, it is possible to recommend advertising content that users are interested in and that the resource party is willing to pay for. In general recommendation scenarios, these estimation algorithms only need to sort the candidate recommended content according to the degree of user interest, and finally comprehensively select the content that users are most interested in for exposure. In the advertising recommendation scenario, since it also involves the issue of deduction by the advertising publisher, it is generally deducted according to ecp m (Effective Cost Per Mille). ecpm = bid * pCTR * pCVR (Conversion Rate, a measure of advertising effectiveness, in short, the conversion rate from users clicking on an ad to becoming an effective activated or registered or even paying user). Therefore, the estimation algorithm also needs to ensure that the distribution of the estimated value is as consistent as possible with the actual distribution, that is, the estimated deviation value is small. However, the advertising recommendation algorithm is affected by factors such as features, samples, traffic, delayed sample return, model training, and updates, which will cause the estimated value to fluctuate. At present, there is no unified and widely agreed model volatility indicator to evaluate the quality of advertising recommendation models.

[0151] The model quality assessment method proposed in the embodiment of the present application proposes an indicator for evaluating the volatility of the fitting ability of the advertising algorithm, so that the indicator of the estimated deviation between different algorithms is comparable. The specific method steps are as follows: Step 1: Set a shorter window aggregation period (first aggregation period), for example 10 minutes, and calculate the first estimated deviation value average(bias,10min)=avg(y1' / y1-1) once each period, where y1' is the sum of the estimated values ​​in the short window, and y1 is the sum of the true values ​​in the short window. Step 2: Set a longer window aggregation period (second aggregation period), for example 1 day, and calculate the second estimated deviation value average(bias,1d)=avg(y2' / y2-1) in the aggregation period, where y2' is the sum of the estimated values ​​in the long window, and y2 is the sum of the true values ​​in the long window. Step 3: Select the first estimated deviation value in the first aggregation period of step 1 (assuming there are N) contained in the second aggregation period of step 2, subtract it from the second estimated deviation value of step 2, take the square, then add the square root, and finally divide by N to calculate the mean. For example, the bias volatility calculation under the 10-minute short window - 1-day long window is as shown in formula (IV).

[0152]

[0153] Step 4: When you need to observe the estimated volatility for a longer period, you can calculate the average of the long-term volatility in step 3. The larger the volatility value, the more unstable the estimated value output by the advertising recommendation algorithm.

[0154] Based on the above steps, the embodiment of the present application can calculate the estimated volatility of each advertising recommendation algorithm in a certain period of time, and can be used in the following two aspects based on this indicator:

[0155] 1) You can set an alarm threshold based on the stability requirements or volatility tolerance of the algorithm application scenario. When the volatility exceeds the threshold, the system will issue an alarm in time to quickly investigate the algorithm abnormality.

[0156] 2) When iterating and upgrading the advertising recommendation algorithm, volatility is used as one of the indicators. In other words, in addition to the common evaluation indicators in ABTest (separate group experiment), the upgraded algorithm adds volatility indicator comparison to ensure the long-term stability of the algorithm fitting. Figure 6 is the bias curve of algorithm A and algorithm B. Figure 6 Intuitively, the overall volatility of algorithm A is more severe than that of algorithm B. The volatility and difference calculated by the two algorithms according to different aggregation time windows are shown in Table 1.

[0157] Table 1 Volatility of Algorithm A and Algorithm B

[0158] Aggregation time window (1min,1d) (5min,1d) (10min,1d) (30m,1d) (1h,1d) Algorithm A 3.033 2.991 2.925 2.752 2.652 Algorithm B 2.474 1.783 1.581 1.359 1.245 Volatility (1-B / A) -18％ -40％ -45％ -50％ -53％

[0159] It can be seen that the volatility of algorithm A in each aggregation time window is higher than that of algorithm B. The volatility of the upgraded algorithm B is reduced and the model quality is higher. Algorithm A can be upgraded to obtain algorithm B.

[0160] The embodiment of the present application proposes volatility as an indicator, which is a supplement to the evaluation of the quality of the advertising recommendation algorithm. In the algorithm optimization work, strengthening the quality monitoring of the estimated volatility and using it as a guide for the optimization direction is conducive to the stable iteration of the advertising system. It avoids the technical problem that the algorithm operates benignly 99% of the time, and fluctuates violently 1% of the time, resulting in the entire advertising recommendation effect falling short. The embodiment of the present application combines the deviation of the estimated value of the advertising recommendation algorithm in the long and short time windows to evaluate the stability (volatility) of the fitting ability of the advertising recommendation algorithm, so as to compare the fitting stability between different algorithms.

[0161] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0162] The following further describes the exemplary structure of the model quality assessment device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the model quality evaluation device 455 of the memory 440 may include: a data acquisition module 4551, which is used to obtain estimated value distribution data and true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object marked in advance for the recommended target in the time dimension. An aggregation processing module 4552 is used to respectively use a first aggregation cycle with a first time window and a second aggregation cycle with a second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, and correspondingly obtain the estimated value aggregation results within multiple first aggregation cycles, the true value aggregation results within multiple first aggregation cycles, the estimated value aggregation results within multiple second aggregation cycles, and the true value aggregation results within multiple second aggregation cycles. A first deviation determination module 4553 is used to determine the first estimated deviation value of the information recommendation model in each first aggregation cycle based on the estimated value aggregation results within the first aggregation cycle and the true value aggregation results within the first aggregation cycle. The second deviation determination module 4554 is used to determine the second estimated deviation value of the information recommendation model in each second aggregation period based on the estimated value aggregation result in the second aggregation period and the real value aggregation result in the second aggregation period. And the quality evaluation module 4555 is used to perform model quality evaluation on the information recommendation model based on the first estimated deviation value and the second estimated deviation value.

[0163] In some embodiments, the aggregation processing module is also used to use a first aggregation period with a first time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, respectively, to obtain the estimated value aggregation results within multiple first aggregation periods and the true value aggregation results within multiple first aggregation periods; use a second aggregation period with a second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, respectively, to obtain the estimated value aggregation results within multiple second aggregation periods and the true value aggregation results within multiple second aggregation periods; the first time window is smaller than the second time window.

[0164] In some embodiments, the aggregation processing module 4552 is also used to divide the estimated value distribution data and the true value distribution data in the time dimension according to the first aggregation period, respectively, to obtain estimated value aggregation results within multiple first aggregation periods and true value aggregation results within multiple first aggregation periods; wherein the estimated value aggregation result within each first aggregation period includes multiple estimated values ​​within the first aggregation period, and the true value aggregation result within each first aggregation period includes multiple true values ​​within the first aggregation period.

[0165] In some embodiments, the aggregation processing module 4552 is also used to divide the estimated value distribution data and the true value distribution data in the time dimension according to the second aggregation period, respectively, to obtain estimated value aggregation results within multiple second aggregation periods and true value aggregation results within multiple second aggregation periods; wherein the estimated value aggregation result within each second aggregation period includes multiple estimated values ​​within the second aggregation period, and the true value aggregation result within each second aggregation period includes multiple true values ​​within the second aggregation period.

[0166] In some embodiments, the first deviation determination module 4553 is also used to obtain, for the estimated value aggregation results and the true value aggregation results within any first aggregation period, multiple estimated values ​​in the estimated value aggregation results within the first aggregation period and multiple true values ​​in the true value aggregation results within the first aggregation period; wherein each estimated value corresponds to a true value; determine the ratio of the sum of each estimated value to the sum of the corresponding true values, and determine the difference between the ratio and 1 as the ratio difference between the estimated value and the true value; average the ratio difference between the estimated value and the true value to obtain the first estimated deviation value of the information recommendation model within the first aggregation period.

[0167] In some embodiments, the second deviation determination module 4554 is also used to obtain, for the estimated value aggregation results and the true value aggregation results within any second aggregation period, multiple estimated values ​​in the estimated value aggregation results within the second aggregation period and multiple true values ​​in the true value aggregation results within the second aggregation period; wherein each estimated value corresponds to a true value; determine the ratio of the sum of each estimated value to the sum of the corresponding true values, and determine the difference between the ratio and 1 as the ratio difference between the estimated value and the true value; average the ratio difference between the estimated value and the true value to obtain the second estimated deviation value of the information recommendation model within the second aggregation period.

[0168] In some embodiments, the data collection module 4551 is also used to predict the estimated value of the target object for the recommended target at each moment in the time dimension through the information recommendation model; perform data distribution statistics on the estimated value at each moment according to the time increasing law to obtain estimated value distribution data; obtain the true value of the pre-marked target object for the recommended target at each moment; perform data distribution statistics on the true value at each moment according to the time increasing law to obtain true value distribution data.

[0169] In some embodiments, the quality assessment module 4555 is also used to determine the volatility of the information recommendation model in the first aggregation period and the second aggregation period based on the first estimated deviation value and the second estimated deviation value; the volatility is used to characterize the algorithm stability of the information recommendation model in the first aggregation period and the second aggregation period; the volatility is used to perform model quality assessment on the information recommendation model.

[0170] In some embodiments, the quality assessment module 4555 is also used to determine multiple first estimated deviation values ​​corresponding to the second estimated deviation value within the second aggregation period; determine the difference between each first estimated deviation value and the second estimated deviation value in the multiple first estimated deviation values; calculate the square value of each difference, and perform a square root operation on the square value to obtain a square root operation result corresponding to each first estimated deviation value; and average all the square root operation results to obtain the volatility of the information recommendation model in the first aggregation period and the second aggregation period.

[0171] In some embodiments, the model quality assessment device further includes an alarm module for generating an alarm message when the volatility of the information recommendation model in the first aggregation period and the second aggregation period is greater than a volatility threshold; and outputting the alarm message.

[0172] In some embodiments, the model quality assessment device also includes a model upgrade module, which is used to respond to a model upgrade request for the information recommendation model, obtain an upgrade module corresponding to the information recommendation model; determine the upgrade volatility of the upgraded model under the first aggregation period and the second aggregation period; if the volatility of the information recommendation model is less than the upgrade volatility of the upgraded model, prohibit the upgrade of the information recommendation model.

[0173] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the model quality assessment method described in the embodiment of the present application.

[0174] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the model quality assessment method provided by the present application embodiment, for example, Figure 3A The model quality assessment method shown.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0176] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0177] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0178] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0179] In summary, the volatility is proposed as an indicator through the embodiment of the present application, which is a supplement to the evaluation of the quality of the advertising recommendation algorithm. In the algorithm optimization work, strengthening the quality monitoring of the estimated volatility and using it as a guide for the optimization direction is conducive to the stable iteration of the advertising system. It is not a technical problem that the algorithm operates benignly 99% of the time, and fluctuates violently 1% of the time, resulting in the entire advertising recommendation effect falling short. The embodiment of the present application combines the deviation of the estimated value of the advertising recommendation algorithm in the long and short time windows to evaluate the stability (volatility) of the fitting ability of the advertising recommendation algorithm, so as to compare the fitting stability between different algorithms.

[0180] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model quality assessment method, characterized in that: The method comprises: Obtain estimated value distribution data and true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object for the recommended target in the time dimension that is pre-labeled; A first aggregation cycle with a first time window and a second aggregation cycle with a second time window are respectively used to aggregate the estimated value distribution data and the true value distribution data, and correspondingly obtain estimated value aggregation results within multiple first aggregation cycles, true value aggregation results within multiple first aggregation cycles, estimated value aggregation results within multiple second aggregation cycles, and true value aggregation results within multiple second aggregation cycles; Determining a first estimated deviation value of the information recommendation model in each of the first aggregation cycles based on an estimated value aggregation result in the first aggregation cycle and an actual value aggregation result in the first aggregation cycle; Determining a second estimated deviation value of the information recommendation model in each of the second aggregation periods based on an estimated value aggregation result in the second aggregation period and an actual value aggregation result in the second aggregation period; Based on the first estimated deviation value and the second estimated deviation value, a model quality assessment is performed on the information recommendation model.

2. The method according to claim 1, characterized in that The first aggregation cycle with a first time window and the second aggregation cycle with a second time window are respectively used to aggregate the estimated value distribution data and the true value distribution data, and correspondingly obtain estimated value aggregation results within multiple first aggregation cycles, true value aggregation results within multiple first aggregation cycles, estimated value aggregation results within multiple second aggregation cycles, and true value aggregation results within multiple second aggregation cycles, including: Using a first aggregation cycle with the first time window, the estimated value distribution data and the true value distribution data are respectively aggregated to obtain estimated value aggregation results within multiple first aggregation cycles and true value aggregation results within multiple first aggregation cycles; Using a second aggregation period with the second time window, respectively performing aggregation processing on the estimated value distribution data and the true value distribution data to obtain estimated value aggregation results within multiple second aggregation periods and true value aggregation results within multiple second aggregation periods; The first time window is smaller than the second time window.

3. The method according to claim 2, characterized in that The method of using a first aggregation cycle having the first time window to respectively aggregate the estimated value distribution data and the true value distribution data to obtain estimated value aggregation results within multiple first aggregation cycles and true value aggregation results within multiple first aggregation cycles includes: According to the first aggregation period, the estimated value distribution data and the true value distribution data are respectively divided in the time dimension to obtain estimated value aggregation results within multiple first aggregation periods and true value aggregation results within multiple first aggregation periods; Among them, each estimated value aggregation result within the first aggregation period includes multiple estimated values ​​within the first aggregation period, and each real value aggregation result within the first aggregation period includes multiple real values ​​within the first aggregation period.

4. The method according to claim 2, characterized in that: The method of using a second aggregation cycle with the second time window to respectively aggregate the estimated value distribution data and the true value distribution data to obtain estimated value aggregation results within multiple second aggregation cycles and true value aggregation results within multiple second aggregation cycles includes: According to the second aggregation period, the estimated value distribution data and the true value distribution data are respectively divided in the time dimension to obtain estimated value aggregation results within multiple second aggregation periods and true value aggregation results within multiple second aggregation periods; Among them, each estimated value aggregation result within the second aggregation period includes multiple estimated values ​​within the second aggregation period, and each real value aggregation result within the second aggregation period includes multiple real values ​​within the second aggregation period.

5. The method according to claim 1, characterized in that The determining, based on the estimated value aggregation result in the first aggregation period and the real value aggregation result in the first aggregation period, a first estimated deviation value of the information recommendation model in each of the first aggregation periods includes: For an estimated value aggregation result and a real value aggregation result in any first aggregation period, obtain multiple estimated values ​​in the estimated value aggregation result in the first aggregation period and multiple real values ​​in the real value aggregation result in the first aggregation period; wherein each estimated value corresponds to one real value; Determine a ratio of a sum of the estimated values ​​to a sum of the corresponding true values, and determine a difference between the ratio and 1 as a ratio difference between the estimated value and the true value; The ratio difference between the estimated value and the true value is averaged to obtain a first estimated deviation value of the information recommendation model in the first aggregation period.

6. The method according to claim 1, characterized in that The determining, based on the estimated value aggregation result in the second aggregation period and the real value aggregation result in the second aggregation period, a second estimated deviation value of the information recommendation model in each of the second aggregation periods includes: For any estimated value aggregation result and true value aggregation result in a second aggregation period, obtain multiple estimated values ​​in the estimated value aggregation result in the second aggregation period and multiple true values ​​in the true value aggregation result in the second aggregation period; wherein each estimated value corresponds to one true value; Determine a ratio of a sum of the estimated values ​​to a sum of the corresponding true values, and determine a difference between the ratio and 1 as a ratio difference between the estimated value and the true value; The ratio difference between the estimated value and the true value is averaged to obtain a second estimated deviation value of the information recommendation model in the second aggregation period.

7. The method according to claim 1, characterized in that The obtaining of estimated value distribution data and true value distribution data includes: In the time dimension, predicting the estimated value of the target object for the recommendation target at each moment through the information recommendation model; Performing data distribution statistics on the estimated value at each moment according to the time increasing law to obtain the estimated value distribution data; Obtaining the pre-marked true value of the target object for the recommended target at each moment; The real value at each moment is subjected to data distribution statistics according to the time increasing law to obtain the real value distribution data.

8. The method according to any one of claims 1 to 7, characterized in that: The performing model quality assessment on the information recommendation model based on the first estimated deviation value and the second estimated deviation value includes: Based on the first estimated deviation value and the second estimated deviation value, determining the volatility of the information recommendation model in the first aggregation period and the second aggregation period; the volatility is used to characterize the algorithm stability of the information recommendation model in the first aggregation period and the second aggregation period; The volatility is used to evaluate the model quality of the information recommendation model.

9. The method according to claim 8, characterized in that The determining, based on the first estimated deviation value and the second estimated deviation value, the volatility of the information recommendation model in the first aggregation period and the second aggregation period includes: determining a plurality of first estimated deviation values ​​corresponding to second estimated deviation values ​​within the second aggregation period; determining a difference between each first estimated deviation value of the plurality of first estimated deviation values ​​and the second estimated deviation value; Calculating a square value of each of the difference values, and performing a square root operation on the square value to obtain a square root operation result corresponding to each of the first estimated deviation values; The average of all square root operation results is calculated to obtain the volatility of the information recommendation model in the first aggregation period and the second aggregation period.

10. The method according to claim 8, characterized in that The method further comprises: When determining the volatility under the third aggregation period, a plurality of second aggregation periods corresponding to the third aggregation period are determined, and the volatility under each second aggregation period of the plurality of second aggregation periods is obtained; the third time window corresponding to the third aggregation period is greater than the second time window; The volatility under the plurality of second aggregation cycles is averaged to obtain the volatility under the third aggregation cycle.

11. The method according to claim 8, characterized in that The method further comprises: When the volatility of the information recommendation model in the first aggregation period and the second aggregation period is greater than a volatility threshold, generating an alarm message; Output the warning information.

12. The method according to claim 8, characterized in that The method further comprises: In response to a model upgrade request for the information recommendation model, acquiring an upgraded model corresponding to the information recommendation model; determining an upgrade volatility of the upgrade model in the first aggregation period and the second aggregation period; If the volatility of the information recommendation model is smaller than the upgrade volatility of the upgrade model, upgrading the information recommendation model is prohibited.

13. A model quality assessment device, characterized in that: The device comprises: A data collection module is used to obtain estimated value distribution data and true value distribution data; the estimated value distribution data is the distribution data of the estimated value of the target object output by the information recommendation model to be evaluated for the recommended target in the time dimension, and the true value distribution data is the distribution data of the true value of the target object for the recommended target in the time dimension that is pre-labeled; An aggregation processing module, used to respectively use a first aggregation period with a first time window and a second aggregation period with a second time window to perform aggregation processing on the estimated value distribution data and the true value distribution data, and correspondingly obtain estimated value aggregation results within multiple first aggregation periods, true value aggregation results within multiple first aggregation periods, estimated value aggregation results within multiple second aggregation periods, and true value aggregation results within multiple second aggregation periods; A first deviation determination module, configured to determine a first estimated deviation value of the information recommendation model in each of the first aggregation periods based on an estimated value aggregation result in the first aggregation period and a real value aggregation result in the first aggregation period; A second deviation determination module, configured to determine a second estimated deviation value of the information recommendation model in each of the second aggregation periods based on an estimated value aggregation result in the second aggregation period and a real value aggregation result in the second aggregation period; A quality assessment module is used to perform a model quality assessment on the information recommendation model based on the first estimated deviation value and the second estimated deviation value.

14. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, used to implement the model quality assessment method according to any one of claims 1 to 12 when executing the computer executable instructions or computer program stored in the memory.

15. A computer-readable storage medium storing a computer-executable instruction or a computer program for a method for evaluating the quality of a computer model, characterized in that: When the computer executable instructions or computer program are executed by a processor, the model quality assessment method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model quality assessment method according to any one of claims 1 to 12 is implemented.

Images