Advertising Configuration Verification Method and Its Device, Equipment, Medium

Abstract

The present application discloses an advertisement configuration verification method, its device, equipment, and medium. The method includes: allocating advertisement samples in an advertisement library to an experimental pool and a control pool according to a preset new advertisement random allocation rule, so as to conduct a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration; obtaining the true impact indicators of the new and old configuration comparison experiment, so as to generate corresponding prior conditions according to the true impact indicators; generating multiple pairs of experimental pools and control pools based on the advertisement samples in the advertisement library according to a preset historical advertisement random allocation rule to conduct old configuration comparison experiments respectively, and retaining the multiple pairs of experimental pools and control pools whose experimental impact indicators meet the prior conditions, so as to statistically generate impact indicator distribution data; verifying whether the new advertisement configuration meets the configuration validity rule based on the impact indicator distribution data. If it meets, the new advertisement configuration is a valid new configuration. The present application can enhance the comparability of experimental samples in a comparison experiment and improve the authenticity of experimental results.

Description

Technical Field

[0001] The present application relates to the field of online advertising, and in particular to an advertisement configuration verification method. In addition, it also relates to a corresponding device, equipment, and non-volatile storage medium for this method. Background Art

[0002] An online advertising platform can push advertisements sent by advertisers to the media side for users to view, so that users on the media side can understand the corresponding products or services through the advertisements and then use and consume them; the online advertising platform usually formulates corresponding advertisement configurations for the advertisements that advertisers need to place to improve the exposure of the advertisements it places. And when formulating new advertisement configurations, corresponding control experiments are usually carried out, such as AB tests, etc., to evaluate the placement effect of the new advertisement configuration, and corresponding AA tests are carried out to verify the effect gap between the new advertisement configuration and the old advertisement configuration, and then verify the credibility of the AB test. However, if the number of advertisements owned by the online advertising platform is limited, resulting in non-uniform samples in the experimental pool and control pool in the AB test and AA test carried out by it, the experimental results are not comparable and have poor reference value, so the placement effect of the new advertisement configuration cannot be effectively determined.

[0003] In view of the problems existing in the existing control experiments for new advertisement configurations, the applicant has made corresponding explorations in consideration of solving this problem. Summary of the Invention

[0004] The purpose of the present application is to provide an advertisement configuration verification method to meet user needs. In addition, it also relates to a corresponding device, equipment, non-volatile storage medium, and computer program product for this method.

[0005] To achieve the purpose of the present application, the following technical solutions are adopted:

[0006] An advertisement configuration verification method proposed to adapt to the purpose of the present application includes the following steps:

[0007] According to a preset new advertisement random distribution rule, distribute the advertisement samples in the advertisement library to the experimental pool and the control pool, so as to conduct a new and old configuration control experiment on the experimental pool and the control pool based on the new advertisement configuration;

[0008] Obtain the true impact index of the new and old configuration control experiment, so as to generate a corresponding prior condition according to the true impact index;

[0009] According to a preset historical advertisement random distribution rule, generate multiple pairs of experimental pools and control pools based on the historical advertisement samples in the advertisement library to conduct old configuration control experiments respectively, and retain the multiple pairs of experimental pools and control pools whose experimental impact indexes meet the prior conditions, so as to statistically generate corresponding impact index distribution data;

[0010] Verify whether the new advertisement configuration meets the configuration validity rule based on the impact index distribution data. If it meets, the new advertisement configuration is a valid new configuration.

[0011] In a further embodiment, the step of allocating advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule to conduct a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration includes the following steps:

[0012] Randomly allocate the existing advertisement samples in the advertisement library to the experimental pool and the control pool, where the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range;

[0013] The new advertisement configuration and the old advertisement configuration are correspondingly configured in the experimental pool and the control pool to start the new and old configuration comparison experiment;

[0014] Continuously store the advertisement samples pushed by one or more advertiser terminals in the advertisement library, and hierarchically allocate them to the experimental pool and the control pool according to the advertiser terminals to which these advertisement samples belong.

[0015] In a further embodiment, the step of obtaining the true impact index of the new and old configuration comparison experiment to generate a corresponding prior condition based on the true impact index includes the following steps:

[0016] When it is monitored that the current time exceeds the experimental time acting on the new and old configuration comparison experiment, count the true impact index of the new and old configuration comparison experiment, and the true impact index is determined according to the index parameters of the experimental pool and the control pool respectively;

[0017] Generate an impact index value interval containing this value according to the value represented by the true impact index, and use this impact index value interval as the judgment threshold interval of the prior condition.

[0018] In a further embodiment, according to a preset historical advertisement random allocation rule, generate multiple pairs of experimental pools and control pools based on the historical advertisement samples in the advertisement library to conduct old configuration comparison experiments respectively, and retain the multiple pairs of experimental pools and control pools whose experimental impact indexes meet the prior conditions, including the following steps:

[0019] Obtain the historical advertisement samples with earlier warehousing times in the advertisement library, and randomly allocate these historical advertisement samples to the experimental pool and the control pool to configure the same old advertisement configuration for the experimental pool and the control pool to conduct the first old configuration comparison experiment;

[0020] Statistically analyze the first experimental impact index of the first old configuration control experiment, and verify whether the first experimental impact index is within the judgment threshold range of the prior condition. If it is, allocate the advertisement samples in the advertisement library to the experimental pool and the control pool according to the advertisers' ends to continue the second old configuration control experiment. If not, stop the first old configuration control experiment;

[0021] When the second old configuration control experiment ends, statistically analyze the final second experimental impact index of the second old configuration control experiment, and store the second experimental impact index and the first experimental impact index in the experimental impact index list in a corresponding manner.

[0022] In a preferred embodiment, in the step of retaining multiple pairs of experimental pools and control pools whose experimental impact indexes meet the prior conditions and statistically generating corresponding impact index distribution data, the following steps are included:

[0023] Obtain multiple second experimental impact indexes stored in the experimental impact index list and their first experimental impact indexes;

[0024] Statistically analyze the impact index increments of each second experimental impact index and its first experimental impact index, and statistically generate corresponding impact index distribution data based on these impact index increments. The impact index distribution data will show the proportion of the data representing each impact index increment.

[0025] In a preferred embodiment, in the step of retaining multiple pairs of experimental pools and control pools whose experimental impact indexes meet the prior conditions and statistically generating corresponding impact index distribution data, the following steps are included:

[0026] Obtain multiple second experimental impact indexes stored in the experimental impact index list and their first experimental impact indexes;

[0027] Calculate the impact index increments of each second experimental impact index and its first experimental impact index, and then calculate the ratio of these impact index increments to their first experimental impact indexes. Take this ratio as the impact index increment rate corresponding to the first experimental impact index;

[0028] Statistically analyze each impact index increment rate to generate corresponding impact index distribution data.

[0029] In a further embodiment, in the step of verifying whether the new advertisement configuration meets the configuration validity rule based on the impact index distribution data, the following any one or any combination of steps are included:

[0030] Determine the impact indicator increment with the highest data proportion in the impact indicator distribution data, use this impact indicator increment as the expected increment, calculate the difference between the expected increment and the actual impact indicator. If the difference meets the increment threshold of the configured validity rule, then the new advertisement configuration is a valid new configuration;

[0031] Determine the impact indicator increment with the largest value in the impact indicator distribution data, verify whether the actual impact indicator exceeds this impact indicator increment. If it exceeds, then the new advertisement configuration is a valid new configuration.

[0032] An advertisement configuration verification device proposed to meet the objectives of the present application, which includes:

[0033] A new configuration experiment module, configured to allocate advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, so as to conduct a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration;

[0034] A prior condition generation module, configured to obtain the actual impact indicator of the new and old configuration comparison experiment, so as to generate corresponding prior conditions according to the actual impact indicator;

[0035] A distribution data generation module, configured to generate multiple pairs of experimental pools and control pools based on the historical advertisement samples in the advertisement library according to a preset historical advertisement random allocation rule to conduct old configuration comparison experiments respectively, retain the multiple pairs of experimental pools and control pools whose experimental impact indicators meet the prior conditions, so as to statistically generate corresponding impact indicator distribution data;

[0036] A configuration validity verification module, configured to verify whether the new advertisement configuration meets the configuration validity rule based on the impact indicator distribution data. If it meets, then the new advertisement configuration is a valid new configuration.

[0037] In a further embodiment, the new configuration experiment module includes:

[0038] A sample random allocation sub-module, configured to randomly allocate the existing advertisement samples in the advertisement library to the experimental pool and the control pool, wherein the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range;

[0039] A new configuration experiment start sub-module, configured to correspondingly configure the new advertisement configuration and the old advertisement configuration to the experimental pool and the control pool to start the new and old configuration comparison experiment;

[0040] A sample stratified allocation sub-module, configured to continuously store the advertisement samples pushed by one or more advertiser terminals into the advertisement library, and stratify and allocate the advertisement samples to the experimental pool and the control pool according to the advertiser terminals to which the advertisement samples belong.

[0041] In a further embodiment, the prior condition generation module includes:

[0042] An impact index statistics sub-module, configured to, when it is monitored that the current time exceeds the experiment time of the new and old configuration comparison experiment, statistically calculate the true impact index of the new and old configuration comparison experiment, where the true impact index is determined according to the index parameters of the experiment pool and the control pool respectively;

[0043] A judgment threshold determination sub-module, configured to generate an impact index value range including the value according to the value represented by the true impact index, and use the impact index value range as the judgment threshold range of the prior condition.

[0044] In a further embodiment, the distribution data generation module includes:

[0045] A historical sample allocation sub-module, configured to obtain historical advertisement samples with earlier warehousing times in the advertisement library, and randomly allocate these historical advertisement samples to the experiment pool and the control pool, so as to configure the same old advertisement configuration for the first old configuration comparison experiment for the experiment pool and the control pool;

[0046] An experimental index verification sub-module, configured to statistically calculate the first experimental impact index of the first old configuration comparison experiment, verify whether the first experimental impact index is within the judgment threshold range of the prior condition, and if it is, allocate the advertisement samples in the advertisement library to the experiment pool and the control pool according to the advertisement owner end to which they belong respectively to continue the second old configuration comparison experiment, and if not, stop the first old configuration comparison experiment;

[0047] An experimental index storage sub-module, configured to, when the second old configuration comparison experiment ends, statistically calculate the final second experimental impact index of the second old configuration comparison experiment, and store the second experimental impact index and the first experimental impact index corresponding to each other in the experimental impact index list.

[0048] In a preferred embodiment, the distribution data generation module further includes:

[0049] An experimental index acquisition sub-module, configured to acquire multiple second experimental impact indexes stored in the experimental impact index list and their first experimental impact indexes;

[0050] An index increment statistics sub-module, configured to statistically calculate the impact index increments of each second experimental impact index and its first experimental impact index, and statistically calculate these impact index increments to generate corresponding impact index distribution data, where the data ratio representing each impact index increment is in the impact index distribution data.

[0051] In a preferred embodiment, the distribution data generation module further includes:

[0052] An index parameter sub-module for obtaining multiple second experimental impact indices and their first experimental impact indices stored in the experimental impact index list;

[0053] An index value-added rate sub-module for calculating the impact index value-added of each of the second experimental impact indices and their first experimental impact indices, and then calculating the ratio of these impact index value-addeds to their first experimental impact indices, and taking this ratio as the impact index value-added rate corresponding to the first experimental impact index;

[0054] An index distribution data generation sub-module for statistically analyzing each of the impact index value-added rates to generate corresponding impact index distribution data.

[0055] In a further embodiment, the configuration validity verification module includes:

[0056] An expected value-added verification sub-module for determining the impact index value-added with the highest data proportion in the impact index distribution data, taking this impact index value-added as the expected value-added, calculating the difference between the expected value-added and the true impact index, and if this difference meets the value-added threshold of the configuration validity rule, then the new advertisement configuration is a valid new configuration;

[0057] A maximum value-added verification sub-module for determining the impact index value-added with the largest value in the impact index distribution data, and verifying whether the true impact index exceeds this impact index value-added, and if it exceeds, then the new advertisement configuration is a valid new configuration.

[0058] To solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory and a processor, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned advertisement configuration verification method.

[0059] To solve the above technical problems, an embodiment of the present application further provides a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, one or more processors execute the steps of the above-mentioned advertisement configuration verification method.

[0060] To solve the above technical problems, an embodiment of the present application further provides a computer program product, including a computer program and computer instructions, and when the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned advertisement configuration verification method.

[0061] Compared with the prior art, the advantages of the present application are as follows:

[0062] This application can provide a control experiment for condition verification for a network advertising placement platform to solve the problem that the credibility of the experimental results of the network advertising placement platform is low due to a small experimental sample size. By constructing corresponding prior conditions with reference to the true indicators output by the online new and old configuration control experiment for the new advertisement configuration, an old configuration control experiment with prior conditions for experimental indicator verification is constructed. Furthermore, the experimental indicators output by the old configuration control experiment that meet the prior statistics are counted to generate indicator distribution data that conforms to the prior statistics, so as to construct the indicator distribution data to verify whether the new advertisement configuration is effective. By constructing prior conditions to verify the sample quality of the experimental group and the control group in the control experiment, and then determining an experimental group and a control group with homogeneous samples for the control experiment, a control experiment with a small sample size provides a more credible experimental result to evaluate the placement effect of the new advertisement configuration.

[0063] Secondly, this application screens the old configuration control experiment by constructing prior conditions. The old configuration control experiment is the AA experiment, so as to ensure the data validity of the experimental indicators generated by the AA experiment carried out by the network advertising placement platform for the new advertisement configuration, and construct relatively highly referential distribution data to verify the placement effect of the new advertisement configuration in the platform, and improve the evaluation effect of the new advertisement configuration.

[0064] In addition, this application can evaluate the experimental effect of the AA experiment with the help of the AB experiment. By constructing corresponding conditions with reference to the experimental results output by the AA experiment to evaluate the experimental effect of the AA experiment, compared with the traditional experimental method of evaluating the AB experiment through the AA experiment, in this application, the AB experiment and the AA experiment can evaluate the experimental effects of each other, forming a data closed-loop between the AB experiment and the AA experiment to improve the credibility of the experimental results of both sides. Brief Description of the Drawings

[0065] The above and / or additional aspects and advantages of this application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0066] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;

[0067] Figure 2 A flowchart of a typical embodiment of the advertisement configuration verification method of this application;

[0068] Figure 3 A data distribution diagram formed by visualizing the influence on the indicator increment of this application;

[0069] Figure 4 A data distribution diagram formed by visualizing the influence on the indicator increment rate of this application;

[0070] Figure 5It is a flow diagram formed by the specific implementation manner of implementing the new and old advertisement configuration comparison experiment in this application;

[0071] Figure 6 It is a flow diagram formed by the specific implementation manner of generating prior conditions in this application;

[0072] Figure 7 It is a flow diagram formed by the specific implementation manner of verifying the old configuration comparison experiment according to the prior conditions in this application;

[0073] Figure 8 It is a flow diagram formed by the specific implementation manner of statistically calculating the increment of influence indicators to generate influence indicator distribution data in this application;

[0074] Figure 9 It is a flow diagram formed by the specific implementation manner of statistically calculating the increment rate of influence indicators to generate influence indicator distribution data in this application;

[0075] Figure 10 It is a flow diagram formed by the specific implementation manner of verifying whether the new advertisement configuration meets the configuration validity rule based on the influence indicator distribution data in this application;

[0076] Figure 11 It is a principle block diagram of a typical embodiment of the advertisement configuration verification device of this application;

[0077] Figure 12 It is a basic structure block diagram of a computer device according to an embodiment of this application. Specific implementation manner

[0078] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and should not be construed as limiting the present application.

[0079] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application means the presence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more of the associated listed items.

[0080] Those skilled in the art can understand that, unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as the general understanding of those of ordinary skill in the art to which the present application pertains. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0081] Those skilled in the art can understand that the "client", "terminal", and "terminal device" used herein include both devices with a wireless signal receiver that only has the ability to receive and no ability to transmit, and devices with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such devices can include: cellular or other communication devices such as personal computers, tablets, etc., which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which can include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; traditional laptop and / or palm-top computers or other devices, which are traditional laptop and / or palm-top computers or other devices with and / or including a radio frequency receiver. The "client", "terminal", and "terminal device" used herein can be portable, transportable, installed in a vehicle (air, sea, and / or land), or suitable for and / or configured to run locally and / or in a distributed manner at any other location on the earth and / or in space. The "client", "terminal", and "terminal device" used herein can also be a communication terminal, an Internet access terminal, a music / video playback terminal, such as a PDA, MID (Mobile Internet Device), and / or a mobile phone with music / video playback function, or can also be devices such as a smart TV, a set-top box, etc.

[0082] The hardware referred to by names such as "server", "client", and "working node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer, and is a hardware device with the necessary components disclosed by the von Neumann principle, including a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit loads the program stored in the external memory into the memory for execution, executes the instructions in the program, and interacts with the input / output devices to complete specific functions.

[0083] It should be noted that the concept of "server" as referred to in this application can similarly be extended to apply to server clusters. According to the network deployment principles understood by those skilled in the art, the various servers should be logically divided. Physically, these servers can either be independent of each other but can be invoked through interfaces, or integrated into a single physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility and should not be restricted by this in the implementation of the network deployment method of this application.

[0084] Please refer to Figure 1 , the hardware foundation required for the implementation of the related technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers providing relevant support, etc., thereby constituting a logically related service cluster to provide services for relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can both access the Internet through well-known network access methods and establish a data communication link with the server 80 in the cloud to run the terminal application programs related to the services provided by the server.

[0085] For the server, the application programs are usually built as service processes, opening corresponding program interfaces for remote invocation by the application programs running on various terminal devices. The related technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0086] The application programs refer to the application programs running on the server or terminal devices. These application programs implement the related technical solutions of this application through programming. Their program codes can be saved in a non-volatile storage medium recognizable by a computer in the form of computer-executable instructions and be loaded into the memory by the central processing unit for running. The related devices of this application are constructed through the running of this application on the computer.

[0087] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus show commonality with each other, unless otherwise specified, these methods can be executed independently. Similarly, for the various embodiments disclosed in this application, they are all proposed based on the same inventive concept. Therefore, for the same-concept expressions, as well as the concepts that are appropriately transformed only for convenience although the concept expressions are different, they should be equivalently understood.

[0088] Please refer to Figure 2 , in a typical embodiment of an advertisement configuration verification method of this application, it includes the following steps:

[0089] Step S11: Allocate the advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, so as to conduct a new-old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration.

[0090] After the online advertising platform sets a new advertisement configuration for advertising delivery, it is necessary to conduct a new-old configuration comparison experiment to determine the advertising delivery effect of using the new advertisement configuration. When conducting the new-old configuration comparison experiment, the advertisement samples in the advertisement library need to be allocated to the experimental pool and the control pool respectively according to the new advertisement random allocation rule, so as to conduct the new-old configuration comparison experiment.

[0091] The advertisement samples in the advertisement library are generally the advertisements that the advertiser needs to deliver or the advertisement samples of the advertisements that have been delivered by the online advertising platform to the corresponding media side. The advertisement samples include advertisement information, advertisement delivery cost information, advertisement conversion rate (CTR, Click Through Rate) information, advertisement turnover (GMV, Gross Merchandise Volume) information, etc. Among them, the advertisement conversion rate information and the advertisement turnover information need to be statistically analyzed and updated after the advertisement is delivered. Therefore, the advertisement conversion rate information and the advertisement turnover information in the advertisement samples of the advertisements that have not been delivered to the media side are generally empty or not set.

[0092] The advertisement samples for conducting the new-old configuration comparison experiment are generally the advertisement samples of the advertisements that need to be delivered to the media side currently, that is, the new-old configuration comparison experiment is generally a launch experiment. The server allocates the currently delivered advertisement samples in the advertisement library to the experimental pool or the control pool according to the new advertisement random allocation rule to conduct the new-old configuration comparison experiment. Specifically, the server randomly allocates the existing new advertisement samples in the advertisement library to the experimental pool and the control pool. The difference in the number of advertisement samples between the experimental pool and the control pool is within a preset threshold range to prevent the difference in the number of advertisement samples between the two from being too large. After that, continuously store one or more advertisement samples pushed by the advertiser to the advertisement library, and allocate them to the experimental pool and the control pool according to the advertiser to which they belong respectively, to prevent a large number of advertisements with good quality from being allocated to the experimental pool or the control pool, so as to ensure that the advertisement samples in the experimental pool and the control pool are homogeneous and improve the experimental credibility of the new-old configuration comparison experiment.

[0093] In one embodiment, the new and old configuration comparison experiment can be carried out using the historical advertisement samples that have been completed in the advertisement library, so that the network advertisement delivery platform can review the delivery effects of its historical advertisement configurations. For the corresponding advertisement sample allocation method, the server can allocate the advertisement owners into different layers according to the advertisement owners to which the historical advertisement samples belong, and then allocate them to the corresponding experimental pool and control pool for the new and old configuration comparison experiment.

[0094] The new and old configuration comparison experiment is generally an AB experiment to evaluate the delivery effect of the new advertisement configuration. That is, the advertisement samples in the experimental pool are generally configured with the new advertisement configuration newly customized by the network advertisement delivery platform, while the old advertisement configuration is generally used in the control pool. The advertisement configuration refers to the advertisement delivery strategy customized by the network advertisement delivery platform for the advertisements it delivers. For example, the network advertisement delivery platform can customize the advertisement configuration to modify the budget of the advertisement it delivers to form a new advertisement budget strategy, customize a new advertisement configuration to replace the media-side platform for delivering advertisements to form a new media-side strategy, or customize a new advertisement configuration to adjust the number of times of delivering advertisements to form a new delivery times strategy, etc. Of course, the indicators to be output corresponding to the new and old configuration comparison experiments for different types of advertisement configurations will also be different. For example, for the new and old configuration comparison experiment corresponding to the new advertisement configuration for the advertisement budget strategy, the indicators to be output generally include the advertisement delivery cost indicator and the return on advertising spend (ROAS) indicator. Among them, the advertisement delivery cost indicator is generally the main evaluation indicator of the new and old configuration comparison experiment, while the advertisement support return indicator is generally the secondary evaluation indicator of the new and old configuration comparison experiment. For the new and old configuration comparison experiment for the advertisement configuration for the new advertisement budget strategy, the network advertisement delivery platform needs to know whether the return on advertising spend (ROAS) is affected when modifying the advertisement delivery cost (budget) of the advertisement. Therefore, the true impact indicator of the new and old configuration comparison experiment is generally also the gap ratio of the advertisement delivery cost indicators between the experimental pool and the control pool. For specific related embodiments, please refer to the subsequent steps, and this step will not be elaborated here; for the advertisement delivery strategy corresponding to the advertisement configuration, those skilled in the art can flexibly design according to the actual business scenario and set the corresponding indicators output by the new and old configuration comparison experiment accordingly, which will not be elaborated here.

[0095] Step S12, obtain the true impact indicator of the new and old configuration comparison experiment, and generate the corresponding prior condition according to this true impact indicator:

[0096] The server listens in real time whether the new and old configuration comparison experiment has been completed. When it is monitored that the new and old configuration comparison experiment has completed the experiment, the true impact indicator of the new and old configuration comparison experiment will be obtained, and the corresponding prior condition will be generated according to this true impact indicator.

[0097] The described true impact indicator is generally related to the indicators used in the main impact assessment to evaluate the advertising delivery effect of the new and old advertising configurations. For example, when the new advertising configuration in the new and old advertising configuration experiment is a new advertising budget strategy, the true impact indicator affecting the advertising delivery effect is generally the ratio of the advertising delivery cost gap between the experimental pool and the control pool. The server calculates the difference in the advertising delivery costs of the experimental pool and the control pool in the new and old configuration control experiment that has completed the experiment by counting, where the advertising delivery cost of the experimental pool is generally the minuend, and then divides the calculated difference by the advertising delivery cost of the control pool to obtain the ratio of the advertising delivery cost gap between the experimental pool and the control pool as the true impact indicator.

[0098] After the server obtains the true impact indicator of the new and old configuration control experiment that has completed the experiment, it will generate corresponding prior conditions based on this true impact indicator. The server generally refers to the value represented by the true impact indicator to generate corresponding prior conditions. For example, the value represented by the true impact indicator can be used as the median, maximum value, or minimum value to generate a corresponding numerical range as the impact indicator numerical range of the prior conditions for subsequent judgment, or the value represented by the true impact indicator can be used as the impact indicator threshold of the prior conditions for numerical size judgments such as greater than, less than, or equal to. Taking the true impact indicator as the ratio of the advertising delivery cost gap as an example, for example, when the true impact indicator is the ratio of the advertising delivery cost gap and its value is represented as 15%, this value can be used as the median to generate a numerical range of 10% to 20% as the impact indicator numerical range of the prior conditions, or the value can be used as the impact indicator threshold to construct a prior condition where the experimental impact indicator is greater than or equal to 15%; of course, those skilled in the art can design other ways to generate prior conditions based on the true impact indicator, which will not be elaborated here.

[0099] Step S13, according to the preset historical advertisement random allocation rule, generate multiple pairs of experimental pools and control pools based on the historical advertisement samples in the advertisement library to conduct old configuration control experiments respectively, and retain the multiple pairs of experimental pools and control pools whose experimental impact indicators meet the prior conditions to statistically generate corresponding impact indicator distribution data:

[0100] After completing the new and old configuration control experiment, the advertisement samples used in the new and old configuration control experiment will be used to conduct the old configuration control experiment, and the old configuration control experiment will be combined with the prior conditions corresponding to the true impact indicator of the new and old configuration control experiment to generate corresponding impact indicator distribution data.

[0101] The described experimental impact indicators are generally divided into the first experimental impact indicator and the second experimental impact indicator. Correspondingly, the first experimental impact indicator corresponds to the first old configuration control experiment, and the second experimental impact indicator corresponds to the second old configuration experiment. Moreover, the prior condition is generally used to verify the first experimental impact indicator.

[0102] The server allocates the historical advertisement samples in the advertisement library according to the historical advertisement random allocation rule. The historical advertisement samples refer to the historical advertisement samples used in the new and old configuration control experiments to evaluate the experimental credibility of the new and old configuration control experiments. Specifically, first obtain the historical advertisement samples with an earlier warehousing time from the advertisement library and randomly allocate them to the experimental pool and the control pool of the old configuration control experiment. For example, generally select the experimental samples on the first day or the first time of the new and old configuration control experiment. For example, when the experimental date of the new and old configuration control experiment is from January 1st to January 15th, obtain the advertisement samples with the date of January 1st in the advertisement library and randomly allocate them. The reason is that the number of advertisements to be launched on the first day when starting a new advertisement launch business in the business scenario of the gateway advertisement delivery platform is relatively large, and the number of experimental samples is sufficient. Moreover, the warehousing time of the advertisement samples in the advertisement library generally corresponds to their delivery time. Therefore, randomly allocate the advertisement samples with the warehousing time or the first day in the advertisement library to the experimental pool and the control pool of the old configuration control experiment. Of course, they can also be hierarchically allocated to the experimental pool and the control pool according to the advertisers to which these advertisement samples belong. It should be noted that the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range to prevent the difference in the number of advertisement samples between the experimental pool and the control pool from being too large.

[0103] After the server distributes the historical advertisement samples with earlier storage times in the advertisement library to the experimental pool and the control pool in the old configuration control experiment, it will conduct the first old configuration control experiment to count the experimental impact indicators of this first old configuration control experiment for the prior condition verification. Specifically, the experimental samples in the first old configuration control experiment only include the historical advertisement samples with earlier storage times. After the server completes the first old configuration control experiment, it will count the first experimental impact indicator corresponding to this first old configuration control experiment to determine whether this first experimental impact indicator meets the prior condition. For example, when the first experimental impact indicator is the advertisement placement cost gap between the experimental pool and the control pool, correspondingly, the prior condition has a numerical interval for verification. When the value represented by the advertisement placement cost gap is within the numerical interval, then this first experimental impact indicator meets the prior condition, and then the server will continue to stratify and distribute the other historical advertisement samples in the advertisement library to the experimental pool and the control pool according to their corresponding advertisers for the second old configuration control experiment; if the first experimental impact indicator does not meet the prior condition, then it will stop this first old configuration control experiment and will not retain this first experimental impact indicator.

[0104] The server will conduct multiple times of the first old configuration control experiment, that is, multiple times distribute the historical advertisement samples with earlier storage times in the advertisement library to the experimental pool and the control pool to generate multiple pairs of the experimental pool and the control pool to form multiple first old configuration control experiments, and only retain the first old configuration control experiments whose first experimental impact indicators meet the prior condition for the second old configuration control experiment, and then obtain the second experimental impact indicators corresponding to these second old configuration control experiments respectively, so as to generate the corresponding impact indicator distribution data by counting based on multiple groups of first experimental impact indicators and second experimental impact indicators; generally, the server will conduct 50,000 - 100,000 times of the first configuration control experiment for experimental screening. Of course, those skilled in the art can also flexibly design the number of times of this experimental screening, which will not be elaborated here.

[0105] When the first experiment of any of the first old configuration control experiments meets the prior conditions, the server obtains the unused advertisement samples in the advertisement library for this first old configuration control experiment, and randomly stratifies and assigns these advertisement samples to the experimental pool and the control pool of this first old configuration control experiment according to the advertisers to which these advertisement samples belong, so as to conduct the second old configuration control experiment and then count the corresponding second experimental impact indicators at the end of the experiment. Among them, counting the second experimental impact indicators will combine the historical advertisement samples experimented by the first old configuration control experiment with the newly assigned historical advertisement samples of the second old configuration control experiment; the server stores the obtained first experimental impact indicators and second experimental impact indicators as mapping relationship data in the experimental impact indicator list correspondingly, so as to generate the corresponding impact indicator distribution data subsequently.

[0106] Please refer to Figure 3 , after completing the specified number of the first old configuration control experiments, the server obtains all the first experimental impact indicators from the experimental impact indicator list and their correspondingly stored second experimental impact indicators, and then calculates the difference between each first experimental impact indicator and its corresponding second experimental impact indicator as the corresponding impact indicator increment. Among them, the second experimental impact indicator will be used as the minuend, and then counts these impact indicator increments to form the impact indicator distribution data. The impact indicator distribution data will be sorted according to each impact indicator distribution data, and the data proportion of each impact indicator increment in the impact indicator distribution data will be counted. For example, as Figure 3 shown, for the convenience of browsing, Figure 3 in the data distribution diagram shown is the visualization data of the impact indicator distribution data described above. Among them, the abscissa of the shown data distribution diagram represents the impact indicator increment, the left ordinate represents the quantity of the impact indicator increment, and the right ordinate represents the proportion of the impact indicator increment. Correspondingly, the shown columns in the shown data distribution diagram correspond to the quantity of the impact indicator increment, and the shown lines in the shown data distribution diagram correspond to the proportion of the impact indicator increment.

[0107] Please refer to Figure 4 , in addition to having the impact indicator increment, the impact indicator distribution data may also have the corresponding impact indicator increment rate. Generally, the impact indicator increment rate refers to the ratio of the impact indicator increment to its corresponding first experimental impact indicator. Specifically, the server obtains multiple second experimental impact indicators and their first experimental impact indicators stored in the experimental impact indicator list, calculates the impact indicator increment of each second experimental impact indicator and its first experimental impact indicator, and then calculates the ratio of these impact indicator increments to their first experimental impact indicators, and takes this ratio as the impact indicator increment rate corresponding to this first experimental impact indicator, so as to count these impact indicator increment rates to generate the corresponding impact indicator distribution data, such asFigure 4 As shown Figure 4 The data distribution diagram shown in the figure is the visualization data of the impact index distribution data with the impact index value-added rate. Among them, the abscissa of the data distribution diagram shown represents the impact index value-added rate, the left ordinate represents the quantity of the impact index value-added rate, and the right ordinate represents the proportion of the impact index value-added rate. Correspondingly, the columns shown in the data distribution diagram correspond to the quantity of the impact index value-added rate, and the lines shown in the data distribution diagram correspond to the proportion of the impact index value-added rate.

[0108] Step S14: Based on the impact index distribution data, verify whether the new advertisement configuration meets the configuration validity rule. If it meets, the new advertisement configuration is a valid new configuration:

[0109] After the server generates the impact index distribution data, it will verify whether the new advertisement configuration meets the configuration validity rule based on the impact index distribution data to determine whether the new advertisement configuration is a valid new configuration.

[0110] The valid new configuration refers to the advertisement configuration that meets the configuration validity rule. The configuration validity rule generally verifies whether the new advertisement configuration is a valid new configuration based on the impact index value-added in the impact index distribution data. Specifically, determine the impact index value-added with the highest data proportion in the impact index distribution data, and use this impact index value-added as the expected value-added, or calculate the average value corresponding to all impact index value-added in the impact index distribution data as the expected value-added, and then calculate the difference between the expected value-added and the actual impact index. Among them, the actual impact index is the minuend. When the difference meets the preset value-added threshold of the configuration validity rule, the new advertisement configuration will be verified as a valid new configuration. The value-added threshold is generally set within the range of 0-100, and those skilled in the art can also design the value range of the value-added threshold according to the actual application scenario.

[0111] In addition, the server calls the configuration validity rule to determine the impact index value-added with the largest value in the impact index distribution data, and verify whether the actual impact index exceeds this impact index value-added. If it exceeds, the new advertisement configuration is a valid new configuration.

[0112] It can be understood that in addition to verifying the new advertisement configuration based on the impact index increment in the impact index distribution data, the effective configuration rule can also be determined according to the impact index increment rate in the impact index distribution data. Specifically, first, determine the expected increment based on the impact index increment to calculate the difference between the expected increment and the actual impact index, and then calculate the ratio of the difference to the total impact index obtained from the control experiment of the new and old configurations; the total impact index is the total value corresponding to the actual impact index. For example, when the actual impact index is the average difference in advertising costs obtained from the control experiment of the new and old configurations, correspondingly, the total impact index will be the total advertising cost value. After calculating the ratio of the difference to the total impact index obtained from the control experiment of the new and old configurations, use this ratio as the actual impact index increment rate, and determine the impact index increment rate with the largest value in the impact index distribution data to determine whether the actual impact index increment rate is greater than or equal to the impact index increment rate. If it exceeds, the new advertisement configuration will be determined as a valid new configuration.

[0113] In addition to verifying whether the new advertisement configuration is a valid new configuration based on the impact index distribution data, the sample quality of the advertisement samples used in the control experiment of the new and old configurations can also be determined through the effective index distribution data. Specifically, determine the impact index increment with the highest data proportion in the impact index distribution data, calculate the average value corresponding to all impact index increments in the impact index distribution data, or determine the impact index increment rate with the largest data proportion, and by judging the difference between any or all of the obtained values and 0, the smaller the difference, the higher the sample quality of the advertisement samples used in the control experiment of the new and old configurations.

[0114] From the typical embodiments of this method, it can be seen that this method can provide a control experiment for conditional verification for the online advertising platform to solve the problem that the credibility of the experimental results is low due to the small sample size of the experiment in the online advertising platform. By referring to the actual indicators output from the online control experiment of the new and old configurations for the new advertisement configuration to construct corresponding prior conditions, a control experiment of the old configuration with prior conditions for experimental indicator verification is constructed. Then, the experimental indicators output from the control experiment of the old configuration that meet the prior statistics are statistically analyzed to generate an indicator distribution data that conforms to the prior statistics, and this indicator distribution data is used to verify whether the new advertisement configuration is effective. By constructing prior conditions to verify the sample quality of the experimental group and the control group in the control experiment, and then determining an experimental group and a control group with homogeneous samples for the control experiment, a control experiment with a small sample size is provided with a highly credible experimental result to evaluate the placement effect of the new advertisement configuration.

[0115] Secondly, this method conducts experimental screening of the old configuration control experiment by constructing prior conditions. The old configuration control experiment is the AA experiment, so as to ensure the data validity of the experimental indicators generated by the AA experiment carried out by the network advertising delivery platform for the new advertisement configuration, construct distribution data with relatively high reference value for verifying the delivery effect of the new advertisement configuration in the platform, and improve the evaluation effect of the new advertisement configuration.

[0116] In addition, this method can evaluate the experimental effect of the AA experiment by means of the AB experiment. By referring to the experimental results output by the AA experiment, corresponding conditions are constructed to evaluate the experimental effect of the AA experiment. Compared with the traditional experimental method of evaluating the AB experiment through the AA experiment, in this method, the AB experiment and the AA experiment can mutually evaluate the experimental effects of each other, forming a data closed-loop between the AB experiment and the AA experiment to improve the credibility of the experimental results of both sides.

[0117] The above typical embodiments and their variant embodiments fully disclose the implementation scheme of the advertisement configuration verification method of this application. However, various variant embodiments of this method can still be deduced through the transformation and amplification of some technical means. Other embodiments are briefly described as follows:

[0118] In one embodiment, please refer to Figure 5 In the step of allocating the advertisement samples in the advertisement library to the experimental pool and the control pool according to the preset random allocation rule of new advertisements, and conducting the old and new configuration control experiment on the experimental pool and the control pool based on the new advertisement configuration, the following steps are included:

[0119] Step S111, randomly allocate the existing advertisement samples in the advertisement library to the experimental pool and the control pool, wherein the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range:

[0120] The server configures the corresponding experimental pool and control pool for the old and new configuration control experiment. First, the server randomly allocates the currently existing advertisement samples in the advertisement library to the experimental pool and the control pool, and controls the difference in the number of advertisement samples in the experimental pool and the control pool to be within the preset threshold range. The threshold range is generally set within the range of 4 - 10.

[0121] Step S112, configure the new advertisement configuration and the old advertisement configuration for the experimental pool and the control pool respectively to start the old and new configuration control experiment:

[0122] After the server completes the allocation of the advertisement samples in the experimental pool and the control pool, it configures the new advertisement configuration for the experimental pool and the old advertisement configuration for the control pool to conduct the old and new configuration control experiment based on the experimental pool and the control pool.

[0123] Step S113: Continuously store one or more advertisement samples pushed by advertisers into the advertisement library, and hierarchically allocate them to the experimental pool and the control pool according to the advertisers to which these advertisement samples belong:

[0124] After starting the new and old configuration control experiment, advertisement samples of advertisements that advertisers push to the current online advertisement delivery platform will also be continuously stored in the advertisement library, so that the server hierarchically allocates them to the experimental pool and the control pool of the new and old configuration control experiment according to the advertisers to which these advertisement samples belong, thereby enriching the sample size of the new and old configuration control experiment to improve the experimental credibility of the new and old configuration control experiment.

[0125] In this embodiment, in the new and old configuration control experiment, the samples for the experiment will, in addition to having the advertisement samples that the online advertisement delivery platform already has, also allocate the advertisement samples newly pushed by advertisers later to be used in the new and old configuration control experiment, and the allocation of the new advertisement samples will be hierarchically allocated based on the advertisers to ensure the homogenization of samples between the experimental pool and the control pool.

[0126] In one embodiment, please refer to Figure 6 In the step of obtaining the true impact index of the new and old configuration control experiment to generate the corresponding prior conditions according to this true impact index, the following steps are included:

[0127] Step S121: When it is monitored that the current time exceeds the experimental time for the new and old configuration control experiment, count the true impact index of the new and old configuration control experiment, and the true impact index is determined according to the index parameters of the experimental pool and the control pool respectively:

[0128] The server monitors in real time whether the new and old configuration control experiment has been completed. When it is monitored that the new and old configuration control experiment has completed the experiment, the true impact index of the new and old configuration control experiment will be obtained. Specifically, the server determines whether the current time exceeds the experimental time for the new and old configuration control experiment by monitoring. If it exceeds, the new and old configuration control experiment will be ended to obtain the true impact index corresponding to this experiment.

[0129] Since the new and old configuration control experiment is a control experiment, therefore, the true impact index generally will be determined according to the index parameters of the experimental pool and the control pool in the new and old control experiment. For example, when the index parameter between the experimental pool and the control pool is the advertisement delivery cost, the true impact index will be the difference between the advertisement delivery costs of the two, where the advertisement delivery cost of the experimental pool is the minuend.

[0130] Step S122: Generate an impact index value range containing the value represented by the true impact index, and use this impact index value range as the judgment threshold range for the prior condition.

[0131] After the server obtains the true impact index of the new and old configuration control experiment with completed experiments, it will generate the corresponding prior condition according to this true impact index. Use the value represented by the true impact index as the median, maximum value or minimum value to generate the corresponding value range as the impact index value range of the prior condition for subsequent judgment. For example, when the true impact index is the advertising investment cost gap ratio and its value is 15%, this value can be used as the median, and a value range of 10% to 20% can be generated as the impact index value range of the prior condition.

[0132] In this embodiment, by generating the corresponding prior condition according to the index output from the experimental results of the new and old configuration control experiment, it is convenient to perform experimental screening in the subsequent old configuration control experiment to ensure the experimental validity of AA experiments such as the old configuration control experiment due to insufficient sample size.

[0133] In one embodiment, please refer to Figure 7 , generate multiple pairs of experimental pools and control pools respectively for the old configuration control experiment based on the historical advertisement samples in the advertisement library according to the preset historical advertisement random allocation rule, and retain the multiple pairs of experimental pools and control pools whose experimental impact index meets the prior condition, including the following steps:

[0134] Step S131: Obtain the historical advertisement samples with earlier storage times in the advertisement library, and randomly allocate these historical advertisement samples to the experimental pool and the control pool to configure the same old advertisement configuration for the first old configuration control experiment for the experimental pool and the control pool.

[0135] The server allocates historical advertisement samples in the advertisement library according to the historical advertisement random allocation rule. The historical advertisement samples refer to the historical advertisement samples used in the new and old configuration comparison experiment to evaluate the experimental credibility of the new and old configuration comparison experiment. Specifically, first, obtain historical advertisement samples with an earlier storage time from the advertisement library and randomly allocate them to the experimental pool and the control pool of the old configuration comparison experiment. For example, generally select the experimental samples on the first day or the first time of the new and old configuration comparison experiment. For example, when the experimental date of the new and old configuration comparison experiment is from January 1st to January 15th, obtain the advertisement samples with the date of January 1st in the advertisement library and randomly allocate them. The reason is that in the business scenario of the gateway advertisement delivery platform, the number of advertisements to be delivered on the first day when starting a new advertisement delivery business is large, the number of experimental samples is sufficient, and the storage time of the advertisement samples in the advertisement library generally corresponds to their delivery time. Therefore, randomly allocate the advertisement samples with the storage time or the first day in the advertisement library to the experimental pool and the control pool of the old configuration comparison experiment. Of course, they can also be stratified and allocated to the experimental pool and the control pool according to the advertisers to which these advertisement samples belong. It should be noted that the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range to prevent the difference in the number of advertisement samples between the two from being too large.

[0136] Step S132, count the first experimental impact index of the first old configuration comparison experiment, and check whether the first experimental impact index is within the judgment threshold range of the prior condition. If it is, then stratify and allocate according to the advertisers to which the advertisement samples in the advertisement library belong to the experimental pool and the control pool respectively to continue the second old configuration comparison experiment. If not, stop the first old configuration comparison experiment:

[0137] The server will conduct multiple first old configuration comparison experiments, that is, after allocating the historical advertisement samples with an earlier storage time in the advertisement library to the experimental pool and the control pool multiple times to generate multiple pairs of the experimental pool and the control pool to form multiple first old configuration comparison experiments, and only retain the first old configuration comparison experiments whose first experimental impact index meets the prior condition for the second old configuration comparison experiment, and then obtain the second experimental impact index corresponding to each of these second old configuration comparison experiments, so as to generate the corresponding impact index distribution data according to the statistics of multiple groups of first experimental impact indexes and second experimental impact indexes; generally, the server will conduct 50,000 - 100,000 first configuration comparison experiments for experimental screening. Of course, those skilled in the art can also flexibly design the number of times of the experimental screening, which will not be elaborated here.

[0138] Step S133, when the second-oldest configuration control experiment ends, count the final second experimental impact metric of the second-oldest configuration control experiment, and store the second experimental impact metric and the first experimental impact metric corresponding to each other in the experimental impact metric list:

[0139] When the first experiment of any one of the first-oldest configuration control experiments meets the prior condition, the server obtains the unused ad samples in the ad library for this first-oldest configuration control experiment, and stratifies and randomly assigns these ad samples to the experimental pool and the control pool of the first-oldest configuration control experiment according to the advertisers to which these ad samples belong, so as to conduct the second-oldest configuration control experiment and then count the corresponding second experimental impact metric at the end of the experiment. Among them, counting the second experimental impact metric will combine the historical ad samples experimented by the first-oldest configuration control experiment and the newly assigned historical ad samples of the second-oldest configuration control experiment; the server stores the obtained first experimental impact metric and second experimental impact metric as mapping relationship data in the experimental impact metric list correspondingly, so as to generate the corresponding impact metric distribution data subsequently.

[0140] In this embodiment, the experimental screening of the old configuration control experiment is carried out through the prior condition to ensure the data validity of the subsequent generated impact metric distribution data, and further ensure the verification validity of the subsequent new ad configuration verification based on the impact metric distribution data.

[0141] In one embodiment, please refer to Figure 3 and Figure 8 In the step of retaining the experimental pools and control pools with experimental impact metrics that meet the prior condition and statistically generating the corresponding impact metric distribution data, the following steps are included:

[0142] Step S131', obtain multiple second experimental impact metrics stored in the experimental impact metric list and their first experimental impact metrics:

[0143] When the specified number of the first-oldest configuration control experiments is completed, the server obtains all the first experimental impact metrics and their correspondingly stored second experimental impact metrics from the experimental impact metric list.

[0144] Step S133', count the impact metric increments of each second experimental impact metric and its first experimental impact metric, and statistically generate the corresponding impact metric distribution data, and the data ratio representing each impact metric increment will be in the impact metric distribution data:

[0145] Please refer to Figure 3, calculate the difference between each of the first experimental impact indicators and its corresponding second experimental impact indicator as the corresponding impact indicator increment. Among them, the second experimental impact indicator will be used as the minuend, and then count these impact indicator increments to form the impact indicator distribution data. The impact indicator distribution data will be sorted according to each impact indicator distribution data, and the data proportion of each impact indicator increment in the impact indicator distribution data will be statistically calculated. For example, as Figure 3 shown, for the convenience of browsing, Figure 3 in the data distribution diagram shown is the visualized data of the impact indicator distribution data mentioned above. Among them, the abscissa of the shown data distribution diagram represents the impact indicator increment, the left ordinate represents the quantity of the impact indicator increment, and the right ordinate represents the proportion of the impact indicator increment. Correspondingly, the shown bar in the shown data distribution diagram corresponds to the quantity of the impact indicator increment, and the shown line in the shown data distribution diagram corresponds to the proportion of the impact indicator increment.

[0146] In this embodiment, by statistically calculating the differences of the experimental impact indicators corresponding to each old configuration control experiment that meet the prior conditions, an impact indicator distribution data representing the data proportion of each impact indicator increment is formed, so as to subsequently verify the true impact indicators obtained from the new advertisement configuration through the new advertisement configuration control experiment according to the data proportion of each impact indicator increment, and then determine the configuration effectiveness of the new advertisement configuration.

[0147] In one embodiment, please refer to Figure 4 and Figure 9 , in the step of retaining multiple pairs of experimental pools and control pools where the experimental impact indicators meet the prior conditions and statistically generating the corresponding impact indicator distribution data, the following steps are included:

[0148] Step S131”, obtain multiple second experimental impact indicators and their first experimental impact indicators stored in the experimental impact indicator list:

[0149] After completing the specified number of the first old configuration control experiments, the server will obtain all the first experimental impact indicators and their corresponding stored second experimental impact indicators from the experimental impact indicator list.

[0150] Step S132”, calculate the impact indicator increment between each of the second experimental impact indicators and their first experimental impact indicators, and then calculate the ratio of these impact indicator increments to their first experimental impact indicators, and use this ratio as the impact indicator increment rate corresponding to the first experimental impact indicator:

[0151] Calculate the difference between each of the first experimental impact indicators and their corresponding second experimental impact indicators as the corresponding impact indicator increment. Among them, the second experimental impact indicator will be used as the minuend, and then calculate the ratio of these impact indicator increments to their first experimental impact indicators, and take this ratio as the impact indicator increment rate corresponding to the first experimental impact indicator.

[0152] Step S133”, count the impact indicator increment rates to generate corresponding impact indicator distribution data:

[0153] Please refer to Figure 4 , Figure 4 In the figure, the data distribution diagram is the visualization data of the impact indicator distribution data with the impact indicator increment rate. Among them, the abscissa of the shown data distribution diagram represents the impact indicator increment rate, the left ordinate represents the quantity of the impact indicator increment rate, the right ordinate represents the proportion of the impact indicator increment rate. Correspondingly, the shown columns in the shown data distribution diagram correspond to the quantity of the impact indicator increment rate, and the shown lines in the shown data distribution diagram correspond to the proportion of the impact indicator increment rate.

[0154] In this embodiment, in addition to having impact indicator increments, the impact indicator distribution data can also determine the corresponding impact indicator increment rate through the first experimental impact data, so as to subsequently verify whether the new advertisement configuration is a valid new configuration according to the impact indicator increment rate.

[0155] In one embodiment, please refer to Figure 10 , in the step of verifying whether the new advertisement configuration meets the configuration validity rule based on the impact indicator distribution data, it includes any one or any combination of the following steps:

[0156] Step S141, determine the impact indicator increment with the highest data proportion in the impact indicator distribution data, take this impact indicator increment as the expected increment, calculate the difference between the expected increment and the true impact indicator. If the difference meets the increment threshold of the configuration validity rule, then the new advertisement configuration is a valid new configuration:

[0157] The configuration validity rule generally verifies whether the new advertisement configuration is a valid new configuration according to the impact indicator increments in the impact indicator distribution data. Specifically, determine the impact indicator increment with the highest data proportion in the impact indicator distribution data, take this impact indicator increment as the expected increment, and then calculate the difference between the expected increment and the true impact indicator. Among them, the true impact indicator is the minuend. When the difference meets the preset increment threshold of the configuration validity rule, then the new advertisement configuration will be verified as a valid new configuration. The increment threshold is generally set within the range of 0 - 100, and those skilled in the art can also design the numerical range of the increment threshold according to the actual application scenario.

[0158] Step S142: Determine the impact indicator increment with the largest value in the impact indicator distribution data, and verify whether the true impact indicator exceeds this impact indicator increment. If it exceeds, the new advertisement configuration is determined as a valid new configuration:

[0159] In addition to determining the impact indicator increment with the highest data proportion in the impact indicator distribution data to verify whether the new advertisement configuration is a valid new configuration, it is also possible to judge whether the true effective indicator corresponding to the new and old configuration comparison experiment exceeds the impact indicator increment with the largest value in the impact indicator distribution data, so as to verify whether the new advertisement configuration is a valid new configuration.

[0160] Of course, it is also possible to combine the respective configuration validity rules of step S141 and step S142 to verify whether the new advertisement configuration is a valid new configuration. When the true effective indicator meets any or all of the configuration validity rules, the new advertisement configuration is determined as a valid new configuration.

[0161] In this embodiment, the configuration validity verification of the new advertisement configuration is performed based on the impact indicator distribution data generated by statistically analyzing the experimental impact indicators output by the old configuration comparison experiment that meets the prior statistics. It can be understood that the comparability of the experimental samples in the comparison experiment can be effectively enhanced, the authenticity of the experimental results can be improved, and the verification result of the configuration validity of the final verification of the new advertisement configuration is more credible.

[0162] Furthermore, by functionalizing each step in the methods disclosed in the above embodiments, an advertisement configuration verification device of the present application can be constructed. According to this idea, please refer to Figure 11 , in one typical embodiment, the device includes: a new configuration experiment module 11, configured to allocate advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, so as to perform a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration; a prior condition generation module 12, configured to obtain the true impact indicator of the new and old configuration comparison experiment, so as to generate a corresponding prior condition according to the true impact indicator; a distribution data generation module 13, configured to perform old configuration comparison experiments on multiple pairs of experimental pools and control pools respectively based on the historical advertisement samples in the advertisement library according to a preset historical advertisement random allocation rule, retain the multiple pairs of experimental pools and control pools whose experimental impact indicators meet the prior condition, and statistically generate corresponding impact indicator distribution data; a configuration validity verification module 14, configured to verify whether the new advertisement configuration meets the configuration validity rule based on the impact indicator distribution data. If it meets, the new advertisement configuration is a valid new configuration.

[0163] In one embodiment, the new configuration experiment module 11 includes: a sample random allocation sub-module for randomly allocating existing advertisement samples in the advertisement library to an experimental pool and a control pool, wherein the difference in the number of advertisement samples in the experimental pool and the control pool is within a preset threshold range; a new configuration experiment start sub-module for correspondingly configuring the new advertisement configuration and the old advertisement configuration in the experimental pool and the control pool to start the control experiment of the new and old configurations; a sample stratified allocation sub-module for continuously storing advertisement samples pushed by one or more advertiser terminals in the advertisement library and stratifiedly allocating them to the experimental pool and the control pool according to the advertiser terminals to which these advertisement samples belong respectively.

[0164] In one embodiment, the prior condition generation module 12 includes: an influence index statistics sub-module for, when it is monitored that the current time exceeds the experimental time acting on the control experiment of the new and old configurations, statistically calculating the true influence index of the control experiment of the new and old configurations, where the true influence index is determined according to the index parameters of the experimental pool and the control pool respectively; a judgment threshold determination sub-module for generating an influence index value range containing this value according to the value represented by the true influence index, and using this influence index value range as the judgment threshold range of the prior condition.

[0165] In one embodiment, the distribution data generation module 13 includes: a historical sample allocation sub-module for obtaining historical advertisement samples with earlier storage times in the advertisement library and randomly allocating these historical advertisement samples to the experimental pool and the control pool to configure the same old advertisement configuration for the experimental pool and the control pool to conduct a first old configuration control experiment; an experimental index verification sub-module for statistically calculating the first experimental influence index of the first old configuration control experiment and verifying whether the first experimental influence index is within the judgment threshold range of the prior condition. If it is, then stratify and allocate according to the advertiser terminals to which the advertisement samples in the advertisement library belong respectively to the experimental pool and the control pool to continue the second old configuration control experiment. If not, stop the first old configuration control experiment; an experimental index storage sub-module for, when the second old configuration control experiment ends, statistically calculating the final second experimental influence index of the second old configuration control experiment and storing the second experimental influence index and the first experimental influence index corresponding to each other in the experimental influence index list.

[0166] In another embodiment, the distribution data generation module 13 further includes: an experimental index acquisition sub-module for obtaining multiple second experimental influence indexes stored in the experimental influence index list and their first experimental influence indexes; an index increment statistics sub-module for statistically calculating the influence index increments of each second experimental influence index and its first experimental influence index, and statistically calculating these influence index increments to generate corresponding influence index distribution data, where the data proportion representing each influence index increment is in the influence index distribution data.

[0167] In another embodiment, the distribution data generation module 13 further includes: an index parameter sub-module, configured to obtain a plurality of second experimental impact indices stored in the experimental impact index list and their first experimental impact indices; an index value-added rate sub-module, configured to calculate the impact index value-added of each of the second experimental impact indices and their first experimental impact indices, and then calculate the ratio of these impact index value-addeds to their first experimental impact indices, and use this ratio as the impact index value-added rate corresponding to the first experimental impact index; an index distribution data generation sub-module, configured to count each of the impact index value-added rates to generate corresponding impact index distribution data.

[0168] In one embodiment, the configuration validity verification module 14 includes: an expected value-added verification sub-module, configured to determine the impact index value-added with the highest data proportion in the impact index distribution data, use this impact index value-added as the expected value-added, calculate the difference between the expected value-added and the actual impact index, and if this difference meets the value-added threshold of the configuration validity rule, then the new advertisement configuration is a valid new configuration; a maximum value-added verification sub-module, configured to determine the impact index value-added with the largest value in the impact index distribution data, and verify whether the actual impact index exceeds this impact index value-added, and if it exceeds, then the new advertisement configuration is a valid new configuration.

[0169] To solve the above technical problems, an embodiment of the present application further provides a computer device for running a computer program implemented according to the advertisement configuration verification method. For details, please refer to Figure 12 , Figure 12 which is the basic structural block diagram of the computer device in this embodiment.

[0170] As Figure 12 shown in the internal structural schematic diagram of the computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected through a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database can store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement an advertisement configuration verification method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device can store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute an advertisement configuration verification method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that Figure 12The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0171] In this embodiment, the processor is used to execute the specific functions of each module / sub-module in the advertisement configuration verification device of this application. The memory stores the program code and various types of data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. The memory in this embodiment stores the program code and data required to execute all modules / sub-modules in the advertisement configuration verification device, and the server can call the program code and data of the server to execute the functions of all sub-modules.

[0172] This application also provides a non-volatile storage medium. The advertisement configuration verification method is written as a computer program and stored in this storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means the running of this program in the computer, thereby enabling one or more processors to execute the steps of the advertisement configuration verification method in any of the above embodiments.

[0173] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing related hardware through a computer program. This computer program can be stored in a computer-readable storage medium. When this program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disc, a read-only memory (ROM), etc., or a random access memory (RAM), etc.

[0174] In summary, this application can enhance the comparability of experimental samples in a controlled experiment and improve the authenticity of experimental results.

[0175] It should be understood that although the steps in the flowchart of the accompanying drawings are shown in sequence according to the indication of the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily need to be executed at the same moment, but can be executed at different moments. Their execution order does not necessarily need to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0176] Those skilled in the art can understand that the various operations, methods, steps, measures, and solutions in the processes discussed in this application can be alternated, changed, combined, or deleted. Further, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, those in the prior art that have steps, measures, and solutions in the various operations, methods, and processes disclosed in this application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0177] The above are only partial embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. An advertisement configuration verification method, characterized in that, It includes the following steps: Allocate the advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, so as to conduct a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration; the new advertisement configuration and the old advertisement configuration are correspondingly configured in the experimental pool and the control pool to start the new and old configuration comparison experiment; Obtain the true influence index of the new and old configuration comparison experiment, so as to generate a corresponding prior condition according to the true influence index; the true influence index is determined according to the index parameters of the experimental pool and the control pool respectively; Based on a preset historical advertisement random allocation rule, generate multiple pairs of experimental pools and control pools from the historical advertisement samples in the advertisement library to conduct old configuration comparison experiments respectively, and retain the multiple pairs of experimental pools and control pools whose experimental influence indexes meet the prior conditions, so as to statistically generate corresponding influence index distribution data; Verify whether the new advertisement configuration meets the configuration validity rule based on the influence index distribution data. If it meets, the new advertisement configuration is a valid new configuration.

2. The method according to claim 1, characterized in that, In the step of allocating the advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, so as to conduct a new and old configuration comparison experiment on the experimental pool and the control pool based on the new advertisement configuration, it includes the following steps: Randomly allocate the existing advertisement samples in the advertisement library to the experimental pool and the control pool, where the difference in the number of advertisement samples of the experimental pool and the control pool is within a preset threshold range; Continuously store the advertisement samples pushed by one or more advertiser terminals into the advertisement library, and layer and allocate them to the experimental pool and the control pool according to the advertiser terminals to which these advertisement samples belong respectively.

3. The method according to claim 1, characterized in that, In the step of obtaining the true influence index of the new and old configuration comparison experiment, so as to generate a corresponding prior condition according to the true influence index, it includes the following steps: When it is monitored that the current time exceeds the experimental time acting on the new and old configuration comparison experiment, count the true influence index of the new and old configuration comparison experiment; Generate an influence index value interval containing this value according to the value represented by the true influence index, so as to use this influence index value interval as the judgment threshold interval of the prior condition.

4. The method according to claim 1, characterized in that, Based on a preset historical advertisement random allocation rule, generate multiple pairs of experimental pools and control pools from the historical advertisement samples in the advertisement library to conduct old configuration comparison experiments respectively, and retain the multiple pairs of experimental pools and control pools whose experimental influence indexes meet the prior conditions, including the following steps: Obtain the historical advertisement samples with earlier warehousing time in the advertisement library, and randomly allocate these historical advertisement samples to the experimental pool and the control pool to configure the same old advertisement configuration for the experimental pool and the control pool to conduct the first old configuration comparison experiment; Count the first experimental influence index of the first old configuration comparison experiment, and verify whether the first experimental influence index is within the judgment threshold interval of the prior condition. If it is, layer and allocate them to the experimental pool and the control pool according to the advertiser terminals to which the advertisement samples in the advertisement library belong respectively to continue the second old configuration comparison experiment. If not, stop the first old configuration comparison experiment; When the second-oldest configuration control experiment ends, the final second experimental impact metric of the second-oldest configuration control experiment is statistically analyzed, and the second experimental impact metric and the first experimental impact metric are stored correspondingly in the experimental impact metric list.

5. The method according to claim 4, characterized in that, In the step of retaining pairs of experimental pools and control pools whose experimental impact metrics meet the prior conditions and statistically generating corresponding impact metric distribution data, the following steps are included: Obtain multiple second experimental impact metrics stored in the experimental impact metric list and their first experimental impact metrics; Statistically analyze the impact metric increments of each of the second experimental impact metrics and their first experimental impact metrics, and statistically generate corresponding impact metric distribution data based on these impact metric increments. The impact metric distribution data will show the proportion of the data representing each impact metric increment.

6. The method according to claim 4, characterized in that, In the step of retaining pairs of experimental pools and control pools whose experimental impact metrics meet the prior conditions and statistically generating corresponding impact metric distribution data, the following steps are included: Obtain multiple second experimental impact metrics stored in the experimental impact metric list and their first experimental impact metrics; Calculate the impact metric increments of each of the second experimental impact metrics and their first experimental impact metrics, and then calculate the ratio of these impact metric increments to their first experimental impact metrics. Take this ratio as the impact metric increment rate corresponding to the first experimental impact metric; Statistically analyze each of the impact metric increment rates to generate corresponding impact metric distribution data.

7. The method according to claim 5, characterized in that, In the step of verifying whether the new advertisement configuration meets the configuration validity rule based on the impact metric distribution data, any one or more of the following steps are included: Determine the impact metric increment with the highest data proportion in the impact metric distribution data, take this impact metric increment as the expected increment, calculate the difference between the expected increment and the true impact metric. If this difference meets the increment threshold of the configuration validity rule, then the new advertisement configuration is a valid new configuration; Determine the impact metric increment with the largest value in the impact metric distribution data, and verify whether the true impact metric exceeds this impact metric increment. If it exceeds, then the new advertisement configuration is a valid new configuration.

8. An advertisement configuration verification device, characterized in that, Including: A new configuration experiment module, used to allocate advertisement samples in the advertisement library to the experimental pool and the control pool according to a preset new advertisement random allocation rule, and conduct a new-old configuration control experiment on the experimental pool and the control pool based on the new advertisement configuration; the new advertisement configuration and the old advertisement configuration are correspondingly configured in the experimental pool and the control pool to start the new-old configuration control experiment; A prior condition generation module, used to obtain the true impact metric of the new-old configuration control experiment and generate corresponding prior conditions according to this true impact metric; the true impact metric is determined according to the index parameters of the experimental pool and the control pool respectively; A distribution data generation module, used to generate multiple pairs of experimental pools and control pools based on the historical advertisement samples in the advertisement library according to a preset historical advertisement random allocation rule, conduct old configuration control experiments respectively, retain pairs of experimental pools and control pools whose experimental impact metrics meet the prior conditions, and statistically generate corresponding impact metric distribution data; Configure a validity verification module to verify whether the new advertisement configuration meets the configuration validity rule based on the impact index distribution data. If it meets, the new advertisement configuration is a valid new configuration.

9. An electronic device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, It stores in the form of computer-readable instructions a computer program implemented according to the method according to any one of claims 1 to 7. When the computer program is called and run by the computer, it executes the steps included in the method.

Images