An advertisement experiment method, device, equipment, storage medium, product and system

Abstract

The application provides an advertisement experiment method, device, equipment, storage medium, product and system, the method comprises the following steps: creating a plurality of experimental advertisements according to the experimental elements of advertisement delivery experiment, the experimental elements comprise at least one experimental variable and a plurality of parameter groups of each experimental variable; according to the preset traffic distribution strategy, determine the first advertisement delivery strategy corresponding to the first traffic request in the advertisement delivery strategy corresponding to each experimental advertisement, the first traffic request is used to trigger the delivery advertisement; the first experimental advertisement corresponding to the first advertisement delivery strategy is delivered, and the experimental advertisement delivery data corresponding to the first traffic request is collected, and the experimental advertisement delivery data represents the advertisement effect. The application can effectively improve the efficiency and accuracy of the advertisement experiment, reduce the artificial error, at the same time, ensure the fairness of the experimental environment, and then can provide more reliable advertisement experiment data for optimizing the advertisement delivery strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an advertisement experiment method, device, equipment, storage medium, product and system. BACKGROUND

[0002] In the current digital era, advertisement delivery has become an important means for enterprises to promote products, services and brands. With the rapid development of Internet technology, the forms and delivery channels of advertisements are increasingly diversified. How to effectively evaluate and optimize advertisement delivery strategies to improve advertisement effectiveness and reduce costs has become the focus of attention of both advertisers and advertisement platforms.

[0003] The existing advertisement experiment method usually creates multiple experimental advertisements by operation personnel, each advertisement containing different advertisement delivery strategy parameter groups, so as to test the effect of each parameter group separately while controlling other variables. This process is extremely cumbersome and cannot guarantee that all advertisements are in a fair comparison environment throughout the process, nor can it avoid various errors caused by manual operation. SUMMARY

[0004] Therefore, the embodiments of the present application aim to provide an advertisement experiment method, device, equipment, storage medium, product and system, which can effectively improve the efficiency and accuracy of advertisement experiments, reduce human errors, and ensure the fairness of the experimental environment, thereby providing more reliable advertisement experiment data for optimizing advertisement delivery strategies.

[0005] According to a first aspect of the embodiments of the present application, an advertisement experiment method is provided, applied to an advertisement experiment system, and the method comprises:

[0006] According to an experiment element of the advertisement delivery experiment, multiple experimental advertisements are created, the experiment element comprising at least one experimental variable and multiple parameter groups of each experimental variable, and the advertisement delivery strategies corresponding to each experimental advertisement being different, wherein one advertisement delivery strategy comprises one parameter group corresponding to each experimental variable;

[0007] According to a preset traffic distribution strategy, a first advertisement delivery strategy corresponding to a first traffic request is determined in the advertisement delivery strategies corresponding to each experimental advertisement, and the first traffic request is used to trigger the delivery of an advertisement;

[0008] The first experimental advertisement corresponding to the first advertisement delivery strategy is delivered, and experimental advertisement delivery data corresponding to the first traffic request is collected, the experimental advertisement delivery data representing advertisement effectiveness.

[0009] Optionally, the method further comprises:

[0010] performing data analysis on the experimental advertisement delivery data corresponding to each traffic request to obtain an analysis result, the analysis result including advertisement effects corresponding to different parameter groups of a same experimental variable respectively and / or differences in advertisement effects of different parameter groups of the same experimental variable.

[0011] Optionally, the traffic distribution strategy includes a hierarchical traffic orthogonal strategy, wherein different experimental variables are located in different traffic layers, and multiple parameter groups of a same experimental variable are located in a same traffic layer.

[0012] According to the preset traffic distribution strategy, a first advertisement delivery strategy corresponding to the first traffic request is determined from among the advertisement delivery strategies of the experimental advertisements, including:

[0013] According to the preset traffic distribution strategy, a parameter group matching the first traffic request in each traffic layer is determined.

[0014] The parameter groups matching the first traffic request in each traffic layer are combined to obtain the first advertisement delivery strategy corresponding to the first traffic request.

[0015] Optionally, the method further includes storing a mapping relationship between an advertisement delivery strategy corresponding to a historical traffic request and identification information of the historical traffic request into a mapping relationship database.

[0016] According to the preset traffic distribution strategy, a first advertisement delivery strategy corresponding to the first traffic request is determined from among the advertisement delivery strategies of the experimental advertisements, including:

[0017] The identification information of the first traffic request is used to search in the mapping relationship database, and when a target advertisement delivery strategy having a mapping relationship with the identification information of the first traffic request is stored in the mapping relationship database, the target advertisement delivery strategy is determined as the first advertisement delivery strategy.

[0018] Optionally, the method further includes:

[0019] According to the attribute information of the first traffic request, a first advertisement set is determined from among a current advertisement set of the advertisement experiment system, the first traffic request satisfying an advertisement delivery strategy of each advertisement in the first advertisement set, the current advertisement set including experimental advertisements and / or non-experimental advertisements.

[0020] When the first experimental advertisement corresponding to the first advertisement delivery strategy is not included in the first advertisement set, a non-experimental advertisement is selected from among the non-experimental advertisements of the first advertisement set for delivery.

[0021] Optionally, the advertisement experiment system stores a setting parameter template of each experiment quantitative in the advertisement delivery strategy.

[0022] According to the experiment elements of the advertisement delivery experiment, a plurality of experiment advertisements are created, including:

[0023] The plurality of parameter groups of each experiment variable in the experiment elements are combined with the setting parameter template corresponding to the experiment quantitative, to obtain a plurality of advertisement delivery strategies.

[0024] An experiment advertisement is created according to each advertisement delivery strategy respectively.

[0025] Optionally, the experiment elements further include at least one of an experiment budget, an experiment period, and an advertisement effect determination index.

[0026] Optionally, the experiment elements further include an application flow of each parameter group in each flow layer.

[0027] According to a second aspect of an embodiment of the present application, an advertisement experiment device is provided, including:

[0028] A first unit is configured to create a plurality of experiment advertisements according to experiment elements of an advertisement delivery experiment, the experiment elements including at least one experiment variable and a plurality of parameter groups of each experiment variable, and each experiment advertisement corresponding to a different advertisement delivery strategy, wherein one of the advertisement delivery strategies includes one parameter group corresponding to each experiment variable.

[0029] A second unit is configured to determine a first advertisement delivery strategy corresponding to a first flow request in the advertisement delivery strategies corresponding to the experiment advertisements according to a preset flow distribution strategy, the first flow request being used to trigger the delivery of an advertisement.

[0030] A third unit is configured to deliver a first experiment advertisement corresponding to the first advertisement delivery strategy, and collect experiment advertisement delivery data corresponding to the first flow request, the experiment advertisement delivery data representing an advertisement effect.

[0031] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor.

[0032] The memory is connected with the processor, and is configured to store a program.

[0033] The processor is configured to realize the advertisement experiment method according to any one of the first aspect of the embodiments of the present application by running the program in the memory.

[0034] According to a fourth aspect of the embodiments of the present application, a computer program product is provided, comprising computer program instructions which, when executed by a processor, cause the processor to implement the advertisement experiment method according to any one of the first aspect of the embodiments of the present application.

[0035] According to a fifth aspect of the embodiments of the present application, an advertisement experiment system is provided, comprising:

[0036] a server and a client,

[0037] the client is configured to acquire user input information and send the user input information to the server, the user input information comprising experiment elements of a first experiment, the experiment elements comprising at least one experiment variable and a plurality of parameter sets of each experiment variable, the experiment variable comprising one sub-strategy in an advertisement delivery strategy;

[0038] the server is configured to implement the advertisement experiment method according to any one of the first aspect of the embodiments of the present application.

[0039] The advertisement experiment method provided by the embodiments of the present application first creates a plurality of experimental advertisements according to experiment elements of an advertisement delivery experiment, the experiment elements comprising at least one experiment variable and a plurality of parameter sets of each experiment variable, each experimental advertisement corresponding to a different advertisement delivery strategy, wherein one advertisement delivery strategy comprises one parameter set corresponding to each experiment variable; then determines a first advertisement delivery strategy corresponding to a first traffic request in each advertisement delivery strategy of each experimental advertisement according to a preset traffic distribution strategy, the first traffic request being used to trigger the delivery of an advertisement; and finally delivers a first experimental advertisement corresponding to the first advertisement delivery strategy and collects experimental advertisement delivery data corresponding to the first traffic request, the experimental advertisement delivery data representing an advertisement effect.

[0040] The advertisement experiment method provided by the embodiments of the present application can automatically create a plurality of experimental advertisements according to experiment elements of an advertisement delivery experiment and automatically deliver advertisements according to a preset traffic distribution strategy, thereby significantly improving the efficiency of the experiment and improving the accuracy of the experiment due to the reduction of manual intervention. On the other hand, by designing a suitable traffic distribution strategy, each experimental advertisement can obtain relatively balanced traffic, thereby avoiding experimental result deviation caused by uneven traffic distribution and ensuring the fairness and comparability of the experimental results. On the other hand, by collecting experimental advertisement delivery data, the effect data of different experimental advertisements can be provided to operators and advertisers, so that they can more accurately evaluate the effects of different advertisement delivery strategies, thereby optimizing the advertisement delivery strategy, improving the advertisement effect and reducing the cost. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only belong to the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0042] FIG. 1 A structural schematic diagram of an advertisement experiment system provided by an embodiment of the present application.

[0043] FIG. 2 A flowchart of an advertisement experiment method provided by an embodiment of the present application.

[0044] FIG. 3 A flowchart of another advertisement experiment method provided by an embodiment of the present application.

[0045] FIG. 4 A processing flowchart of determining a first advertisement putting strategy corresponding to a first traffic request provided by an embodiment of the present application.

[0046] FIG. 5 A processing flowchart of determining a first advertisement putting strategy corresponding to a first traffic request provided by another embodiment of the present application.

[0047] FIG. 6 A flowchart of another advertisement experiment method provided by an embodiment of the present application.

[0048] FIG. 7 A structural schematic diagram of an advertisement experiment device provided by an embodiment of the present application.

[0049] FIG. 8 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical scheme of the embodiment of the application is suitable for being applied in various scenes requiring experiments on advertisement launching strategies, such as Internet marketing, e-commerce promotion, mobile application promotion and the like, and is widely applicable to the fields of digital media, marketing, brand building and the like. In these application scenes, there are many factors influencing the advertisement effect in the advertisement launching strategy, including but not limited to the positioning of target groups, the pricing strategy of advertisements, the selected media channels, the regional range of launching, the design and optimization of advertisement materials and landing pages and the like. In order to find the best advertisement launching strategy and make the advertisement obtain better advertisement effect, the operation personnel need to make careful strategy formulation and adjustment with respect to each key influencing factor (i.e. each sub-strategy) in the advertisement launching strategy, and therefore, the advertisement experiment needs to be performed with respect to each sub-strategy in the advertisement launching strategy. The technical scheme of the embodiment of the application can effectively improve the efficiency and accuracy of the advertisement experiment, reduce the artificial error, ensure the fairness of the experimental environment, and further provide more reliable advertisement experiment data for optimizing the advertisement launching strategy, by automatically creating a plurality of experimental advertisements, launching the experimental advertisements based on a preset traffic distribution strategy, and collecting experimental advertisement launching data representing the advertisement effect.

[0051] The technical scheme provided by the embodiment of the application can be exemplarily applied to a hardware device such as a processor, an electronic device, a server (including a cloud server) or a software program packaged for running. When the hardware device executes the processing process of the technical scheme of the application or the software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required by the task can be realized, and the purpose of the target task can be achieved. The embodiment of the application only exemplarily introduces the specific processing process of the technical scheme of the application, and does not limit the specific implementation form of the technical scheme of the application. Any technical implementation form that can execute the processing process of the technical scheme of the application can be adopted by the embodiment of the application.

[0052] The technical scheme in the embodiment of the application will be described clearly and completely in combination with the drawings in the embodiment of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0053] Before introducing the scheme of the application, the related art is introduced first.

[0054] In the current digital era, advertisement launching has become an important means for enterprises to promote products, services and brands. With the rapid development of Internet technology, the forms and launching channels of advertisements are increasingly diversified. How to effectively evaluate and optimize advertisement launching strategies to improve advertisement effect and reduce costs has become the focus of attention of both advertisers and advertisement platforms.

[0055] The existing advertisement experiment method is usually created by an operator to create multiple experimental advertisements, each of which contains different advertisement delivery strategy parameter groups, so as to test the effect of each parameter group separately while controlling other variables. This process is extremely cumbersome and cannot guarantee that all advertisements are in a fair comparison environment throughout the process, nor can it avoid various error effects caused by manual operation.

[0056] Therefore, the embodiments of the present application are dedicated to providing an advertisement experiment method, device, equipment, storage medium, product and system, which can effectively improve the efficiency and accuracy of the advertisement experiment, reduce manual errors, and ensure the fairness of the experimental environment, thereby providing more reliable advertisement experiment data for optimizing the advertisement delivery strategy.

[0057] Exemplary system

[0058] In order to facilitate understanding, first, the implementation environment of the advertisement experiment method provided by the embodiments of the present application is exemplarily introduced, please refer to FIG. 1 , FIG. 1 The structure diagram of an advertisement experiment system provided by the embodiments of the present application, the advertisement experiment method provided by the present application can be exemplarily applied to the advertisement experiment system.

[0059] As shown in FIG. 1 , the advertisement experiment system includes a client 110 and a server 120, and the client 110 and the server 120 are in communication connection.

[0060] Among them, the client 110 is used to obtain user input information and send the user input information to the server 120.

[0061] The user input information includes experimental elements of a first experiment, the experimental elements include at least one experimental variable and multiple parameter groups of each experimental variable, and the experimental variable includes one sub-strategy in the advertisement delivery strategy.

[0062] The server 120 is configured to execute the advertisement experiment method provided in the embodiments of the present application after receiving the experiment elements of the advertisement launching experiment sent by the client 110, including: creating a plurality of experiment advertisements according to the experiment elements of the advertisement launching experiment, the experiment elements including at least one experiment variable and a plurality of parameter groups of each experiment variable, and each experiment advertisement corresponding to different advertisement launching strategies, wherein one advertisement launching strategy includes one parameter group corresponding to each experiment variable; determining a first advertisement launching strategy corresponding to a first traffic request in the advertisement launching strategies corresponding to each experiment advertisement according to a preset traffic distribution strategy, the first traffic request being used to trigger the launching of an advertisement; launching a first experiment advertisement corresponding to the first advertisement launching strategy and collecting experiment advertisement launching data corresponding to the first traffic request, the experiment advertisement launching data representing the effect of the advertisement.

[0063] The specific content of the advertisement experiment method will be described in subsequent embodiments, and can be referred to the content of the subsequent embodiments, and will not be described in detail here.

[0064] The client 110 can be various terminal devices with a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer and a desktop computer, etc., and can also be an application program running on a terminal device, such as a web browser, a desktop application program, a mobile application program, etc., and the present application does not limit this.

[0065] The server 120 can be a stand-alone physical server, a server cluster composed of a plurality of physical servers, a cloud server providing cloud computing services or a virtual machine, and the present application does not limit this.

[0066] The communication network can be any form of wireless communication network or wired communication network, or any combination thereof, and the present application does not limit this.

[0067] Exemplary method

[0068] FIG. 2 A flowchart of an advertisement experiment method provided in the embodiments of the present application is shown in FIG. 2. As shown in FIG. 2, the advertisement experiment method can be exemplarily applied to the advertisement experiment system in FIG. 1, and is specifically executed by the server in the advertisement experiment system. The method includes steps S201-S203. FIG. 2 FIG. 1

[0069] ​​S201, create a plurality of experimental advertisements according to experimental elements of the advertisement launching experiment, the experimental elements including at least one experimental variable and a plurality of parameter groups of each experimental variable, each experimental advertisement corresponding to a different advertisement launching strategy, wherein the advertisement launching strategy includes a parameter group corresponding to each experimental variable.

[0070] The advertisement launching strategy can be understood as a series of plans and methods taken in the advertisement activity in order to achieve the marketing target, aiming to maximize the advertisement effect, so that the advertisement budget can accurately hit the target and optimize the cost efficiency.

[0071] The advertisement launching strategy usually includes multiple dimensions, including but not limited to advertisement materials (including pictures, videos, scripts, etc.), landing page design (the page jumped to by the user after clicking the advertisement), launching time (the specific time period of the advertisement display), media selection (the platform or channel of the advertisement launching), target audience (the sub-group of the advertisement audience), RTA (Real-Time Advertising, real-time advertisement optimization) strategy, JS template (JavaScript template for dynamically generating advertisement content), etc. Each dimension can be regarded as a sub-strategy of the advertisement launching strategy, and a complete advertisement launching strategy includes multiple sub-strategies. Different settings or combinations of each sub-strategy of the advertisement launching strategy will have an impact on the advertisement effect, and therefore are the key research objects in the advertisement launching experiment.

[0072] The experimental variable can be understood as a factor that can be changed in the advertisement launching experiment and will affect the advertisement effect due to the change. By changing these factors, the specific influence of the advertisement effect can be observed and analyzed.

[0073] Optionally, one experimental variable corresponds to one adjustable sub-strategy in the advertisement launching strategy. For example, in the advertisement launching experiment, the experimental variable can correspond to different designs of the advertisement materials, different versions of the landing page, differences in the launching time, selection of the target audience, distribution of the media channel, adjustment of the real-time bidding (RTA) strategy, change of the JavaScript (JS) template, etc.

[0074] The parameter group of the experimental variable can be understood as the different states or settings of the experimental variable in the advertisement launching experiment. By changing the parameter value of the experimental variable, the specific influence of the experimental variable on the advertisement effect can be observed and analyzed. Each experimental variable corresponds to a plurality of parameter groups. For example, if the experimental variable is the advertisement material, the parameter group can include different picture, video or animation designs; if the experimental variable is the launching time, the parameter group can include different time periods or dates; if the experimental variable is the target audience, the parameter group can include different audience characteristics such as age, gender, interest, etc.

[0075] The experimental elements can be obtained in various ways depending on the needs of the advertiser, available data resources, and the functions of the advertising experiment system, etc., which are not limited in the present application. Optionally, the design principles of A / B testing or multi-element testing can be followed to select the experimental elements, including ensuring the independence of experimental variables, controlling other potential influencing factors, and setting up appropriate control groups, etc.

[0076] Optionally, the experimental elements can be obtained by user input through the client. The advertising experiment system usually provides a user-friendly interface to facilitate users to manually input experimental elements through the client interface of the advertising experiment system, which includes selecting or defining experimental variables and setting different parameter groups for each experimental variable.

[0077] Optionally, the advertising experiment system can analyze historical advertising activity data to identify key factors that may have a significant impact on advertising effectiveness, determine these factors as experimental variables, and set parameter groups for each experimental variable based on historical performance.

[0078] Optionally, the advertising experiment system can use machine learning algorithms to automatically identify and select experimental variables. The algorithm analyzes a large amount of advertising data to find key factors related to advertising effectiveness and suggests these factors as experimental variables. In addition, the algorithm can also predict the potential effects of different parameter groups based on data patterns, thereby optimizing the experimental design.

[0079] After obtaining the experimental elements of the advertising experiment, multiple experimental advertisements can be created based on the experimental elements of the advertising experiment.

[0080] Based on the selected experimental variables and parameter groups, multiple experimental advertisements can be created. Each experimental advertisement corresponds to a unique advertising delivery strategy, which includes a parameter group corresponding to each experimental variable. For example, if the experimental variables include advertising materials and delivery time, and each variable has two parameter groups, there will be a total of 4 (2x2) different advertising delivery strategies, i.e. 4 experimental advertisements, each of which will have a specific advertising material version and a specific delivery time period.

[0081] As an optional implementation, the advertising experiment system stores a setting parameter template for each experimental variable in the advertising delivery strategy.

[0082] In this implementation, multiple experimental advertisements are created based on the experimental elements of the advertising experiment, including steps A1-A2:

[0083] A1, combine the multiple parameter groups of each experimental variable in the experimental elements with the setting parameter template corresponding to the experimental variable to obtain multiple advertising delivery strategies.

[0084] Specifically, the advertising experiment system stores setting parameter templates of each sub-strategy in the advertising delivery strategy. The setting parameter templates are predefined and include setting parameters of each sub-strategy in common or recommended cases. Optionally, the setting parameter templates can be summarized based on historical data, industry best practices, or the advertiser's own experience, etc.

[0085] The experiment quantifies other types of sub-strategies in the advertising delivery strategy except for the sub-strategy corresponding to the experiment variable. The setting parameters of these sub-strategies are generally stable and do not change frequently, so they can be managed through templating.

[0086] Optionally, the experiment quantifies other types of sub-strategies in the advertising delivery strategy except for the sub-strategy corresponding to the experiment variable.

[0087] After obtaining the experiment elements and determining the experiment variable, the experiment quantification can be determined according to the experiment variable, and then the setting parameter templates corresponding to the experiment quantification can be retrieved from the advertising experiment system. These templates provide a basic framework and recommended values for the experiment quantification. Of course, users can also adaptively modify the setting parameter templates of the experiment quantification according to actual needs to meet specific experimental purposes.

[0088] After obtaining the setting parameter templates corresponding to the experiment quantification, the system automatically combines each parameter group of the experiment variables in the experiment elements with the setting parameter templates corresponding to the experiment quantification. In this way, multiple advertising delivery strategies with differences can be quickly generated. Each combination forms a complete advertising delivery strategy, and each advertising delivery strategy includes a parameter group of each experiment variable and the setting parameter templates of the experiment quantification.

[0089] A2, create an experimental advertisement according to each advertising delivery strategy, respectively.

[0090] After obtaining multiple advertising delivery strategies, an experimental advertisement is created according to each advertising delivery strategy, respectively.

[0091] Optionally, a new advertising instance is created for each strategy in the advertising experiment system, and the properties of the advertisement are configured according to the setting parameters in the strategy. For example, according to the requirements of the advertising materials in the advertising delivery strategy, the corresponding advertising content such as pictures, videos, and copywriting is made or selected; for example, configure the corresponding delivery settings for each experimental advertisement, including the budget, bidding method, delivery channel, and target audience of the advertisement.

[0092] Optionally, before formal delivery, preview and test each experimental advertisement to ensure that the content, format, and delivery settings of the advertisement meet the requirements and expectations of the advertiser. If problems or adjustments are found, modify and optimize in a timely manner.

[0093] In this implementation, by storing and using the setting parameter template, multiple advertising strategies with differences can be quickly generated without manually setting each parameter one by one, not only reducing the time and error risk of manual configuration, but also improving the diversity and flexibility of the advertising strategies, greatly improving the efficiency and accuracy of creating experimental advertisements. In addition, the recommended values in the template are based on historical data or best practices, which helps to reduce the blindness and trial-and-error cost in the experiment.

[0094] S202, according to the preset traffic allocation strategy, determine the first advertising strategy corresponding to the first traffic request in the advertising strategies corresponding to each experimental advertisement, the first traffic request is used to trigger the advertising of the advertisement.

[0095] The traffic allocation strategy can be understood as a rule or method used to decide how to allocate traffic to different advertising strategies in the advertising experiment. It ensures the fairness and effectiveness of the experiment, and by reasonably allocating traffic, it can make each experimental advertisement corresponding to the advertising strategy have the opportunity to obtain sufficient exposure and testing.

[0096] The traffic request can be understood as a request issued by the system to display an advertisement when a user accesses a webpage or uses an application. In the advertising experiment process, the traffic request is the trigger point of advertising, and whenever a user's behavior triggers the conditions for advertising display (such as opening a new page, scrolling to a specific position, clicking a button, etc.), the advertising experiment system will receive the traffic request, and then trigger the advertising corresponding to the traffic request.

[0097] After receiving the first traffic request, the advertising experiment system will comprehensively analyze the first traffic request and the advertising strategies corresponding to each experimental advertisement to determine which advertising strategy the first traffic request should be allocated to according to the preset traffic allocation strategy, and which advertising strategy should be used to advertise the experimental advertisement corresponding to it.

[0098] Optionally, the traffic allocation strategy can include user feature matching, experimental condition judgment, random or deterministic selection, and other multi-dimensional content. For example, the system may analyze the user features corresponding to the first traffic request, such as geographic location, age, gender, interest preferences, etc., in order to find the most matched advertising strategy for the user. For example, according to the experimental design, the system will judge whether the current traffic request meets the conditions of a certain specific experiment (such as a specific time window, a specific page access, etc.). For example, according to the specific design of the traffic allocation strategy, the system may use random selection or deterministic selection to determine the first advertising strategy, where random selection is usually used to ensure the randomness and fairness of the experiment, and deterministic selection may be based on specific rules or algorithms.

[0099] As an optional implementation manner, the traffic allocation strategy comprises at least one of a hierarchical traffic orthogonal strategy, a same-layer traffic mutual exclusion strategy, and a traffic recycling strategy, which will be described in subsequent content and will not be described in detail here.

[0100] After comprehensive analysis and evaluation based on the traffic allocation strategy, the system selects an advertising strategy that best matches the first traffic request as the first advertising strategy, which determines which experimental advertisement is displayed to the user triggering the first traffic request.

[0101] Optionally, after determining the first advertising strategy, the system records the decision-making process and results, so that subsequent analysis of experimental effects can trace how each traffic request is allocated, which helps to ensure the transparency and repeatability of the experiment.

[0102] S203, the first experimental advertisement corresponding to the first advertising strategy is put into the market, and the experimental advertisement data corresponding to the first traffic request is collected, which represents the advertising effect.

[0103] After determining the first advertising strategy, the system will put the first experimental advertisement corresponding to the first advertising strategy into the market.

[0104] Optionally, putting the first experimental advertisement into the market can be understood as sending the first experimental advertisement to the media that generates the first traffic request. After putting the first experimental advertisement into the market, the bidding strategy in the first advertising strategy is automatically participated in the advertising bidding of the media. If the bidding is successful, that is, the bid is higher than other competitors and meets the display requirements of the first traffic request, the first experimental advertisement will be displayed to the user who generates the first traffic request.

[0105] After putting the first experimental advertisement into the market, the system will collect data related to the advertising effect in real time. These data may include but are not limited to the number of exposures, the number of clicks, the click-through rate, the conversion rate, and other key indicators of the advertisement. The collected experimental advertisement data will be used for subsequent advertising effect analysis and evaluation. By comparing the experimental effects under different advertising strategies, it can be identified which strategies are more effective and which strategies need to be improved. At the same time, data analysis can also help understand the behavior habits and preferences of target audiences, providing strong support for future advertising strategies.

[0106] Optionally, when the experimental elements of the advertising experiment include an experimental period and / or monitoring indicators, the advertising effect is monitored and data is collected in real time according to the preset experimental period and monitoring indicators.

[0107] The advertisement experiment method provided in the embodiments of the present application first creates a plurality of experimental advertisements according to experimental elements of advertisement launching experiments, the experimental elements include at least one experimental variable and a plurality of parameter groups of each experimental variable, and the advertisement launching strategies corresponding to each experimental advertisement are all different, wherein one advertisement launching strategy includes one parameter group corresponding to each experimental variable; then determines a first advertisement launching strategy corresponding to a first traffic request in the advertisement launching strategies corresponding to each experimental advertisement according to a preset traffic distribution strategy, the first traffic request is used to trigger the launching of an advertisement; finally launches a first experimental advertisement corresponding to the first advertisement launching strategy and collects experimental advertisement launching data corresponding to the first traffic request, the experimental advertisement launching data represents an advertisement effect.

[0108] The advertisement experiment method provided in the embodiments of the present application can automatically create a plurality of experimental advertisements according to experimental elements of advertisement launching experiments and automatically launch advertisements according to a preset traffic distribution strategy, thereby significantly improving the efficiency of experiments and improving the accuracy of experiments due to the reduction of manual intervention; on the other hand, by designing a suitable traffic distribution strategy, it can ensure that each experimental advertisement can obtain relatively balanced traffic, thereby avoiding the deviation of experimental results caused by uneven traffic distribution and ensuring the fairness and comparability of experimental results; on the other hand, by collecting experimental advertisement launching data, it can provide effect data of different experimental advertisements for operators and advertisers, so that they can more accurately evaluate the effects of different advertisement launching strategies, thereby optimizing the advertisement launching strategy, improving the advertisement effect and reducing the cost.

[0109] As an optional implementation manner, as shown in FIG. 3 The method further includes step S301:

[0110] S301, data analysis is performed on the experimental advertisement launching data corresponding to each traffic request to obtain an analysis result, the analysis result includes an advertisement effect corresponding to each parameter group of a same experimental variable and / or an advertisement effect difference between different parameter groups of the same experimental variable.

[0111] The object of data analysis is the experimental advertisement launching data corresponding to each traffic request, according to the foregoing content, the experimental advertisement launching data can represent an advertisement effect, including but not limited to the number of exposures, the number of clicks, the click rate, the conversion rate and other data of the advertisement.

[0112] The main target of data analysis is to evaluate the advertisement effect corresponding to each parameter group of a same experimental variable and / or the advertisement effect difference between these parameter groups.

[0113] The analysis result of the data analysis will provide detailed insights into the effects of different parameter groups on the advertisement, for example, it may be found that the click rate of an advertisement of a certain color is higher, while the conversion rate of an advertisement of another color is higher. The analysis result will serve as an important basis for subsequent advertisement optimization strategies, for example, the color, layout, copy, etc. of the advertisement can be adjusted according to the analysis result to improve the effect of the advertisement.

[0114] Optionally, when the experimental elements of the advertisement launching experiment include the advertisement effect determination index, the data analysis is performed based on the advertisement effect determination index as the core index, and the analysis result uses the related data analysis result (core index value, difference relative value, confidence interval, etc.) of the advertisement effect determination index of different experimental advertisements to represent the respective advertisement effects of different parameter groups of the same experimental variable and / or the advertisement effect differences of different parameter groups of the same experimental variable.

[0115] Optionally, the analysis result of the data analysis further includes a final experimental conclusion determined based on the respective advertisement effects of different parameter groups of the same experimental variable and / or the advertisement effect differences of different parameter groups of the same experimental variable. For example, the experimental conclusion is "statistically significant, experimental version 2 is the winning version, and it is recommended to put this configuration online to improve the core index".

[0116] Optionally, data visualization tools can be used to visually display the analysis result, such as bar charts, line charts, scatter plots, etc.

[0117] Specifically, after determining the analysis object and the analysis target, a suitable data analysis method is determined according to the actual scene and demand, and the experimental advertisement launching data corresponding to each traffic request is analyzed to obtain the analysis result. According to the result of the data analysis, the respective advertisement effects of different parameter groups of the same experimental variable and the advertisement effect differences between these parameter groups can be explained, which helps to understand which parameter groups perform better in advertisement launching and why such differences occur.

[0118] The data analysis method is determined according to the actual application scene and demand, which is not limited in the present application.

[0119] Through the traffic division corresponding to the traffic allocation strategy in step S202, the experimental advertisements corresponding to each advertisement launching strategy will obtain the same traffic, but due to the parameter differences of the advertisement launching strategy itself (such as different bidding sub-strategies), the experimental advertisements may not obtain the same exposure. To avoid such sample quantity differences, as an optional implementation manner, the data analysis method can use the Welch's t-test evaluation method.

[0120] Optionally, when the Welch's t-test evaluation method is used to analyze the data, the following steps are followed to ensure the accuracy and reliability of the result.

[0121] First, according to the experimental advertisement delivery data, the mean and variance of each parameter group (such as advertisements of different colors, layouts or copy) are calculated, as well as the respective sample size. These data are the basis for evaluating the differences in advertising effectiveness. Next, the calculated mean, variance and sample size are used to calculate the t-distribution, degrees of freedom and standard error. Finally, based on the t-distribution, a suitable confidence level (e.g. 95%) is selected, the corresponding critical value is found by looking up the table, and the confidence interval is calculated.

[0122] The formula for calculating the t-distribution is

[0123]

[0124] The formula for calculating the degrees of freedom is

[0125]

[0126] The formula for calculating the standard error is

[0127]

[0128] The formula for calculating the confidence interval is

[0129]

[0130] where m A , m B are the means, is the variance, and n A , n B are the sample sizes.

[0131] Finally, we can determine the experimental conclusion based on the evaluation results of Welch's t-test. For example, if the advertising effectiveness of version 2 is statistically significantly better than version 1, we can conclude that "the statistics are significant, version 2 is the winning version, and it is recommended to go online with this configuration to improve the core indicators." Such a conclusion will provide strong data support for the formulation of advertising strategies. In addition, in order to more intuitively display the analysis results, we can also use data visualization tools such as bar charts and line charts to present the differences in advertising effectiveness of different parameter groups in a graphical manner, helping decision-makers quickly understand the meaning behind the data.

[0132] In the field of advertising delivery, evaluation indicators can be roughly divided into the following types:

[0133] Normal distribution indicators, such as cpm (thousand exposure cost), which can be directly calculated by substituting the corresponding mean and variance into the formula.

[0134] Binomial distribution indicators, such as click-through rate and conversion rate, can be represented by 0 or 1 for each sample, naturally conforming to the binomial distribution, so the sample mean and variance can be quickly calculated. Assuming the click-through rate of sample A is P A , then its m A = P A , and

[0135] Skewed indicators, such as conversion cost indicators, do not conform to the normal distribution on each exposure and are likely to fall on high-priced traffic, specific audience traffic, and other indicators. Secondly, most exposures do not bring conversions, and it is impossible to define how much conversion cost is. According to statistical rules, we use the method of extracting samples, such as extracting 5000 exposures as a sample data, which still conforms to the normal distribution, and through repeated sampling to form a sample set to calculate significance.

[0136] The experimental system completes the strict analysis of the experimental group data according to statistical rules, so that the execution personnel can directly and intuitively see whether the experimental group effect has a significant increase or decrease.

[0137] As an optional implementation manner, the experimental elements further include at least one of an experimental budget, an experimental period, and an advertisement effect judgment index.

[0138] The experimental budget can be understood as the total amount of funds allocated for the advertisement delivery experiment. It determines the scale, duration, and delivery intensity of the experiment.

[0139] The experimental period can be understood as the time period from the start to the end of the advertisement delivery experiment. A reasonable experimental period can ensure that the experiment has enough time to collect data, analyze effects, and optimize, so as to obtain reliable experimental conclusions.

[0140] The advertisement effect judgment index is a standard or criterion for measuring the effect of the advertisement delivery experiment. They can help advertisers understand the performance of the advertisement among the target audience and whether the expected target has been achieved. Common advertisement effect judgment indexes include click-through rate (CTR), conversion rate (CVR), exposure, cost-effectiveness ratio (ROI), etc. These indicators can be selected and combined according to the experimental purpose and the needs of the advertiser to comprehensively and accurately evaluate the effect of the advertisement.

[0141] In the advertisement delivery experiment, the experimental budget, the experimental period and the advertisement effect determination index are interrelated and influence each other. The experimental budget determines the scale and duration of the experiment, and further influences the collection and analysis of experimental data; the length of the experimental period directly influences the sufficiency and reliability of experimental data; and the advertisement effect determination index is an important basis for measuring the experimental effect, and their selection and setting need to be carefully considered based on the experimental purpose and the demand of the advertiser. By reasonably setting these experimental elements, the effective implementation of the advertisement delivery experiment can be ensured, and valuable reference and basis can be provided for the advertiser.

[0142] As an optional implementation manner, the traffic distribution strategy includes a hierarchical traffic orthogonal strategy, wherein different experimental variables are located in different traffic layers, and multiple parameter groups of the same experimental variable are located in the same traffic layer.

[0143] As shown in FIG. 4 According to the preset traffic distribution strategy, the first advertisement delivery strategy corresponding to the first traffic request is determined in the advertisement delivery strategy corresponding to each experimental advertisement, including steps S401-S402:

[0144] S401, according to the preset traffic distribution strategy, determining the parameter group matched with the first traffic request in each traffic layer.

[0145] In step S201, multiple experimental advertisements have been created according to the experimental elements of the advertisement delivery experiment, and the experimental elements include at least one experimental variable and multiple parameter groups of each experimental variable. Each experimental advertisement corresponds to a unique advertisement delivery strategy, which is composed of a parameter group corresponding to each experimental variable.

[0146] In step S202, the first advertisement delivery strategy corresponding to the first traffic request needs to be determined according to the preset traffic distribution strategy. An optional traffic distribution strategy is provided in this implementation manner, that is, a hierarchical traffic orthogonal strategy.

[0147] In this strategy, different experimental variables are distributed to different traffic layers, and each traffic layer corresponds to an experimental variable. For example, if there are two experimental variables (such as material and bid) in the experiment, the two variables will be placed in two different traffic layers. In addition, multiple parameter groups of the same experimental variable are located in the same traffic layer, and the parameter groups are orthogonal in traffic distribution, that is, each parameter group has the opportunity to be independently distributed to the traffic without being affected by other parameter groups. This strategy ensures that each traffic request will match a parameter group of each experimental variable when passing through each traffic layer, thereby realizing the orthogonal design of the advertisement delivery strategy.

[0148] Specifically, in step S401, for each traffic layer, we have a preset traffic allocation rule that determines which parameter group will be assigned to the current traffic request. When the first traffic request arrives, the system needs to traverse each traffic layer in turn, and determine a parameter group that matches the first traffic request in each traffic layer according to the preset traffic allocation rule. Finally, the parameter groups that match the first traffic request in each traffic layer will be determined.

[0149] As an optional implementation, a hash algorithm is used to determine the parameter group that matches the first traffic request in each traffic layer.

[0150] A hash algorithm is an algorithm that maps input data (such as strings, numbers, etc.) to a fixed-size output value (i.e., a hash value). In the context of traffic allocation, we can use the properties of hash algorithms to generate a hash value from certain attributes of the first traffic request (such as user ID, session ID, timestamp, etc.), and then use this hash value to determine which parameter group to match.

[0151] First, a suitable hash function needs to be selected, which should have good hash performance, i.e., it should be able to map different inputs to different hash values as much as possible to reduce the possibility of hash collisions. Second, for each traffic layer, the relevant attributes of the first traffic request are extracted and used as inputs to the hash function to calculate the corresponding hash value. Next, the calculated hash value is mapped to the preset parameter group, which can be achieved by taking the modulus operation or looking up the table. If necessary, the mapping relationship between the identification information of the first traffic request and the determined parameter group can also be stored in the mapping relationship database for subsequent use or analysis.

[0152] In this implementation, on the one hand, the computational complexity of the hash algorithm is relatively low, which can quickly process a large number of traffic requests, on the other hand, by reasonably designing the hash function and mapping strategy, it can ensure that the traffic is evenly distributed among the parameter groups, thereby avoiding the situation that some parameter groups are overloaded while others are idle, and on the other hand, when the number of experimental variables or parameter groups increases, only the hash function and mapping strategy need to be adjusted accordingly, without the need to make major changes to the entire traffic allocation model.

[0153] S402, combine the parameter groups that match the first traffic request in each traffic layer to obtain the first ad placement strategy corresponding to the first traffic request.

[0154] After determining the parameter groups that match the first traffic request in each traffic layer, the system combines these parameter groups to form a complete ad placement strategy, which is the first ad placement strategy corresponding to the first traffic request.

[0155] The first ad serving strategy determines which experimental ad should be served to the current traffic request under the first traffic request, and the parameters of the experimental ad (such as ad copy, bidding strategy, targeting condition, etc.).

[0156] This optional implementation realizes fine-grained control of ad serving strategies through hierarchical traffic orthogonal strategies. It ensures that each traffic request can be matched with each parameter group of experimental variables, thereby obtaining a comprehensive and orthogonal ad serving strategy. This strategy helps to more accurately evaluate the impact of different parameter groups on ad effectiveness, providing strong support for subsequent ad optimization. Through hierarchical traffic orthogonal strategies, each experimental variable and parameter group can be independently evaluated, thereby more accurately understanding their specific contributions to ad effectiveness. This method helps us more effectively optimize ads and improve ad delivery effectiveness.

[0157] As an optional implementation, the experimental elements also include the application traffic of each parameter group in each traffic layer.

[0158] Application traffic can be understood as the proportion or quantity of traffic that each parameter group expects to obtain in an ad serving experiment, reflecting the experimenter's expectations and preferences for the effectiveness of each parameter group, and is one of the important bases for traffic allocation strategies. In the hierarchical traffic orthogonal strategy, application traffic is an important part of the traffic allocation strategy. Each parameter group in each traffic layer will obtain the corresponding traffic proportion according to its application traffic. By reasonably setting the application traffic and using effective traffic allocation strategies (such as hash algorithms), each parameter group can be independently evaluated, thereby more accurately understanding their specific contributions to ad effectiveness.

[0159] As an optional implementation, the method further includes storing the mapping relationship between the historical traffic request corresponding ad serving strategy and the identification information of the historical traffic request into a mapping relationship library.

[0160] As shown in FIG. 5 According to the preset traffic allocation strategy, the first traffic request corresponds to the first ad serving strategy in the ad serving strategy corresponding to each experimental ad, including step S501:

[0161] S501, using the identification information of the first traffic request to search in the mapping relationship library, when the target ad serving strategy corresponding to the identification information of the first traffic request is stored in the mapping relationship library, the target ad serving strategy is determined as the first ad serving strategy.

[0162] A mapping relationship library is pre-established to store the mapping relationship between historical traffic requests and corresponding advertising strategies. This library can be a database, a cache system, or any data structure that can efficiently store and retrieve data.

[0163] Whenever a traffic request arrives, the system stores the identification information of the traffic request (such as user ID, session ID, timestamp, etc.) and the corresponding advertising strategy (including a parameter set for each experimental variable) in the mapping relationship library. Over time, new traffic requests and corresponding advertising strategies are continuously added to the mapping relationship library.

[0164] When the first traffic request arrives, the system first extracts the identification information of the first traffic request and uses this information to search in the mapping relationship library. In the mapping relationship library, the system searches for whether there is a target advertising strategy that has a mapping relationship with the identification information of the first traffic request. If such a target advertising strategy exists, the system directly determines it as the first advertising strategy and performs advertising.

[0165] If there is no mapping relationship, the system may need to dynamically determine the first advertising strategy according to the pre-set traffic allocation strategy (such as the hierarchical traffic orthogonal strategy) and store it in the mapping relationship library together with the identification information of the first traffic request for subsequent use.

[0166] In this implementation, on the one hand, by storing and retrieving the mapping relationship, the system can more quickly determine the corresponding advertising strategy for the traffic request, thereby improving the efficiency of advertising; on the other hand, for the same traffic request, the system always returns the same advertising strategy, ensuring the consistency and stability of advertising; on the other hand, the stored mapping relationship can be used as historical records for subsequent data analysis, problem troubleshooting, and auditing.

[0167] In the AB testing scenario in the field of advertising, due to the strict bidding requirements of advertising for traffic and the limited traffic resources, it is necessary to efficiently utilize each piece of traffic. When a piece of traffic does not meet the requirements of either A experimental advertising or B experimental advertising, in order to avoid waste of traffic, the system will use other non-experimental advertising to fill this piece of traffic, which is called traffic recycling.

[0168] In the process of programmatic advertising, the calculation time window of advertising recall and filling is very short. If the system first tries to recall experimental advertising and then recalls non-experimental advertising for filling after the recall fails, the entire processing process may fail due to timeout, resulting in the waste of this piece of traffic.

[0169] To avoid this situation, the present application proposes another optional advertising experiment method, as shown in FIG. 4. FIG. 6As shown, the method further includes steps S601-S602:

[0170] S601, according to the attribute information of the first traffic request, determine a first ad set in the current ad set of the ad experiment system, the first traffic request meets the ad serving strategy of each ad in the first ad set, and the current ad set includes experimental ads and / or non-experimental ads.

[0171] The system first determines a first ad set in the current ad set of the ad experiment system according to the attribute information of the first traffic request. The first ad set contains all ads whose ad serving strategies are met by the first traffic request, whether they are experimental ads or non-experimental ads.

[0172] Among them, the attribute information of the first traffic request can be understood as a series of characteristics and parameters related to the traffic request, which are usually used to describe the characteristics, source, user behavior and possible delivery target of the traffic request. The attribute information of the first traffic request may include but not limited to user information (such as user ID, user portrait, user device information, geographic location, etc.), request information (such as request time, request source, request type, etc.), context information (such as current page information, browsing history, pre-sequence behavior, etc.), delivery target (such as ad position information, bidding requirements, etc.).

[0173] The current ad set is the set of all available ads in the ad experiment system, including experimental ads being tested (such as A version and B version in AB test) and regular non-experimental ads. These ads may be targeted at different user groups and ad serving strategies. Experimental ads are usually versions of ads being tested for comparing the effects of different ad strategies. Non-experimental ads are regular ads for delivery, not for experimental comparison. In the first ad set, there may be both experimental ads and non-experimental ads, depending on the attribute information of the traffic request and the ad serving strategy.

[0174] The first ad set contains all ads that may be suitable for the current traffic request. The screening process usually involves matching the attribute information of the traffic request with the ad serving strategy.

[0175] S602, when the first ad set does not include the first experimental ad corresponding to the first ad serving strategy, select a non-experimental ad from the non-experimental ads in the first ad set for delivery.

[0176] The system checks whether the first ad set contains the first experimental ad corresponding to the first ad serving strategy.

[0177] If the first ad set contains the first experimental ad, the system will select this ad for delivery.

[0178] If the first experimental ad corresponding to the first ad serving strategy is not included in the first ad set, it means that the attribute information of the current traffic request does not match the serving strategy of the first experimental ad, or the first experimental ad is currently unavailable due to certain reasons (such as budget restrictions, serving time restrictions, etc.). Therefore, the system will select one of the non-experimental ads in the first ad set for serving.

[0179] The selection of a non-experimental ad from the non-experimental ads in the first ad set is usually based on a series of algorithms and strategies, aiming to maximize the effectiveness of the ad (such as click-through rate, conversion rate, etc.) and user experience. For example, the system may consider various factors such as the relevance of the ad, the user's interest preferences, the ad's bid, etc. By considering these factors comprehensively, the system can display the ad that best meets the user's needs and interests.

[0180] Step S602 ensures that when the first experimental ad is not included in the first ad set, the system can quickly and accurately select one of the remaining non-experimental ads for serving. This mechanism not only guarantees the effective use of traffic, but also ensures the continuity and consistency of user experience. At the same time, by continuously optimizing the serving strategy of non-experimental ads, the effectiveness of the ad and the return on investment of the advertiser can be further improved.

[0181] This implementation, on the one hand, by considering both experimental ads and non-experimental ads, the system can complete the recall and filling of ads within a time window, avoiding the waste of traffic due to timeout; on the other hand, it can flexibly select the most suitable ad for serving according to the attribute information of the traffic request and the actual situation of the current ad set. Since the system can find the most suitable ad for each piece of traffic in a short time, it can improve the user's ad experience and reduce user churn due to irrelevant or slow-loading ads.

[0182] It should be noted that there can be multiple ad serving experiments in the ad experiment system. Optionally, considering that different experiments cannot be cross-tested, the concept of traffic domain is introduced.

[0183] As an optional implementation, a unique traffic domain is allocated to each advertising experiment, and the traffic domains are allocated in mutual exclusion. The entire traffic is divided into a plurality of experimental traffic domains and a common traffic domain. The common traffic domain is directly used to recall non-experimental advertisements. Each traffic domain is divided into a plurality of traffic layers according to the experimental variables of the advertising experiment corresponding to the traffic domain. Each traffic layer corresponds to an experimental variable of the advertising experiment. Each traffic bucket in the traffic layer corresponds to a parameter group of the experimental variable. For each parameter group in each traffic layer in each traffic domain, the parameter groups are allocated in mutual exclusion. For the parameter groups of different experimental variables of the same advertising experiment, a hierarchical traffic orthogonal allocation strategy is used. Specifically, for each traffic request, a first hash operation is performed according to the device number identification information to determine the target traffic domain of the traffic, and then a second hash operation is performed according to the device number identification information to determine the target traffic bucket of the second traffic layer of the target traffic domain to which the traffic falls. The step is continued until the target traffic bucket of each traffic layer in the target traffic domain is determined.

[0184] Exemplary apparatus

[0185] Corresponding to the advertising experiment method described above, the embodiment of the application further provides an advertising experiment device. FIG. 7 is a structural schematic diagram of an advertising experiment device provided by the embodiment of the application, as FIG. 7 shown, the advertising experiment device provided by the embodiment of the application comprises:

[0186] The first unit 701 is configured to create a plurality of experimental advertisements according to experimental elements of an advertising experiment, the experimental elements comprising at least one experimental variable and a plurality of parameter groups of each experimental variable, and the advertising strategies corresponding to each experimental advertisement being different from each other, wherein one advertising strategy comprises one parameter group corresponding to each experimental variable.

[0187] The second unit 702 is configured to determine a first advertising strategy corresponding to a first traffic request in the advertising strategies corresponding to each experimental advertisement according to a preset traffic allocation strategy, the first traffic request being used to trigger the advertising.

[0188] The third unit 703 is configured to advertise the first experimental advertisement corresponding to the first advertising strategy, and collect experimental advertising data corresponding to the first traffic request, the experimental advertising data representing the advertising effect.

[0189] The advertisement experiment device provided by the embodiments of the present application can automatically create multiple experimental advertisements according to the experimental elements of advertisement launching experiments, and automatically launch the advertisements according to a preset traffic distribution strategy, thereby significantly improving the efficiency of the experiments, and improving the accuracy of the experiments due to the reduction of manual intervention. On the other hand, by designing a suitable traffic distribution strategy, it can be ensured that each experimental advertisement can obtain relatively balanced traffic, thereby avoiding the experimental result deviation caused by uneven traffic distribution, and ensuring the fairness and comparability of the experimental results. On the other hand, by collecting experimental advertisement launching data, the effect data of different experimental advertisements can be provided for the operators and advertisers, so that they can more accurately evaluate the effect of different advertisement launching strategies, thereby optimizing the advertisement launching strategy, improving the advertisement effect and reducing the cost.

[0190] Optionally, the apparatus further comprises:

[0191] The fourth unit is configured to perform data analysis on the experimental advertisement launching data corresponding to each traffic request, and obtain an analysis result, wherein the analysis result comprises the advertisement effect corresponding to each parameter group of a same experimental variable and / or the advertisement effect difference of different parameter groups of the same experimental variable.

[0192] Optionally, the traffic distribution strategy comprises a hierarchical traffic orthogonal strategy, wherein different experimental variables are located in different traffic layers, and multiple parameter groups of a same experimental variable are located in a same traffic layer.

[0193] The second unit 702 can be specifically configured to:

[0194] According to the preset traffic distribution strategy, determine the parameter group in each traffic layer that matches the first traffic request;

[0195] Combine the parameter groups in each traffic layer that match the first traffic request, to obtain a first advertisement launching strategy corresponding to the first traffic request.

[0196] Optionally, the apparatus further comprises:

[0197] The fifth unit is configured to store a mapping relationship between the advertisement launching strategy corresponding to a historical traffic request and the identification information of the historical traffic request into a mapping relationship database.

[0198] The second unit 702 can be specifically configured to:

[0199] Use the identification information of the first traffic request to search in the mapping relationship database, and when the mapping relationship database stores a target advertisement launching strategy that has a mapping relationship with the identification information of the first traffic request, determine the target advertisement launching strategy as the first advertisement launching strategy.

[0200] Optionally, the apparatus further comprises:

[0201] A sixth unit configured to determine a first advertisement set from a current advertisement set of the advertisement experiment system according to attribute information of the first traffic request, wherein the first traffic request satisfies an advertisement delivery strategy of each advertisement in the first advertisement set, and the current advertisement set comprises experimental advertisements and / or non-experimental advertisements; and when the first experimental advertisement corresponding to the first advertisement delivery strategy is not included in the first advertisement set, select a non-experimental advertisement from non-experimental advertisements in the first advertisement set for delivery.

[0202] Optionally, the advertisement experiment system stores a setting parameter template of each experimental quantitative in the advertisement delivery strategy.

[0203] The first unit 701 can be specifically configured to:

[0204] Combine the multiple parameter groups of each experimental variable in the experimental element with the setting parameter template corresponding to the experimental quantitative to obtain multiple advertisement delivery strategies.

[0205] Create an experimental advertisement according to each advertisement delivery strategy.

[0206] Optionally, the experimental element further comprises at least one of an experimental budget, an experimental period, and an advertisement effect determination index.

[0207] Optionally, the experimental element further comprises application traffic of each parameter group in each traffic layer.

[0208] The advertisement experiment apparatus provided in the embodiment belongs to the same application concept as the advertisement experiment method provided in the embodiments of the present application, can execute the advertisement experiment method provided in any of the embodiments of the present application, has the corresponding function modules and beneficial effects of executing the advertisement experiment method. Technical details not described in the embodiment can refer to the specific processing content of the advertisement experiment method provided in the embodiments of the present application, which will not be described here.

[0209] The functions implemented by the first unit 701, the second unit 702, and the third unit 703 can be implemented by the same or different processors, and the embodiments of the present application are not limited.

[0210] It should be understood that the units in the above apparatus can be implemented in the form of processor calling software. For example, the apparatus includes a processor connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the apparatus, wherein the processor can be a general processor such as CPU or microprocessor, and the memory can be an internal memory or an external memory of the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units can be realized by the design of the hardware circuit, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is ASIC, and the functions of part or all of the units are realized by the design of the logical relationship of the elements in the circuit. For another example, in another implementation, the hardware circuit can be realized by PLD, and taking FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the units. All the units of the above apparatus can be realized in the form of processor calling software, or realized in the form of hardware circuit, or part of them is realized in the form of processor calling software, and the remaining part is realized in the form of hardware circuit.

[0211] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as CPU, microprocessor, GPU, or DSP, etc. In another implementation, the processor can realize certain functions through the logical relationship of hardware circuit, which is fixed or can be reconfigured, such as ASIC or PLD implemented hardware circuit, such as FPGA, etc. In the reconfigurable hardware circuit, the process of the processor loading configuration document to realize hardware circuit configuration can be understood as the process of the processor loading instructions to realize the functions of part or all of the units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a kind of ASIC, such as NPU, TPU, DPU, etc.

[0212] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above method, such as CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0213] In addition, all or part of each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, the units are integrated together to be implemented in the form of a SOC. The SOC can include at least one processor for implementing the functions of any of the above methods or implementing the functions of each unit of the apparatus. The at least one processor can be of different types, such as including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0214] Exemplary electronic device

[0215] The embodiments of the present application also propose an electronic device, as shown in FIG. 8 The device includes:

[0216] a memory 200 and a processor 210;

[0217] The memory 200 is connected with the processor 210, and is configured to store a program.

[0218] The processor 210 is configured to realize the advertisement experiment method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0219] Specifically, the electronic device can further include a bus, a communication interface 220, an input device 230, and an output device 240.

[0220] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are connected with each other through the bus. Among them:

[0221] The bus can include a path for transmitting information between various components of a computer system.

[0222] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0223] The processor 210 can include a main processor, and can also include a baseband chip, a modem, etc.

[0224] The memory 200 stores programs for implementing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, and the like.

[0225] The input device 230 can include devices that receive data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, a gravity sensor, and the like.

[0226] The output device 240 can include devices that allow information to be output to a user, such as a display screen, a printer, a speaker, and the like.

[0227] The communication interface 220 can include devices of the transceiver type or the like to communicate with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0228] The processor 210 executes programs stored in the memory 200 and calls other devices, which can be used to implement each step of any of the advertisement experiment methods provided by the above-described embodiments.

[0229] The embodiments of the present application also propose a chip including a processor and a data interface, the processor reads and runs programs stored on the memory through the data interface to execute the advertisement experiment methods introduced in any of the above-described embodiments, and the specific processing process and its beneficial effects can be referred to the above-described embodiment introduction of the advertisement experiment method.

[0230] Exemplary computer program product and storage medium

[0231] In addition to the above methods and devices, the embodiments of the present application can also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the advertisement experiment methods according to various embodiments of the present application described in any of the above-described embodiments.

[0232] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0233] In addition, the embodiments of the present application can also be storage media, which store computer programs, and the computer programs are executed by processors to perform the steps of the advertisement experiment method according to various embodiments of the present application described in any embodiment of the present specification. The following steps can be implemented:

[0234] S201, creating a plurality of experimental advertisements according to experimental elements of advertisement launching experiments, the experimental elements including at least one experimental variable and a plurality of parameter groups of each experimental variable, and each experimental advertisement corresponding to different advertisement launching strategies, wherein one of the advertisement launching strategies includes one parameter group corresponding to each experimental variable.

[0235] S202, determining a first advertisement launching strategy corresponding to a first traffic request in the advertisement launching strategies corresponding to each experimental advertisement according to a preset traffic distribution strategy, the first traffic request being used to trigger the launching of an advertisement.

[0236] S203, launching a first experimental advertisement corresponding to the first advertisement launching strategy, and collecting experimental advertisement launching data corresponding to the first traffic request, the experimental advertisement launching data representing an advertisement effect.

[0237] For the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action order described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0238] It should be noted that each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment can be referred to. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0239] The steps in the methods of the embodiments of the present application can be adjusted in sequence, combined, and deleted according to actual needs, and the technical features described in the embodiments can be replaced or combined.

[0240] The modules and sub-modules in the devices and terminals in the embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0241] In several embodiments provided by the present application, it should be understood that the disclosed terminal, device and method can be implemented by other ways. For example, the terminal embodiments described above are only schematic, for example, the division of the modules or sub-modules is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0242] The modules or sub-modules described as separate components can or can not be physically separate, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, that is, they can be located in one place or distributed on a plurality of network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0243] In addition, each functional module or sub-module in the embodiments of the present application can be integrated in one processing module, or each module or sub-module can exist physically, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or software functional module or sub-module.

[0244] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0245] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software executed by a processor, or in a combination of the two. A software unit can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM or EEPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The storage medium can be loaded into the execution system by a manufacturer, a seller, or a user of an electronic system.

[0246] Finally, it is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It is to be understood that the use of relational terms such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Additionally, it is to be understood that the use of the term "including", "comprising", or "having" the described elements does not limit the application to the listed elements, but rather, is intended to cover the description of stated elements or integers or groups thereof, without requiring the presence of other elements or integers.

[0247] The above description of disclosed embodiments provides enough information to enable those skilled in the art to make and use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Accordingly, the application is not to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An advertising experiment method characterized by, The method is applied to an advertisement experiment system, and the method comprises the following steps: According to an experiment element of an advertisement delivery experiment, a plurality of experimental advertisements are created, the experiment element comprises at least one experiment variable and a plurality of parameter groups of each experiment variable, and an advertisement delivery strategy corresponding to each experimental advertisement is different, wherein the advertisement delivery strategy comprises one parameter group corresponding to each experiment variable; According to a preset traffic distribution strategy, a first advertisement delivery strategy corresponding to a first traffic request is determined from the advertisement delivery strategies corresponding to the experimental advertisements, the first traffic request is used to trigger the delivery of an advertisement, wherein the traffic request is a request sent by a system in order to display an advertisement when a user accesses a webpage or uses an application; and the traffic distribution strategy comprises at least one dimension of content such as random or deterministic selection, user feature matching and experiment condition judgment; The first experimental advertisement corresponding to the first advertisement delivery strategy is delivered, and experimental advertisement delivery data corresponding to the first traffic request is collected, wherein the experimental advertisement delivery data represents an advertisement effect.

2. The method of claim 1, wherein, The method further comprises the following steps: Data analysis is performed on the experimental advertisement delivery data corresponding to each traffic request, and an analysis result is obtained, wherein the analysis result comprises an advertisement effect corresponding to each parameter group of a same experiment variable and / or an advertisement effect difference of different parameter groups of the same experiment variable.

3. The method of claim 1, wherein, The traffic distribution strategy comprises a hierarchical traffic orthogonal strategy, wherein different experiment variables are located in different traffic layers, and a plurality of parameter groups of a same experiment variable are located in a same traffic layer; According to the preset traffic distribution strategy, the first advertisement delivery strategy corresponding to the first traffic request is determined from the advertisement delivery strategies corresponding to the experimental advertisements, comprising the following steps: According to the preset traffic distribution strategy, a parameter group matched with the first traffic request in each traffic layer is determined; The parameter groups matched with the first traffic request in each traffic layer are combined to obtain the first advertisement delivery strategy corresponding to the first traffic request.

4. The method of claim 1, wherein, The method further comprises the following step: A mapping relationship between an advertisement delivery strategy corresponding to a historical traffic request and identification information of the historical traffic request is stored in a mapping relationship database; According to the preset traffic distribution strategy, the first advertisement delivery strategy corresponding to the first traffic request is determined from the advertisement delivery strategies corresponding to the experimental advertisements, comprising the following steps:

5. The method of claim 1, wherein, The identification information of the first traffic request is used to search in the mapping relationship database, and when it is determined that a target advertisement delivery strategy having a mapping relationship with the identification information of the first traffic request is stored in the mapping relationship database, the target advertisement delivery strategy is determined as the first advertisement delivery strategy. The method further comprises the following steps: According to attribute information of the first traffic request, a first advertisement set is determined from a current advertisement set of the advertisement experiment system, the first traffic request satisfies an advertisement delivery strategy of each advertisement in the first advertisement set, and the current advertisement set comprises experimental advertisements and / or non-experimental advertisements. determining that the first experimental advertisement corresponding to the first advertisement delivery strategy is not included in the first advertisement set, selecting one non-experimental advertisement in the non-experimental advertisements of the first advertisement set for delivery.

6. The method of claim 1, wherein, The advertisement experiment system stores a setting parameter template of each experimental quantitative in the advertisement delivery strategy; According to the experimental elements of the advertisement delivery experiment, a plurality of experimental advertisements are created, including: Combining a plurality of parameter groups of each experimental variable in the experimental elements with the setting parameter template corresponding to the experimental quantitative of the advertisement delivery experiment, obtaining a plurality of advertisement delivery strategies; According to each advertisement delivery strategy, an experimental advertisement is created.

7. The method of claim 1, wherein, The experimental elements further include at least one of an experimental budget, an experimental period, and an advertisement effect determination index.

8. The method of claim 3, wherein, The experimental elements further include the application traffic of each parameter group in each traffic layer.

9. An advertising experiment apparatus characterized by comprising: Including: The first unit is configured to create a plurality of experimental advertisements according to experimental elements of an advertisement delivery experiment, the experimental elements including at least one experimental variable and a plurality of parameter groups of each experimental variable, and each experimental advertisement corresponding to an advertisement delivery strategy is different, wherein one of the advertisement delivery strategies includes one parameter group corresponding to each experimental variable; The second unit is configured to determine a first advertisement delivery strategy corresponding to a first traffic request in the advertisement delivery strategies corresponding to each experimental advertisement according to a preset traffic distribution strategy, the first traffic request being used to trigger the delivery of an advertisement, wherein the traffic request is a request sent by a system to display an advertisement when a user accesses a webpage or uses an application; and the traffic distribution strategy includes at least one dimension of content in random or deterministic selection, user feature matching, and experimental condition judgment; The third unit is configured to deliver a first experimental advertisement corresponding to the first advertisement delivery strategy and collect experimental advertisement delivery data corresponding to the first traffic request, the experimental advertisement delivery data representing an advertisement effect.

10. An electronic device, comprising: Including a memory and a processor; The memory is connected with the processor and is configured to store programs; The processor is configured to realize the advertisement experiment method in any one of claims 1-8 by running the programs in the memory.

11. A computer program product, characterised in that, Including computer program instructions, which, when executed by a processor, cause the processor to realize the advertisement experiment method in any one of claims 1-8.

12. An advertising experimentation system characterized by, Including: A server and a client, wherein The client is configured to obtain user input information and send the user input information to the server, the user input information including experimental elements of an advertisement delivery experiment, the experimental elements including at least one experimental variable and a plurality of parameter groups of each experimental variable, the experimental variable corresponding to one sub-strategy in the advertisement delivery strategy; The server is configured to realize the advertisement experiment method in any one of claims 1-8.

Images