Experiment strategy evaluation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, in particular to the technical field of artificial intelligence, and provides an experimental strategy evaluation method and device, electronic equipment and a storage medium, which are used for improving the accuracy of experimental strategy evaluation. The method comprises the following steps: acquiring each experiment browsing index corresponding to a control group in an experiment period, and acquiring each experiment browsing index corresponding to a test group after executing an exposure experiment strategy of multimedia content; obtaining reference browsing indexes corresponding to the control group and the test group in the non-experimental period; based on the experiment browsing indexes and the reference browsing indexes, respectively obtaining experiment index differences and reference index differences between the control group and the test group in each time unit; and obtaining a strategy evaluation result based on a difference evaluation value between each experiment index difference and each reference index difference. The original reference index difference between the control group and the test group is also added into the difference comparison, so that the influence of the difference on the final evaluation result is eliminated, and the evaluation accuracy is improved.

Description

Background Art

[0002] At present, Internet technology is developing rapidly, and various Internet products are emerging one after another. In order to retain the users of Internet products and promote their enthusiasm, Internet products need to continuously launch new strategies, such as updating the style of an interactive button, updating the recommended method of information, etc. If it can be proved that the new strategy can indeed bring significant beneficial effects, then the new strategy will be promoted to all users. Therefore, how to judge whether the new strategy can bring significant beneficial effects has become an urgent problem to be solved.

[0003] In related technologies, a part of the objects are usually selected as sample objects, and the sample objects are further divided into a control group and a test group for experiments, and an observation indicator is set based on the experimental scenario; then the new strategy is used for the test group, while the new strategy is not used for the control group; finally, the effect of the new strategy is evaluated by comparing the observation indicators of each sample object in the control group and the test group.

[0004] However, the above observation indicators are based on object-granular data, and experiments often require multiple days of data and cumulative calculations, which results in a large amount of calculations and low experimental efficiency.

[0005] On the other hand, if there are original differences in observed indicators between the test group and the control group before the new strategy is applied, the above method alone cannot eliminate this original difference, which may lead to inaccurate evaluation of the effectiveness of the new strategy. Summary of the invention

[0006] The embodiments of the present application provide an experimental strategy evaluation method, device, electronic device and storage medium to improve the accuracy of evaluating the experimental strategy.

[0007] An evaluation method for an experimental strategy provided in an embodiment of the present application includes:

[0008] For the experimental period, obtaining the experimental browsing index of each sample object in the control group, and obtaining the experimental browsing index of each sample object in the test group after executing the exposure experimental strategy of the multimedia content, wherein the experimental period is the first preset time period after the exposure experimental strategy is applied to the test group;

[0009] For the non-experimental period, obtaining the reference browsing index of each sample object in the control group and the test group; the non-experimental period is a second preset time period before applying the exposure experimental strategy to the test group;

[0010] Based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group, respectively obtaining the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period;

[0011] Based on the difference evaluation values ​​obtained between the differences of each experimental indicator and the differences of each reference indicator, a strategy evaluation result of the exposure experiment strategy is obtained.

[0012] An experimental strategy evaluation device provided in an embodiment of the present application includes:

[0013] A first indicator acquisition unit is used to acquire, for the experimental period, an experimental browsing indicator of each sample object in the control group, and, after the exposure experimental strategy of the multimedia content is executed, an experimental browsing indicator of each sample object in the test group, wherein the experimental period is a first preset time period after the exposure experimental strategy is applied to the test group;

[0014] A second index acquisition unit is used to acquire a reference browsing index of each sample object in the control group and the test group for a non-experimental period; the non-experimental period is a second preset time period before applying the exposure experimental strategy to the test group;

[0015] A difference obtaining unit, configured to obtain, based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group, the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period;

[0016] An evaluation acquisition unit is used to obtain a strategy evaluation result of the exposure experiment strategy based on the difference evaluation values ​​between the obtained experimental indicator differences and the reference indicator differences.

[0017] Optionally, the difference acquisition unit is specifically used to acquire the experimental indicator difference in the following manner:

[0018] Based on the difference between the comprehensive value of each experimental index corresponding to the test group and the comprehensive value of each experimental index corresponding to the control group, the difference of the experimental index between the control group and the test group in each time unit included in the experimental period is obtained; wherein the comprehensive value of the experimental index is obtained based on the average value of each experimental browsing index corresponding to the control group or the test group in a time unit in the experimental period;

[0019] The difference acquisition unit is specifically used to acquire the reference indicator difference in the following manner:

[0020] Based on the difference between the comprehensive value of each reference indicator corresponding to the test group and the comprehensive value of each reference indicator corresponding to the control group, the difference in reference indicators between the control group and the test group in each time unit included in the non-experimental period is obtained; wherein, the comprehensive value of reference indicators is obtained based on the mean value of each reference browsing indicator corresponding to the control group or the test group in a time unit in the non-experimental period.

[0021] Optionally, the difference acquisition unit is specifically used for:

[0022] The difference between the comprehensive value of the experimental index corresponding to the test group and the comprehensive value of the experimental index corresponding to the control group in each time unit included in the experimental period is taken as the experimental index difference between the control group and the test group.

[0023] Optionally, the difference acquisition unit is specifically used for:

[0024] The difference between the comprehensive value of the reference index corresponding to the test group and the comprehensive value of the reference index corresponding to the control group in each time unit included in the non-experimental period is taken as the reference index difference between the control group and the test group.

[0025] Optionally, the evaluation acquisition unit is specifically used for:

[0026] Based on the ranks of the differences between the experimental indicators and the differences between the reference indicators, obtaining the digit difference between the differences between the experimental indicators and the differences between the reference indicators;

[0027] Based on the digit difference, a difference evaluation value between the experimental indicator differences and the reference indicator differences is obtained, and based on the difference evaluation value, a strategy evaluation result of the exposure experiment strategy is obtained.

[0028] Optionally, the evaluation acquisition unit is specifically used for:

[0029] Sort the differences of the experimental indicators in order from small to large to obtain the rank of the differences of the experimental indicators;

[0030] Sort the differences of the reference indicators in order from small to large to obtain the rank of the differences of the reference indicators;

[0031] The differences between the experimental indicators and the reference indicators are comprehensively ranked in order from small to large to obtain the comprehensive rankings of the differences between the experimental indicators and the reference indicators;

[0032] Based on the rank of the differences between the experimental indicators, the rank of the differences between the reference indicators, the comprehensive rank, the number of time units included in the non-experimental period and the number of time units included in the experimental period, the number of digits difference between the differences between the experimental indicators and the differences between the reference indicators is obtained.

[0033] Optionally, the data is stored and processed by referring to browsing indexes, implicit partitions, time travel characteristics, and dynamic partition pruning.

[0034] Optionally, the device further comprises:

[0035] The allocation unit is used to confirm each object entering the experiment during the experimental period as a sample object before obtaining the experimental browsing index of each sample object in the control group, and allocate each sample object to the control group or the test group.

[0036] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the evaluation method of any one of the above-mentioned experimental strategies.

[0037] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned evaluation methods for the experimental strategies.

[0038] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of the evaluation method of any one of the above-mentioned experimental strategies.

[0039] The beneficial effects of this application are as follows:

[0040] The embodiments of the present application provide an evaluation method, device, electronic device and storage medium for an experimental strategy. Since the present application obtains the experimental browsing index corresponding to each sample object in the control group during the experimental period, and the experimental browsing index of each sample object in the test group under the influence of the exposure experimental strategy, and obtains the reference browsing index of each sample object in the control group and the test group during the non-experimental period, it does not directly compare the differences based on a large number of experimental browsing indicators and a large number of reference browsing indicators, but first converts the sample data at the sample object level, that is, the experimental browsing indicators and the reference browsing indicators, into sample data at the time unit level, and obtains the experimental indicator differences and reference indicator differences between the control group and the test group in each time unit included in the experimental period and the non-experimental period, thereby greatly reducing the magnitude of the sample data and saving the computing cost.

[0041] Furthermore, the present application also applies the reference indicator difference to the evaluation process of the experimental strategy, so as to compare the reference indicator difference with the experimental indicator difference; since the experimental indicator difference is the difference between the control group and the test group based on the indicator obtained after the experimental strategy is issued, and the reference difference indicator is the difference between the control group and the test group based on the indicator obtained before the experimental strategy is issued, therefore, the difference evaluation value obtained by comparing the reference indicator difference with the experimental indicator difference can eliminate the original difference that originally existed between the control group and the test group, thereby making the strategy evaluation result of the exposure experiment strategy more accurate.

[0042] Other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 A schematic diagram of a double difference detection principle provided in an embodiment of the present application;

[0045] Figure 2 A comparison chart of the differences between a T test, a chi-square test, and a time unit level difference test when testing different types of indicators provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of an application scenario of an evaluation method for an experimental strategy provided in an embodiment of the present application;

[0047] Figure 4 An overall flow chart of an evaluation method for an experimental strategy provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of grouping and aggregating sample objects provided in an embodiment of the present application;

[0049] Figure 6 A comparison chart of the results obtained by obtaining indicators and performing experimental strategy evaluation by means of offline diversion and backtracking respectively, provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a reference browsing index of each sample object in a non-experimental period in retrospect provided by an embodiment of the present application;

[0051] Figure 8 A logic diagram for obtaining experimental indicator differences provided in an embodiment of the present application;

[0052] Fig. 9 A logic diagram for obtaining a reference indicator difference provided in an embodiment of the present application;

[0053] Fig.10 A data conversion logic diagram provided for an embodiment of the present application;

[0054] Fig.11 A flow chart for testing non-North Star indicators of a conventional experiment provided in an embodiment of the present application;

[0055] Fig.12 A flow chart of heterogeneous treatment effect analysis of a conventional experiment provided in an embodiment of the present application;

[0056] Fig.13 A comparison diagram of a fixed window and a sliding window provided in an embodiment of the present application;

[0057] Fig.14A A comparison chart of the test results of the time unit level difference test and the T test in the first case provided in the embodiment of the present application;

[0058] Fig. 14B A comparison chart of the test results of the time unit level difference test and the T test in the second case provided in the embodiment of the present application;

[0059] Fig. 14C A comparison chart of the test results of the time unit level difference test and the T test in the third case provided in the embodiment of the present application;

[0060] Fig.14D A comparison chart of the test results of the time unit level difference test and the T test in the fourth case provided in the embodiment of the present application;

[0061] Fig.15 The detection result of DCT-DID test using a sliding window in an actual application scenario provided in an embodiment of the present application;

[0062] Fig.16 The detection result of DCT-DID test using a fixed window in an actual application scenario provided in an embodiment of the present application;

[0063] Fig.17 An overall flow chart of another evaluation method of an experimental strategy provided in an embodiment of the present application;

[0064] Fig.18 A schematic diagram of the composition structure of an evaluation device for an experimental strategy provided in an embodiment of the present application;

[0065] Fig.19 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0066] Fig. 20 A schematic diagram of the hardware structure of another electronic device provided for application of an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0068] The following is an introduction to some concepts involved in the embodiments of the present application.

[0069] Reference indicator difference: reflects the difference between the corresponding reference browsing indicators of the control group and the test group in the non-experimental period before the experimental strategy is issued. If the reference indicator difference is not zero, it means that there was an original difference between the control group and the test group before the experimental strategy was issued. Ideally, the reference indicator difference should be zero.

[0070] Experimental indicator difference: reflects the difference between the corresponding experimental browsing indicators of the control group and the test group during the experimental period after the experimental strategy is issued. The experimental indicator difference will be affected by the difference in experimental strategy and reference indicator.

[0071] Heterogeneous Treatment Effects (HTE): In experiments with control and test groups, the effect of the strategy on different individuals or groups in subgroups varies; that is, the effect of the strategy is not evenly distributed in all samples of the entire experiment, but may differ between different individuals or subgroups, such as a significant effect in the male group, but not in the female group, etc.

[0072] Data drill-down: used to deeply study and explore the details of the data to understand the deeper information behind the data; data drill-down can further analyze the analysis results of the experimental strategy and segment the data according to different subgroups, time periods and other conditions to find out possible hidden patterns, trends or THE; through data drill-down, you can find phenomena or information that may be obscured in the overall effect, so as to understand the analysis results and effects of the experimental strategy in more detail and comprehensively.

[0073] Rank: It is the position number corresponding to each data after a set of data is arranged in order from small to large. For example, if a set of data is [7, 3, 9], the three numbers in it should be 3, 7, 9 in order from small to large. That is, the rank corresponding to the number 7 is 2, the rank corresponding to the number 3 is 1, and the rank corresponding to the number 9 is 3.

[0074] The following is a brief introduction to the design concept of the embodiment of the present application:

[0075] Current Internet products usually need to frequently propose new strategies to update Internet products in order to attract more users, retain current users, or increase exposure, etc. For example, the style of an interactive button, the method of recommending information, and the location of some important information on the page can be updated. If it can be confirmed that the new strategy can indeed bring significant beneficial effects, the new strategy can be promoted to all users. Therefore, how to judge whether a new strategy can bring significant beneficial effects has become an urgent problem to be solved.

[0076] In related technologies, a part of the users is usually selected as sample objects, and the sample objects are further divided into a control group and a test group for experiments, and an observation indicator is set based on the experimental scenario; then the new strategy is used for the test group, but not for the control group; finally, T-test, chi-square test, etc. are used to detect whether there is a difference between the control group and the test group, and whether the difference is significant. If the difference is obvious, it indicates that the new strategy can bring significant beneficial effects.

[0077] In the above, the T test is a commonly used hypothesis testing method based on T distribution or normal distribution. It first proposes a null hypothesis for the experimental strategy, assuming that there is no difference between the data mean corresponding to the control group and the data mean corresponding to the test group, and then performs the following calculations:

[0078]

[0079] in, and Respectively represent the data means corresponding to the control group and the test group, n c With n v Represent the data volume of the control group and the test group respectively, and are the variances corresponding to the control group and the test group, respectively.

[0080] Then calculate the degrees of freedom:

[0081]

[0082] Based on the degrees of freedom, we can find the corresponding critical value in the T table of the T distribution. Finally, we can determine whether the original hypothesis is true based on the relationship between the t calculated above and the critical value.

[0083] The chi-square test is based on the χ 2 A common hypothesis testing method based on the distribution, the null hypothesis is that the observed frequency is no different from the expected frequency. Then, assuming that the null hypothesis is true, the test statistic χ is calculated based on this premise. 2 , χ 2 Indicates the degree of deviation between the observed value and the theoretical value. 2 The corresponding p-values ​​of distribution and degrees of freedom can be obtained by looking up the table. The p-value indicates the probability of obtaining the current statistic and more extreme situations when the null hypothesis is established. If the p-value is very small, it means that the observed value deviates too much from the theoretical value, there is a significant difference between the data, and the experimental strategy is effective. At this time, the null hypothesis should be rejected, otherwise the null hypothesis cannot be rejected. 2 The calculation formula is:

[0084]

[0085] Among them, A l is the observed frequency at level l, E l is the expected frequency of level l, n is the total frequency, P l is the expected frequency of level l. l Equal to the expected probability p of the total frequency n×l level i , h is the number of cells. When n is large, χ 2The statistic approximately follows a chi-square distribution with h-1 degrees of freedom.

[0086] However, both the T test and the chi-square test are based on data at the sample object granularity. They require the accumulation and deduplication of data from multiple days during the experiment. The data processing process is complicated, and since the number of sample objects is often large, the required amount of calculation is large, resulting in low experimental efficiency and high long-term experimental costs. It also causes high cost problems when processing data drill-down.

[0087] On the other hand, if there are original differences in the observed indicators between the test group and the control group before the new strategy is implemented, the T test and chi-square test mentioned above alone cannot eliminate this original difference, which may lead to the evaluation results being affected by the original differences when evaluating the effectiveness of the new strategy, and the evaluation results are not accurate enough.

[0088] Based on this, the embodiments of the present application provide an evaluation method, device, electronic device and storage medium for an experimental strategy. The present application first obtains the experimental browsing indicators corresponding to each sample object in the control group during the experimental period, and the experimental browsing indicators of each sample object in the test group under the influence of the exposure experimental strategy; then obtains the reference browsing indicators of each sample object in the control group and the test group during the non-experimental period. After that, the present application does not directly compare the differences between the control group and the test group based on a large number of experimental browsing indicators and a large number of reference browsing indicators, but first converts the sample data at the sample object level, that is, the experimental browsing indicators and the reference browsing indicators, into sample data at the time unit level, and obtains the experimental indicator differences and reference indicator differences between the control group and the test group in each time unit included in the experimental period and the non-experimental period, thereby greatly reducing the magnitude of the sample data, achieving double compression of the sample data volume and the variance of the test statistic, saving computing costs, and reducing test costs. Therefore, the detection method proposed in the present application can be called a time unit level differential test; when 1 day is 1 time unit, the detection method proposed in the present application can also be called a day level differential test (English: Data Compression Test-Differences in Differences, DCT-DID).

[0089] Furthermore, the present application obtains a reference indicator difference based on the reference browsing indicator obtained before the start of the experiment, and also applies the reference indicator difference to the evaluation process of the experimental strategy, so as to compare the reference indicator difference with the experimental indicator difference; since the experimental indicator difference is the difference between the control group and the test group based on the indicator obtained after the experimental strategy is issued, and the reference difference indicator is the difference between the control group and the test group based on the indicator obtained before the experimental strategy is issued, therefore, the difference evaluation value obtained by comparing the reference indicator difference with the experimental indicator difference can correct the original difference that originally existed between the control group and the test group, thereby making the strategy evaluation result of the exposure experiment strategy more accurate.

[0090] like Figure 1 As shown, it is a schematic diagram of the principle of double difference detection provided by an embodiment of the present application. Since there may be an original difference P1-S1 between the test group and the control group before the experiment starts, if the difference between the test group and the control group is directly detected after the experiment starts, P2-S2 is obtained. Obviously, the test result is affected by the original difference P1-S1; therefore, the double difference (English: Differences in Differences, DID) is adopted to eliminate the influence of the original difference before the experiment starts, that is, before the experimental strategy is issued, and obtain the true difference P2-Q between the test group and the control group after the revised experimental strategy is issued. Double difference detection requires the parallel trend hypothesis to be met, that is, if there is no experimental strategy, the absolute difference between the two groups before and after the experiment starts should be consistent. The basic form of double difference detection is a regression model, which can test the original difference and the revised experimental effect at the same time.

[0091] In addition, both the T test and the chi-square test have certain limitations. Different test methods need to be selected for different types of indicators. That is, there are types of indicators that the T test and the chi-square test cannot handle, and the process quantities required to handle different types of indicators are also different.

[0092] For example Figure 2 As shown, it is a comparison diagram of the differences between a T test, a chi-square test and a time unit level difference test provided in an embodiment of the present application when testing different types of indicators, wherein β1 is a time unit level difference test, β2 is a T test, and β3 is a chi-square test; it can be seen from the figure that the chi-square test can only test counting indicator types, the T test is applicable to mean, sum, proportion, and ratio indicator types other than counting, and the time unit level difference test can be applicable to all of the above indicator types.

[0093] In terms of process quantities, when the T test is used to test mean indicators, the process quantities that need to be used are: indicator value, numerator sum, denominator sum, sample size, and numerator square sum; when testing sum indicators, the process quantities that need to be used are: indicator value, numerator sum, sample size, and numerator square sum; when testing proportion indicators, the process quantities that need to be used are: indicator value, numerator sum, denominator sum, and sample size; when testing ratio indicators, the process quantities that need to be used are: indicator value, numerator sum, sample size, numerator square sum, denominator square sum, and numerator-denominator inner product. When the chi-square test is used to test count indicators, the process quantities that need to be used are: indicator value and sample size. The time unit level difference test only needs to use the indicator value for mean, proportion, ratio, and count indicator types, and only the sum indicator needs to use the sample size additionally, which greatly improves the operability of the test and the consistency of data storage.

[0094] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0095] like Figure 3 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 310 and a server 320.

[0096] In the embodiment of the present application, the terminal device 310 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; the terminal device may be installed with a client related to the evaluation of the experimental strategy, which may be software (such as a browser, experimental strategy evaluation software, etc.), or a web page, a small program, etc. The server 320 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for experimental strategy evaluation, which is not specifically limited in the present application. The server 320 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0097] It should be noted that the evaluation method of the experimental strategy in each embodiment of the present application can be executed by an electronic device, which can be a terminal device 310 or a server 320, that is, the method can be executed by the terminal device 310 or the server 320 alone, or by the terminal device 310 and the server 320. For example, when the server 320 is the execution subject, during the experimental period, the server 320 determines the objects entering the experiment as sample objects, and assigns the sample objects to the test group or the control group. The server 320 obtains the experimental browsing index of each sample object in the control group for the multimedia content, and issues the exposure experimental strategy of the multimedia content to the test group, and obtains the experimental browsing index of each sample object in the test group for the multimedia content after executing the exposure experimental strategy of the multimedia content; after the experimental period ends, based on the grouping of the experimental period, that is, based on each sample object in the test group and the control group, return to the non-experimental period before the exposure experimental strategy is issued, and obtain the reference browsing index of each sample object in the control group and the test group.

[0098] Afterwards, the server 320 obtains the experimental indicator differences between the control group and the test group in each time unit included in the experimental period based on the experimental browsing indicators of each sample object in the control group and the test group, and obtains the reference indicator differences between the control group and the test group in each time unit included in the non-experimental period based on the reference browsing indicators of each sample object in the control group and the test group; finally, the server 320 obtains the strategy evaluation result of the exposure experiment strategy based on the difference evaluation values ​​between the obtained experimental indicator differences and the reference indicator differences, and the server 320 sends the strategy evaluation result to the terminal device 310 for display.

[0099] In this application, multimedia content-related scenarios are mainly used as examples. Multimedia content can be videos, articles, pictures, small games, etc. The main purpose of the exposure experiment strategy is to increase the exposure of multimedia content, such as increasing video views, increasing article views, increasing game play times, etc. In addition to multimedia content-related scenarios, this application is also applicable to other scenarios that require comparative experiments. For example, a website proposes an experimental strategy, hoping that the experimental strategy can increase the number of web page views. Although web pages do not belong to multimedia content, the evaluation experiment corresponding to the experimental strategy can still be completed through the method proposed in this application.

[0100] In an optional implementation, the terminal device 310 and the server 320 may communicate with each other via a communication network.

[0101] In an optional implementation, the communication network is a wired network or a wireless network.

[0102] It should be noted that Figure 3The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0103] In an embodiment of the present application, when there are multiple servers, the multiple servers can be combined into a blockchain, and the servers are nodes on the blockchain; as in the evaluation method of the experimental strategy disclosed in the embodiment of the present application, the sample data and indicator data involved therein can be saved on the blockchain, for example, experimental browsing indicators, reference browsing indicators, sample quantity, experimental indicator differences, reference indicator differences, etc.

[0104] In addition, the embodiments of the present application can be applied to various scenarios, including not only evaluation scenarios of experimental strategies, but also including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0105] The following describes the evaluation method of the experimental strategy provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0106] See also Figure 4 As shown, it is an implementation flow chart of an evaluation method of an experimental strategy provided in an embodiment of the present application, and the specific implementation process of the method is as follows S401 to S404:

[0107] S401: During the experimental period, the server obtains the experimental browsing index of each sample object in the control group, and obtains the experimental browsing index of each sample object in the test group after executing the exposure experimental strategy of the multimedia content.

[0108] In the above, the experimental period is the first preset time period after the exposure experimental strategy is applied to the test group. Assuming that the experiment starts on day T and ends on day T+x-1, the first preset time period is x days starting from day T.

[0109] Before obtaining the experimental browsing index of each sample object in the control group, the server needs to first confirm each object entering the experiment during the experimental period as a sample object and assign each sample object to the control group or the test group.

[0110] That is, the selection and allocation of sample objects are carried out after the experiment begins. The server will use the objects that enter the experiment between T and T+x-1 as sample objects and group them. The grouping method can be random grouping, equal grouping (for example, the odd-numbered sample objects entering the experiment are divided into the control group, and the even-numbered sample objects entering the experiment are divided into the test group), etc. The server determines whether to issue an exposure experiment strategy for them based on the grouping results, and records the corresponding experimental browsing indicators.

[0111] In addition, the experimental period can often be divided into multiple time units, for example, one day as one time unit. For new sample objects that appear during the experimental period, they can be assigned to the control group or the test group by random assignment. However, if a sample object that appears on a certain day in the experimental period has appeared once or more in the previous few days of the experimental period, it needs to be assigned to the group to which it was previously assigned, and random assignment cannot be used.

[0112] Taking a specific scenario as an example, suppose there is a website with a video on its page. In order to increase the number of views of the video, the website decides to adjust the position of the video, that is, an exposure experiment strategy of adjusting the position of the video is proposed. Now it is necessary to evaluate whether the exposure experiment strategy can increase the number of views of the video.

[0113] Assume that the experiment starts on day T and lasts for two days, with one day being a time unit. For the objects that enter the website in each time unit, the server identifies them as sample objects and assigns them to the test group or the control group. For the sample objects assigned to the control group, no exposure experiment strategy is issued, and they are still shown the page before the video position is adjusted; for the sample objects assigned to the test group, an exposure experiment strategy is issued, and they are shown the page after the video position is adjusted. The server also needs to record whether each sample object has browsed the video. The recording method can be: if browsed, it is recorded as 1, and if not browsed, it is recorded as 0. In this case, 1 and 0 are the experimental browsing indicators corresponding to the sample objects.

[0114] It should be noted that the above-mentioned method of recording whether the sample object browses the video is only an example and is not specifically limited in this application.

[0115] Specifically, Figure 5 As shown, a schematic diagram of sample object grouping and aggregation provided in an embodiment of the present application is shown. For objects entering the website on day T, the server confirms them as sample objects and assigns them to the test group or the control group by random assignment. The control group on day T finally aggregates sample objects 1, sample objects 3, sample objects 4..., and the test group aggregates sample objects 2, sample objects 5, sample objects 6...; for objects entering the website on day T+1, the server also confirms them as sample objects and assigns them to the test group or the control group. The control group on day T+1 finally aggregates sample objects 1, sample objects 4, sample objects 7..., and the test group aggregates sample objects 5, sample objects 8, sample objects 9... In summary, the entire control group includes sample objects 1, sample objects 3, sample objects 4, sample objects 7..., and the entire test group includes sample objects 2, sample objects 5, sample objects 6, sample objects 8, sample objects 9...

[0116] For the sample objects in the test group, the server displays the page after the video position is adjusted. For the sample objects in the control group, the server displays the page before the video position is adjusted. Finally, the experimental browsing indicators corresponding to each sample object are recorded on days T and T+1.

[0117] In addition, it can be seen that on day T of the experimental period, sample object 1, sample object 2, sample object 3, sample object 4, sample object 5, and sample object 6 all appeared for the first time during the experimental period, so the server grouped the above sample objects by random allocation; and on day T+1, sample object 1, sample object 4, and sample object 5 did not appear for the first time, but had already appeared on day T of the experimental period, and on day T, the server allocated sample object 1 and sample object 4 to the control group and allocated sample object 5 to the control group test group, then on day T+1, the server still allocated sample object 1 and sample object 4 to the control group and allocated sample object 5 to the control group test group according to the allocation result on day T; and on day T+1 of the experimental period, sample object 7, sample object 8, and sample object 9 entered the website for the first time during the experimental period, so the server can randomly allocate sample object 7, sample object 8, and sample object 9, and finally allocate sample object 7 to the control group and sample object 8 and sample object 9 to the test group.

[0118] S402: During the non-experimental period, the server obtains reference browsing indicators of each sample object in the control group and the test group.

[0119] In the above, the non-experimental period is the second preset time period before applying the exposure experiment strategy to the test group. Assuming that the experiment starts on T day, and y days before the start of the experiment are selected as the non-experimental period, the second preset time period is from Ty day to T-1 day. The server obtains the reference browsing indicators corresponding to the sample objects in the test group and the control group in the non-experimental period by backtracking.

[0120] In the related technology, the offline diversion method is often used to obtain indicators, that is, before the experiment, the sample objects contained in the control group and the test group are selected, and the experimental strategy is issued to the sample objects in the test group, and finally the indicators of the control group and the test group in the experimental period are obtained. Assuming that the experimental period is 7 days, the experimental strategy can be evaluated according to the indicator difference after 7 days to obtain the evaluation results. However, since the present application also needs to obtain the indicators of the control group and the test group in the non-experimental period, if the offline diversion method is continued, it is necessary to start grouping the sample objects before the non-experimental period. Assuming that the non-experimental period is 6 days and the experimental period is 7 days, it is necessary to run empty for 6 days, and then the experimental strategy can be issued to the test group after 6 days, and the evaluation results can be obtained after 7 days, which will greatly increase the time and other costs.

[0121] Therefore, the present application adopts a retrospective approach to obtain reference browsing indicators in the non-experimental period, that is, after entering the experimental period, the objects entering the experiment are confirmed as sample objects, and then grouped and whether to issue an exposure experiment strategy is determined based on the grouping results. After the experimental period ends, based on the sample objects included in the control group and the test group obtained in the experimental period, the reference browsing indicators of each sample object in the non-experimental period before the start of the experiment are obtained through retrospection.

[0122] In addition, all sample objects in the test group and control group obtained during the experimental period can be divided into two categories according to whether they have active behaviors in the non-experimental period. Whether they have active behaviors means whether they have entered the experiment. The first category of sample objects has active behaviors in both the experimental period and the non-experimental period, that is, they have entered the experiment in both the experimental period and the non-experimental period; the second category of sample objects has active behaviors in the experimental period, but not in the non-experimental period, that is, they have entered the experiment in the experimental period, but not in the non-experimental period. The server can obtain the corresponding reference browsing indicators by tracing back the behaviors of the first category of sample objects in the non-experimental period, but cannot trace back the behaviors of the second category of sample objects in the non-experimental period, that is, it cannot obtain the corresponding reference browsing indicators of the second category of sample objects; therefore, the second category of sample objects are not included in the control group and test group in the non-experimental period, that is, in the non-experimental period, the number of all sample objects in the control group and the test group is less than or equal to the number of all sample objects in the control group and the test group in the experimental period.

[0123] Compared with the offline diversion method of obtaining indicators, the use of backtracking to obtain indicators can save time and have little impact on the evaluation results of the final experimental strategy. Figure 6 As shown, a comparison diagram of the results obtained by obtaining indicators and performing experimental strategy evaluation by offline diversion and backtracking respectively provided in an embodiment of the present application. It can be seen from the figure that no matter whether it is the mean indicator type, or the sum type, proportion type, ratio type or counting type, the evaluation result finally obtained by the method of obtaining the reference browsing indicator by offline diversion is very similar to the evaluation result finally obtained by the method of obtaining the reference browsing indicator by backtracking; for example, in the offline diversion method, the evaluation result of the mean class is 0.104, the evaluation result of the sum class is 0.112, the evaluation result of the proportion class is 0.08, the evaluation result of the ratio class is 0.096, and the evaluation result of the counting class is 0.064, while in the backtracking method, the evaluation result of the mean class is 0.114, the evaluation result of the sum class is 0.100, the evaluation result of the proportion class is 0.072, the evaluation result of the ratio class is 0.084, and the evaluation result of the counting class is 0.058.

[0124] In the above, the numerical value corresponding to the evaluation result can reflect whether the experimental strategy is effective. The specific calculation process and specific meaning thereof refer to S403-S404.

[0125] It can be understood that in the specific implementation of the present application, it involves relevant data such as experimental browsing indicators, reference browsing indicators, etc. of each sample object for multimedia content. When the above embodiments of the present application are applied to specific products or technologies, it is necessary to obtain the permission or consent of the sample object, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0126] Using the assumptions in S401, Figure 7 As shown, it is a schematic diagram of a reference browsing index of each sample object in a non-experimental period retrospectively provided by an embodiment of the present application. After the server obtains the experimental browsing indexes of the control group and the test group respectively in the experimental period, the server retrospects the reference browsing indexes corresponding to sample objects 1, sample objects 2, sample objects 3, sample objects 4, sample objects 5, sample objects 6, sample objects 7, sample objects 8, and sample objects 9 in each time unit of the non-experimental period according to the sample objects included in the control group and the test group. Assuming that there are two days, T-1 and T-2, in the non-experimental period, a time unit in the non-experimental period should be the same as that in the experimental period, which is also 1 day. It can be seen from the figure that sample object 1 enters the page on T-2, sample object 2 enters the page on T-1, sample object 3 enters the page on T-1, sample objects 5 and sample objects 6 enter the page on T-2, sample object 7 enters the page on both T-1 and T-2, sample object 8 enters the page on T-1...while sample objects 4 and sample objects 9 have no active behavior in the non-experimental period, that is, they do not enter the page.

[0127] Finally, in the non-experimental period, the entire control group includes sample object 1, sample object 3, sample object 7, ..., and the entire test group includes sample object 2, sample object 5, sample object 6, sample object 8, .... The server can query the reference browsing indicators of sample object 1, sample object 3, sample object 7, sample object 2, sample object 5, sample object 6, sample object 8, etc. on the corresponding dates, but cannot query the reference browsing indicators of sample object 4 and sample object 9 in the non-experimental period.

[0128] S403: The server obtains the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group.

[0129] In the related art, the experimental strategy is often evaluated by directly comparing the differences between the experimental browsing indicators corresponding to the test group and the experimental browsing indicators corresponding to the control group during the experimental period, such as the multimedia content exposure experimental strategy in this application. However, if there are original differences between the sample objects in the test group and the sample objects in the control group before the experimental strategy is executed, then this original difference will affect the final evaluation result, that is, the difference between the test group and the control group during the experimental period is affected by both the experimental strategy and the original difference.

[0130] Assuming that the original difference is a positive difference and the impact of the experimental strategy is also positive, the final evaluation result is that the impact of the experimental strategy is positive, but the impact of the evaluated experimental strategy will be greater than the actual impact due to the existence of the original difference; assuming that the original difference is a positive difference, but the experimental strategy has almost no impact, the final evaluation result will still detect that the impact of the experimental strategy is positive.

[0131] For example, an existing experimental strategy is to change the color of an interactive component related to mini-game G on a web page from red to green, and to evaluate whether changing the color of the interactive component is more attractive to players by recording the number of times players play the game. However, the frequency of players in the current test group playing mini-game G is originally higher than that of players in the control group, that is, the original difference is a positive difference. During the experimental period, after the color of the interactive component was changed for the test group, the number of times players in the test group played mini-game G further increased, that is, the impact of the experimental strategy is positive. If the relevant technology is used to evaluate the experimental strategy by directly comparing the differences between the number of times the test group plays the game and the number of times the control group plays the game during the experimental period, the final evaluation result will be that the impact of the experimental strategy is positive, but the evaluated impact will be greater than the actual impact.

[0132] Therefore, this application not only needs to obtain the differences in indicators between the control group and the test group during the experimental period, that is, the experimental indicator differences, but also needs to obtain the differences in indicators between the control group and the test group during the non-experimental period, that is, the reference indicator differences.

[0133] An optional way to obtain the difference in experimental indicators is: based on the difference between the comprehensive value of each experimental indicator corresponding to the test group and the comprehensive value of each experimental indicator corresponding to the control group, obtain the difference in experimental indicators between the control group and the test group in each time unit included in the experimental period; an optional way to obtain the difference in reference indicators is: based on the difference between the comprehensive value of each reference indicator corresponding to the test group and the comprehensive value of each reference indicator corresponding to the control group, obtain the difference in reference indicators between the control group and the test group in each time unit included in the non-experimental period.

[0134] In the above, the comprehensive value of the experimental index is obtained based on the mean value of each experimental browsing index corresponding to the control group or the test group in a time unit during the experimental period; the comprehensive value of the reference index is obtained based on the mean value of each reference browsing index corresponding to the control group or the test group in a time unit during the non-experimental period.

[0135] Assuming that a time unit is one day, for a certain day t0 in the experimental period, the mean value of each experimental browsing index corresponding to each sample object that participated in the experiment on day t0 and belonged to the control group can be used as the comprehensive value of the experimental index of the control group on day t0; the mean value of each experimental browsing index corresponding to each sample object that participated in the experiment on day t0 and belonged to the test group can be used as the comprehensive value of the experimental index of the test group on day t0; similarly, on each day in the non-experimental period, the comprehensive value of the reference index corresponding to the test group and the comprehensive value of the reference index corresponding to the control group can also be obtained in the above manner.

[0136] In the prior art, the experimental strategy is often evaluated by directly comparing the experimental browsing indicators corresponding to the test group and the experimental browsing indicators corresponding to the control group during the experimental period. For example, the color of an interactive component related to the mini-game G in the webpage is still changed from red to green, and the number of times the players play is recorded to evaluate whether the interactive component can attract players to play more after the color is changed. For this scenario, the null hypothesis and the alternative hypothesis are first proposed. The null hypothesis is: there is no difference in the attractiveness of the green interactive component and the red interactive component to the players, that is, the green interactive component cannot increase the number of times the players play the mini-game G; and the alternative hypothesis is the opposite of the null hypothesis, that is: the green interactive component can increase the number of times the players play the mini-game G. The null hypothesis and the alternative hypothesis can be expressed in the form of formulas as follows:

[0137] H0: E (experimental browsing index corresponding to the test group) = E (experimental browsing index corresponding to the control group) vs H1: E (experimental browsing index corresponding to the test group) ≠ E (experimental browsing index corresponding to the control group).

[0138] In the above, E is the expectation, H0: E (experimental browsing index corresponding to the test group) = E (experimental browsing index corresponding to the control group) is the null hypothesis, and H1: E (experimental browsing index corresponding to the test group) ≠ E (experimental browsing index corresponding to the control group) is the alternative hypothesis.

[0139] This application converts the sample object-level indicators into time unit-level indicators. Assuming that 1 day is a time unit, the number of time units included in the experimental period is recorded as x, then the above formula is equivalent to:

[0140] H0: E (comprehensive value of experimental indicators corresponding to the test group) = E (comprehensive value of experimental indicators corresponding to the control group) v.sH1: E (comprehensive value of experimental indicators corresponding to the test group) ≠ E (comprehensive value of experimental indicators corresponding to the control group).

[0141] Through the above method, the present application converts the indicators at the sample object level into indicators at the time unit level, which greatly saves the calculation cost. Assuming that during the experimental period, the number of all sample objects contained in the test group and the control group is in the millions, then when the difference evaluation value is calculated subsequently, if the calculation is performed directly according to the indicators at the sample object level, the calculation complexity will eventually increase due to the need to process the calculation of data such as variance; however, the present application converts the experimental browsing indicators corresponding to each sample object into two experimental indicator comprehensive values ​​corresponding to each time unit during the experimental period (a comprehensive value of the experimental indicators of the control group and a comprehensive value of the experimental indicators of the test group), thereby achieving order of magnitude compression.

[0142] Furthermore, assuming that one day is a time unit and the number of experimental days is in the hundreds, compared with the prior art, the method proposed in this application directly compresses the order of magnitude of the required data from O(1000000) to O(100). At the same time, the calculation of the variance can be reduced from O(1) to O(1 / 1000000), achieving a great degree of order compression and saving computing costs.

[0143] In addition, considering that there may be original differences between the sample objects in the test group and the sample objects in the control group, and further, the original differences will affect the final evaluation results, therefore, the present application also obtains the reference browsing indicators corresponding to the non-experimental period to participate in the calculation; the reference browsing indicators also need to be converted like the experimental indicators, that is, the reference browsing indicators corresponding to each sample object in the non-experimental period are converted into two reference indicator comprehensive values ​​corresponding to each time unit in the non-experimental period (a reference indicator comprehensive value for the control group and a reference indicator comprehensive value for the test group).

[0144] The comprehensive values ​​of each experimental index corresponding to the test group in the experimental period are recorded as There are x in total, which are the comprehensive values ​​of various experimental indicators corresponding to the control group during the experimental period, recorded as There are x in total, i represents the i-th time unit of the experimental period. In addition, in a certain time unit of the experimental period, the comprehensive value of the experimental index corresponding to the test group is the mean of the experimental browsing indexes corresponding to the test group in that time unit, and the comprehensive value of the experimental index corresponding to the control group is the mean of the experimental browsing indexes corresponding to the control group in that time unit.

[0145] Assuming that the number of time units in the non-experimental period is y, the reference comprehensive indicators corresponding to the test group in the non-experimental period can be recorded as The reference comprehensive indicators corresponding to the control group can be recorded as j represents the jth time unit of the non-experimental period.

[0146] Afterwards, the server takes the difference between the comprehensive value of the experimental indicators corresponding to the test group and the comprehensive value of the experimental indicators corresponding to the control group in each time unit included in the experimental period as the experimental indicator difference between the control group and the test group; and takes the difference between the comprehensive value of the reference indicators corresponding to the test group and the comprehensive value of the reference indicators corresponding to the control group in each time unit included in the non-experimental period as the reference indicator difference between the control group and the test group.

[0147] That is, the server will M1 i -M2 i The difference in experimental indicators corresponding to each time unit included in the experimental period can be recorded as M i , represents the difference in experimental indicators between the test group and the control group on the i-th day of the experimental period. Assuming that there was an original difference between the test group and the control group before the experimental strategy was issued, then M i The specific value of will be affected by both the original difference and the experimental strategy; j -N2 j The difference in reference indicators corresponding to each time unit included in the non-experimental period can be recorded as N j , represents the difference in reference index between the test group and the control group on the jth day of the non-experimental period.

[0148] In summary, the server finally obtains the experimental indicator difference {M1, M2, ..., M x}, abbreviated as And obtain the reference index differences {N1, N2, ..., N y}, abbreviated as It can reflect the original differences between the test group and the control group before the experimental strategy is issued.

[0149] Using the assumptions in S402, Figure 8 As shown, it is a logic diagram for obtaining the difference of experimental indicators provided in an embodiment of the present application. The server obtains the comprehensive experimental indicator value M21 corresponding to the control group on day T according to the mean of the experimental browsing indicators corresponding to sample objects 1, sample objects 3, sample objects 4... on day T, and obtains the comprehensive experimental indicator value M11 corresponding to the test group on day T according to the mean of the experimental browsing indicators corresponding to sample objects 2, sample objects 5, sample objects 6... On day T, M11-M21 is subtracted to obtain the experimental indicator difference M1 corresponding to day T, which represents the difference between the comprehensive experimental indicator values ​​of the test group and the comprehensive experimental indicator values ​​of the control group on the first day of the experimental period.

[0150] Similarly, the server obtains the comprehensive experimental indicator value M22 corresponding to the control group on T+1 day based on the mean of the experimental browsing indicators corresponding to sample objects 1, sample objects 4, sample objects 7, etc. on T+1 day; obtains the comprehensive experimental indicator value M12 corresponding to the test group on T+1 day based on the mean of the experimental browsing indicators corresponding to sample objects 5, sample objects 8, sample objects 9, etc.; and obtains the experimental indicator difference M2 corresponding to T+1 day by subtracting M12 from M22, which represents the difference between the comprehensive experimental indicator value of the test group and the comprehensive experimental indicator value of the control group on the second day of the experimental period.

[0151] like Fig. 9 As shown, it is a logic diagram for obtaining a reference indicator difference provided in an embodiment of the present application. The server obtains a reference indicator comprehensive value N21 corresponding to the control group on day T-2 according to the mean values ​​of the reference browsing indicators corresponding to sample objects 1, sample objects 7, etc. on day T-2. The server obtains a reference indicator comprehensive value N11 corresponding to the test group on day T-2 according to the mean values ​​of the reference browsing indicators corresponding to sample objects 5, sample objects 6, etc. N21-N11 is subtracted to obtain a reference indicator difference N1 corresponding to day T-2, which represents the difference between the reference indicator comprehensive values ​​of the test group and the reference indicator comprehensive values ​​of the control group on the first day of the non-experimental period.

[0152] The server obtains the reference index comprehensive value N22 corresponding to the control group on day T-1 based on the mean values ​​of the reference browsing indicators corresponding to sample objects 3, sample objects 7, etc. on day T-1; obtains the reference index comprehensive value N12 corresponding to the test group on day T-1 based on the mean values ​​of the reference browsing indicators corresponding to sample objects 2, sample objects 8, etc.; and obtains the reference index difference N2 corresponding to day T-1 by subtracting N22 from N12, which represents the difference between the reference index comprehensive value of the test group and the reference index comprehensive value of the control group on the second day of the non-experimental period.

[0153] S404: The server obtains a strategy evaluation result of the exposure experiment strategy based on the obtained difference evaluation values ​​between the experimental indicator differences and the reference indicator differences.

[0154] Since the present application greatly reduces the magnitude of the indicators, and the traditional t-test, chi-square test and other methods have large errors in processing data of smaller magnitude, the present application proposes to use the Brunner-Munzel test method to calculate the difference evaluation value between the difference of each experimental indicator and the difference of each reference indicator. The Brunner-Munzel test is a non-parametric statistical test method for comparing two independent samples. Even when applied to a small sample size and not satisfying the normal distribution, the test result can be made more accurate by comparing the difference in the number of digits of the two groups of sample data. The core idea is to calculate the rank sum of the two groups of sample data and compare the difference in the number of digits of the two groups of sample data based on the rank sum.

[0155] That is, the server obtains the digit difference between each experimental indicator difference and each reference indicator difference based on the rank of the digit difference between each experimental indicator difference and each reference indicator difference; and based on the digit difference, obtains the difference evaluation value between each experimental indicator difference and each reference indicator difference, and finally obtains the strategy evaluation result of the exposure experiment strategy based on the difference evaluation value.

[0156] In the above, the rank of the differences of each experimental indicator is obtained by sorting the differences of each experimental indicator in order from small to large; the rank of the differences of each reference indicator is obtained by sorting the differences of each reference indicator in order from small to large; the comprehensive rank of the differences of each experimental indicator and each reference indicator is obtained by comprehensively sorting the differences of each experimental indicator and each reference indicator in order from small to large.

[0157] The rank of the difference between each experimental index is recorded as The rank of the difference between each reference index is recorded as Will and Merge and splice, and sort the differences between each experimental indicator and each reference indicator in order from small to large to obtain a comprehensive rank, recorded as Among them, x represents the number of time units included in the experimental period, y represents the number of time units included in the non-experimental period, i represents the i-th time unit in the experimental period, and j represents the j-th time unit in the non-experimental period.

[0158] Assuming that in the experimental period, the difference of experimental indicators corresponding to each time unit is [0.14, 0.56, 0.25], and in the non-experimental period, the difference of reference indicators corresponding to each time unit is [0.78, 0.39, 0.91], then R m0 is [1, 3, 2], R n0 is [2, 1, 3], and R is [1, 4, 2, 5, 3, 6].

[0159] Afterwards, the server obtains the digit difference between each experimental indicator difference and each reference indicator difference based on the rank of each experimental indicator difference, the rank of each experimental indicator difference, the comprehensive rank, the number of time units included in the non-experimental period, and the number of time units included in the experimental period.

[0160] Specifically, the server first extracts the ranking results of the differences of each experimental indicator in the comprehensive rank, and obtains the split rank of the differences of each experimental indicator, which is recorded as R m = R[1:x]; extract the ranking results of the differences of each reference indicator in the comprehensive rank, and obtain the split rank of the differences of each reference indicator, recorded as R n =R[x+1:x+y]; Assuming that the experimental index differences corresponding to each time unit in the experimental period are [0.14, 0.56, 0.25], and the reference index differences corresponding to each time unit in the non-experimental period are [0.78, 0.39, 0.91], then R m is [1, 4, 2], R n is [5, 3, 6].

[0161] Based on the above-obtained data, the server calculates the median difference Z between the differences of each experimental indicator and each reference indicator, so as to eliminate the influence of the original difference in the experimental indicator difference (i.e., the difference between the test group and the control group obtained during the experimental period), and obtain the real difference between the test group and the control group that is only affected by the experimental strategy. The specific formula is as follows:

[0162]

[0163] In the above, Represents R n The mean of each rank in , assuming R n is [5, 3, 6], then is (5+3+6)÷3, Similarly; Represents the rank variance of the differences in each experimental indicator, The specific formula for the rank variance of the differences between the reference indicators and the experimental indicators is:

[0164]

[0165] In the above, Represents R m0 The mean of each rank in , Represents R m The mean of the split ranks in , assuming R m0 is [1, 3, 2], then = (1+3+2)÷3; assuming R m is [1, 4, 2], then It is (1+4+2)÷3.

[0166] Afterwards, the server can obtain a difference evaluation value based on the digit difference Z. The difference evaluation value can be a p-value. The calculation formula of the p-value is p=1-norm.cdf(Z), that is, the size of the p-value is the area to the right of the digit difference Z of the standard normal distribution.

[0167] For the p-value, a preset significance level is usually preset, and the preset significance level is usually 0.05. That is, when the p-value is less than 0.05, the null hypothesis can be rejected, indicating that the difference between the experimental indicator difference and the reference indicator difference is significant, which shows that the experimental indicator difference after eliminating the original difference can still show the significant impact of the experimental strategy, that is, the experimental strategy itself can bring significant impact, and the smaller the p-value, the stronger the significance; if the p-value is greater than 0.05, it indicates that the difference between the experimental indicator difference and the reference indicator difference is not large, that is, after eliminating the original difference, the experimental indicator difference can no longer show the significant impact of the experimental strategy, that is, the experimental strategy itself cannot bring significant impact.

[0168] After obtaining the analysis results, the server can transmit the analysis results to the terminal device for display, or directly display them on the relevant experimental platform.

[0169] Following the assumption in S403, after obtaining the experimental indicator difference M1 corresponding to day T, the experimental indicator difference M2 corresponding to day T+1, the reference indicator difference N1 corresponding to day T-2, and the reference indicator difference N2 corresponding to day T-1, the server calculates the rank R of {M1, M2} m0 , the rank R of {N1, N2} n0 , the comprehensive rank R of {M1, M2, N1, N2}, the split rank R of {M1, M2} m , the splitting rank of {N1, N2} is R n Based on the above data, the digit difference is calculated to obtain the digit difference Z. Finally, the server calculates the p value based on the digit difference Z. Assuming that the p value is 0.035, which is less than 0.05, it indicates that adjusting the position of the video can significantly increase the number of video views.

[0170] In summary, the input of the time unit level difference test proposed in this application is mainly: the indicators and the number of indicators at the time unit level of the control group and the test group in the non-experimental period, that is, the number of time units included in the non-experimental period; and the indicators and the number of indicators at the time unit level of the control group and the test group in the experimental period, that is, the number of time units included in the experimental period. In addition, since a website or an application can evaluate multiple experimental strategies at the same time, and the indicators and indicator types required by each experimental strategy during the evaluation process may be different, it is also necessary to input the corresponding indicator identification (English: Identity Document, ID) and indicator type. Each indicator has a unique corresponding indicator id, which can be a number, for example, the indicator id corresponding to page views can be 1000, the indicator id corresponding to game play times can be 1001, and so on. The indicator type represents the specific form of the indicator, such as mean, sum, proportion, ratio, count, etc. For example, the video skip rate is a ratio indicator, which represents the ratio of the number of skipped videos to the number of clicked videos. The object views mean is a mean indicator, and the number of games played mentioned in this application is a count indicator.

[0171] Furthermore, the server mainly relies on the experimental exposure table, the experimental information table and the market business indicator table to obtain the required input data, such as Fig.10 As shown, a data conversion logic diagram provided by an embodiment of the present application is provided, wherein the data exposure table stores data according to date partitions, and mainly stores relevant information of the object participating in the experiment, such as the object id, and the time and location of the object exposure in the experiment, etc. Assuming that the application scenario is a certain website, the information stored in the experimental exposure table may be: at X o'clock X minutes X seconds in the afternoon of a certain month of a certain year, the object corresponding to a certain object id was exposed under the experiment corresponding to a certain experimental id. In addition, the experimental exposure table may involve multiple experiments of the website, such as an experiment to detect the impact of adjusting the video position on the video views, an experiment to detect the impact of changing the color of a component related to the game on the number of times the game is played, etc., so the experimental exposure table also involves the start and end time of the experiment, etc. There are often multiple experimental exposure tables, and multiple experimental exposure tables constitute a cumulative experimental exposure table. Each experimental exposure table can be stored separately according to each time unit, so the experimental exposure table will also store the time of the corresponding data record.

[0172] The experimental information table stores data according to date partitions, such as one month as one partition, or one day as one partition, etc. It mainly stores relevant data of the experiment, such as experiment ID, control group ID, test group ID, experiment start and end time, experimental period time, non-experimental period time, etc.

[0173] The large-scale business indicator table also stores data according to date partitions, and mainly records relevant information of indicators based on statistical summary. For example, a certain object watched a certain video 3 times on a certain day, a certain object played a small game once on a certain day, and a certain object stayed in the navigation bar area of ​​the website homepage for 3 minutes. Furthermore, a business atomic indicator table can be generated based on the large-scale business indicator table. The atomic indicators in the business atomic indicator table are further calculated from the relevant information in the large-scale business indicator table. For example, the large-scale business indicator table records the number of videos watched by objects entering the website on a certain day. Based on this information, the average number of videos watched by objects entering the website on that day can be obtained and stored in the business atomic indicator table. The business atomic indicator table also records information such as object id, indicator id, indicator name, indicator type, data recording date, etc.

[0174] According to the information in the cumulative experiment exposure table, the experiment information table and the business atomic indicator table, the original aggregation table of the non-experimental period and the original aggregation table of the experimental period corresponding to each experiment can be extracted and summarized; the original aggregation table of the non-experimental period stores the data recording date, non-experimental period time, experiment id, object id, various indicators in the non-experimental period, etc., and the original aggregation table of the experimental period stores the data recording date, experiment id, object id, various indicators in the experimental period, etc. The data information of the original aggregation table is aggregated according to the corresponding date and a certain experiment id to obtain an experimental indicator table specifically used for a certain experiment. Finally, according to the experimental indicator table of the experimental period and the experimental indicator table of the non-experimental period, the time unit level difference test is called to calculate and output the corresponding results.

[0175] The output of the time unit level difference test can be: the relative difference between the daily control group and the test group in the non-experimental period; the average daily relative difference in the non-experimental period; the relative difference between the daily control group and the test group in the experimental period; the average daily relative difference in the experimental period; the average daily relative difference after correction by the time unit level difference test; the difference evaluation value calculated based on the time unit level difference test, such as the p-value; the evaluation result of the experimental strategy based on the difference evaluation value calculated by the time unit level difference test.

[0176] In addition, you can also set whether to enable the time unit level difference test independently, and the data related to the independent selection design can be stored in the experiment information table.

[0177] In addition to saving a lot of computing costs in terms of indicator magnitude, this application also makes improvements in data processing. For the cumulative experimental exposure table that needs to be stored for a long time, as well as the original aggregation table corresponding to the non-experimental period and the experimental indicator table corresponding to the non-experimental period, this application uses a combination of time travel characteristics (English: Time travel), implicit partitioning and dynamic partition pruning to store, read and perform other operations on the above data. The time travel characteristics and implicit partitioning can be implemented through Iceberg, and dynamic partition pruning can be implemented through the Spark 3.0 software framework.

[0178] Compared with the full-slice storage method in the existing technology, Iceberg can save more storage space. Full-slice needs to store a full-dimensional data in each time unit partition, that is, it contains all available attributes in the corresponding data set, such as time, location, object, etc. The Time travel feature can directly merge the daily added data into the original dimension table, without the need to repeatedly create the full-dimensional data in each time unit partition. It uses snapshots to access the partition data of the previous time unit and accumulate data, which can ultimately save 40% of storage space.

[0179] The combination of Spark 3.0 and Iceberg can improve data query efficiency. Suppose you need to find the data corresponding to a certain experiment ID. The traditional query method is to traverse each time unit partition based on the experiment ID and filter out the matching data. However, due to the long storage time of the experiment dimension table and the large number of experiments in the process of processing the experimental data, if you still use the traditional query method to filter out the required data by filtering out unnecessary experiment IDs through secondary partitions, you will need to traverse a large number of partitions. Based on the massive experimental data, this method will bring huge pressure to the database metadata for each query. Dynamic partition pruning can directly filter out the time unit partitions that do not meet the conditions by analyzing the query conditions to avoid unnecessary access to these partitions, thereby reducing data input and output operations and improving data query efficiency.

[0180] Iceberg metadata organization is implemented through implicit partitioning, that is, by reading Iceberg metadata information, data files corresponding to the time and filtering conditions are directly read and calculated, which can also avoid accessing a large number of unnecessary partitions and reading a large amount of unnecessary data, thereby reducing the input and output operations of data and reducing the consumption of the central processing unit. The query efficiency will further affect the computing efficiency. The combination of Iceberg and dynamic partition pruning adopted in this application can reduce the running core time by 67% compared with the traditional query method of the same level.

[0181] In addition, in terms of compression, this application adopts a combination of parquet+gzip compression formats. Compared with the original orc+zlib compression combination, parquet+gzip can reduce the underlying storage space of the distributed file system (English: Hadoop Distributed File System, HDFS), save 20% of storage costs, and save a certain amount of computing costs.

[0182] For data compression, this solution will use the data granularity of the time unit level for testing, converting the problem of big data into the problem of small samples. For the test of a certain experimental indicator, if the sample size is in the millions and the number of experimental days is in the hundreds, then compared with the traditional cumulative t-test, the data used in the time unit level difference test can be greatly compressed, and multi-dimensional data drill-down analysis and heterogeneous treatment effect analysis can be easily realized.

[0183] Among them, data drill-down analysis is used to deeply study and explore the details of the data to understand the deeper information behind the data. In random experiments, data drill-down refers to further analyzing the experimental results and segmenting the data according to different dimensions, subgroups, time periods, etc. to reveal possible hidden patterns, trends or HTEs. Through data drill-down, phenomena or information that may be concealed in the overall effect can be discovered, thereby gaining a more comprehensive understanding of the results and effects of the experiment.

[0184] HTE refers to the fact that in a randomized experiment, the effects of the experimental strategy on different individuals or subgroups vary. That is, the effect of the strategy is not evenly distributed in the entire experimental sample, but may vary significantly between different individuals or groups. For example, a strategy may have a greater impact on the elderly and almost no effect on young people.

[0185] For the experimental strategy analysis method provided in this application, when it is specifically put into the inspection scenario of non-North Star indicators for conventional experiments, it can be as follows Fig.11 As shown, it is a flow chart of the inspection of non-North Star indicators of a conventional experiment provided in an embodiment of the present application, and the server specifically executes steps 1101 to 1106:

[0186] Step 1101: The server performs a time unit level differential test to obtain an evaluation result.

[0187] Step 1102: The server determines whether the impact of the experimental strategy is significant based on the evaluation results. If it is significant, execute step 1103; otherwise, execute step 1104.

[0188] For example, the impact of the experimental strategy can be judged to be significant based on whether the p-value is ≤ 0.05.

[0189] Step 1103: The server confirms whether the experimental strategy needs to be paid attention to. If so, execute step 1105, otherwise execute step 1104.

[0190] Step 1104: The server sends the evaluation results to the experimental platform or terminal device for display.

[0191] Step 1105: The server initiates a T test or a chi-square test.

[0192] In the above, non-North Star indicators refer to indicators with low fault tolerance. When the experimenter confirms that the experimental strategy needs attention, the server can receive corresponding attention instructions to confirm that the experimental strategy needs attention. After that, the server conducts a secondary test on the experiment through T test or chi-square test to improve the accuracy of the test.

[0193] Step 1106: The server outputs the test results of the T test or the chi-square test to the experimental platform or the terminal device.

[0194] In another testing scenario for heterogeneous treatment effect analysis in a conventional experiment, one can Fig.12 As shown, it is a flow chart of heterogeneous treatment effect analysis of a conventional experiment provided in an embodiment of the present application, and the server specifically executes steps 1201 to 1206:

[0195] Step 1201: The server performs a time unit level difference check on all sample objects.

[0196] Step 1202: The server performs a time unit level difference test on the drill-down population.

[0197] The drill-down population can be divided according to attributes such as age, gender, and geographic location, and the specific selection can be made according to the actual scenario.

[0198] Step 1203: The server finds valid and significant information in a specific group of people.

[0199] Step 1204: The server confirms whether the experimental strategy needs to be paid attention to. If so, execute step 1205; otherwise, execute step 1206.

[0200] After the experimenter confirms that the experimental strategy needs attention, the server can obtain a corresponding attention instruction to confirm that the experimental strategy needs attention.

[0201] Step 1205: The server initiates a T test or a chi-square test and outputs the test result.

[0202] Step 1206: The server outputs the time unit level difference test result.

[0203] In addition, this application can also flexibly select time windows according to the scenario. Time windows are mainly divided into fixed windows and sliding windows, such as Fig.13 As shown, it is a comparison diagram of a fixed window and a sliding window provided in an embodiment of the present application, wherein the fixed window means that the start and end time of the non-experimental period has been determined in advance, from the 1st day to the Qth day, that is, Q is a fixed constant, the start time of the experimental period is Q+1, and the end time is not fixed. For example, on March 1st of year P, a website hopes to start observing the comprehensive impact of one or more experimental strategies on the frequency of players' play in the next month for a small game on the website. Then an optional implementation method is to use January 1st of year P to January 31st of year P as the non-experimental period, and from February 1st of year P to that day as the experimental period for observation; the fixed window is suitable for the scenario of comprehensive comparison of multiple experimental strategies.

[0204] The sliding window has a fixed length of time between the experimental period and the non-experimental period. The experimental period starts on T and ends on T+x-1, and the non-experimental period starts on Ty and ends on T-1. x and y are both fixed constants, but T can slide along the time axis as time changes. The sliding window is suitable for scenarios of continuous detection and real-time warning of abnormalities.

[0205] Next, the experimental strategy evaluation method proposed in this application is verified by virtual data. According to different situations, the mean values ​​of the experimental browsing indicators of the test group and the control group during the experimental period, as well as the mean values ​​of the reference browsing indicators during the non-experimental period, are set. Normal distribution data is randomly generated in combination with preset parameters to simulate the data of sample object granularity, and finally the test group and the control group are generated, each containing 40,000 data points. The simulation experimental strategy evaluation is repeated 1,000 times in each case, and the p-values ​​of the time unit level difference test and the T test are calculated in each case, and the error rate of the p-values ​​calculated by the time unit level difference test and the T test is further detected.

[0206] Case 1: If Fig.14A As shown, it is a comparison chart of the test results of the time unit level difference test and the T test in the first case provided in the embodiment of the present application. In this case, the difference between the control group and the test group in the experimental period is 0, and the difference between the control group and the test group in the non-experimental period is 0, that is, there is no original difference between the test group and the control group themselves, and the experimental strategy has no positive or negative impact, that is, the effect of the experimental strategy is not significant, and the calculated result of the p value should be greater than 0.05.

[0207] Fig.14AThe figure above shows the test results of the time unit level difference test. The horizontal axis is the size of the p value, and the vertical axis is the number of times the corresponding p value appears in 1000 tests. Among them, the darker columns are the columns corresponding to the p value falling within 0.05, that is, the test results corresponding to the columns show that the experimental strategy is effective. It can be seen that in 1000 evaluations, 52 evaluation results have corresponding p values ​​falling within 0.05, that is, the significance rate is 0.052, but because the expected test result in this case should be insignificant, Fig.14A The above figure shows that 948 evaluation results are not significant, that is, the accuracy of the time unit level difference test proposed in this application is relatively high.

[0208] Fig.14A The figure below shows the test results of the T test. The horizontal axis is the size of the p-value, and the vertical axis is the number of times the corresponding p-value appears in 1000 tests. Among them, the darker columns are the columns corresponding to the p-value falling within 0.05, that is, the test results corresponding to the columns show that the experimental strategy is effective. It can be seen that in 1000 evaluations, 56 evaluation results have corresponding p-values ​​falling within 0.05, that is, the significance rate is 0.056, but because the expected test result in this case should be insignificant, Fig.14A The figure below shows that 944 evaluation results were not significant, that is, compared with the T test, the time unit level difference test proposed in this application has a higher accuracy.

[0209] In summary, the error rate of the time unit level difference test can be controlled at around 0.05, and the error rate is lower than that of the T test. In addition, in the test result bar charts corresponding to cases 2, 3, and 4, the horizontal axis is the size of the p-value, and the vertical axis is the number of times the corresponding p-value appears in 1000 tests. The darker columns are the columns corresponding to the p-value falling within 0.05, that is, the test results corresponding to the column show that the experimental strategy is effective, which will not be repeated below.

[0210] Case 2: If Fig. 14B As shown, it is a comparison chart of the test results of the time unit level difference test and the T test in the second case provided in the embodiment of the present application. In this case, the difference between the control group and the test group in the experimental period is 1%, and the difference between the control group and the test group in the non-experimental period is 1%, that is, there is an original difference of 1% between the test group and the control group themselves. Since the difference between the control group and the test group in the experimental period is equal to the difference between the control group and the test group in the non-experimental period, it shows that the experimental strategy has no positive or negative impact, and the difference between the control group and the test group in the experimental period is caused by the original difference, that is, the effect of the experimental strategy is not significant, and the calculated p-value should be greater than 0.05.

[0211] Fig. 14BThe figure above shows the test results of the time unit level difference test. It can be seen that among the 1000 evaluations, the p-values ​​corresponding to 64 evaluation results fall within 0.05, that is, the significance rate is 0.064. However, since the expected test result in this case should be insignificant, Fig. 14B The above figure shows that 936 evaluation results are not significant. It can be seen that the accuracy of the time unit level difference test proposed in this application is relatively high.

[0212] Fig. 14B The figure below shows the test results of the T test. It can be seen that among the 1000 evaluations, the p-values ​​corresponding to 246 evaluation results fall within 0.05, that is, the significance rate is 0.246. However, since the expected test result in this case should be insignificant, Fig. 14B The figure below shows that only 754 evaluation results were insignificant, that is, compared with the T test, the time unit level difference test proposed in this application has a higher accuracy.

[0213] Case 3: If Fig. 14C As shown, it is a comparison chart of the test results of the time unit level difference test and the T test in the third case provided in the embodiment of the present application. In this case, the difference between the control group and the test group in the experimental period is 3%, and the difference between the control group and the test group in the non-experimental period is 1%, that is, there is an original difference of 1% between the test group and the control group themselves; further, the difference between the control group and the test group in the experimental period is greater than the difference between the control group and the test group in the non-experimental period, indicating that the experimental strategy also produced positive effects.

[0214] Fig. 14C The figure above shows the test results of the time unit level difference test. It can be seen that among the 1000 evaluations, the p-values ​​corresponding to 405 evaluation results fall within 0.05; Fig. 14C The figure below shows the test results of the T test. It can be seen that among the 1000 evaluations, 968 evaluation results have corresponding p-values ​​within 0.05. The number of times the p-value falls within 0.05 is much higher than any other arbitrary unit interval. This is due to the influence of the original difference between the test group and the control group itself, which causes the significance rate to inflate.

[0215] Case 4: If Fig.14D As shown, it is a comparison chart of the test results of the time unit level difference test and the T test in the fourth case provided in the embodiment of the present application. In this case, the difference between the control group and the test group in the experimental period is 0%, and the difference between the control group and the test group in the non-experimental period is 1%, that is, there is an original difference of 1% between the test group and the control group themselves; further, the difference between the control group and the test group in the experimental period is smaller than the difference between the control group and the test group in the non-experimental period, which shows that the experimental strategy has a negative impact.

[0216] Fig.14DThe figure above shows the test results of the time unit level difference test. It can be seen that among the 1000 evaluations, the p-values ​​corresponding to 146 evaluation results fall within 0.05; Fig.14D The figure below shows the test results of the T test. It can be seen that among the 1000 evaluations, only 44 evaluation results have p values ​​that fall within 0.05. Compared with the time unit level difference test of this application, the detection effect of the T test in this case is poor.

[0217] It should be noted that the p-value can only detect whether the experimental strategy is significant, but cannot detect whether its significant direction is positive or negative. That is, the p-value range is [0, 1]. If it is significant, whether it is positively significant or negatively significant, the p-value will be less than 0.05.

[0218] Based on the evaluation results in the above four cases, it can be seen that when the null hypothesis is established (the experimental effect is not significant), the error rate of the time unit level difference test can be controlled at the level of 0.05, the p-value approximately follows a uniform distribution in the interval [0, 1], and can effectively correct the original difference between the control group and the test group in the non-experimental period; when the alternative hypothesis is established (the experimental effect is significant), the p-value of the time unit level difference test no longer follows a uniform distribution. Compared with the T test, the time unit level difference test can effectively correct the expansion or contraction of the significance rate caused by the original difference between the control group and the test group in the non-experimental period.

[0219] After using virtual data to verify the experimental strategy evaluation method proposed in this application, real data is used to verify the experimental strategy evaluation method proposed in this application, and one indicator is selected from each of the five indicator types as an observation indicator. For example, the mean indicator type selects the average per capita comprehensive page views and video views on a certain website, where comprehensive refers to the combination of page views and video views; the sum indicator type selects the total length of information flow, such as the total length of time a sample object stays in a certain scene on the website; the proportion indicator type selects the information penetration rate, for example, for a certain sub-page, for the sample objects that enter the website, what proportion of the sample objects enter the sub-page; the ratio indicator type selects the video skip rate, that is, for a certain sample object, the ratio of the number of videos skipped to the number of videos clicked; the counting indicator type selects the daily active objects in the information flow.

[0220] Assume that the application scenario is a website, and one day is a time unit. The relevant data of the website within F consecutive days are taken, and the objects entering the website within F days are taken as sample objects. Each sample object is randomly divided into one group in multiple traffic layers. Each traffic layer has multiple groups, for example, each traffic layer has 100 groups. Since the sample objects are randomly assigned, there is no difference in the data between each group. Different traffic layers correspond to different experiments. For example, the experiment with the observation indicator of information penetration rate corresponds to traffic layer 1, and the experiment with the observation indicator of per capita comprehensive views corresponds to traffic layer 7, etc. Each sample object can be divided into multiple traffic layers, which means that the same sample object can participate in multiple experiments at the same time.

[0221] In each experiment, two different groups of the same traffic layer were selected as the test group and the control group. The differences in the observed indicators of the sample objects in each group were compared by using the cumulative T test and the DCT-DID test respectively. According to the differences between the control group and the test group in the experimental period, and the differences between the control group and the test group in the non-experimental period, the situations were divided into the following five categories:

[0222] Case 1: Taking the average number of comprehensive views per person, total time of information flow, information penetration rate, video skip rate, and daily active objects in information flow as observation value indicators, the difference between the control group and the test group during the experimental period is 0, and the difference between the control group and the test group during the non-experimental period is 0.

[0223] This situation simulates that there is no significant difference between the control group and the test group in the experimental period and the non-experimental period, that is, there is no original difference between the experimental group and the control group, and the experimental strategy is not effective. At this time, the correct test result should be a p value greater than 0.05, which means that the experimental strategy is not effective.

[0224] As shown in the following Table 1, the detection results of Case 1 provided in the embodiment of the present application are compared. Since the daily active objects of information flow are counting indicator types, and the T test cannot detect counting indicator types, when the observed indicator is the daily active objects of information flow, the DCT-DID test and the chi-square test are used for detection, and other observed indicators are detected by the DCT-DID test and the T test.

[0225] The comparison of test results is divided into significance rate comparison and relative difference comparison. The significance rate is the ratio of the number of times p value ≤ 0.05 appears in the corresponding experiment to the number of times the experiment is repeated. The relative difference is different from the reference indicator difference and experimental indicator difference in this application. The average daily difference under the relative difference is the mean of the daily differences only in the experimental period in the DCT-DID test. The daily difference corresponding to a certain day is: the comprehensive value of the experimental indicators of the test group on that day / the comprehensive value of the experimental indicators of the control group on that day-1; the cumulative difference under the relative difference corresponds to the T test / chi-square test. When conducting the test, the data belonging to the same sample object will be accumulated, and then the cumulative data corresponding to all sample objects in the control group will be averaged, and the cumulative data corresponding to all sample objects in the test group will be averaged. Finally, the ratio between the two means -1 is taken as the cumulative difference. Follow-up Figure 2-Figure 5 The test results in the comparison are the same as above, so Figure 2-Figure 5 I will not go into details here.

[0226] Table 1 Comparison of test results for case 1

[0227]

[0228] It can be seen that the test results of the DCT-DID test and the T test both show that the probability of p value ≤ 0.05 is small, and the experimental strategies corresponding to each observation indicator have no significant effect, and the relative differences are all 0.

[0229] Case 2: Taking the average number of views per person, total information flow duration, and video skip rate as observation indicators, the difference between the control group and the test group during the experimental period was 0.5%, and the difference between the control group and the test group during the non-experimental period was 0.5%.

[0230] This situation simulates that the differences between the control group and the test group in the experimental period and the non-experimental period are the same, that is, there are original differences between the experimental group and the control group, but the experimental strategy did not work; at this time, the correct test result should be that the p value is greater than 0.05, that is, the effect of the experimental strategy is not significant.

[0231] As shown in the following Table 2, a comparison of the detection results of Case 2 provided in the embodiment of the present application is provided.

[0232] Table 2 Comparison of test results for case 2

[0233]

[0234] It can be seen that in this case, the significance rate of the DCT-DID test is basically consistent with the result in Case 1; while the significance rate of the T test is significantly increased and inconsistent with the actual situation.

[0235] In addition, since there was no difference between the test group and the control group, the data corresponding to the test group needed to be adjusted. Taking the video skip rate and the non-experimental period as an example, by adding 0.5% of the average of the daily reference index comprehensive value of the test group to the video skip rate corresponding to each sample object in the test group, the difference between the control group and the test group in the non-experimental period was 0.5%. The same applies to other indicators in the experimental period. However, since the video skip rate corresponding to each sample object is different, and each sample object has an increase of the same value, the final test group and control group in the non-experimental period generally have a small fluctuation around 0.5%. Therefore, the relative differences are also shown in Table 2.

[0236] The relative difference only detects the difference between the test group and the control group during the experimental period. The average daily difference shows that the actual difference between the control group and the test group during the experimental period is about 0.5%. There is a gap between the average daily difference and the cumulative difference due to different calculation methods. When the average daily difference is stable, the cumulative difference is affected by the length of the experimental period.

[0237] Case 3: Taking the average number of views per person, total information flow duration, and video skip rate as observation indicators, the difference between the control group and the test group during the experimental period is 0, and the difference between the control group and the test group during the non-experimental period is 0.5%.

[0238] This situation simulates the situation in which there is no original difference between the control group and the test group in the non-experimental period, and there is a difference between the control group and the test group in the experimental period, that is, the experimental strategy is effective; at this time, the correct test result should be a p-value less than 0.05, which means that the experimental strategy is effective.

[0239] As shown in the following Table 3, a comparison of the detection results of Case 3 provided in the embodiment of the present application is provided.

[0240] Table 3 Comparison of test results for case 3

[0241]

[0242]

[0243] It can be seen that the significance rate of the DCT-DID test is higher than that of the T test, that is, the DCT-DID test is more capable of detecting differences than the T test; the daily average difference shows that the actual difference between the control group and the test group during the experimental period is about 0.5%, and the cumulative difference is greater than the daily average difference.

[0244] Case 4: Taking the average number of views per person, total information flow duration, and video skip rate as observation indicators, the difference between the control group and the test group during the experimental period was 0.5%, and the difference between the control group and the test group during the non-experimental period was 1%.

[0245] This situation simulates the situation in which there is an original difference between the control group and the test group in the non-experimental period, and there is also a difference between the control group and the test group in the experimental period, which is greater than the original difference, that is, the experimental strategy is effective; at this time, the correct test result should be a p-value less than 0.05, which means that the experimental strategy is effective.

[0246] As shown in the following Table 4, a comparison of the detection results of Case 4 provided in the embodiment of the present application is provided.

[0247] Table 4 Comparison of test results for case 4

[0248]

[0249] It can be seen that in this case, the significance rate result of the DCT-DID test is consistent with that of Case 3, while the significance rate of the T test is obviously too high and inconsistent with the actual situation because it cannot eliminate the influence of the original difference; the average daily difference shows that the actual difference between the control group and the test group during the experimental period is about 1%, and the cumulative difference is larger than the average daily difference.

[0250] Case 5: Taking the average number of views per person, total information flow duration, and video skip rate as observation indicators, the difference between the control group and the test group during the experimental period was -0.5%, and the difference between the control group and the test group during the non-experimental period was 0.5%.

[0251] This situation simulates the situation in the non-experimental period, in which there is an original difference between the control group and the test group, but it is a reverse difference. In the experimental period, there is also a difference between the control group and the test group and it is a positive difference, that is, the experimental strategy is very effective; at this time, the correct test result should be a p-value less than 0.05, which means that the experimental strategy is significant.

[0252] As shown in the following Table 5, a comparison of the test results of Case 5 provided in the embodiment of the present application is provided.

[0253] Table 5 Comparison of test results for case 5

[0254]

[0255] It can be seen that the DCT-DID test can eliminate the reverse impact of the original difference and can better detect the significant effect of the experimental strategy, while the T test cannot eliminate the impact of the original difference, so its test result is obviously low; the average daily difference shows that the actual difference between the control group and the test group during the experimental period is about 0.5%, and the cumulative difference is greater than the average daily difference.

[0256] It should be noted that when the experimental strategy evaluation method proposed in this application is verified using actual data, the acquisition of observation indicators, grouping of the control group and the test group, etc. are not exactly the same as the description in S401 and S402 of this application. This is to obtain the required data faster and more conveniently when verifying the method, so as to save verification costs; and when the experimental strategy evaluation method provided by this application is actually applied, the way of obtaining indicators should be as shown in S401 and S402, that is, in the non-experimental period, a retrospective method is used to obtain the corresponding indicators in the non-experimental period, and in the experimental period, the objects entering the experiment are determined as sample objects and grouped.

[0257] Taking the average number of page views and video views per capita in a certain application as the observation indicator, a DCT-DID test is performed under a sliding window from January to July of a certain year; specifically, the p-value of Td1 on a certain day is obtained based on the DCT-DID test by using {Td1-7,…,Td1-1} as the non-experimental period and {Td1,…,Td1+6} as the experimental period. Based on this, the average number of page views per capita is compared with the week before the issuance of the corresponding experimental strategy in each experiment in the long term.

[0258] like Fig.15 As shown, it is the detection result of DCT-DID test using sliding window in an actual application scenario provided by an embodiment of the present application, that is, 1 day is 1 time unit; the horizontal axis of the upper figure is time, and the vertical axis is p value, the horizontal axis of the lower figure is time, and the vertical axis is daily difference, the horizontal dotted line indicates p value = 0.05, and the p value less than 0.05 is cut out by two vertical dotted lines in the figure.

[0259] Since the p-value cannot reflect whether the difference is positively significant or negatively significant, Fig.15 The change of daily difference corresponding to the day is also shown in the figure below. The daily difference is the comprehensive value of experimental indicators of the test group on the day / the comprehensive value of experimental indicators of the control group on the day-1, that is, the vertical axis of the figure below is the difference between the mean value of the indicators of the test group on the day and the mean value of the indicators of the control group, and the horizontal axis is time. Corresponding to the figure above, it can be seen that between the 1st dotted line and the 2nd dotted line, between the 3rd dotted line and the 4th dotted line, between the 9th dotted line and the 10th dotted line, between the 11th dotted line and the 12th dotted line, between the 13th dotted line and the 14th dotted line, and between the 15th dotted line and the 16th dotted line, the daily difference increases, that is, the corresponding test result is positively significant; between the 5th dotted line and the 6th dotted line, between the 7th dotted line and the 8th dotted line, and between the 17th dotted line and the 18th dotted line, the daily difference decreases, that is, the corresponding test result is negatively significant.

[0260] Among them, the optimization strategy 1 of the video recommendation method is issued between the 9th dotted line and the 10th dotted line. It can be seen that through the DCT-DID test, it can be detected that the p-value during this period is less than 0.05 and the daily difference increases by 1.4523%. The detection result is positive and significant, that is, the optimization strategy 1 of the video recommendation method is effective; the optimization strategy 2 of the video recommendation method is issued between the 11th dotted line and the 12th dotted line. The DCT-DID test can still detect that the p-value during this period is less than 0.05 and the daily difference increases by 1.2719%. The detection result is positive and significant, that is, the optimization strategy 2 of the video recommendation method is effective.

[0261] That is, by comparing the results of the DCT-DID test with the results of the experimental strategy during the experimental period, the significant time and direction obtained by the DCT-DID test can correspond to the time when the experimental strategy was issued.

[0262] In addition, this solution also supports the use of fixed windows for DCT-DID testing. For example, with 1 day as a time unit, the 21st day of the beginning of a year is taken as the non-experimental period, and the period from the 22nd to the Td2th day is the experimental period. The p-value of the Td2th day is obtained by bringing the data of {1, ..., 21} as the non-experimental period and the data of {22, ..., 21+Td2} as the experimental period into the DCT-DID test. Based on this, the comprehensive impact of various relevant experimental strategies on the average number of comprehensive page views per person can be tested in the long term; Fig.16 As shown, it is the detection result of DCT-DID test using a fixed window in an actual application scenario provided by an embodiment of the present application. The horizontal axis of the upper figure is time, and the vertical axis is the mean. The horizontal axis of the lower figure is time, and the vertical axis is the cumulative difference between the mean of the index of the test group and the control group. The cumulative difference on day Td2 is: (comprehensive value of experimental indicators corresponding to the test group / comprehensive value of experimental indicators of the control group on the same day-1) / Td2-(comprehensive value of reference indicators corresponding to the test group / comprehensive value of reference indicators of the control group on the same day-1) / 21.

[0263] It can be seen from the above figure that since February of that year, the p-value has begun to decrease below 0.05. The figure below shows that the cumulative difference is on the rise. Combining the above and below figures, it can be seen that compared with the beginning of the year, the comprehensive impact of various relevant experimental strategies on the average number of views per person from February to July of that year was significantly positive.

[0264] An optional implementation is that the Fig.17 The flowchart shown implements S401 to S404, including the following steps:

[0265] S1701: The server obtains the experimental browsing indicators corresponding to the sample objects included in the control group and the test group respectively during the experimental period.

[0266] In the above, the server identifies the objects entering the experiment during the experimental period as sample objects and assigns them to the control group or the test group. During the experimental period, the experimental strategy is issued to each sample object included in the test group, but not to the control group. After that, the server obtains the experimental browsing indicators corresponding to the sample objects included in the test group and the control group respectively.

[0267] S1702: The server obtains reference browsing indicators corresponding to the sample objects included in the control group and the test group respectively during the non-experimental period.

[0268] The server looks back to the non-experimental period based on all sample objects in the test group and the control group obtained during the experimental period. If a sample object also has active behavior in one or more time units included in the non-experimental period, the server obtains one or more corresponding reference browsing indicators.

[0269] S1703: The server obtains the comprehensive value of the experimental indicators corresponding to the test group and the comprehensive value of the experimental indicators corresponding to the control group in each time unit included in the experimental period.

[0270] In the above, the comprehensive value of the experimental indicators of the control group in a certain time unit is obtained based on the average experimental browsing indicators corresponding to all sample objects that have active behaviors and belong to the control group in the time unit; the comprehensive value of the experimental indicators of the test group in a certain time unit is obtained based on the average experimental browsing indicators corresponding to all sample objects that have active behaviors and belong to the test group in the time unit.

[0271] S1704: The server obtains the experimental indicator difference corresponding to each time unit included in the experimental period based on the comprehensive value of each experimental indicator.

[0272] The difference in experimental indicators corresponding to a certain time unit is obtained based on the difference between the comprehensive value of the experimental indicators corresponding to the test group and the comprehensive value of the experimental indicators corresponding to the control group under the time unit.

[0273] S1705: The server obtains the comprehensive value of the reference indicators corresponding to the test group and the comprehensive value of the reference indicators corresponding to the control group in each time unit included in the non-experimental period.

[0274] In the above, the comprehensive value of the reference indicators of the control group in a certain time unit is obtained based on the average value of the reference browsing indicators corresponding to all sample objects that have active behaviors and belong to the reference group in the time unit; the comprehensive value of the reference indicators of the test group in a certain time unit is obtained based on the average value of the reference browsing indicators corresponding to all sample objects that have active behaviors and belong to the test group in the time unit.

[0275] S1706: The server obtains the reference indicator difference corresponding to each time unit included in the non-experimental period based on the comprehensive value of each reference indicator.

[0276] The reference indicator difference corresponding to a certain time unit is obtained based on the difference between the comprehensive value of the reference indicators corresponding to the test group and the comprehensive value of the reference indicators corresponding to the control group in the time unit.

[0277] S1707: The server obtains the digit difference between each experimental indicator difference and each reference indicator difference based on the rank of each experimental indicator difference, the rank of each reference indicator difference, the comprehensive rank, the number of time units included in the non-experimental period, and the number of time units included in the experimental period.

[0278] S1708: The server obtains a difference evaluation value between each experimental indicator difference and each reference indicator difference based on the median difference.

[0279] S1709: The server obtains a policy evaluation result of the exposure experiment policy based on the difference evaluation value.

[0280] In summary, the evaluation method of the experimental strategy provided in this application can be combined with the DID and Brunner-Munzel tests to detect whether the differences between the groups in the experimental period / non-experimental period are significant, so as to achieve the purpose of analyzing whether the effect of the experimental strategy is significant, and the data at the time unit level significantly improves the computational efficiency of the test.

[0281] In addition, when performing the Brunner-Munzel test based on the difference between the experimental period and the non-experimental period, this solution can also combine variance analysis (English: anova) to decompose the intra-group variance and inter-group variance to achieve the effect of further controlling the significance rate; in terms of data storage and processing, for the various indicators covered by the experiment, this application can obtain the offline data of the Tth time unit at the T+1th time unit, and perform analysis and detection on the data at the time unit level. Since the data required by some indicators have overlapping parts, the data processing tasks corresponding to these indicators are also related. A problem with one may affect the data processing tasks of other indicators. Therefore, when the magnitude of the indicator increases, the data processing tasks of the indicators from the same source can be decoupled, and the required other data can be automatically generated according to the completion status of the upstream data and the corresponding tasks can be run, thereby reducing the impact of the upstream data on other tasks when the service level agreement is not met.

[0282] Based on the same inventive concept, the present application also provides an evaluation device for an experimental strategy. Fig.18 As shown, it is a schematic diagram of the structure of the evaluation device of the experimental strategy, which may include:

[0283] The first indicator acquisition unit 1801 is used to acquire the experimental browsing indicator of each sample object in the control group for the experimental period, and acquire the experimental browsing indicator of each sample object in the test group after the exposure experimental strategy of the multimedia content is executed, wherein the experimental period is the first preset time period after the exposure experimental strategy is applied to the test group;

[0284] The second index acquisition unit 1802 is used to acquire the reference browsing index of each sample object in the control group and the test group for the non-experimental period; the non-experimental period is a second preset time period before applying the exposure experimental strategy to the test group;

[0285] The difference obtaining unit 1803 is used to obtain the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group;

[0286] The evaluation acquisition unit 1804 is used to obtain a strategy evaluation result of the exposure experiment strategy based on the difference evaluation values ​​between the obtained experimental indicator differences and the reference indicator differences.

[0287] Optionally, the difference acquisition unit 1803 is specifically configured to acquire the experimental indicator difference in the following manner:

[0288] Based on the difference between the comprehensive value of each experimental indicator corresponding to the test group and the comprehensive value of each experimental indicator corresponding to the control group, the difference of the experimental indicators between the control group and the test group in each time unit included in the experimental period is obtained; wherein the comprehensive value of the experimental indicator is obtained based on the average value of each experimental browsing indicator corresponding to the control group or the test group in a time unit in the experimental period;

[0289] The difference acquisition unit 1803 is specifically used to acquire the reference indicator difference in the following manner:

[0290] Based on the difference between the comprehensive value of each reference indicator corresponding to the test group and the comprehensive value of each reference indicator corresponding to the control group, the difference in reference indicators between the control group and the test group in each time unit included in the non-experimental period is obtained; wherein, the comprehensive value of the reference indicator is obtained based on the mean value of each reference browsing indicator corresponding to the control group or the test group in a time unit in the non-experimental period.

[0291] Optionally, the difference obtaining unit 1803 is specifically used for:

[0292] The difference between the comprehensive value of the experimental index corresponding to the test group and the comprehensive value of the experimental index corresponding to the control group in each time unit included in the experimental period is taken as the experimental index difference between the control group and the test group.

[0293] Optionally, the difference obtaining unit 1803 is specifically used for:

[0294] The difference between the comprehensive value of the reference index corresponding to the test group and the comprehensive value of the reference index corresponding to the control group in each time unit included in the non-experimental period is taken as the reference index difference between the control group and the test group.

[0295] Optionally, the evaluation acquisition unit 1804 is specifically configured to:

[0296] Based on the rank of the difference between each experimental indicator and each reference indicator, the digit difference between the difference between each experimental indicator and each reference indicator is obtained;

[0297] Based on the digit difference, the difference evaluation value between each experimental indicator difference and each reference indicator difference is obtained, and the strategy evaluation result of the exposure experiment strategy is obtained based on the difference evaluation value.

[0298] Optionally, the evaluation acquisition unit 1804 is specifically configured to:

[0299] Sort the differences of each experimental indicator from small to large to obtain the rank of the difference of each experimental indicator;

[0300] Sort the differences of each reference indicator from small to large to obtain the rank of the difference of each reference indicator;

[0301] The differences between the experimental indicators and the reference indicators are comprehensively ranked from small to large to obtain the comprehensive rank of the differences between the experimental indicators and the reference indicators;

[0302] Based on the rank of the difference of each experimental indicator, the rank of the difference of each reference indicator, the comprehensive rank, the number of time units included in the non-experimental period and the number of time units included in the experimental period, the digit difference between the difference of each experimental indicator and the difference of each reference indicator is obtained.

[0303] Optionally, data is stored and processed by referring to the browsing index in a time travel feature, implicit partitioning, and dynamic partition pruning.

[0304] Optionally, the device further comprises:

[0305] The allocating unit 1805 is used to confirm each object entering the experiment during the experimental period as a sample object before obtaining the experimental browsing index of each sample object in the control group, and allocate each sample object to the control group or the test group.

[0306] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0307] After introducing the evaluation method and device of the experimental strategy according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0308] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0309] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 3 In this embodiment, the structure of the electronic device can be as follows: Fig.19 As shown, it includes a memory 1901 , a communication module 1903 and one or more processors 1902 .

[0310] The memory 1901 is used to store computer programs executed by the processor 1902. The memory 1901 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0311] The memory 1901 may be a volatile memory, such as a random-access memory (RAM); the memory 1901 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1901 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1901 may be a combination of the above memories.

[0312] The processor 1902 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1902 is used to implement the evaluation method of the above-mentioned experimental strategy when calling the computer program stored in the memory 1901 .

[0313] The communication module 1903 is used to communicate with terminal devices and other servers.

[0314] The specific connection medium between the memory 1901, the communication module 1903 and the processor 1902 is not limited in the embodiment of the present application. Fig.19 In the embodiment, the memory 1901 and the processor 1902 are connected via a bus 1904. The bus 1904 is Fig.19 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1904 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.19 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0315] The memory 1901 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the evaluation method of the experimental strategy of the embodiment of the present application. The processor 1902 is used to execute the above-mentioned evaluation method of the experimental strategy, such as Figure 4 shown.

[0316] In another embodiment, the electronic device may also be other electronic devices, such as Figure 3 The terminal device 310 is shown in FIG. In this embodiment, the structure of the electronic device can be as follows: Fig. 20 As shown, it includes: a communication component 2010, a memory 2020, a display unit 2030, a camera 2040, a sensor 2050, an audio circuit 2060, a Bluetooth module 2070, a processor 2080 and other components.

[0317] The communication component 2010 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help an object (such as a user) to send and receive information through the WiFi module.

[0318] The memory 2020 can be used to store software programs and data. The processor 2080 executes various functions and data processing of the terminal device 310 by running the software programs or data stored in the memory 2020. The memory 2020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The memory 2020 stores an operating system that enables the terminal device 310 to run. In the present application, the memory 2020 can store an operating system and various application programs, and can also store a computer program for executing the evaluation method of the experimental strategy of the embodiment of the present application.

[0319] The display unit 2030 can also be used to display information input by the object or information provided to the object and a graphical user interface (GUI) of various menus of the terminal device 310. Specifically, the display unit 2030 may include a display screen 2032 disposed on the front of the terminal device 310. The display screen 2032 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 2030 may be used to display the experimental strategy evaluation interface in the embodiment of the present application, etc.

[0320] The display unit 2030 can also be used to receive input digital or character information, and generate signal input related to the object setting and function control of the terminal device 310. Specifically, the display unit 2030 may include a touch screen 2031 set on the front of the terminal device 310, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0321] The touch screen 2031 can be covered on the display screen 2032, or the touch screen 2031 and the display screen 2032 can be integrated to realize the input and output functions of the terminal device 310, and the integrated touch screen can be referred to as a touch display screen. In this application, the display unit 2030 can display the application and the corresponding operation steps.

[0322] The camera 2040 can be used to capture static images, and the object can publish the image taken by the camera 2040 through the application. The camera 2040 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 2080 to convert it into a digital image signal.

[0323] The terminal device may further include at least one sensor 2050, such as an acceleration sensor 2051, a distance sensor 2052, a fingerprint sensor 2053, and a temperature sensor 2054. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0324] The audio circuit 2060, the speaker 2061, and the microphone 2062 can provide an audio interface between the object and the terminal device 310. The audio circuit 2060 can transmit the electrical signal converted from the received audio data to the speaker 2061, which is converted into a sound signal for output. The terminal device 310 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 2062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2060 and converted into audio data, and then the audio data is output to the communication component 2010 to be sent to, for example, another terminal device 310, or the audio data is output to the memory 2020 for further processing.

[0325] The Bluetooth module 2070 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 2070 to exchange data.

[0326] The processor 2080 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 2020 and calling data stored in the memory 2020. In some embodiments, the processor 2080 may include one or more processing units; the processor 2080 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, object interface and application program, and the baseband processor mainly processes wireless communication. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 2080. In the present application, the processor 2080 can run the operating system, application program, object interface display and touch response, as well as the evaluation method of the experimental strategy of the embodiment of the present application. In addition, the processor 2080 is coupled to the display unit 2030.

[0327] In some possible implementations, various aspects of the experimental strategy evaluation method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the experimental strategy evaluation method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 4 Follow the steps shown in .

[0328] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0329] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0330] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0331] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0332] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program may be executed entirely on the subject electronic device, partially on the subject electronic device, as a separate software package, partially on the subject electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device may be connected to the subject electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0333] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0334] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0335] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0336] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0337] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0338] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0339] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0340] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for evaluating an experimental strategy, characterized in that: The method comprises: For the experimental period, obtaining the experimental browsing index of each sample object in the control group, and obtaining the experimental browsing index of each sample object in the test group after executing the exposure experimental strategy of the multimedia content, wherein the experimental period is the first preset time period after the exposure experimental strategy is applied to the test group; For the non-experimental period, obtaining the reference browsing index of each sample object in the control group and the test group; the non-experimental period is a second preset time period before applying the exposure experimental strategy to the test group; Based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group, respectively obtaining the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period; Based on the difference evaluation values ​​obtained between the differences of each experimental indicator and the differences of each reference indicator, a strategy evaluation result of the exposure experiment strategy is obtained.

2. The method according to claim 1, characterized in that The experimental indicator differences are obtained in the following way: Based on the difference between the comprehensive value of each experimental index corresponding to the test group and the comprehensive value of each experimental index corresponding to the control group, the difference of the experimental index between the control group and the test group in each time unit included in the experimental period is obtained; wherein the comprehensive value of the experimental index is obtained based on the average value of each experimental browsing index corresponding to the control group or the test group in a time unit in the experimental period; The reference index difference is obtained in the following way: Based on the difference between the comprehensive value of each reference indicator corresponding to the test group and the comprehensive value of each reference indicator corresponding to the control group, the difference in reference indicators between the control group and the test group in each time unit included in the non-experimental period is obtained; wherein, the comprehensive value of reference indicators is obtained based on the mean value of each reference browsing indicator corresponding to the control group or the test group in a time unit in the non-experimental period.

3. The method according to claim 2, characterized in that The obtaining of the difference between the comprehensive value of each experimental index corresponding to the test group and the comprehensive value of each experimental index corresponding to the control group in each time unit included in the experimental period, includes: The difference between the comprehensive value of the experimental index corresponding to the test group and the comprehensive value of the experimental index corresponding to the control group in each time unit included in the experimental period is taken as the experimental index difference between the control group and the test group.

4. The method according to claim 2, characterized in that The step of obtaining the reference indicator difference between the control group and the test group in each time unit included in the non-experimental period based on the difference between the comprehensive value of each reference indicator corresponding to the test group and the comprehensive value of each reference indicator corresponding to the control group comprises: The difference between the comprehensive value of the reference index corresponding to the test group and the comprehensive value of the reference index corresponding to the control group in each time unit included in the non-experimental period is taken as the reference index difference between the control group and the test group.

5. The method according to any one of claims 1 to 4, characterized in that: The obtaining of the strategy evaluation result of the exposure experiment strategy based on the difference evaluation value between each experimental indicator difference and each reference indicator difference includes: Based on the ranks of the differences between the experimental indicators and the differences between the reference indicators, obtaining the digit difference between the differences between the experimental indicators and the differences between the reference indicators; Based on the digit difference, a difference evaluation value between the experimental indicator differences and the reference indicator differences is obtained, and based on the difference evaluation value, a strategy evaluation result of the exposure experiment strategy is obtained.

6. The method according to claim 5, characterized in that The obtaining the digit difference between the experimental indicator difference and the reference indicator difference based on the rank of the experimental indicator difference and the reference indicator difference includes: Sort the differences of the experimental indicators in order from small to large to obtain the rank of the differences of the experimental indicators; Sort the differences of the reference indicators in order from small to large to obtain the rank of the differences of the reference indicators; The differences between the experimental indicators and the reference indicators are comprehensively ranked in order from small to large to obtain the comprehensive rankings of the differences between the experimental indicators and the reference indicators; Based on the rank of the differences between the experimental indicators, the rank of the differences between the reference indicators, the comprehensive rank, the number of time units included in the non-experimental period and the number of time units included in the experimental period, the number of digits difference between the differences between the experimental indicators and the differences between the reference indicators is obtained.

7. The method according to any one of claims 1 to 4, characterized in that: The reference browsing index stores and processes data in a manner of time travel characteristics, implicit partitioning, and dynamic partition pruning.

8. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the experimental browsing index of each sample object in the control group, the method further includes: Each subject who enters the experiment during the experimental period is identified as a sample subject, and each sample subject is assigned to the control group or the test group.

9. An experimental strategy evaluation device, characterized in that: include: A first indicator acquisition unit is used to acquire, for the experimental period, an experimental browsing indicator of each sample object in the control group, and, after the exposure experimental strategy of the multimedia content is executed, an experimental browsing indicator of each sample object in the test group, wherein the experimental period is a first preset time period after the exposure experimental strategy is applied to the test group; A second index acquisition unit is used to acquire a reference browsing index of each sample object in the control group and the test group for a non-experimental period; the non-experimental period is a second preset time period before applying the exposure experimental strategy to the test group; A difference obtaining unit, configured to obtain, based on the experimental browsing index and the reference browsing index of each sample object in the control group and the test group, the experimental index difference and the reference index difference between the control group and the test group in each time unit included in the experimental period and the non-experimental period; An evaluation acquisition unit is used to obtain a strategy evaluation result of the exposure experiment strategy based on the difference evaluation values ​​between the obtained experimental indicator differences and the reference indicator differences.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: It comprises a computer program. When the computer program is run on an electronic device, the computer program is used to make the electronic device execute the steps of any one of the methods of claims 1 to 8.

12. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 8.

Images