Test report generation method, apparatus, equipment and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for generating test reports. The method includes: obtaining a data query request, the data query request containing test indicator data; obtaining data query results corresponding to the data query request, the data query results including behavioral event data, business data, and test user data matching the test indicator data; and generating a test report based on the data query results. The test report includes at least a confidence interval and statistical power. This application can automatically generate test reports, reducing labor costs and improving the accuracy of test reports.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method, apparatus, device, and storage medium for generating test reports. Background Technology

[0002] A / B testing is an important data-driven approach that is now widely used in areas such as internet products, design, search, recommendation systems, advertising systems, user growth, data analysis, digital operations, and intelligent marketing.

[0003] In the era of big data, decision-making is increasingly driven by data, which is why A / B testing is becoming more and more popular among internet companies, who use data to refine their products.

[0004] As knowledge of A / B testing becomes more widespread, more and more products are implementing A / B testing for new features / iterations. However, the data metrics for each test vary significantly, leading to a substantial increase in data analysts' daily workload in data collection and report generation for A / B tests, resulting in significant manpower costs. Current A / B testing systems have only automated and tooled aspects such as test configuration, data distribution, event / data reporting, and storage. The process of deriving test report conclusions from reported events and data still heavily relies on manual intervention. This has the following drawbacks: all tests require data analyst involvement and manual report generation, resulting in excessively high manpower costs; test report production time is relatively long; and manually generated test reports are subjective and cannot guarantee complete accuracy. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for generating test reports, which can automatically generate test reports, reduce labor costs, and improve the accuracy of test reports.

[0006] On one hand, embodiments of this application provide a method for generating test reports, the method comprising:

[0007] Obtain a data query request, wherein the data query request contains experimental indicator data;

[0008] Obtain the data query results corresponding to the data query request. The data query results include behavioral event data, business data, and test user data that match the test indicator data.

[0009] A test report is generated based on the data query results. The test report includes at least the confidence interval and statistical power.

[0010] On the other hand, embodiments of this application provide a test report generation apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire a data query request, which includes test index data;

[0012] The second acquisition module is used to acquire the data query results corresponding to the data query request. The data query results include behavioral event data, business data, and test user data that match the test indicator data.

[0013] The generation module is used to generate an experimental report based on the data query results. The experimental report includes at least the confidence interval and statistical power.

[0014] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the test report generation method as described in any of the above embodiments by calling the computer program stored in the memory.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the test report generation method as described in any of the above embodiments.

[0016] This embodiment of the application obtains a data query request, which includes experimental indicator data, and obtains the corresponding data query results. The data query results include behavioral event data, business data, and experimental user data that match the experimental indicator data. An experimental report is generated based on the data query results. The experimental report includes at least a confidence interval and statistical power. This embodiment of the application can automatically generate experimental reports, reducing labor costs and improving the accuracy of the experimental reports. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall architecture provided for an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the test report generation method provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram of a first application scenario provided for an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of a second application scenario provided for an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of a third application scenario provided in the embodiments of this application.

[0023] Figure 6 This is a schematic diagram of the fourth application scenario provided in the embodiments of this application.

[0024] Figure 7 This is a schematic diagram of the fifth application scenario provided in the embodiments of this application.

[0025] Figure 8 This is a schematic diagram of the sixth application scenario provided in the embodiments of this application.

[0026] Figure 9 This is a schematic diagram of the test report generation device provided in an embodiment of this application.

[0027] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] This application provides a method, apparatus, device, and storage medium for generating test reports. Specifically, the test report generation method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be a financial client, browser client, or instant messaging client, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network services, and big data and artificial intelligence platforms, but is not limited to these.

[0030] A / B testing (or ABT), also known as A / B experiment, controlled experiments, or double-blind clinical trial, refers to a method that involves taking a portion of online traffic, randomly assigning it to strategy A and strategy B, and then using certain statistical methods to obtain an accurate estimate of the relative effects of the two strategies, thereby determining which strategy is superior.

[0031] A / B test data reports are a visual means of measuring the effectiveness of two experimental strategies. Metrics are concepts and data that describe the overall data characteristics, such as data mainly used to measure the effectiveness of test results, like conversion rates. The test data report includes all observed metrics for the experimental data, including "core metrics." Each metric is quite different, so each test requires a professional data analyst to conduct on-site investigations to understand the data collection logic, run scripts to output data, and finally provide the business team with the test report conclusions.

[0032] As knowledge of A / B testing becomes more widespread, more and more products are starting to implement A / B testing when adding new features or iterations. However, the data metrics for each test are quite different, leading to a significant increase in the daily workload of data analysts in the A / B test data collection and report generation phase, resulting in substantial human resource costs. Therefore, it is necessary to design a system architecture that can automatically generate test reports to meet the different metric requirements of different tests.

[0033] Based on business data, behavioral event data, and test user data, this application presents an architecture that automates A / B test reports. This solution supports the following features:

[0034] 1. Supports WYSIWYG user configuration. Users can configure metrics through the event model. The configuration file automatically generates the data retrieval SQL script and submits the SQL script to the OLAP computing engine (such as Hive, Clifford House, etc.) for calculation on a daily schedule. After calculation, the data is saved in the relational database management system (MySQL). The test report is generated by reading the data from MySQL. The entire process is fully automated without manual data retrieval.

[0035] 2. When the observed data indicators change, manual self-service modification of the experimental indicator data is supported. Users can manually refresh the experimental report (by clicking the refresh button) and the experimental report can be dynamically and automatically updated on a T+1 basis. T+1 means that the data update frequency is the data up to yesterday's update today.

[0036] 3. Cover the needs of basic and common data indicator statistical methods (including but not limited to summation, averaging, deduplication, etc.).

[0037] 4. In the indicator management module, it supports the combination of indicators after four arithmetic operations between different indicators.

[0038] 5. Supports complex scenarios that combine multiple filtering conditions (AND / OR relationships).

[0039] 6. Supports hypothesis testing for all core metrics.

[0040] 7. Supports custom retention metric configuration.

[0041] Please see Figure 1 , Figure 1 This is a schematic diagram of the overall architecture provided for an embodiment of this application. The overall process is as follows: Figure 1 As shown: Data warehouse 10 is responsible for collecting relevant basic data and organizing and storing the basic data in the ABT (Analog-Based Testing) event detail table; ABT system 20 is responsible for collecting test indicator data, which may include indicator configuration information and test configuration information. It converts the test indicator data into SQL query statements using SQL Build (assembling event name / statistical method / filter conditions), and then sends the data query request containing the SQL query statement to data warehouse 10. Data warehouse 10 submits the data query results based on the data query request to the Online Analytical Processing (OLAP) computing engine (such as Clifford House) in ABT system 20 for backend calculation, ultimately generating an automated test report. For example, the test report may include basic test information, core indicators, traffic data, retention data, confidence intervals, and statistical power.

[0042] The basic data can include business data, behavioral event data, and test user data. Behavioral event data is stored in the event model table, test user data is stored in the test user table, and business data is stored in the business data table.

[0043] A wide table typically refers to a database table that links together related metrics, dimensions, and attributes for a single business entity. Using wide tables can improve query performance, enable faster response times, simplify usage, reduce costs, and increase user satisfaction.

[0044] SQL Build is a Go language SQL concatenation library that supports conditional control and is used to generate complete SQL statements.

[0045] OLAP computing engines include, but are not limited to: Hive, Spark SQL, Presto, Kylin, Impala, Druid, Clickhouse, Greeplum, etc.

[0046] Hive operates by converting HQL statements (SQL-like syntax) into MapReduce for execution. Essentially, it is a MapReduce computing framework based on HDFS, which allows you to analyze stored data using HQL.

[0047] SparkSQL, formerly known as Shark, seamlessly integrates SQL queries with Spark programs, allowing structured data to be queried as Spark RDDs. As a member of the Spark ecosystem, SparkSQL continues to evolve, no longer limited to Hive, but merely compatible with it.

[0048] Presto is Facebook's open-source distributed SQL query engine for big data. When a client sends a data query request, it is first parsed by a syntax parser, and then sent to the corresponding node for execution.

[0049] Kylin is an open-source distributed analytics engine that provides SQL query interfaces and OLAP capabilities on top of Hadoop (Hadoop is a software framework that enables distributed processing of large amounts of data) to support ultra-large-scale datasets.

[0050] Impala is an MPP (Massively Parallel Processing) SQL query engine for processing large amounts of data stored in Hadoop clusters. It is open-source software written in C++ and Java. Compared to other Hadoop SQL engines, it offers high performance and low latency.

[0051] Druid is a high-performance data query system primarily designed for aggregated queries on large volumes of time-series data. Data can be ingested in real-time and is immediately queryable after entering Druid; moreover, the data is virtually immutable. It typically consists of time-series factual events; once an event occurs, it is entered into Druid, and external systems can then query that event. Druid's technical features include: high data throughput, support for streaming and real-time data ingestion, and flexible and fast querying.

[0052] ClickHouse is a columnar database management system (DBMS) for Online Analytical Processing (OLAP). A Russian open-source columnar storage database (DBMS), ClickHouse is primarily used for OLAP online analytical processing queries, enabling real-time generation of analytical data reports using SQL queries. When performing data analysis, ClickHouse allows users to directly select specific columns as analytical attributes, resulting in very fast data retrieval with low latency. At the computational layer, ClickHouse provides multi-core parallelism, distributed computing, approximate computation, and support for complex data types, maximizing CPU resource utilization and improving system query speed.

[0053] Greenplum is a distributed database based on the open-source PostgreSQL platform, employing a shared-nothing architecture. This means that each server independently controls its own host, operating system, memory, and storage, with no shared components. Essentially, Greenplum is a relational database cluster, a logical database composed of multiple independent database services. Greenplum's greatest strength lies in its powerful parallel data computing performance and massive data management capabilities, delivered on a low-cost, open platform. This capability primarily refers to its parallel computing power, enabling fast and efficient computation of large and complex tasks.

[0054] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0055] Please see Figures 2 to 8 , Figure 2 This is a flowchart illustrating the test report generation method provided in the embodiments of this application. Figures 3 to 8 These are all schematic diagrams illustrating application scenarios provided in the embodiments of this application. The method includes the following steps 110 to 130.

[0056] Step 110: Obtain a data query request, which contains experimental indicator data.

[0057] In some embodiments, obtaining the data query request includes: collecting the test indicator data, which includes indicator configuration information and test configuration information; converting the test indicator data into an SQL query statement and generating a data query request containing the SQL query statement.

[0058] Please see Figure 1 Before obtaining data query requests, a large wide table of ABT event details needs to be constructed. For example, relevant basic data is collected through data warehouse 10, and the basic data is organized and stored in the large wide table of ABT (AB testing) event details.

[0059] Among them, the A / B test (ABT) event detail table is a key component of the A / B test report. The data sources for the ABT event detail table mainly include behavioral event data, business data, and test user data. A base table is constructed based on behavioral event data, business data, and test user data to obtain the ABT event detail table, which can support all indicators and dimensions and cover all scenarios of business indicator data.

[0060] Behavioral event data primarily originates from user behavior data reported by the SDK, including Who (who), When (when), Where (where), How (how), and What (what happened), such as user information (e.g., number of clicks), behavior occurrence time, behavior location, behavior details, and event details. Because the natively reported event model data lacks many observation dimensions—for example, the SDK only reports stock codes and not stock types or other attribute information—this embodiment, based on the event model data, uses HiveSQLJoin dimension tables within the data warehouse to associate dimensions (e.g., stock names corresponding to stock IDs, categories, etc.) to improve and enrich the attribute information. This supports statistical analysis of most dimension attributes, which is one of the key points enabling the automated generation of most experimental report indicators.

[0061] Business data: Business data generated by backend operations (such as transactions and transaction cancellations). Generally, after a business trial launch, the business focuses not only on relevant user behavior data but also on funnel data (such as account opening and deposits in the financial securities industry, and order placement and payment in the e-commerce industry). Funnel data is data obtained through funnel analysis, one of the most common "programmed" data analysis methods in the data field. It can scientifically evaluate the conversion rate of a business process from start to finish at each stage. Through quantifiable data analysis, it helps the business identify problematic business links and optimize them accordingly. Funnel analysis models are widely used in website and app user behavior analysis and are extensively applied in daily data operations and analysis work such as traffic monitoring and product target conversion. The most commonly used metrics in funnel analysis are conversion rate and churn rate, where churn rate = 1 - conversion rate. Because some business data only contains the login status ID, while some experiments trace the guest status ID, this embodiment of the application needs to match and associate this part of the business data with user behavior data to meet the data metric requirements in A / B testing scenarios. For example, it matches and associates based on the user status ID, traces the most recent user behavior data that generated the business data, and supplements it with the guest status ID, thus improving the statistical data requirements of business data in the guest status. Business data is the most important data for measuring the effectiveness of experiments. Experiments are business-oriented, and most experiments require business data support. Therefore, supporting business data configuration is one of the key points for the automated generation of most experiment reports.

[0062] Trial user data: All information that triggers the experiment when users participate, such as user information, time, trigger scenario, group identifier, etc. A / B test data needs to be highly accurate, so trial user data is needed to define the running period of each experiment, obtain all data from trial users within this trial period, and thus calculate accurate experimental metrics to generate an experimental report.

[0063] Please see Figure 1 After the ABT event detail wide table is built in the data warehouse 10, the AB test system 20 is responsible for collecting test indicator data. This test indicator data may include indicator configuration information and test configuration information. The test indicator data is converted into SQL query statements through SQL Build (assembling event name / statistic method / filter conditions), and then the data query request containing the SQL query statement is sent to the data warehouse 10.

[0064] In some embodiments, collecting the test index data includes: in response to an index configuration command input in the index management interface, collecting test index data generated based on the index configuration command.

[0065] For example, such as Figure 3 or Figure 4 As shown, it can configure indicators in response to the indicator configuration command entered in the indicator management interface, so as to collect test indicator data generated based on the indicator configuration command. The test indicator data includes indicator configuration information and test configuration information.

[0066] The metric configuration supports the business data and user behavior data described above, and allows for filtering and statistics based on any dimension. The metric configuration functionality can include the following parts:

[0067] A) Supports filtering events / business processes to target specific user behaviors or business processes (e.g., transactions); for example, Figure 3 As shown, the indicator management interface allows you to set the business line, indicator name, indicator description, indicator type (business indicator or combined indicator), and set the indicator (for example, when selecting "business indicator" as the indicator type, taking click event as an example, the indicator settings can include the total number of clicks, the current page name, the current subpage name, the current page ID, etc.), with the numerical format being number and the number of decimal places being 2, etc.

[0068] B) Supports multi-level and / or relational filtering; for example, such as Figure 4 The content shown is set under the indicator settings field, and the relationship is set as well as the OR relationship.

[0069] C) Supports various statistical methods for all event attributes (such as user ID, amount, number of times, etc.), including summation, deduplication, and averaging.

[0070] D) Supports combined indicators, i.e., the ability to perform arithmetic operations between different indicators; for example, ... Figure 5 and Figure 6 The content shown, through the setting of associated attributes and event relationships, and by filling in details in the fields such as indicator settings, indicator attributes, numerical format, and decimal places when selecting "Combined Indicator" as the indicator type, can support arithmetic operations on multiple indicators in A / B testing; for example, in Figure 5 While setting up the associated attributes and event relationships, Figure 6The indicator management interface shown is set as follows: Business line: "Zhuanzhuan B2C"; Indicator name: "B2C Exposure_Payment Conversion Rate"; Indicator description: "[B2C Payment UV / B2C Exposure UV]*100%"; Indicator type: "Combined Indicator"; Indicator content: "B2C Payment Order UV / B2C Product Exposure UV"; Indicator attribute: "Conversion Rate"; Numeric format: "Percentage"; Decimal places: 2. Figure 5 and Figure 6 The settings enable arithmetic operations between different metrics.

[0071] E) Supports custom retention configuration, allowing selection of start and return events to observe the impact of different trial versions on user activity.

[0072] Step 120: Obtain the data query results corresponding to the data query request. The data query results include behavioral event data, business data, and test user data that match the test indicator data.

[0073] In some embodiments, obtaining the data query result corresponding to the data query request includes: sending a data query request containing the SQL query statement to a data warehouse, so that the data warehouse outputs a data query result based on the data query request, wherein the data query result includes behavioral event data, business data, and test user data that match the test indicator data, wherein the behavioral event data, the business data, and the test user data originate from the AB test event detail wide table in the data warehouse; and obtaining the data query result returned from the data warehouse.

[0074] Please see Figure 1 The A / B testing system 20 sends a data query request containing an SQL query statement to the data warehouse 10, so that the data warehouse 10 returns the data query result based on the data query request.

[0075] Among them, Data Warehouse 10 queries behavioral event data, business data, and test user data that match the test indicator data from the AB test event details wide table based on data query requests.

[0076] In some embodiments, the business data includes business data corresponding to the login status identity ID and business data corresponding to the guest status identity ID.

[0077] Since some business data only contains the login status ID, but some experiments trace the guest status ID, in order to meet the data indicator requirements in A / B testing scenarios, the business data obtained in this application embodiment includes business data corresponding to the login status ID and business data corresponding to the guest status ID, thus improving the statistical data requirements of business data in the guest status.

[0078] Step 130: Generate an experimental report based on the data query results. The experimental report shall include at least the confidence interval and statistical power.

[0079] Please see Figure 1 The data warehouse 10 submits the data query results output based on the data query request to the online analytical processing (OLAP) computing engine (such as Clickerhouse) in the AB test system 20 for backend calculation, and finally generates an automated test report. For example, the test report may include basic test information, core data, traffic data, retention data, confidence intervals and statistical power.

[0080] In some embodiments, the method further includes: in response to a refresh instruction, obtaining refresh test index data generated based on the refresh instruction; and generating a refreshed test report based on the data query results corresponding to the refresh test index data.

[0081] For example, after constructing a large wide table of ABT event details, an OLAP calculation engine (such as ClickHouse) can be used to support the automation of metrics processes in the A / B testing system.

[0082] The A / B testing system provided in this application supports ad-hoc queries and provides real-time data responses. Ad-hoc queries allow users to flexibly select query conditions according to their needs, and the system can generate corresponding statistical reports based on the user's selections. It also supports detailed data queries and aggregation by any dimension and any aggregation method.

[0083] For example, such as Figure 7 As shown, the test report will begin calculation at 08:00 AM, generating test report data. The test report 'a' opened by the user contains the calculated data. Figure 8As shown, when a user adjusts parameters such as indicators or confidence intervals for hypothesis testing, the experimental report can be refreshed. For example, a refresh command can be triggered by using the "Refresh Historical Data" button on the touch display interface to achieve a refresh. The backend asynchronously completes data calculation and generates a refreshed experimental report b along with a report generation notification to inform the user to view the refreshed experimental report b. The AB testing system provided in this application embodiment can support hypothesis testing of experimental indicator data, pre-calculate and generate experimental reports daily, and support self-service refresh of experimental reports after modifying indicators.

[0084] In some embodiments, generating a test report based on the data query results includes: calculating the confidence interval and statistical power of the core indicators of the experimental group compared with the control group based on the data query results, a preset reliability coefficient and a preset reliability; and outputting a test report that includes at least the confidence interval and the statistical power.

[0085] For example, the formula for calculating the confidence interval can be expressed as the following formula (1):

[0086]

[0087] The confidence interval calculation formula is used to represent the range of variation between the experimental group sample mean and the control group sample mean. These experimental group and control group sample means are determined based on data query results.

[0088] in:

[0089] (1) These represent the sample means of the core indicators for the experimental group and the control group, respectively.

[0090] (2) n and m represent the sample size of the experimental group and the control group, respectively;

[0091] (3)Z (1-a) / 2 σ represents the confidence coefficient (e.g., the preset confidence coefficient is 1.96), a represents the confidence level (e.g., the preset confidence level is 95%), and σ represents the standard deviation.

[0092] (4) This indicates the overall error.

[0093] The range of variation of the sample mean of the experimental group compared with that of the control group can be obtained by calculation using the formula (1) above. Then, the confidence interval of the rate of change of the core indicator of the experimental group compared with that of the control group can be obtained at a 95% pre-set confidence level.

[0094] For example, the formula for calculating statistical power can be expressed as the following formula (2):

[0095]

[0096] in:

[0097] (1) n1 and n2 represent the sample sizes of groups A and B, respectively;

[0098] (2) a represents the probability of a Type I error, which is usually taken as 0.05;

[0099] (3) Z represents the quantile function of the normal distribution. This represents the area to the right of the quantile in a two-tailed test;

[0100] (4) Δ represents the difference between the values ​​in groups A and B. For example, if the registration conversion rate is 50% to 60%, then Δ is 10%.

[0101] (5) σ1 and σ2 represent the standard deviations of the populations A and B, respectively;

[0102] (6) φ(X) represents the function value that returns the standard normal cumulative distribution function.

[0103] For example, the following example illustrates how confidence intervals and statistical power are calculated:

[0104] For example, if the core indicator of the trial is "average payment amount per person", it is necessary to calculate the confidence interval range of the core indicator between the experimental group and the control group within the trial period from June 1, 2022 to June 10, 2022, and to calculate the statistical power. This can be achieved through the following steps 1 to 5.

[0105] For example, in the control group, there were 239 users participating in the trial, with a total payment of 121,392 yuan; in the experimental group, there were 640 users participating in the trial, with a total payment of 504,795 yuan.

[0106] Step 1: Calculate the average payment per person for the control group and the experimental group:

[0107] Control group sample mean:

[0108] Sample mean of the experimental group:

[0109] Difference of sample means:

[0110] Step 2, calculate the standard deviation of average payment per person:

[0111] The formula for calculating the standard deviation can be expressed as follows (3):

[0112]

[0113] Where σ represents the standard deviation; n represents the sample size; x i Σ represents the corresponding sample value; μ represents the mean of the sample; Σ represents the summation of the results after taking all sample values; √ represents the square root.

[0114] The sample variance of the control group is: σ y 2 =107062.237;

[0115] The sample variance of the experimental group is: σ x 2 =241502.664.

[0116] Step 3, calculate the two-sample standard deviation:

[0117] The standard deviation for two samples is:

[0118]

[0119] Step 4: Calculate the confidence interval at a 95% pre-set confidence level (Note: the pre-set confidence level α is 95%, and the pre-set confidence coefficient Z...). (1-a) / 2 (The Z-value is 1.96; the Z-value differs for other confidence intervals.)

[0120]

[0121] Converted to percentages, it is [224.5187103 / 507.9163, 337.1330897 / 507.9163] = [44.20%, 66.38%]; therefore, the confidence interval of this core indicator is [44.20%, 66.38%].

[0122] Step 5, calculate statistical power (Note: the pre-set reliability α is 95%, and the pre-set reliability coefficient Z...). (1-a) / 2 (The Z-value is 1.96; the Z-value differs for other confidence intervals.)

[0123]

[0124] For example, in the AB test system 20, the OLAP calculation engine performs backend calculations based on the data query results, preset reliability coefficients, and preset reliability to calculate the confidence intervals and statistical power of the core indicators of the experimental group compared to the control group. It then outputs a test report that includes at least the confidence intervals and the statistical power. The output test report can be used as a reference. Figure 7 or Figure 8 The content shown.

[0125] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0126] The test report generation method provided in this application involves obtaining a data query request containing test indicator data, and obtaining corresponding data query results. The data query results include behavioral event data, business data, and test user data that match the test indicator data. A test report is then generated based on the data query results. The test report includes at least a confidence interval and statistical power. This application embodiment can automatically generate test reports, reducing labor costs and improving the accuracy of the test reports.

[0127] To facilitate better implementation of the test report generation method of this application, this application also provides a test report generation apparatus. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of the structure of the test report generation device provided in an embodiment of this application. The test report generation device 200 may include:

[0128] The first acquisition module 210 is used to acquire a data query request, wherein the data query request contains experimental index data;

[0129] The second acquisition module 220 is used to acquire the data query result corresponding to the data query request. The data query result includes behavioral event data, business data and test user data that match the test indicator data.

[0130] The generation module 230 is used to generate an experimental report based on the data query results. The experimental report includes at least the confidence interval and statistical power.

[0131] In some embodiments, the first acquisition module 210 is configured to: collect the test index data, the test index data including index configuration information and test configuration information; convert the test index data into an SQL query statement, and generate a data query request containing the SQL query statement.

[0132] In some embodiments, when the first acquisition module 210 collects the test index data, it is used to: collect test index data generated based on the index configuration instruction input in the index management interface.

[0133] In some embodiments, the second acquisition module 220 is configured to: send a data query request containing the SQL-based query statement to a data warehouse, so that the data warehouse outputs data query results based on the data query request, the data query results including behavioral event data, business data, and test user data that match the test indicator data, the behavioral event data, the business data, and the test user data originating from the AB test event detail wide table in the data warehouse; and acquire the data query results returned from the data warehouse.

[0134] In some embodiments, the generation module 230 is configured to: calculate the confidence interval and statistical power of the core indicators of the experimental group compared with the control group based on the data query results, the preset reliability coefficient and the preset reliability; and output an experimental report that includes at least the confidence interval and the statistical power.

[0135] In some embodiments, the first acquisition module 210 is further configured to acquire refresh test index data generated based on the refresh instruction in response to the refresh instruction;

[0136] The generation module 230 is also used to generate a refreshed test report based on the data query results corresponding to the refreshed test index data.

[0137] In some embodiments, the business data includes business data corresponding to the login status identity ID and business data corresponding to the guest status identity ID.

[0138] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0139] It should be understood that the embodiments of the test report generation device and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details are omitted here. Specifically, Figure 9 The test report generation apparatus shown can execute the test report generation method embodiments described above, and the aforementioned and other operations and / or functions of each unit in the test report generation apparatus respectively implement the corresponding processes of the above method embodiments. For the sake of brevity, they will not be described in detail here.

[0140] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0141] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal or a server. Figure 10As shown, the computer device 300 may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, memory 302, and processor 303 communicate with each other via the communication bus 304. The communication interface 301 is used for data communication between the computer device 300 and external devices. The memory 302 can be used to store software programs and modules, and the processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for the corresponding operations in the foregoing method embodiments.

[0142] Optionally, the processor 303 can invoke software programs and modules stored in the memory 302 to perform the following operations:

[0143] Obtain a data query request, which includes experimental indicator data; obtain the data query results corresponding to the data query request, which include behavioral event data, business data, and experimental user data that match the experimental indicator data; generate an experimental report based on the data query results, which includes at least a confidence interval and statistical power.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0145] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple computer programs that can be loaded by a processor to execute the steps of any of the test report generation methods provided in this application. Specific implementations of the above operations can be found in the preceding embodiments and will not be repeated here.

[0146] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0147] Since the computer program stored in the storage medium can execute the steps in any of the test report generation methods provided in the embodiments of this application, the beneficial effects that any of the test report generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0148] This application also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in any of the test report generation methods described in this application. For simplicity, further details are omitted here.

[0149] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in any of the test report generation methods described in this application. For brevity, further details are omitted here.

[0150] The above provides a detailed description of a test report generation method, client, server, equity incentive system, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A test report generation method characterized by comprising: The method comprises: The data warehouse acquires a data query request, the data query request containing test index data; The data warehouse acquires data query results corresponding to the data query request from an ABT event detail wide table, the data query results including behavior event data, business data and test user data matched with the test index data; The data warehouse generates a test report according to the data query results, the content of the test report at least including a confidence interval and statistical power; Before the data query request is acquired, the method further comprises: the data warehouse constructs an ABT event detail wide table; the ABT event detail wide table includes behavior event data, business data and test user data; an AB test system is responsible for collecting test index data, the test index data including index configuration information and test configuration information, the test index data being converted into a SQL query statement through SQL Build, and a data query request containing the SQL query statement being sent to the data warehouse; The test report is generated according to the data query results, including: Based on the data query results, a preset confidence coefficient and a preset confidence, a confidence interval and statistical power of a core index of a test group compared with a control group are calculated; A test report at least including the confidence interval and the statistical power is output.

2. The test report generation method of claim 1, wherein, The test index data is collected, including: In response to an index configuration instruction input in an index management interface, test index data generated based on the index configuration instruction is collected.

3. The test report generating method of claim 1, wherein, The data query results corresponding to the data query request are acquired, including: The data query request containing the SQL query statement is sent to the data warehouse, so that the data warehouse outputs data query results based on the data query request, the data query results including behavior event data, business data and test user data matched with the test index data, the behavior event data, the business data and the test user data being derived from an AB test event detail wide table in the data warehouse; The data query results returned from the data warehouse are acquired.

4. The test report generating method of claim 1, wherein, The method further comprises: In response to a refresh instruction, refresh test index data generated based on the refresh instruction is acquired; A refreshed test report is generated according to data query results corresponding to the refresh test index data.

5. The test report generation method of claim 1, wherein, The business data includes business data corresponding to an identity ID in a login state and business data corresponding to an identity ID in a visitor state.

6. A test report generating apparatus characterized by comprising: The device is used to execute the test report generation method in any one of claims 1-5, and the device comprises: A first acquisition module is configured to acquire a data query request, the data query request containing test index data; A second acquisition module is configured to acquire data query results corresponding to the data query request, the data query results including behavior event data, business data and test user data matched with the test index data; A generation module is configured to generate a test report according to the data query results, the content of the test report at least including a confidence interval and statistical power.

7. A computer device, characterized by The computer device comprises a processor and a memory, the memory storing a computer program, and the processor is configured to execute the test report generation method according to any one of claims 1-5 by calling the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the test report generation method according to any one of claims 1-5.

Images