A business scenario comparison method, device, terminal and storage medium

Abstract

The embodiments of the present invention provide a business scenario comparison method, device, terminal and storage medium; experimental test data obtained based on the business operations of experimental group users in experimental business scenarios and control test data obtained based on the business operations of control group users in control business scenarios can be obtained, and based on the experimental test data, the experimental test indicators of the experimental group users are determined, and based on the control test data, the control test indicators of the control group users are determined. If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, the discrete degree information of the difference between the control test indicator and the experimental test indicator is determined, and the relative conversion effect information of the business resources is determined based on the discrete degree information and the difference between the experimental test indicator and the control test indicator. In this way, the influence of different business resource delivery situations in different business scenarios can be avoided, the accuracy of business scenario comparison can be improved, and the correctness of business scenario test results can be guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a business scenario comparison method, device, terminal and storage medium. Background Art

[0002] In recent years, with the rapid development of internet technology, scenarios where data can directly impact business have exploded, making data-driven growth strategies a core competitive advantage across industries and businesses. To fully implement this consensus on data-driven business growth, many companies are using business scenario testing during product development to validate hypotheses and leverage the test data generated to inform business decisions.

[0003] Currently, indicators in business scenario testing are divided into two categories: mean indicators and ratio indicators. However, ratio indicators are often affected by different testing strategies. Directly using relevant methods for hypothesis testing may lead to inaccurate conclusions from business scenario testing and inability to make correct decisions on the business being developed. Summary of the Invention

[0004] The embodiments of the present invention provide a business scenario comparison method, device, terminal and storage medium, which can avoid the influence of different business resource deployment situations in different business scenarios, improve the accuracy of business scenario comparison, and ensure the correctness of business scenario test results.

[0005] An embodiment of the present invention provides a business scenario comparison method, including:

[0006] Obtaining user test data, the user test data including experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario has the same business type as the experimental business scenario;

[0007] Determining an experimental test indicator for the experimental group of users based on the experimental test data, the experimental test indicator being used to indicate a ratio of a target business operation triggered by a user in the experimental business scenario to a quantity of business resources delivered corresponding to the target business operation;

[0008] Determining a control test indicator for the control group users based on the control test data, the control test indicator being used to indicate a ratio of the target business operation triggered by the user in the control business scenario to the amount of business resources delivered corresponding to the target business operation;

[0009] If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, determining the discrete degree information of the difference between the control test indicator and the experimental test indicator;

[0010] Based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, the relative conversion effect information of the business resource is determined. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario.

[0011] Accordingly, an embodiment of the present invention further provides a business scenario comparison device, the business scenario comparison device comprising:

[0012] a data acquisition unit, configured to acquire user test data, the user test data including experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, the control business scenario being of the same business type as the experimental business scenario;

[0013] An experimental indicator determination unit, configured to determine an experimental test indicator for the experimental group of users based on the experimental test data, wherein the experimental test indicator indicates a ratio of a target business operation triggered by a user in the experimental business scenario to a quantity of business resources allocated to the target business operation;

[0014] a control indicator determination unit, configured to determine a control test indicator for the control group users based on the control test data, wherein the control test indicator is used to indicate a ratio of the target business operation triggered by the user in the control business scenario to the amount of business resources delivered corresponding to the target business operation;

[0015] a dispersion degree determining unit, configured to determine dispersion degree information of a difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator meets a preset deviation condition;

[0016] An effect determination unit is used to determine the relative conversion effect information of the business resources based on the discrete degree information and the difference between the experimental test indicator and the control test indicator. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resources in the experimental business scenario and the control business scenario.

[0017] In an optional example, the effect determination unit may be further configured to normalize the difference between the experimental test index and the control test index based on the discrete degree information to obtain a normalized difference;

[0018] Based on the standardized difference, the dispersion information and the user test data, a confidence interval of the difference between the experimental test indicator and the control test indicator is determined, and the confidence interval is the relative conversion effect information of the business resource.

[0019] In an optional example, the business scenario comparison device provided by an embodiment of the present invention further includes a user division unit, configured to divide the experimental group users into a first preset number of experimental groups and divide the control group users into the first preset number of control groups based on a preset user allocation algorithm;

[0020] The experimental indicator determination unit is used to determine the experimental test indicator of each experimental group based on the experimental test data of each experimental group;

[0021] The control index determination unit is used to determine the experimental test index of each control group based on the experimental test data of each control group;

[0022] The discrete degree determination unit is used to calculate the standard deviation of the difference between the control test index and the experimental test index based on the experimental test index of each experimental group and the control test index of each control group, and the standard deviation is the discrete degree information.

[0023] In an optional example, the discrete degree determining unit is configured to determine first discrete degree information of the experimental test indicator based on the experimental test data;

[0024] Determining second discrete degree information of the control test indicator based on the control test data;

[0025] The discrete degree information of the experimental test indicator and the difference between the experimental test indicators is determined according to the first discrete degree information and the difference between the second discrete degree information.

[0026] In an optional example, the experimental indicator determination unit includes an experimental data acquisition unit and an experimental indicator calculation unit, wherein the experimental data acquisition unit is used to obtain, based on the experimental test data, the first number of users who trigger the target business operation in the experimental business scenario and the first number of users who deploy the business resources in the experimental business scenario;

[0027] The experimental indicator calculation unit is used to calculate the ratio of the first number of users to the first number of users for delivery as an experimental test indicator;

[0028] The control indicator determination unit includes a control data acquisition unit and a control indicator calculation unit, wherein the control data acquisition unit is used to obtain, based on the control test data, the number of second users who trigger the target business operation in the control business scenario and the number of second users for the business resource in the control business scenario;

[0029] The control index calculation unit is used to calculate the ratio of the second number of users to the second number of delivery users as a control test index.

[0030] In an optional example, the experimental business scenario is a content push scenario that adopts an experimental push strategy, and the control business scenario is a content push scenario that adopts a control push strategy;

[0031] The experimental data acquisition unit is configured to acquire, based on the experimental test data, a first number of users who click on the pushed content in the experimental business scenario and a first number of users for delivering the pushed content in the experimental business scenario;

[0032] The control data acquisition unit is configured to acquire, based on the control test data, the second number of users who click on the pushed content in the control business scenario and the second number of users for delivering the pushed content in the control business scenario.

[0033] In an optional example, before the discrete degree determination unit, the business scenario comparison device provided by the embodiment of the present invention further includes a second discrete degree determination unit, which is configured to not perform the step of determining the discrete degree information of the difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator does not meet the preset deviation condition;

[0034] Determining at least two differences between the control test indicator and the experimental test indicator based on the control test indicator and the experimental test indicator, and performing continuity conversion based on the at least two differences to obtain a continuity difference;

[0035] Calculating the mean of the continuity difference values, and estimating the limiting distribution of the continuity difference values ​​when the continuity difference values ​​increase infinitely;

[0036] Calculating the discrete degree information of the difference between the control test index and the experimental test index based on the limit distribution;

[0037] Based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, the relative conversion effect information of the business resource is determined. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario.

[0038] Correspondingly, an embodiment of the present invention also provides an electronic device, including a memory and a processor; the memory stores an application, and the processor is used to run the application in the memory to perform operations in any business scenario comparison method provided in an embodiment of the present invention.

[0039] In addition, an embodiment of the present invention further provides a storage medium storing a plurality of instructions suitable for loading by a processor to execute the steps in any one of the business scenario comparison methods provided in the embodiment of the present invention.

[0040] By adopting the scheme of the embodiment of the present invention, user test data can be obtained, and the user test data includes experimental test data obtained based on the business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on the business operations of at least one group of control group users in a control business scenario, and the control business scenario has the same business type as the experimental business scenario. Based on the experimental test data, the experimental test index of the experimental group users is determined, and based on the control test data, the control test index of the control group users is determined. If the deviation between the experimental test index and the control test index meets the preset deviation condition, the discrete degree information of the difference between the control test index and the experimental test index is determined. Based on the discrete degree information and the difference between the experimental test index and the control test index, Determine the relative conversion effect information of the business resource, and the relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario; since the embodiment of the present invention can determine the discrete degree of the difference between the control test indicator and the experimental test indicator based on the obtained experimental test indicator and the control test indicator that meet the preset deviation conditions, the relative conversion effect information of the business resource is determined based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, and the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario is obtained from the relative conversion effect information. Therefore, the influence of different business resource delivery situations in different business scenarios can be avoided, the accuracy of business scenario comparison can be improved, and the correctness of business scenario test results can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 Schematic diagram of a business scenario comparison method provided by an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a business scenario comparison method provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a business scenario comparison system provided by an embodiment of the present invention;

[0045] Figure 4 is another flow chart of the business scenario comparison method provided by an embodiment of the present invention;

[0046] Figure 5 is a structural diagram of a business scenario comparison device provided by an embodiment of the present invention;

[0047] Figure 6 is another structural diagram of the business scenario comparison device provided by an embodiment of the present invention;

[0048] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The embodiments of the present application provide a business scenario comparison method. This business scenario comparison method can be implemented by a business scenario comparison system, which can be installed in an electronic device. The electronic device can include at least one of a terminal and a server. That is, the business scenario comparison system can be deployed in a terminal, installed in a server, or installed in a terminal and a server that can communicate with each other.

[0051] The electronic device can be a terminal or other device, including but not limited to a mobile terminal and a fixed terminal. For example, a mobile terminal includes but is not limited to a smart phone, a smart watch, a tablet computer, a laptop computer, a smart car terminal, etc., wherein a fixed terminal includes but is not limited to a desktop computer, a smart TV, etc.

[0052] The electronic device can also be a server or other device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms, but is not limited to these.

[0053] In one embodiment, if Figure 3 As shown, the business scenario comparison system can include three major units, namely a test access unit, a test traffic management unit and a test data analysis unit.

[0054] The test access unit may include a test access subunit and a traffic screening subunit. The test access subunit may include a client, a client processing (AppSvr) module, an application programming interface (API), a business logic module, and an agent module. The traffic screening subunit may include a profile module, a configuration (ProfileSvr) module, an accuracy (AccurateSvr) module, and a cookie module.

[0055] The test traffic management unit may include an Internet Data Center Proxy (IDC Proxy) module, a traffic distribution (RuleSvr) module, a synchronization module and a traffic management tool.

[0056] Among them, the test data analysis unit may include a portrait module, a test log module, a business log module, a task scheduling module, a data storage center and a test inspection module.

[0057] The business scenario comparison system can be connected to the World Wide Web (Web) system. The Web system serves as a platform for testers to design, configure, observe, and manage tests, and serves as the entry point to the business scenario comparison system. When conducting business scenario comparisons, testers can use the Web system to design, configure, observe, and manage tests. Furthermore, testers can use the Web system to visualize data, manage permissions, and generate test reports.

[0058] After the tester completes the web system configuration test, the web system submits the test configuration to the traffic allocation module in the business scenario comparison system. The traffic allocation module then allocates traffic for the test using a random traffic allocation algorithm based on the test configuration. The traffic allocation module then sends the test configuration to the proxy service in the IDC Proxy module via the synchronization module. The proxy service then distributes the test configuration to the proxy module in the test terminal.

[0059] The random traffic distribution algorithm may be any random traffic distribution algorithm, such as an orthogonal table algorithm, a hash algorithm, and the like.

[0060] The test terminal may include a terminal of the test object, for example, a tablet computer, laptop computer, personal computer (PC), smart home device, wearable electronic device, VR / AR device, vehicle-mounted computer, etc.

[0061] When comparing business scenarios, the test system's traffic management tool can divide the overall traffic in the test into several buckets, that is, divide the total number of testers in the test into several parts, and configure the corresponding number of buckets for each sub-test.

[0062] For example, if a product verification test has four subtests, the test system's traffic management tool can divide the overall test traffic into 10,000 buckets and assign a corresponding number of buckets to each subtest. For example, each subtest can be assigned 2,500 buckets, and so on.

[0063] Then, the test system's traffic distribution module samples traffic for each subtest in proportion to the number of configured buckets.

[0064] For example, if the traffic management tool divides the overall test traffic into 10,000 buckets and configures 2,500 buckets for each of the four subtests, the traffic allocation module can select the corresponding test object for each subtest based on the traffic management tool's configuration.

[0065] Then, the business logic module in the business scenario comparison system needs to deploy an agent module. The business logic module can read the test configuration and traffic distribution information from the shared content in the agent module through the application programming interface (API). It can randomly sample test objects through the API to select a test population, and then test the test objects using the test configuration.

[0066] When there are special requirements for the test population, such as requiring female college students over 24 years old, the business logic module can read the test subject's profile (cookie) information through the profiling module in the product verification test system and pass the cookie information to the API, thereby ensuring that the test subjects meet the filtering conditions when randomly sampling. The cookie information of these test subjects can then be written to the product verification test system through the ProfileSvr module.

[0067] In addition, if testers need to strictly control the number of test subjects, for example, a maximum of 100,000 test subjects can participate in the test, the number limit can be synchronized to the cookie through the accurateSvr module in the business scenario comparison system, so that the number of test subjects will not exceed this limit during API sampling.

[0068] After configuring the test configuration and allocating traffic, you can compare business scenarios, collect test feedback data, and store it in the data storage center. When comparing business scenarios, you can send the test feedback data from the data storage center to the test data analysis module. After receiving the test feedback data, the test data analysis unit can use the test verification module in the test data analysis unit to compare the business scenarios.

[0069] In one embodiment, if Figure 1 As shown, the business scenario comparison system can be integrated into an electronic device such as a terminal or server to implement the business scenario comparison method proposed in the embodiments of the present application. Specifically, the terminal 10 can send user test data to the server 20, and the server 20 can obtain the user test data, which includes experimental test data obtained based on the business operations of at least one group of experimental group users in the experimental business scenario, and control test data obtained based on the business operations of at least one group of control group users in the control business scenario, where the control business scenario has the same business type as the experimental business scenario.

[0070] Based on the experimental test data, the server 20 may determine an experimental test indicator for the experimental group of users, where the experimental test indicator indicates the ratio of the target business operation triggered by the user to the number of business resources delivered to the target business operation in the experimental business scenario. The server 20 may determine a control test indicator for the control group of users based on the control test data, where the control test indicator indicates the ratio of the target business operation triggered by the user to the number of business resources delivered to the target business operation in the control business scenario.

[0071] If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, the server 20 can determine the discrete degree information of the difference between the control test indicator and the experimental test indicator, and determine the relative conversion effect information of the business resources based on the discrete degree information and the difference between the experimental test indicator and the control test indicator. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resources in the experimental business scenario and the control business scenario.

[0072] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0073] The embodiment of the present invention will be described from the perspective of a business scenario comparison device, which can be integrated into a server or a terminal.

[0074] like Figure 2 As shown, the specific process of the business scenario comparison method of this embodiment can be as follows:

[0075] 201. Obtain user test data, where the user test data includes experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario has the same business type as the experimental business scenario.

[0076] A business scenario involves providing users with potentially needed or related products or services during a specific phase of their product use. For example, a business scenario could include marketing related products to specific product users, placing advertisements to product users, or modifying the product's front-end display interface.

[0077] When designing and operating business scenarios or application products, technical personnel often have many intuitive ideas, suspecting that certain designs and strategies may be better suited to user needs. However, validating these assumptions requires data. Controlled testing is often used to identify differences in metrics between experimental groups using different strategies and to determine whether these differences are statistically significant.

[0078] Among them, control testing, also known as AB testing, is an effective means of refined operations. AB testing can be to create two (A / B) or multiple (A / B / n) versions of a web page or application interface or process. In the same time dimension, visitor groups (users) with the same (or similar) composition are randomly allowed to access these versions, and user experience data and business data of each group are collected. Finally, the best version is analyzed and evaluated for formal adoption. For example, when conducting AB testing on a certain product, two plans can be formulated for the same optimization goal. Let some users use Plan A, while other users use Plan B. Statistics and comparison of conversion rates, click-through rates, retention rates and other indicators of different plans can be made to judge the pros and cons of different plans and make decisions, thereby improving conversion rates.

[0079] The experimental and control business scenarios can be business scenarios with the same test objective that employ different testing strategies. For example, if the business scenario involves changing the position of a control on a display page, the experimental business scenario could correspond to Test Plan A (control position in the lower left corner of the display interface), Test Plan B (control position in the upper right corner of the display interface), and Test Plan C (control position in the lower right corner of the display interface). The control business scenario could correspond to Test Plan D (control position in the upper left corner of the display interface).

[0080] In other examples, the experimental and control business scenarios can be different testing strategies for the same business scenario. For example, the business scenario might be delivering an advertisement to users of an application. The experimental business scenario could correspond to Test Plan A, "delivering advertisements to male users over 18 years of age," while the control business scenario could correspond to Test Plan B, "delivering advertisements to female users over 18 years of age," and so on.

[0081] It's understandable that to directly compare the effectiveness of a test solution, the control group uses a control business scenario. Generally, the solution currently used in the current business scenario or product is used as the control business scenario. Comparing the results of the experimental group with those of the control group can directly determine the pros and cons of the experimental group's test solution.

[0082] In some embodiments, when conducting a control test, there may be only one group of experimental group users and one group of control group users corresponding to the experimental group users, or there may be multiple groups of experimental group users and multiple groups of control group users corresponding to the experimental group users, or there may be multiple groups of experimental group users and one group of control group users corresponding to the experimental group users.

[0083] For example, a mutual control scheme can also be adopted in the process of control testing, that is, instead of setting up a separate control group, several experiments are used as controls for each other.

[0084] Among them, the experimental test data is the data obtained based on the business operations of the experimental group users in the experimental business scenario, such as the clicks, browsing time, etc. of the experimental group users. The control test data is the data obtained based on the business operations of the control group users in the control business scenario, such as the clicks, browsing time, etc. of the control group users.

[0085] In an example, the business scenario may be delivering advertisements to users, and the experimental test data and the control test data may be as shown in Table 1.

[0086] Test Group flow Hit exposure Click Control group A <![CDATA[p1]]> <![CDATA[n1]]> <![CDATA[v1]]> <![CDATA[c1]]> Experimental Group B <![CDATA[p2]]> <![CDATA[n2]]> <![CDATA[v2]]> <![CDATA[c2]]> … … … …

[0087] Table 1

[0088] Traffic, hits, exposures, clicks, and so on are all collected test data. Traffic refers to the number of test subjects (users) assigned to each test group. For example, control group A uses strategy A to deliver ads to 500 users, while experimental group B uses strategy B to deliver ads to 300 users. Hits refer to the number of test subjects assigned to each test group who were actually able to participate in the test. For example, 490 users assigned to control group A successfully received an ad, while 290 users assigned to experimental group B successfully received an ad. Exposures refer to the number of test subjects assigned to each test group who actually participated in the test. For example, 450 users in control group A viewed the ad, while 280 users in experimental group B viewed the ad. Clicks refer to the number of test subjects who actually clicked on the ad link. For example, 200 users in the control group clicked on the ad link, while 200 users in experimental group B clicked on the ad link.

[0089] 202. Determine an experimental test indicator for the experimental group of users based on the experimental test data. The experimental test indicator is used to indicate the ratio of the target business operation triggered by the user in the experimental business scenario to the amount of business resources allocated corresponding to the target business operation.

[0090] The target business operation may be a specific operation type determined by a technical person, such as a click on a control or a link, no operation on a displayed page, the duration of manipulation of a certain control, and the like.

[0091] Among them, the specific number of business resources delivered can be the number of test objects (users) allocated to each test group, or the number of test objects that can actually participate in the test among the test objects allocated to each test group, or the ratio of the test objects allocated to each test group to all test objects, etc. The specific type of data selected from the experimental test data as the number of business resources delivered can be determined by technical personnel based on actual conditions, and the embodiments of the present invention do not limit this.

[0092] In some embodiments, the step of “determining experimental test indicators for users in the experimental group based on the experimental test data” includes:

[0093] Based on the experimental test data, obtain the first number of users who trigger the target business operation in the experimental business scenario, and the first number of users who deploy business resources in the experimental business scenario;

[0094] The ratio of the number of first users to the number of first delivery users is calculated as the experimental test indicator.

[0095] For example, the experimental business scenario could be a new front-end display interface for an app. The target business operation is continuous app usage for more than 10 minutes. The first user population for business resources is the number of app users who were pushed the new front-end display interface. The experimental test metric would be the percentage of app users who used the new front-end display interface and who used the app for more than 10 minutes.

[0096] In one example, the experimental business scenario may be a content push scenario that adopts an experimental push strategy. In this case, the step of "obtaining, based on experimental test data, the first number of users who trigger the target business operation in the experimental business scenario and the first number of users who deliver business resources in the experimental business scenario" may include:

[0097] Based on the experimental test data, obtain the first number of users who clicked on the pushed content in the experimental business scenario, as well as the first number of users who delivered the pushed content in the experimental business scenario.

[0098] The experimental push strategy can push content to users who meet preset conditions. For example, it can push book reading content to women over 20 years old, or push content to all users during a certain period of time. The pushed content can be product promotion links, product display pages, etc.

[0099] 203. Determine a control test index for the control group users based on the control test data. The control test index is used to indicate the ratio of the target business operation triggered by the user in the control business scenario to the amount of business resources delivered corresponding to the target business operation.

[0100] In AB testing, control test indicators and experimental test indicators may include click-through rate, usage time, etc. In this embodiment, specific indicators can also be set according to the needs of relevant personnel, or according to the specific business scenarios that need to be tested. For example: if the program is a news APP, the indicator may be click-through rate or browsing time; if the program is a game APP, the indicator may be user usage time or traffic peak. The specific indicator may depend on the actual scenario, and the present invention does not limit this.

[0101] It is understandable that the generation of indicator information can be input by the user, that is, the user configures various indicator information based on relevant experience before AB testing; the generation of indicator information can also be automatically generated by the system, that is, the AB testing system refers to the setting records of historical indicator information and selects indicator information with a higher selection rate.

[0102] For example, control test indicators and experimental test indicators can be mean indicators, such as user usage time, traffic peak, etc., or they can be proportion indicators, such as click-through rate, etc.

[0103] In some embodiments, if the control test indicator and / or the experimental test indicator is not a ratio indicator but a mean indicator, then to compare the business scenarios, it is not necessary to perform steps 203, 204, and 205. Instead, the steps of calculating the statistic corresponding to the mean indicator based on the mean indicator and determining the relative expected effect between the control business scenario and the experimental business scenario based on the statistic are performed. That is, the control test indicator is not the ratio between the execution information of the target type business operation in the control business scenario and the business resources pre-allocated to the control group, and / or the experimental test indicator cannot indicate the ratio between the target business operation triggered by the user in the experimental business scenario and the number of business resources corresponding to the target business operation;

[0104] After the step of "determining the control test indicators corresponding to the control group users and the experimental test indicators corresponding to the experimental group users", the following steps are also included:

[0105] If the deviation between the experimental test indicator and the control test indicator satisfies the preset deviation condition, the step of determining the dispersion degree information of the difference between the control test indicator and the experimental test indicator is not performed, and the assumption that there is no significant difference between the means of the control test indicator and the experimental test indicator is performed;

[0106] Calculate the paired sample statistics corresponding to the user test data according to the preset paired sample statistics calculation method;

[0107] Based on paired sample statistics, relative conversion effect information of business resources is determined. The relative conversion effect information is used to indicate the quality of conversion effects of business resources in the experimental business scenario and the control business scenario.

[0108] Among them, the calculation method of paired sample statistics can be shown as the following formula:

[0109]

[0110] in, is the mean of the paired sample differences, is the standard deviation of the paired sample differences, and n is the number of paired samples.

[0111] In some optional examples, step 203 may include:

[0112] Based on the control test data, obtain the number of second users who trigger the target business operation in the control business scenario and the number of second users who deploy business resources in the control business scenario;

[0113] The ratio of the number of second users to the number of second delivery users is calculated as the control test indicator.

[0114] For example, the control business scenario could be the original front-end display interface of an app, the target business operation is continuous app usage for more than 10 minutes, and the secondary user number of business resources is the number of app users who have not been pushed the new front-end display interface. The control test indicator is the proportion of app users who have continuously used the app for more than 10 minutes among those who have not been pushed the new front-end display interface.

[0115] In an optional example, specifically, the control business scenario is a content push scenario that adopts a control push strategy. The step of "obtaining, based on the control test data, the number of second users who trigger the target business operation in the control business scenario and the number of second users who deliver the business resources in the control business scenario" may include:

[0116] Based on the control test data, the number of second users who click on the pushed content in the control business scenario and the number of second delivery users of the pushed content in the control business scenario are obtained.

[0117] The push strategy may be to push content to all users, or to push content to users who meet another preset condition, etc. For example, book content may be pushed to male users who are older than 20.

[0118] 204. If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, determine the discrete degree information of the difference between the control test indicator and the experimental test indicator.

[0119] Understandably, if the sample ratios of the experimental and control groups deviate from expectations, for example, if two different front-end display pages are tested for users, even if the test samples are distributed simultaneously to both groups, there's no guarantee that the distribution speed will be consistent. It's likely that inconsistent network conditions, differences in user activity, and other factors during the two distribution periods will trigger unnecessary changes in variables, ultimately leading to deviations between the test and reference ratios for the experimental and control groups. Using the reference ratio in analysis can distort the comparison results and the actual situation, and in serious cases, lead to erroneous conclusions.

[0120] Therefore, a deviation calculation can be performed based on the test ratio and the reference ratio. If the deviation does not meet the preset deviation condition, for example, there is no deviation between the test ratio and the reference ratio, or the deviation between the test ratio and the reference ratio does not affect the results of the business scenario comparison, step 204 can be skipped and the difference information can be calculated based on the test ratio. That is, before the step of "calculating the discrete degree information of the difference between the control test indicator and the experimental test indicator based on the control test indicator and the experimental test indicator", the following steps are also included:

[0121] If the deviation between the experimental test indicator and the control test indicator does not meet the preset deviation condition, the step of determining the discrete degree information of the difference between the control test indicator and the experimental test indicator is not performed;

[0122] Based on the control test index and the experimental test index, determining at least two differences between the control test index and the experimental test index, and performing continuity conversion based on the at least two differences to obtain a continuity difference;

[0123] Calculate the mean of the continuity difference and estimate the limiting distribution of the continuity difference when the continuity difference increases infinitely;

[0124] Based on the extreme distribution and the difference between the experimental test indicators and the control test indicators, the relative conversion effect information of the business resources is determined. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0125] The continuity difference value can be obtained by performing Taylor expansion on the difference between the control test index and the experimental test index. It is understandable that the value of the continuity difference value is not necessarily a true or exact value, and may also be a value with a certain estimation error.

[0126] In other embodiments, based on a preset user allocation algorithm, the users in the experimental group may be divided into a first preset number of experimental groups, and the users in the control group may be divided into a first preset number of control groups.

[0127] The user allocation algorithm may be a random traffic allocation algorithm such as a hash algorithm, an orthogonal table algorithm, etc., which is not limited in the embodiment of the present invention.

[0128] Accordingly, after dividing the experimental group users into experimental groups, step 202 may include:

[0129] Based on the experimental test data of each experimental group, the experimental test indicators of each experimental group are determined.

[0130] The number of experimental test indicators in each experimental group is the same as the first preset number.

[0131] Correspondingly, after dividing the control group users into the control group, step 203 may include:

[0132] Based on the experimental test data of each control group, the experimental test indicators of each control group are determined.

[0133] The number of control test indicators in each control group is the same as the first preset number.

[0134] In an optional example, step 204 may include: calculating the standard deviation of the difference between the control test index and the experimental test index based on the experimental test index of each experimental group and the control test index of each control group, where the standard deviation is the degree of dispersion information.

[0135] It is understandable that in addition to the standard deviation, parameters such as variance and mean value can also be used as the degree of dispersion information. Technicians can determine appropriate parameters as the degree of dispersion information based on actual conditions.

[0136] In another embodiment, the discrete degree information of the experimental test indicator and the control test indicator may be calculated separately, and then the discrete degree information of the difference between the experimental test indicator and the control test indicator may be calculated. That is, the step of "determining the discrete degree information of the difference between the control test indicator and the experimental test indicator" may also include:

[0137] Determining first discrete degree information of the experimental test indicator based on the experimental test data;

[0138] Determining second discrete degree information of the control test indicator based on the control test data;

[0139] The discrete degree information of the difference between the experimental test index and the control test index is determined according to the first discrete degree information and the difference between the second discrete degree information.

[0140] For example, taking the data in Table 1 as an example, the dispersion information of the control group A can be calculated through the following process:

[0141]

[0142] Among them, E[X]=c1 / n1, E[N]=n1, Var[N]=n1(1-p1), so the discrete degree information of control group A can be

[0143] Where N is a statistic that represents the number of samples required during the test. n is the number of observed samples, and se is the sample standard deviation.

[0144] 205. Based on the discrete degree information and the difference between the experimental test index and the control test index, determine the relative conversion effect information of the business resources. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resources in the experimental business scenario and the control business scenario.

[0145] It is understandable that since the difference between the experimental test indicators and the control test indicators is used as an intermediate quantity in the calculation during the comparison of business scenarios, the relative conversion effect information finally calculated cannot directly reflect the conversion effect of the actual business resources of the experimental business scenario and / or the control business scenario, but can only reflect the relative conversion effect of the experimental business scenario relative to the control business scenario, or the relative conversion effect of the control business scenario relative to the experimental business scenario.

[0146] Optionally, if the actual conversion effect of the control business scenario or the experimental business scenario is known, the actual conversion effect of another business scenario can be calculated based on the relative conversion effect information.

[0147] In some embodiments, the dispersion information may be standardized to facilitate calculation. That is, the step of "determining the relative conversion effect information of the business resource based on the dispersion information and the difference between the experimental test indicator and the control test indicator" may include:

[0148] Based on the discrete degree information, the difference between the experimental test index and the control test index is standardized to obtain the standardized difference;

[0149] Based on the standardized difference, as well as the dispersion information and user test data, the confidence interval of the difference between the experimental test indicator and the control test indicator is determined. The confidence interval is the relative conversion effect information of the business resources.

[0150] The calculation process of the standardized difference can be shown as follows:

[0151]

[0152] Among them, t represents the standardized difference, and ∑ represents the degree of dispersion information.

[0153] The calculation process of the critical point of the confidence interval can be shown in the following formula:

[0154]

[0155] Here, α represents the confidence level. Generally, a confidence interval of 95% can be taken, that is, α = 0.05. The value of Z can be determined by looking up the table based on the value of α. For example, when α = 0.05, the value of Z can be determined by looking up the table to be 1.645, and so on.

[0156] As can be seen from the above, the embodiments of the present invention can avoid the influence of different deployment conditions of business resources in different business scenarios, improve the accuracy of business scenario comparison, and ensure the correctness of business scenario test results.

[0157] The method described in the above embodiments will be further described in detail below with examples.

[0158] In this embodiment, the Figure 1 system is described.

[0159] like Figure 4 As shown, the business scenario comparison method of this embodiment may have the following specific process:

[0160] 401. The terminal sends test data obtained based on the business operation of the terminal user in the business scenario to the server.

[0161] Among them, multiple terminals can send corresponding test data directly to the server, or the terminals can send the test data collected by each terminal to the cloud server. When the server needs to compare business scenarios, it can obtain test data from the cloud server.

[0162] 402. The server obtains test data from at least two terminals to obtain user test data.

[0163] The test data may include experimental group user identifiers and control group user identifiers. After the server obtains the test data, it may classify the test data according to the identifier information of the test data to obtain user test data including experimental test data and control test data.

[0164] 403. The server determines the experimental test indicators for the experimental group users based on the experimental test data, and determines the control test indicators for the control group users based on the control test data.

[0165] Optionally, the control test indicators and experimental test indicators can be mean indicators, such as user usage time, traffic peak, etc., or ratio indicators, such as click-through conversion rate, etc. Specific indicators can also be set according to the needs of relevant personnel or based on the specific business scenarios to be tested.

[0166] In one example, if the control test indicator and / or the experimental test indicator is not a ratio indicator but a mean indicator, the step of determining the dispersion degree information of the difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition may not be performed, and the assumption that there is no significant difference between the means of the control test indicator and the experimental test indicator may be performed;

[0167] Calculate the paired sample statistics corresponding to the user test data according to the preset paired sample statistics calculation method;

[0168] Based on paired sample statistics, relative conversion effect information of business resources is determined. The relative conversion effect information is used to indicate the quality of conversion effects of business resources in the experimental business scenario and the control business scenario.

[0169] 404. The server calculates the deviation between the experimental test indicator and the control test indicator. If the deviation between the experimental test indicator and the control test indicator meets a preset deviation condition, the server determines the discrete degree information of the difference between the control test indicator and the experimental test indicator.

[0170] In one example, if the deviation does not meet the preset deviation condition, for example, there is no deviation between the test ratio and the reference ratio, or the deviation between the test ratio and the reference ratio does not affect the result of the business scenario comparison, the step of determining the discrete degree information of the difference between the control test indicator and the experimental test indicator may not be performed;

[0171] Based on the control test index and the experimental test index, determining at least two differences between the control test index and the experimental test index, and performing continuity conversion based on the at least two differences to obtain a continuity difference;

[0172] Calculate the mean of the continuity difference and estimate the limiting distribution of the continuity difference when the continuity difference increases infinitely;

[0173] Based on the extreme distribution and the difference between the experimental test indicators and the control test indicators, the relative conversion effect information of the business resources is determined. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0174] In another example, if the deviation satisfies a preset deviation condition, a step of determining discrete degree information of the difference between the control test indicator and the experimental test indicator is performed.

[0175] 405. The server normalizes the difference between the experimental test indicator and the control test indicator based on the discrete degree information to obtain a normalized difference.

[0176] The calculation process of the standardized difference can refer to the process of calculating the standardized random variable for a random variable with a mathematical expectation of 0 and a variance equal to 1. For any random variable X, X*=(X-EX) / √DX is the standardized random variable.

[0177] In the embodiment of the present invention, a null hypothesis may be made for the difference between the control test index and the experimental test index, that is, H0: c1 / p1-c2 / p2=0.

[0178] Therefore, the central limit theorem can be applied, and it is believed that the difference between the control test index and the experimental test index satisfies the distribution: c1 / p1-c2 / p2~N(0,∑ 2 ). Therefore, the calculation process of the standardized difference can be as follows:

[0179]

[0180] 406. The server determines a confidence interval of the difference between the experimental test indicator and the control test indicator based on the standardized difference, the dispersion information, and the user test data. The confidence interval is information about the relative conversion effect of the business resource.

[0181] The calculation process of the critical point of the confidence interval can be shown in the following formula:

[0182]

[0183] Wherein, α represents the confidence level, and generally the confidence interval can be set to 95%, that is, α = 0.05. The value of Z can be determined by looking up the value of α in the table. α can also be set to 0.1, 0.01, etc., and can be set by technicians according to their needs. The present invention does not limit this.

[0184] As can be seen from the above, the embodiments of the present invention can avoid the influence of different deployment conditions of business resources in different business scenarios, improve the accuracy of business scenario comparison, and ensure the correctness of business scenario test results.

[0185] In order to better implement the above method, accordingly, an embodiment of the present invention further provides a business scenario comparison device.

[0186] refer to Figure 5 , the device comprises:

[0187] The data acquisition unit 601 is configured to acquire user test data, wherein the user test data includes experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario has the same business type as the experimental business scenario.

[0188] An experimental indicator determination unit 602 is configured to determine an experimental test indicator for the experimental group of users based on the experimental test data. The experimental test indicator indicates the ratio of the target business operation triggered by the user to the amount of business resources allocated to the target business operation in the experimental business scenario.

[0189] A control index determining unit 603 is configured to determine a control test index for users in the control group based on the control test data. The control test index indicates the ratio of target business operations triggered by users in the control business scenario to the amount of business resources allocated to the target business operations.

[0190] The dispersion degree determining unit 604 is configured to determine the dispersion degree information of the difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator meets a preset deviation condition;

[0191] The effect determination unit 605 is used to determine the relative conversion effect information of the business resources based on the discrete degree information and the difference between the experimental test indicators and the control test indicators. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0192] In an optional example, the effect determination unit 605 may also be configured to normalize the difference between the experimental test index and the control test index based on the discrete degree information to obtain a normalized difference;

[0193] Based on the standardized difference, dispersion information and user test data, the confidence interval of the difference between the experimental test indicator and the control test indicator is determined. The confidence interval is the relative conversion effect information of the business resources.

[0194] In an alternative example, Figure 6 As shown, the service scenario comparison device provided by the embodiment of the present invention further includes a user division unit 606, which is used to divide the experimental group users into a first preset number of experimental groups and the control group users into a first preset number of control groups based on a preset user allocation algorithm;

[0195] An experimental indicator determination unit 602 is configured to determine an experimental test indicator for each experimental group based on the experimental test data of each experimental group;

[0196] A control index determination unit 603 is used to determine the experimental test index of each control group based on the experimental test data of each control group;

[0197] The dispersion degree determining unit 604 is used to calculate the standard deviation of the difference between the control test index and the experimental test index based on the experimental test index of each experimental group and the control test index of each control group. The standard deviation is the dispersion degree information.

[0198] In an optional example, the discrete degree determining unit 604 is configured to determine first discrete degree information of the experimental test indicator based on the experimental test data;

[0199] Determining second discrete degree information of the control test indicator based on the control test data;

[0200] The discrete degree information of the experimental test index and the difference between the experimental test indexes is determined according to the first discrete degree information and the difference between the second discrete degree information.

[0201] In an optional example, the experimental indicator determination unit 602 includes an experimental data acquisition unit 607 and an experimental indicator calculation unit 608. The experimental data acquisition unit 607 is used to obtain, based on the experimental test data, the first number of users who trigger the target business operation in the experimental business scenario and the first number of users who deploy business resources in the experimental business scenario;

[0202] The experimental index calculation unit 608 is used to calculate the ratio of the first number of users to the first number of users for delivery as the experimental test index;

[0203] The control indicator determination unit 603 includes a control data acquisition unit 609 and a control indicator calculation unit 610. The control data acquisition unit 609 is used to obtain the number of second users who trigger the target business operation in the control business scenario and the number of second users for business resources in the control business scenario based on the control test data.

[0204] The control index calculation unit 610 is used to calculate the ratio of the second number of users to the second delivery user number as a control test index.

[0205] In an optional example, the experimental business scenario is a content push scenario that adopts an experimental push strategy, and the control business scenario is a content push scenario that adopts a control push strategy;

[0206] The experimental data acquisition unit 607 is used to acquire the first number of users who click on the pushed content in the experimental business scenario and the first number of users who receive the pushed content in the experimental business scenario based on the experimental test data;

[0207] The control data acquisition unit 609 is configured to acquire, based on the control test data, the second number of users who click on the pushed content in the control business scenario and the second number of users who receive the pushed content in the control business scenario.

[0208] In an optional example, before the discrete degree determination unit 604, the business scenario comparison device provided by the embodiment of the present invention further includes a second discrete degree determination unit 611, which is configured to not perform the step of determining the discrete degree information of the difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator does not meet the preset deviation condition;

[0209] Based on the control test index and the experimental test index, determining at least two differences between the control test index and the experimental test index, and performing continuity conversion based on the at least two differences to obtain a continuity difference;

[0210] Calculate the mean of the continuity difference and estimate the limiting distribution of the continuity difference when the continuity difference increases infinitely;

[0211] Based on the limit distribution, the discrete degree information of the difference between the control test index and the experimental test index is calculated;

[0212] Based on the discrete degree information and the difference between the experimental test indicators and the control test indicators, the relative conversion effect information of the business resources is determined. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0213] As can be seen from the above, the business scenario comparison device can avoid the influence of different deployment conditions of business resources in different business scenarios, improve the accuracy of business scenario comparison, and ensure the correctness of business scenario test results.

[0214] In addition, an embodiment of the present invention further provides an electronic device, which may be a terminal or a server. Figure 7 , which shows a schematic structural diagram of an electronic device involved in an embodiment of the present invention, specifically:

[0215] The electronic device may include components such as a radio frequency (RF) circuit 701, a memory 702 including one or more computer-readable storage media, an input unit 703, a display unit 704, a sensor 705, an audio circuit 706, a wireless fidelity (WiFi) module 707, a processor 708 including one or more processing cores, and a power supply 709. It will be understood by those skilled in the art that Figure 7 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0216] The RF circuit 701 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is handed over to one or more processors 708 for processing; in addition, uplink data is sent to the base station. Generally, the RF circuit 701 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 701 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0217] The memory 702 can be used to store software programs and modules. The processor 708 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, a phone book, etc.). In addition, the memory 702 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 708 and the input unit 703 with access to the memory 702.

[0218] The input unit 703 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, in one embodiment, the input unit 703 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can detect user touch operations on or near it (for example, operations performed by a user using a finger, stylus, or any other suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connected devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects signals generated by the touch operation, transmitting the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 708. It can also receive and execute commands from the processor 708. In addition, touch-sensitive surfaces can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 703 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0219] The display unit 704 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 704 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 708 to determine the type of touch event. The processor 708 then provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 7 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.

[0220] The terminal may also include at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that verify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration verification related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the terminal, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0221] Audio circuit 706, a speaker, and a microphone provide an audio interface between the user and the terminal. Audio circuit 706 converts received audio data into electrical signals and transmits them to the speaker, which then converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 706 and converted into audio data. The audio data is then processed by processor 708 and transmitted via RF circuit 701 to, for example, another terminal. Alternatively, the audio data is output to memory 702 for further processing. Audio circuit 706 may also include an earphone jack to allow communication between an external headset and the terminal.

[0222] WiFi is a short-range wireless transmission technology. The terminal can help users send and receive emails, browse web pages and access streaming media through the WiFi module 707. It provides users with wireless broadband Internet access. Figure 7 A WiFi module 707 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as needed without changing the essence of the invention.

[0223] Processor 708 is the terminal's control center, connecting all components of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 702 and accessing data stored in memory 702, it executes various terminal functions and processes data, thereby performing overall phone testing. Optionally, processor 708 may include one or more processing cores; preferably, processor 708 may integrate an application processor and a modem processor, with the application processor primarily handling the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 708.

[0224] The terminal also includes a power supply 709 (e.g., a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 708 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 709 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0225] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 708 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 708 will run the applications stored in the memory 702 to implement various functions as follows:

[0226] Obtaining user test data, where the user test data includes experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario is of the same business type as the experimental business scenario;

[0227] Based on the experimental test data, determine the experimental test indicators for the experimental group of users. The experimental test indicators are used to indicate the ratio of the target business operations triggered by users in the experimental business scenario to the number of business resources allocated to the target business operations.

[0228] Based on the control test data, determine the control test index for the control group users, where the control test index indicates the ratio of the target business operations triggered by the users in the control business scenario to the amount of business resources delivered corresponding to the target business operations;

[0229] If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, the discrete degree information of the difference between the control test indicator and the experimental test indicator is determined;

[0230] Based on the discrete degree information and the difference between the experimental test indicators and the control test indicators, the relative conversion effect information of the business resources is determined. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0231] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0232] To this end, an embodiment of the present invention provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the business scenario comparison methods provided in an embodiment of the present invention. For example, the instructions can execute the following steps:

[0233] Obtaining user test data, where the user test data includes experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario is of the same business type as the experimental business scenario;

[0234] Based on the experimental test data, determine the experimental test indicators for the experimental group of users. The experimental test indicators are used to indicate the ratio of the target business operations triggered by users in the experimental business scenario to the number of business resources allocated to the target business operations.

[0235] Based on the control test data, determine the control test index for the control group users, where the control test index indicates the ratio of the target business operations triggered by the users in the control business scenario to the amount of business resources delivered corresponding to the target business operations;

[0236] If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, the discrete degree information of the difference between the control test indicator and the experimental test indicator is determined;

[0237] Based on the discrete degree information and the difference between the experimental test indicators and the control test indicators, the relative conversion effect information of the business resources is determined. The relative conversion effect information is used to indicate the quality of the conversion effects of the business resources in the experimental business scenario and the control business scenario.

[0238] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0239] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0240] Since the instructions stored in the storage medium can execute the steps in any business scenario comparison method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any business scenario comparison method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0241] According to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the above embodiments.

[0242] The above is a detailed introduction to a business scenario comparison method, device, terminal and storage medium provided in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A business scenario comparison method, characterized in that: include: Obtaining user test data, the user test data including experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, where the control business scenario has the same business type as the experimental business scenario; Determining an experimental test indicator for the experimental group of users based on the experimental test data, the experimental test indicator being used to indicate a ratio of a target business operation triggered by a user in the experimental business scenario to a quantity of business resources delivered corresponding to the target business operation; Determining a control test indicator for the control group users based on the control test data, the control test indicator being used to indicate a ratio of the target business operation triggered by the user in the control business scenario to the amount of business resources delivered corresponding to the target business operation; If the deviation between the experimental test indicator and the control test indicator meets the preset deviation condition, determining the discrete degree information of the difference between the control test indicator and the experimental test indicator; Determining relative conversion effect information of the business resource based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, wherein the relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario; When the control test indicator is not used to indicate the ratio between the execution information of the target type business operation in the control business scenario and the business resources pre-allocated to the control group, and the experimental test indicator is not used to indicate the ratio between the target business operation triggered by the user in the experimental business scenario and the number of business resources corresponding to the target business operation, the statistic corresponding to the mean indicator is calculated based on the mean indicator, and the relative expected effect between the control business scenario and the experimental business scenario is determined based on the statistic.

2. The business scenario comparison method according to claim 1, characterized in that: The determining of the relative conversion effect information of the business resource based on the dispersion degree information and the difference between the experimental test indicator and the control test indicator includes: Based on the dispersion degree information, normalizing the difference between the experimental test index and the control test index to obtain a standardized difference; Based on the standardized difference, the dispersion information and the user test data, a confidence interval of the difference between the experimental test indicator and the control test indicator is determined, and the confidence interval is the relative conversion effect information of the business resource.

3. The business scenario comparison method according to claim 1, characterized in that: Also includes: Based on a preset user allocation algorithm, the users in the experimental group are divided into a first preset number of experimental groups, and the users in the control group are divided into the first preset number of control groups; Determining the experimental test indicators of the experimental group users based on the experimental test data includes: Determine the experimental test indicators of each experimental group based on the experimental test data of each experimental group; Determining the control test indicators of the control group users based on the control test data includes: Determine the experimental test indicators of each control group based on the experimental test data of each control group; The determining of the discrete degree information of the difference between the control test index and the experimental test index includes: Based on the experimental test index of each experimental group and the control test index of each control group, the standard deviation of the difference between the control test index and the experimental test index is calculated, and the standard deviation is the dispersion degree information.

4. The business scenario comparison method according to claim 1, characterized in that: The determining of the discrete degree information of the difference between the control test index and the experimental test index includes: Determining first discrete degree information of the experimental test indicator based on the experimental test data; Determining second discrete degree information of the control test indicator based on the control test data; The discrete degree information of the difference between the experimental test index and the control test index is determined according to the first discrete degree information and the difference between the second discrete degree information.

5. The business scenario comparison method according to claim 1, characterized in that: Determining the experimental test indicators of the experimental group users based on the experimental test data includes: Based on the experimental test data, obtaining the first number of users who trigger the target business operation in the experimental business scenario and the first number of users who deploy the business resources in the experimental business scenario; Calculate the ratio of the first number of users to the first number of users for delivery as an experimental test indicator; Determining the control test index of the control group users based on the control test data includes: Based on the control test data, obtaining the second number of users who trigger the target business operation in the control business scenario and the second number of users who deploy the business resources in the control business scenario; The ratio of the second number of users to the second number of users for delivery is calculated as a control test indicator.

6. The business scenario comparison method according to claim 5, characterized in that: The experimental business scenario is a content push scenario that adopts the experimental push strategy, and the control business scenario is a content push scenario that adopts the control push strategy; The obtaining, based on the experimental test data, the first number of users triggering the target business operation in the experimental business scenario and the first number of users delivering the business resource in the experimental business scenario includes: Based on the experimental test data, obtaining a first number of users who clicked on the pushed content in the experimental business scenario and a first number of users who delivered the pushed content in the experimental business scenario; The obtaining, based on the control test data, the second number of users who trigger the target business operation in the control business scenario and the second number of users for delivering the business resources in the control business scenario includes: Based on the control test data, the second number of users who click on the pushed content in the control business scenario and the second number of users for delivering the pushed content in the control business scenario are obtained.

7. The business scenario comparison method according to any one of claims 1 to 6, characterized in that: Before determining the discrete degree information of the difference between the control test index and the experimental test index, the method further includes: If the deviation between the experimental test indicator and the control test indicator does not meet the preset deviation condition, the step of determining the discrete degree information of the difference between the control test indicator and the experimental test indicator is not performed; Determining at least two differences between the control test indicator and the experimental test indicator based on the control test indicator and the experimental test indicator, and performing continuity conversion based on the at least two differences to obtain a continuity difference; Calculating the mean of the continuity difference values, and estimating the limiting distribution of the continuity difference values ​​when the continuity difference values ​​increase infinitely; Calculating the discrete degree information of the difference between the control test index and the experimental test index based on the limit distribution; Based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, the relative conversion effect information of the business resource is determined. The relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario.

8. A business scenario comparison device, characterized in that: include: a data acquisition unit, configured to acquire user test data, the user test data including experimental test data obtained based on business operations of at least one group of experimental group users in an experimental business scenario, and control test data obtained based on business operations of at least one group of control group users in a control business scenario, the control business scenario being of the same business type as the experimental business scenario; An experimental indicator determination unit, configured to determine an experimental test indicator for the experimental group of users based on the experimental test data, wherein the experimental test indicator indicates a ratio of a target business operation triggered by a user in the experimental business scenario to a quantity of business resources allocated to the target business operation; a control indicator determination unit, configured to determine a control test indicator for the control group users based on the control test data, wherein the control test indicator is used to indicate a ratio of the target business operation triggered by the user in the control business scenario to the amount of business resources delivered corresponding to the target business operation; a dispersion degree determining unit, configured to determine dispersion degree information of a difference between the control test indicator and the experimental test indicator if the deviation between the experimental test indicator and the control test indicator meets a preset deviation condition; an effect determination unit, configured to determine relative conversion effect information of the business resource based on the discrete degree information and the difference between the experimental test indicator and the control test indicator, wherein the relative conversion effect information is used to indicate the quality of the conversion effect of the business resource in the experimental business scenario and the control business scenario; When the control test indicator is not used to indicate the ratio between the execution information of the target type business operation in the control business scenario and the business resources pre-allocated to the control group, and the experimental test indicator is not used to indicate the ratio between the target business operation triggered by the user in the experimental business scenario and the number of business resources corresponding to the target business operation, the device is used to calculate the statistic corresponding to the mean indicator based on the mean indicator, and determine the relative expected effect between the control business scenario and the experimental business scenario based on the statistic.

9. A terminal, characterized in that: It comprises a memory and a processor; the memory stores an application, and the processor is used to run the application in the memory to execute the steps in the business scenario comparison method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute the steps in the business scenario comparison method according to any one of claims 1 to 7.

Images