A method, system, device and storage medium for implementing an AB test

Abstract

The present application proposes a method, system, device and storage medium for implementing an AB test. The method includes: receiving a request sent by a terminal, which includes the unique identifier of the terminal; determining the experimental combination version corresponding to the terminal according to the unique identifier and the experimental grouping information of the current application; allocating an experimental version identifier corresponding to the experimental combination version, and sending the experimental version identifier to the terminal; the terminal accesses the experimental combination version according to the experimental version identifier, and obtains the experimental configuration information of the experimental combination version; different experimental tests are implemented according to the experimental configuration information. The present application automatically determines the experimental combination version and the experimental version identifier corresponding to the terminal, and transmits the experimental configuration through the experimental version identifier. All requests carry the experimental version identifier, which makes it more convenient to count experimental data and eliminates the need for customized report development. It realizes automatic counting of buried points, reports and metrics, and has high experimental efficiency. Multiple experiments can be carried out simultaneously without limitation on the number of experiments, enabling more AB experiments to be implemented.

Description

Technical Field

[0001] This application belongs to the technical field of software development, and particularly relates to a method, system, device, and storage medium for implementing an AB test. Background Art

[0002] An AB test is to build two or more versions for an application or system. In the same time dimension, different versions are randomly or regularly accessed by groups of visitors with the same or similar structures. Then, a data collection tool is used to collect multiple groups of user data and business data. Finally, the optimal version is evaluated based on business metrics and officially adopted.

[0003] With the rapid development of the Internet, more and more product technology iterations start to use AB tests to evaluate benefits. However, using AB tests adds extra workload and a longer product iteration cycle. At the same time, AB tests are often unable to be implemented due to long development processes and insufficient R & D resources, resulting in low efficiency of current AB tests, difficulty in conducting multiple tests simultaneously, and obvious upper limits on the number of tests. Summary of the Invention

[0004] This application proposes a method, system, device, and storage medium for implementing an AB test. The test server automatically determines the experimental combined version and experimental version identifier corresponding to the terminal, and transmits the experimental configuration information through the experimental version identifier. All subsequent requests carry the experimental version identifier to achieve automated statistics of data collection, reports, and metrics, improving the test efficiency. Moreover, multiple tests can be conducted simultaneously without obvious limitations on the number of tests, enabling more AB tests to be implemented.

[0005] The first aspect embodiment of this application proposes a method for implementing an AB test, which is applied to a test server and includes:

[0006] Receiving a request sent by the terminal, where the request includes a unique identifier corresponding to the terminal;

[0007] Determining the experimental combined version corresponding to the terminal according to the unique identifier and the current experimental grouping information of the application;

[0008] Allocating a corresponding experimental version identifier according to the experimental combined version and sending the experimental version identifier to the terminal;

[0009] The terminal accesses the experimental combined version of the current application according to the experimental version identifier and obtains the experimental configuration information of the currently accessed experimental combined version;

[0010] Implementing different experimental tests according to the experimental configuration information.

[0011] In an embodiment of the present application, the experiment grouping information includes experiment identification information and / or experiment time information. Determining the experiment combination version corresponding to the terminal according to the unique identifier and the experiment grouping information of the current application includes:

[0012] Determining the experiment corresponding to the terminal in all experimental groups of the current application according to the unique identifier and the experiment grouping information;

[0013] Determining the set formed by the experiments corresponding to the terminal as the experiment combination version corresponding to the terminal.

[0014] In an embodiment of the present application, the experiment grouping information includes the experiment identification information, and the experiment identification information includes the experiment type of the current application, the scene identifier of the experiment scene, and the experiment group identifiers of all experimental groups included in the experiment scene; determining the experiment combination version corresponding to the terminal according to the unique identifier and the experiment grouping information of the current application includes:

[0015] If the experiment type is a single-scene experiment, determining the experiment corresponding to the terminal in each experimental group included in the experiment scene according to the unique identifier, the scene identifier of the experiment scene, and the experiment group identifiers of all experimental groups included in the experiment scene;

[0016] Determining the set formed by the experiments corresponding to the terminal in each experimental group as the experiment combination version corresponding to the terminal.

[0017] In an embodiment of the present application, the method further includes:

[0018] If the experiment type is a multi-scene experiment, selecting the experiment scene corresponding to the terminal from all the experiment scenes according to the unique identifier and the scene identifiers of all the experiment scenes of the current application;

[0019] Determining the experiment corresponding to the terminal from each experimental group included in the selected experiment scene respectively according to the unique identifier, the scene identifier of the selected experiment scene, and the experiment group identifiers of all experimental groups included in the selected experiment scene;

[0020] Determining the set formed by the experiments corresponding to the terminal in each experimental group included in the selected experiment scene as the experiment combination version corresponding to the terminal.

[0021] In an embodiment of the present application, determining the experiment corresponding to the terminal in each experimental group included in the experiment scene according to the unique identifier, the scene identifier of the experiment scene, and the experiment group identifiers of all experimental groups included in the experiment scene includes:

[0022] Concatenate the unique identifier, the scenario identifier of the experimental scenario, and the experimental group identifier of the first experimental group included in the experimental scenario into a first string according to a preset concatenation order, where the first experimental group is any experimental group in the experimental scenario;

[0023] Convert the first string into a second string with a preset data length through a preset algorithm;

[0024] Calculate the hash value of the second string using a digital hash algorithm and determine the preset numerical range to which the hash value belongs;

[0025] Determine the experiment corresponding to the preset numerical range as the experiment corresponding to the terminal in the first experimental group.

[0026] In the embodiment of the present application, after sending the experiment version identifier to the terminal, it further includes:

[0027] Receive the experiment version confirmation message sent by the terminal at preset time intervals, where the experiment version confirmation message includes the unique identifier;

[0028] Determine the experiment version identifier currently required to correspond to the terminal according to the unique identifier and the current experiment grouping information;

[0029] If the experiment version identifier currently required to correspond to the terminal is different from the experiment version identifier sent to the terminal last time, then send the experiment version identifier currently required to correspond to the terminal to the terminal, so that the terminal switches to the experiment combination version corresponding to the experiment version identifier currently required to correspond to.

[0030] In the embodiment of the present application, the method further includes:

[0031] Detect that the experiment grouping information of the current application has changed;

[0032] Obtain the unique identifiers of all terminals currently accessing the current application;

[0033] According to the unique identifier of each terminal and the changed experiment grouping information, respectively re-determine the experiment version identifier corresponding to each terminal;

[0034] Send the re-determined experiment version identifier corresponding to each terminal to each terminal respectively, so that each terminal currently accessing switches to the experiment combination version corresponding to the re-determined experiment version identifier.

[0035] In the embodiment of the present application, after sending the experiment version identifier to the terminal, it further includes:

[0036] Receive the buried point information sent by the terminal, where the buried point information includes the experiment version identifier;

[0037] Obtain the experiment configuration information corresponding to the experiment version identifier;

[0038] Correspondingly store the buried point information and the experiment configuration information in a preset database.

[0039] In an embodiment of the present application, the method further includes:

[0040] Obtain the experiment configuration information and the corresponding buried point information of different experiment combination versions from the preset database;

[0041] Generate an experiment test report according to preset analysis indicators, the experiment configuration information of different experiment combination versions, and the corresponding buried point information.

[0042] An embodiment of the second aspect of the present application provides a method for implementing an AB experiment, which is applied to a terminal and includes:

[0043] Send a request to a test server, where the request includes the unique identifier of the terminal;

[0044] Receive the experiment version identifier returned by the test server, where the experiment version identifier is the identifier of the experiment combination version corresponding to the terminal determined by the test server according to the unique identifier;

[0045] According to the experiment version identifier, obtain the experiment configuration information of the experiment combination version, and display the experiment combination version;

[0046] Implement different experiment tests according to the experiment configuration information.

[0047] In an embodiment of the present application, the step of obtaining the experiment configuration information of the experiment combination version according to the experiment version identifier and displaying the experiment combination version includes:

[0048] Send a loading request of the current application to a business server, where the loading request includes the experiment version identifier;

[0049] Receive the experiment configuration information corresponding to the experiment version identifier returned by the business server;

[0050] According to the experiment configuration information, display the application interface corresponding to the experiment combination version of the current application.

[0051] In an embodiment of the present application, the method further includes:

[0052] Send an experiment version confirmation message to the test server at preset time intervals, where the experiment version confirmation message includes the unique identifier;

[0053] Receive the new experiment version identifier returned by the test server when it confirms that the experiment combination version corresponding to the terminal has changed;

[0054] Load the experiment combination version corresponding to the new experiment version identifier.

[0055] In the embodiments of the present application, the method further includes:

[0056] Receive the experiment version identifier re-determined by the test server for the terminal when it detects that the experiment grouping information of the current application has changed;

[0057] Load the experiment combination version corresponding to the re-determined experiment version identifier.

[0058] In the embodiments of the present application, the method further includes:

[0059] Send the buried point information to the test server, where the buried point information includes the experiment version identifier, so that the test server generates an experiment test report according to the buried point information.

[0060] An embodiment of the third aspect of the present application provides a system for implementing an AB experiment, including: a terminal and a test server;

[0061] The terminal is used to send a request to the test server, and the request includes the unique identifier of the terminal;

[0062] The test server is used to determine the experiment combination version corresponding to the terminal according to the unique identifier and the experiment grouping information of the current application; allocate the corresponding experiment version identifier according to the experiment combination version, and send the experiment version identifier to the terminal;

[0063] The terminal is used to obtain the experiment configuration information of the experiment combination version according to the experiment version identifier, and display the experiment combination version; implement different experiment tests according to the experiment configuration information.

[0064] An embodiment of the fourth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor runs the computer program to implement the method described in the first aspect or the second aspect above.

[0065] An embodiment of the fifth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by the processor to implement the method described in the first aspect or the second aspect above.

[0066] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0067] In the embodiment of the present application, the test server automatically determines the experimental combination version corresponding to the terminal according to the unique identifier of the terminal and the experimental grouping information of the application program, and assigns a corresponding experimental version identifier to the terminal, and transmits the experimental configuration information through the experimental version identifier. All requests generated by the user using the application program subsequently carry the experimental version identifier, so that all business data and user data generated during the user's use of the application program are associated with the experimental version identifier. In this way, when generating an experimental report subsequently, it is possible to conveniently count the data corresponding to the same experimental combination version according to the experimental version identifier, without customizing the development of reports, realizing an automated process for statistical data embedding, statistical reports, and statistical indicators, and greatly improving the experimental efficiency. Moreover, multiple experiments can be carried out simultaneously, without obvious limitations on the number of experiments, enabling more A / B tests to be implemented.

[0068] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.

[0070] In the drawings:

[0071] Figure 1 shows a flowchart of a method for implementing an A / B test provided by an embodiment of the present application;

[0072] Figure 2 shows a shunt schematic diagram for a single-scenario multi-layer orthogonal experiment provided by an embodiment of the present application;

[0073] Figure 3 shows a shunt schematic diagram for a multi-scenario experiment provided by an embodiment of the present application;

[0074] Figure 4 shows a flowchart of another method for implementing an A / B test provided by an embodiment of the present application;

[0075] Figure 5 shows a structural schematic diagram of a system for implementing an A / B test provided by an embodiment of the present application;

[0076] Figure 6 shows a structural schematic diagram of a device for implementing an A / B test provided by an embodiment of the present application;

[0077] Figure 7 The figure shows a schematic structural diagram of another implementation device of the AB experiment provided by an embodiment of the present application;

[0078] Figure 8 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0079] Figure 9 The figure shows a schematic diagram of a storage medium provided by an embodiment of the present application. Detailed implementation manners

[0080] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully communicated to those skilled in the art.

[0081] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should have the ordinary meanings understood by those skilled in the art to which the present application belongs.

[0082] A method, system, device and storage medium for implementing an AB experiment proposed according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0083] Currently, in the process of application development and product iteration, AB experiments are often used to test different experimental versions, and the optimal experimental version is evaluated based on the user data, business data and set business indicators of different experimental versions, and finally the optimal experimental version is officially adopted. Since the AB experiments in the related technologies are usually self-built AB experiment services by different R & D teams, there will be a situation where multiple non-orthogonal experiments are executed simultaneously and affect each other, resulting in a low confidence level of the final experimental results. Moreover, after the AB experiment is launched, in the related technologies, customized development reports are also required, and there is a lack of reusable automated experimental reports, resulting in waste of human and computing resources. When operations such as AB experiment traffic adjustment or experiment online / offline are required, customized development and deployment are needed to achieve them, and the efficiency is low. And the AB experiment solutions in product operation often cannot be realized due to lack of buried point fields, too long development processes or insufficient R & D resources.

[0084] The embodiment of the present application provides a method for implementing an AB test. This method automatically assigns an experimental combination version to a user and uniquely identifies the experimental combination version corresponding to the user through an experimental version identifier. By using the experimental version identifier to transmit experimental configuration information, multiple experiments can be carried out simultaneously without an obvious limit on the number of experiments. It realizes the automated processes of statistical data embedding, statistical reports, and statistical metrics, greatly improving the experimental efficiency and enabling more AB tests to be implemented.

[0085] The above experimental combination version is the combination of all AB tests corresponding to the application. For example, assume that the application includes two groups of orthogonal AB tests. One group of AB tests includes two experiments, experiment a and experiment b, and the other group of AB tests includes two experiments, experiment c and experiment d. Then the experimental combination versions of this application can include four different experimental combination versions: experiment a and c, a and d, b and c, and b and d.

[0086] See Figure 1 , and the method specifically includes the following steps:

[0087] Step 101: The terminal sends a request to the test server, and the request includes the unique identifier of the terminal.

[0088] An application is installed on the terminal. The provider of the application can design one or more groups of AB tests corresponding to the application. For example, the background color of a certain button on the application's home page can be red or green, and the product information on a certain product details page of the application can be displayed in different forms such as pictures or videos.

[0089] Before the user opens the application on the terminal and the page of the application is displayed, it is necessary to first determine the experimental combination version corresponding to the user. When the terminal detects the opening instruction triggered by the user clicking the application, the terminal sends a request to the test server and carries the unique identifier of the terminal in the request. The unique identifier of the terminal can be the product serial number of the terminal, the MAC (Media Access Control Address) address, or the user identifier, etc. The user identifier can be the name or nickname of the user, etc.

[0090] Step 102: The test server receives the request sent by the terminal and determines the experimental combination version corresponding to the terminal according to the experimental grouping information of the current application and the unique identifier included in the request.

[0091] The experiment grouping information includes experiment identification information and / or experiment time information. Among them, the experiment identification information may include the current experiment type applied, the scene identification of the experiment scene, and the experiment group identification of all experiment groups included in the experiment scene, etc. Among them, the experiment type includes single-scene experiment and multi-scene experiment. A single-scene experiment means that only one experiment scene is set in the application program. Under this experiment scene, there are at least one group of experiment groups, and at least two experiments are included in one experiment group. A multi-scene experiment means that two or more experiment scenes are set in the application program. Each experiment scene includes at least one experiment group, and at least two experiments are included in each experiment group. The experiment time information is used to represent the implementation time of the experiment, and the implementation time can be the specific date of implementing the experiment. For at least two experiments included in the same experiment group, each experiment can be separately assigned to different users within the same implementation time to implement the experiment test. It can also be separately assigned to the same or different users within different implementation times to implement the experiment test.

[0092] The experiment scene can be a combination of one page or multiple pages in the current application. The scene identification can be the identification of the page corresponding to the experiment scene, and the identification of the page can be the name or address of the page, etc. Or, the scene identification can also be a random string that uniquely identifies the experiment scene randomly assigned to each experiment scene. The experiment group identification can be a random string that uniquely identifies the experiment group randomly assigned to the experiment group.

[0093] After the test server receives the request sent by the terminal, according to the experiment grouping information of the current application and the unique identification of the terminal carried by the request, it determines the experiment corresponding to the terminal in all experiment groups of the current application; and determines the set composed of the experiments corresponding to the terminal as the experiment combination version corresponding to the terminal.

[0094] Among them, the experiment grouping information may only include the above-mentioned experiment identification information. In this application scenario, the test server first determines whether the experiment type of the current application is a single-scene experiment or a multi-scene experiment. If the experiment type of the current application is a single-scene experiment, then according to the unique identification of the terminal, the scene identification of the experiment scene, and the experiment group identification of all experiment groups included in the experiment scene, it respectively determines the experiment corresponding to the terminal in each experiment group; and determines the set composed of the experiments corresponding to the terminal in each experiment group as the experiment combination version corresponding to the terminal.

[0095] For any experiment group in the experiment scene, the experiment corresponding to the terminal in the experiment group is determined in the same way. The following takes the first experiment group as an example to illustrate how to determine the experiment corresponding to the terminal in the experiment group, and any other experiment group in this experiment scene is determined in the same way as the first experiment group. Among them, the first experiment group is any experiment group in the experiment scene.

[0096] Specifically, the test server first concatenates the unique identifier of the terminal, the scene identifier of the experimental scene, and the experimental group identifier of the first experimental group included in the experimental scene into a first string according to a preset concatenation order. The preset concatenation order can be any concatenation order obtained by arranging the positions of the unique identifier, the scene identifier, and the experimental group identifier in combination, such as the order of the unique identifier, the scene identifier, and the experimental group identifier, or the order of the unique identifier, the experimental group identifier, and the scene identifier, etc.

[0097] After obtaining the above first string, the first string is converted into a second string with a preset data length through a preset algorithm. The preset algorithm can be a hash algorithm, MD4 (Message Digest Algorithm 4), or MD5 (Message Digest Algorithm 5) algorithm, etc. The preset data length can be 32 bits, 128 bits, etc. Calculate the hash value of the second string through digital hashing operations.

[0098] In the embodiment of the present application, the shunt interval corresponding to the terminal is determined by calculating the hash value. For each experiment in the experimental group, a preset numerical interval corresponding to each experiment is set in advance. For example, an experimental group includes experiments a and b, and the preset numerical interval corresponding to experiment a is set to [0, 500), and the preset numerical interval corresponding to experiment b is set to [500, +∞).

[0099] After calculating the hash value corresponding to the terminal in the above manner, determine the preset numerical interval to which the hash value belongs. Determine the experiment corresponding to the terminal in the first experimental group as the experiment corresponding to the preset numerical interval to which the hash value belongs.

[0100] As Figure 2 shown, for a single-scene multi-layer orthogonal experiment, that is, only one experimental scene is included, and in this experimental scene, there are independent experimental groups A to experimental group N. The test server receives a terminal request, calculates the hash value corresponding to the terminal, determines the shunt interval corresponding to the terminal according to the hash value, and thus determines the experiment corresponding to the terminal in experimental group A. Similarly, for experimental groups B to experimental group N, the experiment corresponding to the terminal in each experimental group is determined in the same manner.

[0101] For each experimental group included in the experimental scene, the experiment corresponding to the terminal in each experimental group is determined respectively in the above manner. The set composed of all the experiments corresponding to the terminal is the experimental combination version corresponding to the terminal.

[0102] If the current experiment type of the application is a multi-scenario experiment, then according to the unique identifier of the terminal and the scenario identifiers of all the experiment scenarios of the current application, select the experiment scenario corresponding to the terminal from all the experiment scenarios; according to the unique identifier, the scenario identifier of the selected experiment scenario, and the experiment group identifiers of all the experiment groups included in the selected experiment scenario, respectively determine the experiment corresponding to the terminal from each experiment group included in the selected experiment scenario; determine the set composed of the experiments corresponding to the terminal in each experiment group included in the selected experiment scenario as the experiment combination version corresponding to the terminal.

[0103] In a multi-scenario experiment, after the test server receives a request from a terminal, it first determines which experiment scenarios the terminal needs to be shunted to according to the unique identifier of the terminal and the scenario identifiers of each experiment scenario in the current application. Specifically, the unique identifier of the terminal and the scenario identifier of each experiment scenario are concatenated into a string according to a preset concatenation rule. The preset concatenation rule stipulates the concatenation order of the unique identifier and each scenario identifier. The string is converted into a string with a preset data length through a preset algorithm. The hash value of the converted string is calculated through a digital hash operation.

[0104] In the embodiments of the present application, it is judged which experiment scenarios the terminal needs to be shunted to through the calculated hash value. For each experiment scenario in the application program, a numerical range corresponding to each experiment scenario is preset. For example, the preset numerical range corresponding to experiment scenario A is [0, 800), and the preset numerical range corresponding to experiment scenario B is [800, +∞). After calculating the hash value in the above manner, determine the numerical range to which the hash value belongs. Determine the experiment scenario corresponding to the numerical range to which the hash value belongs as the experiment scenario corresponding to the terminal.

[0105] For any experiment scenario selected in the above manner, in the manner of the above single experiment scenario, calculate the hash value to respectively determine the experiment corresponding to the terminal in each experiment group included in the selected experiment scenario. The set composed of all the experiments corresponding to the terminal is the experiment combination version corresponding to the terminal.

[0106] As Figure 3 shown, for a multi-scenario experiment, calculate the hash value to determine which experiment scenarios the terminal is shunted to, and for the determined experiment scenarios, use the above Figure 2 shown manner to respectively determine the experiment corresponding to the terminal in each experiment group included in each experiment scenario.

[0107] In some other embodiments of the present application, the experiment grouping information may also include the above-mentioned experiment identification information and experiment time information at the same time. In this application scenario, similar to the processing method of the application scenario where the above-mentioned experiment grouping information only includes experiment identification information, single-scenario experiments and multi-scenario experiments are also distinguished. For each experimental group, the experiment corresponding to the terminal is determined in the same way as the above hash operation. In still some other embodiments of the present application, in the application scenario where the experiment grouping information may only include experiment time information, similar to the processing method of the above single-scenario experiment, for each experimental group, hash operation is performed according to the experiment time information to determine the experiment corresponding to the terminal.

[0108] Step 103: The test server assigns a corresponding experiment version identifier according to the experiment combination version and sends the experiment version identifier to the terminal.

[0109] After determining the experiment combination version corresponding to the terminal through the above step 102, an experiment version identifier is also assigned to the experiment combination version to uniquely identify the experiment combination version. And the experiment configuration information corresponding to the experiment combination version is obtained. The experiment configuration information includes the configuration information of each experiment included in the experiment combination version. The configuration information of the experiment includes page data corresponding to the experiment, implementation time, the user scale required for testing, etc. For example, assuming that the experiment combination version includes a group of experiments where the background color of a button on the home page is red or green, the experiment configuration information includes page data such as the page identifier of the home page and the identifier, position information, color attribute, etc. of the button, and also includes the implementation time "from January 1st to January 5th" and the user scale required for testing is 300 users, etc. The test server also stores the unique identifier of the terminal, the user identifier of the user, the experiment version identifier, and the experiment configuration information corresponding to the experiment combination version. Specifically, the test server can cache the unique identifier, user identifier, experiment version identifier, and the corresponding experiment configuration information in Redis (Remote Dictionary Server) correspondingly. Further, the test server can also store the unique identifier, user identifier, experiment version identifier, and the corresponding experiment configuration information in a MySQL (relational data management system) server in the cloud correspondingly.

[0110] The test server sends the experiment version identifier corresponding to the determined experiment combination version to the terminal so that the terminal can log in to the experiment combination version of the current application. The terminal stores the experiment version identifier locally.

[0111] Since the provider of the current application may modify the experimental groups in the current application, such as adding some experimental groups, deleting some experimental groups, or modifying the configuration information of some experimental groups. After modifying the experimental groups in the current application, the corresponding experimental grouping information of the current application will also be modified. In this case, the corresponding experimental grouping version of the terminal should also be updated in a timely manner.

[0112] In the embodiments of the present application, the terminal can periodically poll the test server through a thread to determine whether the experimental combination version corresponding to it has changed, and obtain the experimental version identifier corresponding to the new experimental combination version when it has changed. Specifically, the terminal sends an experimental version confirmation message to the test server every preset duration, and the experimental version confirmation message includes the unique identifier of the terminal. The test server receives the experimental version confirmation message sent by the terminal every preset duration, and re-determines the experimental version identifier that the terminal currently needs to correspond to according to the unique identifier included in the experimental version confirmation message and the current experimental grouping information in the manner of step 102. If the determined experimental version identifier that the terminal currently needs to correspond to is different from the experimental version identifier sent to the terminal last time, the determined experimental version identifier that the terminal currently needs to correspond to is sent to the terminal. If the determined experimental version identifier that the terminal currently needs to correspond to is the same as the experimental version identifier sent to the terminal last time, it indicates that the experimental grouping situation of the current application has not changed. The above preset duration can be 15 minutes, 30 minutes, 1 hour, etc.

[0113] In the case where the determined experimental version identifier that the terminal currently needs to correspond to is different from the experimental version identifier sent to the terminal last time, the terminal receives the new experimental version identifier returned by the test server when confirming that the experimental version corresponding to the terminal has changed, and loads the experimental combination version corresponding to the new experimental version identifier according to the operation of step 104 below.

[0114] In some other embodiments of the present application, the test server can also actively push the re-determined experimental version identifier to the terminal when detecting that the experimental grouping information of the current application has changed. Specifically, when the test server detects that the experimental grouping information of the current application has changed, it obtains the unique identifiers of all terminals currently accessing the current application from the stored unique identifiers, user identifiers, experimental version identifiers, and corresponding experimental configuration information. According to the unique identifier of each terminal and the changed experimental grouping information, the experimental version identifier corresponding to each terminal is re-determined respectively according to the operation of the above step 102. Then the test server sends the re-determined experimental version identifier corresponding to each terminal to each terminal respectively. Each terminal currently accessing the current application receives the re-determined experimental version identifier for the terminal by the test server when detecting that the experimental grouping information of the current application has changed, and loads the experimental combination version corresponding to the re-determined experimental version identifier according to the following operation of step 104.

[0115] Step 104: The terminal receives the experiment version identifier returned by the test server. According to this experiment version identifier, it obtains the experiment configuration information of the corresponding experiment combined version, and displays this experiment combined version, and implements different experiment tests according to this experiment configuration information.

[0116] The terminal receives the experiment version identifier sent by the test server, and sends a loading request for the current application to the business server. This loading request includes this experiment version identifier and the unique identifier of the terminal, etc. After receiving this loading request, the business server obtains the experiment configuration information corresponding to this experiment version identifier from the test server according to this experiment version identifier. According to the unique identifier of this terminal, the obtained experiment configuration information is sent to this terminal. The terminal receives the experiment configuration information corresponding to this experiment version identifier returned by the business server, and performs page rendering according to this experiment configuration information, and displays the application interface corresponding to the experiment combined version of the current application.

[0117] Through the above method, after the terminal logs in to the corresponding experiment combined version and displays the application interface of the current application, the user can operate on any interface in the currently displayed application interface, such as clicking some buttons or playing the video in the current interface, etc. In the embodiment of the present application, a data logging program is set at each interface included in the application interface of the application program. The data logging program is used to count the operation data of the user. After the terminal logs in to the current application, every preset time period, it synthesizes the operation data of the user counted by each data logging program, and sends the data logging information to the test server. This data logging information includes the experiment version identifier and the counted operation data of the user.

[0118] The test server receives the data logging information sent by the terminal, and according to the experiment version identifier included in the data logging information, obtains the experiment configuration information corresponding to this experiment version identifier from the cache or cloud storage. The data logging information and the experiment configuration information are stored in a preset database correspondingly. The preset database can be an analytical database ClickHouse (a columnar storage analytical database).

[0119] In the embodiment of the present application, when the terminal sends the data logging information to the test server, it can first send the data logging information to the message queue in the kafka server, and then the kafka server forwards the data logging information to the test server.

[0120] When a user needs to view the experimental effects of different experimental combination versions, the test server obtains the experimental configuration information and corresponding buried point information of different experimental combination versions from a preset database; according to the preset analysis metrics and the experimental configuration information and corresponding buried point information of different experimental combination versions, an experimental test report is generated. Among them, the preset analysis metrics may include common metrics and / or custom metrics. The common metrics may include user retention rate, order placement number, advertising revenue, etc. The custom metrics may be metrics customized by the provider of the application according to the business characteristics of the application. For example, the custom metrics in a short video application may be the number of video plays, the number of likes, the number of collections, etc.

[0121] In the embodiments of the present application, the experimental version identifier is carried in all message transmissions triggered by the interfaces and buried point programs of the application. After the system is developed once, there is no need to consider modifying the buried point program when new AB experiments are conducted for new services in the future. The data reports based on common metrics also do not require customized development. The common metrics are directly specified as the metrics in the experimental data reports, and a data analysis template is built in to achieve automatic generation of AB experiment reports. Moreover, through the experimental version identifier carried in the request sent by the terminal, the corresponding experimental shunt situation of the terminal can also be retrieved.

[0122] To facilitate understanding of the method provided by the embodiments of the present application, the following will be described with reference to the accompanying drawings. As Figure 4 shown, when the APP (i.e., the application) is started or called every 5 minutes, the test server returns the experimental version identifier to the client, and the client locally stores the experimental version identifier. The test server caches the experimental version identifier and the corresponding experimental configuration information in the Redis database, and permanently stores the experimental version identifier and the corresponding experimental configuration information in the MySQL server in the cloud. When the APP requests an interface, it carries the experimental version identifier, and the business server obtains the corresponding experimental configuration information from the Redis database or the MySQL server in the cloud in the test server and gives it to the APP. When the user generates business data during the use of the application, since buried point programs are set at the interfaces of the application. The exposure, clicks, etc. of the user are all counted through the buried point program, and the buried point information is uploaded. The buried point information carries the experimental version identifier. The buried point information is forwarded to the kafka message queue. The Kafka message determines the experimental configuration information corresponding to the experimental version identifier according to the experimental version identifier included in the buried point information, and stores the buried point information and the corresponding experimental configuration information in the analytical database ClickHouse. By analyzing the data stored in the analytical database ClickHouse, experimental data statistical reports, effect data statistical reports, etc. can be obtained. Figure 4The dashed line part in the figure indicates that the request log and return result can also be sent to the test server through the kafka message queue.

[0123] In the embodiment of the present application, the test server automatically determines the experimental combination version corresponding to the terminal according to the unique identifier of the terminal and the experimental grouping information of the application program, and assigns a corresponding experimental version identifier to the terminal, and transmits the experimental configuration information through the experimental version identifier. All requests generated by the subsequent user using the application program carry the experimental version identifier, so that all business data and user data generated during the user's use of the application program are associated with the experimental version identifier. In this way, when generating an experimental report later, it is convenient to count the data corresponding to the same experimental combination version according to the experimental version identifier, without customizing the development of reports, realizing the automated processes of statistical data collection, statistical reports, and statistical indicators, and greatly improving the experimental efficiency. Moreover, multiple experiments can be carried out simultaneously, without obvious limitations on the number of experiments, and more AB experiments can be implemented.

[0124] The embodiment of the present application also provides a system for implementing an AB experiment. Refer to Figure 5 , the system includes a terminal 501 and a test server 502;

[0125] The terminal 501 is used to send a request to the test server 502, and the request includes the unique identifier of the terminal;

[0126] The test server 502 is used to determine the experimental combination version corresponding to the terminal 501 according to the unique identifier and the experimental grouping information of the current application; allocate a corresponding experimental version identifier according to the experimental combination version, and send the experimental version identifier to the terminal 501;

[0127] The terminal 501 is used to obtain the experimental configuration information of the experimental combination version according to the experimental version identifier, and display the experimental combination version; implement different experimental tests according to the experimental configuration information.

[0128] In the embodiment of the present application, the operations performed by the test server can all refer to the operations performed by the test server in the method for implementing an AB experiment provided in any of the above embodiments, and will not be elaborated here. Similarly, the operations performed by the terminal in the embodiment of the present application can all refer to the operations performed by the terminal in the method for implementing an AB experiment provided in any of the above embodiments, and will not be elaborated here.

[0129] The system for implementing an AB experiment provided in the above embodiment of the present application and the method for implementing an AB experiment provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the stored application program.

[0130] The embodiment of the present application further provides an implementation device for an AB test, which is used to execute the operations performed by the test server in the AB test implementation method provided in any of the above embodiments. Refer to Figure 6 , the device includes:

[0131] A receiving module 601, configured to receive a request sent by a terminal, where the request includes a unique identifier corresponding to the terminal;

[0132] A determining module 602, configured to determine an experimental combination version corresponding to the terminal according to the unique identifier and the experimental grouping information of the current application;

[0133] A sending module 603, configured to allocate a corresponding experimental version identifier according to the experimental combination version, and send the experimental version identifier corresponding to the experimental combination version to the terminal;

[0134] The terminal accesses the experimental combination version of the current application according to the experimental version identifier, and obtains the experimental configuration information of the currently accessed experimental combination version; different experimental tests are implemented according to the experimental configuration information.

[0135] In the embodiment of the present application, the experimental grouping information includes the experimental type of the current application, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene; the determining module 602 is configured to, if the experimental type is a single-scene experiment, determine the experiment corresponding to the terminal in each experimental group included in the experimental scene according to the unique identifier, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene; determine the set composed of the experiments corresponding to the terminal in each experimental group as the experimental combination version corresponding to the terminal; and, if the experimental type is a multi-scene experiment, select the experimental scene corresponding to the terminal from all experimental scenes according to the unique identifier and the scene identifiers of all experimental scenes of the current application; determine the experiment corresponding to the terminal from each experimental group included in the selected experimental scene according to the unique identifier, the scene identifier of the selected experimental scene, and the experimental group identifiers of all experimental groups included in the selected experimental scene; determine the set composed of the experiments corresponding to the terminal in each experimental group included in the selected experimental scene as the experimental combination version corresponding to the terminal.

[0136] The determining module 602 is specifically configured to splice the unique identifier, the scene identifier of the experimental scene, and the experimental group identifier of the first experimental group included in the experimental scene into a first string in a preset splicing order, where the first experimental group is any experimental group in the experimental scene; convert the first string into a second string with a preset data length through a preset algorithm; calculate the hash value of the second string by using a digital hash algorithm, and determine the preset numerical interval to which the hash value belongs; determine the experiment corresponding to the terminal in the first experimental group as the experiment corresponding to the hash value in the preset numerical interval.

[0137] The device further includes an experimental version determination module, configured to receive the experimental version confirmation message sent by the terminal at each preset time interval, where the experimental version confirmation message includes a unique identifier; determine the experimental version identifier corresponding to the terminal currently required according to the unique identifier and the current experimental grouping information; if the experimental version identifier corresponding to the current requirement is different from the experimental version identifier sent to the terminal last time, send the experimental version identifier corresponding to the current requirement to the terminal, so that the terminal switches to the experimental combined version corresponding to the experimental version identifier corresponding to the current requirement.

[0138] The device further includes a push module, configured to detect that the experimental grouping information of the current application has changed; obtain the unique identifiers of all terminals currently accessing the current application; respectively re-determine the experimental version identifier corresponding to each terminal according to the unique identifier of each terminal and the changed experimental grouping information; send the re-determined experimental version identifier corresponding to each terminal to each terminal respectively, so that each currently accessed terminal switches to the experimental combined version corresponding to the re-determined experimental version identifier.

[0139] The device further includes a buried point information receiving module, configured to receive the buried point information sent by the terminal, where the buried point information includes an experimental version identifier; obtain the experimental configuration information corresponding to the experimental version identifier; store the buried point information and the experimental configuration information correspondingly in a preset database.

[0140] The device includes a report generation module, configured to obtain the experimental configuration information and the corresponding buried point information of different experimental combined versions from the preset database; generate an experimental test report according to the preset analysis metrics and the experimental configuration information and the corresponding buried point information of different experimental combined versions.

[0141] The implementation device of the AB experiment provided in the above embodiments of the present application and the implementation method of the AB experiment provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the stored application programs.

[0142] The embodiments of the present application further provide an implementation device of an AB experiment, and this device is used to execute the operations performed by the terminal in the implementation method of the AB experiment provided in any of the above embodiments. See Figure 7 ,This device includes:

[0143] A sending module 701, configured to send a request to a test server, where the request includes the unique identifier of the terminal;

[0144] A receiving module 702, configured to receive the experimental version identifier returned by the test server, where the experimental version identifier is the identifier of the experimental combined version corresponding to the terminal determined by the test server according to the unique identifier;

[0145] A loading module 703, configured to obtain the experiment configuration information of the experiment combined version according to the experiment version identifier, and display the experiment combined version; and implement different experiment tests according to the experiment configuration information.

[0146] The loading module 703 is specifically configured to send a loading request of the current application to the service server, where the loading request includes the experiment version identifier; receive the experiment configuration information corresponding to the experiment version identifier returned by the service server; and display the application interface corresponding to the experiment combined version of the current application according to the experiment configuration information.

[0147] The apparatus further includes: an experiment version update module, configured to send an experiment version confirmation message to the test server at preset intervals, where the experiment version confirmation message includes a unique identifier; receive the new experiment version identifier returned by the test server when it confirms that the experiment version corresponding to the terminal has changed; and load the experiment combined version corresponding to the new experiment version identifier.

[0148] The experiment version update module is further configured to receive the experiment version identifier re-determined by the test server for the terminal when it detects that the experiment grouping information of the current application has changed; and load the experiment combined version corresponding to the re-determined experiment version identifier.

[0149] The apparatus further includes: a data logging and reporting module, configured to send data logging information to the test server, where the data logging information includes the experiment version identifier, so that the test server generates an experiment test report according to the data logging information.

[0150] The apparatus for implementing the AB experiment provided in the above embodiments of the present application and the method for implementing the AB experiment provided in the embodiments of the present application are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run or implemented by the stored application programs.

[0151] The embodiments of the present application further provide an electronic device to execute the method for implementing the AB experiment as described above. Please refer to Figure 8 , which shows a schematic diagram of an electronic device provided in some embodiments of the present application. As Figure 8 shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802, and a communication interface 803. The processor 800, the communication interface 803, and the memory 801 are connected through the bus 802; a computer program that can run on the processor 800 is stored in the memory 801, and when the processor 800 runs the computer program, it executes the method for implementing the AB experiment provided in any of the foregoing embodiments of the present application.

[0152] Among them, the memory 801 may include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk memory. The communication connection between this system network element and at least one other network element is realized through at least one communication interface 803 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0153] The bus 802 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 801 is used to store programs. After receiving the execution instruction, the processor 800 executes the program. The implementation method of the AB experiment disclosed in any implementation manner of the foregoing embodiments of the present application can be applied to the processor 800 or implemented by the processor 800.

[0154] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 800 or the instructions in the form of software. The above-mentioned processor 800 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. This storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and combines its hardware to complete the steps of the above method.

[0155] The electronic device provided by the embodiments of the present application and the implementation method of the AB experiment provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by it.

[0156] The embodiments of the present application also provide a computer-readable storage medium corresponding to the implementation method of the AB experiment provided by the foregoing embodiments. Please refer toFigure 9 It shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., program product) is stored. When the computer program is run by a processor, it will execute the implementation method of the AB experiment provided in any of the foregoing embodiments.

[0157] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical and magnetic storage media, which will not be elaborated here one by one.

[0158] The computer-readable storage medium provided in the above embodiments of the present application and the implementation method of the AB experiment provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0159] It should be noted that:

[0160] In the specification provided here, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this specification.

[0161] Similarly, it should be understood that, in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the following schematic: that the claimed present application requires more features than those expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim itself serves as a separate embodiment of the present application.

[0162] In addition, those skilled in the art can understand that, although some of the embodiments described herein include certain features included in other embodiments but not other features, the combination of the features of different embodiments means that it is within the scope of the present application and forms different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0163] As described above, it is only the preferred specific implementation manner of the present application. However, the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A method for implementing an AB test, characterized in that, Applied to a test server, including: Receiving a request sent by a terminal, where the request includes a unique identifier corresponding to the terminal; Determining an experimental combined version corresponding to the terminal according to the unique identifier and the experimental grouping information of the currently applied experiment; Allocating a corresponding experimental version identifier according to the experimental combined version, and sending the experimental version identifier to the terminal; The terminal accesses the experimental combined version of the currently applied application according to the experimental version identifier, and obtains experimental configuration information of the currently accessed experimental combined version; Implementing different experimental tests according to the experimental configuration information; The experimental grouping information includes experimental identification information and / or experimental time information. Determining the experimental combined version corresponding to the terminal according to the unique identifier and the experimental grouping information of the currently applied experiment includes: Determining the experiment corresponding to the terminal in all experimental groups of the currently applied application according to the unique identifier and the experimental grouping information; Determining the set formed by the experiments corresponding to the terminal as the experimental combined version corresponding to the terminal; The experimental grouping information includes the experimental identification information, and the experimental identification information includes the experimental type of the currently applied application, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene. Determining the experimental combined version corresponding to the terminal according to the unique identifier and the experimental grouping information of the currently applied experiment includes: If the experimental type is a single-scene experiment, determining the experiment corresponding to the terminal in each experimental group included in the experimental scene according to the unique identifier, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene; Determining the set formed by the experiments corresponding to the terminal in each experimental group as the experimental combined version corresponding to the terminal; Determining the experiment corresponding to the terminal in each experimental group included in the experimental scene according to the unique identifier, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene includes: Concatenating the unique identifier, the scene identifier of the experimental scene, and the experimental group identifier of the first experimental group included in the experimental scene into a first string in a preset concatenation order, where the first experimental group is any experimental group in the experimental scene; Converting the first string into a second string with a preset data length through a preset algorithm; Calculating the hash value of the second string using a digital hash algorithm, and determining the preset numerical range to which the hash value belongs; Determining the experiment corresponding to the preset numerical range as the experiment corresponding to the terminal in the first experimental group.

2. The method according to claim 1, characterized in that, The method further includes: If the experimental type is a multi-scene experiment, selecting the experimental scene corresponding to the terminal from all experimental scenes according to the unique identifier and the scene identifiers of all experimental scenes of the currently applied application; Determining the experiment corresponding to the terminal in each experimental group included in the selected experimental scene respectively according to the unique identifier, the scene identifier of the selected experimental scene, and the experimental group identifiers of all experimental groups included in the selected experimental scene; Determine the set of experiments corresponding to the terminals in each experimental group included in the selected experimental scenario as the experimental combination version corresponding to the terminals.

3. The method according to any one of claims 1-2, characterized in that, After sending the experimental version identifier to the terminal, it further includes: Receiving an experimental version confirmation message sent by the terminal at every preset time interval, where the experimental version confirmation message includes the unique identifier; Determining the experimental version identifier that the terminal currently needs to correspond to according to the unique identifier and the current experimental grouping information; If the experimental version identifier that the terminal currently needs to correspond to is different from the experimental version identifier sent to the terminal last time, then send the experimental version identifier that the terminal currently needs to correspond to to the terminal, so that the terminal switches to the experimental combination version corresponding to the experimental version identifier that the terminal currently needs to correspond to.

4. The method according to any one of claims 1-2, characterized in that, The method further includes: Detecting that the current experimental grouping information of the current application has changed; Obtaining the unique identifiers of all terminals currently accessing the current application; Respectively re-determining the experimental version identifier corresponding to each terminal according to the unique identifier of each terminal and the changed experimental grouping information; Sending the re-determined experimental version identifier corresponding to each terminal to each terminal respectively, so that each terminal currently accessing switches to the experimental combination version corresponding to the re-determined experimental version identifier.

5. The method according to any one of claims 1-2, characterized in that, After sending the experimental version identifier to the terminal, it further includes: Receiving the buried point information sent by the terminal, where the buried point information includes the experimental version identifier; Obtaining the experimental configuration information corresponding to the experimental version identifier; Correspondingly storing the buried point information and the experimental configuration information in a preset database.

6. The method according to claim 5, wherein The method further includes: Obtaining the experimental configuration information and the corresponding buried point information of different experimental combination versions from the preset database; Generating an experimental test report according to the preset analysis indicators and the experimental configuration information and the corresponding buried point information of different experimental combination versions.

7. A method for implementing an AB test, wherein Applied to a terminal, it includes: Sending a request to the test server, where the request includes the unique identifier of the terminal, and where the test server is used to execute the method according to any one of claims 1 to 6; Receiving the experimental version identifier returned by the test server, where the experimental version identifier is the identifier of the experimental combination version corresponding to the terminal determined by the test server according to the unique identifier; Obtaining the experimental configuration information of the experimental combination version according to the experimental version identifier, and displaying the experimental combination version; Implementing different experimental tests according to the experimental configuration information.

8. The method according to claim 7, wherein The obtaining the experimental configuration information of the experimental combination version according to the experimental version identifier, and displaying the experimental combination version includes: Sending a loading request of the current application to the business server, where the loading request includes the experimental version identifier; Receiving the experimental configuration information corresponding to the experimental version identifier returned by the business server; Displaying the application interface corresponding to the experimental combination version of the current application according to the experimental configuration information.

9. The method according to claim 7 or 8, wherein The method further includes: Sending an experimental version confirmation message to the test server at every preset time interval, where the experimental version confirmation message includes the unique identifier; Receive the new experimental version identifier returned by the test server when it confirms that the experimental combination version corresponding to the terminal has changed; Load the experimental combination version corresponding to the new experimental version identifier.

10. The method according to claim 7 or 8, wherein The method further includes: Receive the experimental version identifier re-determined by the test server for the terminal when it detects that the experimental grouping information of the current application has changed; Load the experimental combination version corresponding to the re-determined experimental version identifier.

11. The method according to claim 7 or 8, wherein The method further includes: Send the buried point information to the test server, where the buried point information includes the experimental version identifier, so that the test server generates an experimental test report according to the buried point information.

12. An implementation system for an AB test, wherein Includes: A terminal and a test server; The terminal is used to send a request to the test server, and the request includes the unique identifier of the terminal; The test server is used to determine the experimental combination version corresponding to the terminal according to the unique identifier and the experimental grouping information of the current application; allocate the corresponding experimental version identifier according to the experimental combination version, and send the experimental version identifier to the terminal; The experimental grouping information includes experimental identification information and / or experimental time information. Determining the experimental combination version corresponding to the terminal according to the unique identifier and the experimental grouping information of the current application includes: determining the experiment corresponding to the terminal in all experimental groups of the current application according to the unique identifier and the experimental grouping information; determining the set composed of the experiments corresponding to the terminal as the experimental combination version corresponding to the terminal; The experimental grouping information includes the experimental identification information, and the experimental identification information includes the experimental type of the current application, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene; determining the experimental combination version corresponding to the terminal according to the unique identifier and the experimental grouping information of the current application includes: if the experimental type is a single-scene experiment, determining the experiment corresponding to the terminal in each experimental group included in the experimental scene according to the unique identifier, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene; determining the set composed of the experiments corresponding to the terminal in each experimental group as the experimental combination version corresponding to the terminal; determining the experiment corresponding to the terminal in each experimental group included in the experimental scene according to the unique identifier, the scene identifier of the experimental scene, and the experimental group identifiers of all experimental groups included in the experimental scene includes: splicing the unique identifier, the scene identifier of the experimental scene, and the experimental group identifier of the first experimental group included in the experimental scene into a first string according to a preset splicing order, where the first experimental group is any experimental group in the experimental scene; converting the first string into a second string with a preset data length through a preset algorithm; calculating the hash value of the second string by using a digital hash algorithm, and determining the preset numerical interval to which the hash value belongs; determining the experiment corresponding to the first experimental group as the experiment corresponding to the terminal in the first experimental group; The terminal is used to obtain the experimental configuration information of the experimental combined version according to the experimental version identifier, and display the experimental combined version; and implement different experimental tests according to the experimental configuration information.

13. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein The processor runs the computer program to implement the method according to any one of claims 1-11.

14. A computer-readable storage medium, on which a computer program is stored, whereinThe program is executed by the processor to implement the method according to any one of claims 1-11.

Images