AB test method and equipment

Abstract

The invention provides an AB test method and equipment, and the method comprises the steps: receiving a data obtaining request sent by a client, and enabling the request to be used for requesting test data from a server; and based on the data acquisition request, acquiring test data corresponding to a plurality of target to-be-tested modules in a target application program configured by the server. And performing data conversion processing on the test data corresponding to the plurality of target to-be-tested modules to generate first JSON data corresponding to the plurality of target to-be-tested modules. And issuing the first JSON data to a client, so that the client analyzes the first JSON data according to target parameter keys corresponding to a plurality of to-be-tested modules configured by the client to obtain key value pairs corresponding to the plurality of to-be-tested modules, and performs an AB test experiment on the plurality of to-be-tested modules according to the key value pairs corresponding to the plurality of to-be-tested modules. The AB test experiment can be carried out on a plurality of modules to be tested at the same time, and the test experiment efficiency and the consistency of experiment statistical results are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an AB testing method and device. Background Art

[0002] AB testing is to create two or more experimental versions of a certain function in an application. In the same time dimension, similar target users are allowed to randomly access these experimental versions, and the corresponding user experience data and application performance data of each user are collected to determine the best experimental version and officially adopt it.

[0003] In the traditional AB testing scheme, when the client enters the test scenario corresponding to each test function point, the corresponding test function point obtains test data from the server. After the client receives the test data sent by the server, it performs the operation corresponding to the group according to the test data and the test group information to which the current user device belongs. However, this AB test usually drives the corresponding test experiment separately by each test function point, and reports the corresponding point information separately, which prolongs the statistical time for the experimental results corresponding to multiple test versions and is inefficient. In addition, each test scenario needs to request test data from the server separately, and the obtained test data may be inconsistent, resulting in inconsistent statistical results. Therefore, a new AB testing scheme is urgently needed. Summary of the invention

[0004] The embodiment of the present invention provides an AB testing method and device, which are used to implement AB testing experiments on multiple modules to be tested in an application program at the same time, thereby improving the test experiment efficiency and the real-time and consistency of the experimental statistical results.

[0005] In a first aspect, an embodiment of the present invention provides an AB testing method, which is applied to a server, and the method includes:

[0006] Receive a data acquisition request sent by a client, wherein the data acquisition request is used to request test data corresponding to a target application from the server;

[0007] Based on the data acquisition request, acquiring test data corresponding to a plurality of target modules to be tested in the target application configured by the server;

[0008] Performing data conversion processing on the test data corresponding to the multiple target modules to be tested to generate first JSON data corresponding to the multiple target modules to be tested;

[0009] The first JSON data is sent down to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, and obtains the key-value pairs corresponding to the multiple modules to be tested; and according to the key-value pairs corresponding to the multiple modules to be tested, an AB testing experiment is performed on the multiple modules to be tested in the target application, where the multiple modules to be tested include the multiple target modules to be tested.

[0010] In a second aspect, an embodiment of the present invention provides an AB testing device, the device is located at a server, and the device includes:

[0011] A receiving module, used to receive a data acquisition request sent by a client, wherein the data acquisition request is used to request the server for test data corresponding to a target application;

[0012] An acquisition module, configured to acquire, based on the data acquisition request, test data corresponding to a plurality of target modules to be tested in the target application configured by the server;

[0013] A generating module, used for performing data conversion processing on the test data corresponding to the multiple target modules to be tested, and generating first JSON data corresponding to the multiple target modules to be tested;

[0014] A sending module is used to send the first JSON data to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, obtains the key-value pairs corresponding to the multiple modules to be tested, and performs AB testing experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested, wherein the multiple modules to be tested include the multiple target modules to be tested.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the AB testing method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the AB testing method described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present invention provides an AB testing method, which is applied to a client, and the method includes:

[0018] When the target application is started, a data acquisition request is sent to the server, so that the server acquires test data corresponding to multiple target modules to be tested in the target application configured by the server based on the data acquisition request, and performs data conversion processing on the multiple test data to generate first JSON data;

[0019] Receive first JSON data sent by the server;

[0020] Acquire a plurality of modules to be tested in a target application configured by the client and target parameter keys corresponding to the plurality of modules to be tested, wherein the plurality of modules to be tested include the plurality of target modules to be tested;

[0021] Parsing the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested to obtain key-value pairs corresponding to the multiple modules to be tested;

[0022] An AB testing experiment is performed on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0023] In a sixth aspect, an embodiment of the present invention provides an AB testing device, the device is located at a client, and the device includes:

[0024] A request module, configured to send a data acquisition request to a server when starting a target application, so that the server acquires test data corresponding to a plurality of target modules to be tested in the target application configured by the server based on the data acquisition request, and performs data conversion processing on the plurality of test data to generate first JSON data;

[0025] A receiving module, used for receiving the first JSON data sent by the server;

[0026] An acquisition module, used for acquiring a plurality of modules to be tested in a target application configured by the client and target parameter keys corresponding to the plurality of modules to be tested, wherein the plurality of modules to be tested include the plurality of target modules to be tested;

[0027] A parsing module, used to parse the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested, to obtain key-value pairs corresponding to the multiple modules to be tested;

[0028] The testing module is used to perform an AB testing experiment on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0029] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the AB testing method as described in the fifth aspect.

[0030] In an eighth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the AB testing method described in the fifth aspect.

[0031] In the AB test scheme provided in the embodiment of the present invention, it is applied to the server and the client to realize the AB test experiment of multiple modules to be tested in the application corresponding to the client at the same time, which can not only improve the experimental test efficiency, but also improve the real-time and consistency of the experimental statistical data. Specifically, first, the server receives the data acquisition request sent by the client, and the data acquisition request is used to request the test data corresponding to the target application from the server. Based on the data acquisition request, the test data corresponding to the multiple target modules to be tested in the target application configured by the server is obtained. Then, the test data corresponding to the multiple target modules to be tested are processed by data conversion, and the first JSON data corresponding to the multiple target modules to be tested are generated, and the first JSON data is sent to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, obtains the key-value pairs corresponding to the multiple modules to be tested, and performs AB test experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested. Among them, the multiple modules to be tested configured by the client include the multiple target modules to be tested configured by the server.

[0032] In the above scheme, the server can centrally manage the test data corresponding to all target modules to be tested configured by the server. When the server receives the data acquisition request triggered by the client, the test data corresponding to all target modules to be tested can be sent to the client. In this way, AB testing can be performed on multiple modules to be tested in the application corresponding to the client at the same time, which can not only improve the efficiency of experimental testing, but also improve the real-time and consistency of experimental statistical data. In addition, before sending the test data, the server converts all the test data into JSON data. By using the JSON data structure to transmit all the test data, the data transmission efficiency and the operating performance of the client application are improved. At the same time, the consistency of the data can also be improved to avoid the situation in which the test data corresponding to a module to be tested is updated during the test, while the data of other modules to be tested are not updated because they are not touched. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A schematic diagram of a hardware execution environment of an AB testing method provided by an embodiment of the present invention;

[0035] Figure 2 A flowchart of an AB testing method provided by an embodiment of the present invention;

[0036] Figure 3 A flowchart for managing AB test experiments corresponding to multiple modules to be tested in a target application program provided by an embodiment of the present invention;

[0037] Figure 4 A flowchart of another AB testing method provided by an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of an AB testing method provided by an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of a statistical chart of AB test experimental results provided by an embodiment of the present invention;

[0040] Figure 7 A schematic diagram of the structure of an AB testing device provided by an embodiment of the present invention;

[0041] Figure 8 A schematic diagram of the structure of an electronic device provided in this embodiment;

[0042] Fig. 9 A schematic diagram of the structure of another AB testing device provided by an embodiment of the present invention;

[0043] Fig.10 A schematic diagram of the structure of an electronic device provided in this embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the step timing in the following method embodiments is only an example, not a strict limitation.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0046] Definition of terms:

[0047] AB testing (A / B testing): is a commonly used experimental design method used to compare two or more versions of products, services or functions to determine which version performs better in terms of user experience, conversion rate, click-through rate, etc. In AB testing, the versions being tested are usually called Group A and Group B, representing different experimental conditions. Users are randomly assigned to Group A and Group B and given different versions for testing. Then, user feedback data is collected and the performance differences between the two groups are compared to determine which version is more popular with users or more effective.

[0048] JSON (JavaScript Object Notation) is a lightweight data exchange format that can flexibly represent complex data structures.

[0049] In order to facilitate those skilled in the art to understand the technical solutions provided by the embodiments of the present invention, the following briefly describes the related technologies:

[0050] In the traditional AB testing scheme, each test function point is used to disperse the requests, and the experimental data corresponding to each test function point is dispersedly reported. When the client enters the test scenario corresponding to each test function point, the corresponding test function point obtains the test data from the server. After the client receives the test data sent by the server, it executes the operation corresponding to the group according to the test data and the test group information to which the current user device belongs. In addition, when each test function point completes the corresponding test experiment, it also reports the buried point data corresponding to the test function point to the server separately. The server can perform experimental results statistics on the buried point data corresponding to each test function point reported by multiple clients, so as to realize the monitoring of the AB test experiments corresponding to multiple test function points.

[0051] However, this AB testing solution usually has each test function point drive its own corresponding test experiment separately, and report its own corresponding point data separately, which undoubtedly increases the number of requests and the overall client performance overhead will become larger. At the same time, when the request corresponding to the test function point is reached, it is necessary to wait for the data acquisition request result to be completed before normal operation can be performed, which prolongs the response time corresponding to the test function point, and the user experience is poor. In addition, this also prolongs the statistical time of the experimental results corresponding to multiple test versions, which is inefficient.

[0052] In addition, when a user is assigned to multiple experimental conditions at the same time, conflicts may occur between the multiple experimental conditions, resulting in application anomalies or instability. Moreover, since the test experiments corresponding to each test function point are conducted separately, if the overall test strategy of multiple test function points needs to be adjusted, each test scenario needs to be adjusted separately, which undoubtedly increases the complexity of the test work.

[0053] In addition, each test scenario needs to request test data from the server separately, and the obtained test data may be inconsistent. For example, the test data of a certain test function point is updated, but other test function points are not updated because the request is not received, resulting in inconsistent statistical results.

[0054] In order to solve the above technical problems, the embodiment of the present invention provides a new AB test solution, the server can centrally manage the test data corresponding to all the target modules to be tested configured by the server, and when receiving the data acquisition request sent by the client, the test data corresponding to all the target test modules can be integrated into a JSON data, and the integrated distribution is distributed to the client. The client can parse the JSON data according to the target parameter key corresponding to the module to be tested registered by the client, and obtain the key-value pairs corresponding to multiple modules to be tested, so as to complete the AB test experiments corresponding to all the target modules to be tested configured by the server at the same time, and can also complete the AB test experiments corresponding to the test modules configured by the client, thereby improving the efficiency of the test experiments. After the client parses the key-value pairs corresponding to the multiple modules to be tested, the key-value pairs corresponding to the multiple modules to be tested can be integrated into a JSON data and centrally reported to the server, so as to realize the real-time monitoring of the test experiments corresponding to each module to be tested and the statistics of the experimental results, and timely offline processing of redundant test experiments or test experiments with longer test cycles.

[0055] In the following, some embodiments of the present invention are described in detail in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and features in the embodiments can be combined with each other.

[0056] Figure 1 A schematic diagram of a hardware execution environment of an AB testing method provided by an embodiment of the present invention, such as Figure 1 As shown, the hardware execution environment of the AB test method can be composed of multiple client devices 101 and server devices 102, and the multiple client devices are connected to the server device 102 for communication. Figure 1 As shown, the plurality of client devices include client device 101A, client device 101B, client device 101C, . . . and so on.

[0057] The server device may be a cluster server, a conventional server, a cloud server, a cloud host, a virtual center, etc. The client device may be a laptop, a tablet computer, a PC, etc. Optionally, the execution process of the above AB test method may be: when the user starts the target application installed in the client device 101A, the client device 101A sends a data acquisition request to the server device 102. Based on the data acquisition request, the server device 102 acquires the test data corresponding to the multiple target modules to be tested in the target application configured by the server, and then performs data conversion processing on the test data corresponding to the multiple target modules to be tested, generates the first JSON data corresponding to the multiple target modules to be tested, and then sends the first JSON data to the client device 101A.

[0058] It should be noted that the execution process of AB testing of multiple client devices is consistent with that of client device 101A. Here, only the AB testing process of client device 101A and server device 102 is described as an example. The implementation process of AB testing of other client devices refers to client device 101A.

[0059] Next, after the client device 101A receives the first JSON data sent by the server device 102, it first obtains the multiple modules to be tested registered by the client. Among them, the multiple modules to be tested may only include the multiple target modules to be tested configured by the server, or may also include the modules to be tested configured by the client in addition to the multiple target modules to be tested configured by the server, which can be set according to actual needs. Then, the target parameter keys corresponding to the multiple modules to be tested configured by the client are obtained. Among them, before conducting the AB test experiment, the multiple modules to be tested can be pre-registered by each client and the target parameter keys corresponding to each module to be tested can be pre-configured. When actually testing, the multiple modules to be tested registered by each client and the target parameter keys corresponding to each module to be tested can be directly obtained. Among them, the target parameter key refers to the name corresponding to the parameter required by the module to be tested.

[0060] Next, the client device 101A parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested, obtains key-value pairs corresponding to the multiple modules to be tested, and performs AB testing experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0061] In addition, in actual applications, developers need to monitor multiple modules to be tested in the target application in real time to conduct AB test experiments, and promptly complete the test experiments with a longer test period and complete offline processing as soon as possible, or promptly offline the redundant test experiments. Then, in order to facilitate the management of the AB test experiments corresponding to the multiple modules to be tested in the target application, the client device 101A parses the first JSON data, obtains the key-value pairs corresponding to the multiple modules to be tested, and can also directly perform data conversion processing on the key-value pairs corresponding to the multiple modules to be tested, so as to integrate and obtain the second JSON data, and upload the second JSON data to the server device 102.

[0062] After the server device 102 receives the second JSON data reported by the client device 101A, the second JSON data can be stored in a database, in which identifiers corresponding to different clients and corresponding second JSON data are stored. Furthermore, the server device 102 can read the second JSON data corresponding to each client device from the database in sequence, and parse the second JSON data to obtain multiple key-value pairs of multiple modules to be tested corresponding to each of the multiple clients. By counting the number of each key-value pair, the number of experimental groups corresponding to each module to be tested and the diversion ratio corresponding to each module to be tested can be determined, so as to timely understand the AB test experimental results corresponding to each module to be tested, and timely manage the AB test experiments corresponding to each module to be tested according to the AB test experimental results corresponding to each module to be tested.

[0063] In an embodiment of the present invention, the server centrally manages the test data corresponding to all target modules to be tested configured by the server, and can centrally integrate the test data corresponding to all target modules to be tested into a JSON data, and distribute it to the corresponding client in the form of a JSON data structure. In this way, when the client sends a data acquisition request to the server once, the test data corresponding to multiple target modules to be tested can be obtained, so as to complete multiple test experiments at the same time.

[0064] In addition, after the client receives the JSON data, it can obtain the pre-registered multiple modules to be tested and the target parameter keys corresponding to the multiple modules to be tested, parse the JSON data according to the target parameter keys, and obtain the key-value pairs corresponding to the multiple modules to be tested. In this way, the modules to be tested configured by the client and the modules to be tested configured by the server can be tested. At the same time, the key-value pairs corresponding to the multiple modules to be tested can be integrated into one JSON data and uploaded to the server while conducting the experiment. In this way, the server can monitor and process the data statistics of each module to be tested in real time as a whole. This not only makes it possible to test multiple modules to be tested at the same time and improve the experimental test efficiency and experimental statistical efficiency, but also improve the real-time and consistency of the experimental statistical data. Then, the test experiments corresponding to the multiple modules to be tested can be managed in time according to the experimental statistical data, and the redundant test experiments and the corresponding experimental version codes can be taken offline in time.

[0065] From the above description, it can be seen that: in an embodiment of the present invention, a JSON data structure is used to send test data and store experimental data uploaded by each client, and by centrally distributing test data corresponding to all target modules to be tested and centrally reporting experimental data corresponding to each module to be tested, real-time monitoring of the overall experimental results of each module to be tested and experimental result statistics are achieved, and the AB test experiments corresponding to each module to be tested are managed based on the experimental result statistics.

[0066] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0067] Figure 2 A flowchart of an AB testing method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the execution subject of the method may be an AB test device, which may be located at the server end. It is understandable that the device may be implemented as software, or a combination of software and hardware. Specifically, the method includes the following steps:

[0068] 201. Receive a data acquisition request sent by a client, where the data acquisition request is used to request test data corresponding to a target application from a server.

[0069] 202. Based on the data acquisition request, obtain test data corresponding to multiple target modules to be tested in the target application configured by the server.

[0070] 203. Perform data conversion processing on the test data corresponding to the multiple target modules to be tested to generate first JSON data corresponding to the multiple target modules to be tested.

[0071] 204. Send the first JSON data to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, obtains the key-value pairs corresponding to the multiple modules to be tested, and performs AB testing experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0072] In actual applications, in order to determine the optimal experimental version corresponding to each of the multiple modules to be tested in the target application, when the user starts the target application, the server can randomly set the experimental group corresponding to the user and allow the user to access the experimental version corresponding to the experimental group to complete the AB test experiments corresponding to each module to be tested.

[0073] In specific implementation, when the user starts the target application through the client, the client triggers a data acquisition request to the server. The data acquisition request is used to request the test data corresponding to the target application from the server. In addition, the data acquisition request may include information such as the client identifier and the target application identifier.

[0074] After the server receives the data acquisition request sent by the client, it can obtain the test data corresponding to multiple target modules to be tested in the target application configured by the server based on the data acquisition request. Among them, the target modules to be tested refer to multiple test modules configured by the server. The test data includes the data required for the operation of each module to be tested, the grouping information of the test group where the client is located, etc. For example, the test data includes the key-value pairs required for the operation of each target module to be tested, the parameter key in the key-value pair refers to the parameter name required for the operation of the target module to be tested, and the key value in the key-value pair represents the parameter value corresponding to the parameter key, that is, the specific parameter value required for the operation of the target module to be tested.

[0075] In an optional embodiment, the server can pre-configure the key-value pairs corresponding to each target module to be tested according to the configuration information corresponding to the target application, so as to use the key-value pairs corresponding to each target module to be tested as the test data corresponding to each target module to be tested. At the same time, the test data corresponding to each target module to be tested can also be stored in the corresponding test data table. After receiving the data acquisition request sent by the client, the test data corresponding to multiple target modules to be tested in the target application can be directly obtained from the test data table, which can improve the data acquisition speed and thus improve the response speed of the target application.

[0076] During specific implementation, other methods may also be used to obtain test data corresponding to multiple target modules to be tested in the target application program, which is not limited in the embodiment of the present invention.

[0077] In addition, in actual applications, during the development phase of the target application, the server pre-configures the trigger conditions and execution methods corresponding to each functional module. However, some functional modules in the target application may be triggered by the client. For example, the target application includes functional module A, functional module B, and functional module C, wherein functional module C can only be triggered after functional module A and functional module B are triggered, that is, the triggering (running) of functional module C is determined by the client. Therefore, for functional module C, there may be situations where it runs or does not run in different clients. Then, when it is necessary to perform an AB test experiment on functional module C, not all clients need to perform a test experiment on the functional module. Therefore, in order to more effectively test multiple functional modules in each client to avoid testing experiments that are not their own, before performing AB test experiments on multiple modules to be tested corresponding to each client, the client can pre-register multiple modules to be tested, and the server only configures multiple target modules to be tested controlled by the server.

[0078] The multiple modules to be tested corresponding to the client may only include the multiple target modules to be tested configured by the server, or may include modules to be tested other than the multiple modules to be tested configured by the server.

[0079] Then, in the specific implementation, when the server receives the data acquisition request sent by the client, it can obtain the test data corresponding to all the target modules to be tested in the target application configured by the server, and integrate the test data corresponding to all the target modules to be tested into a JSON data structure, and send it to the corresponding client in a centralized manner.

[0080] In an embodiment of the present invention, the specific implementation method of integrating the test data corresponding to all target modules to be tested into a JSON data structure may be: performing data conversion processing on the test data corresponding to multiple target modules to be tested to generate first JOSN data corresponding to multiple target modules to be tested.

[0081] Wherein, each JSON key in the first JSON data corresponds to a parameter key corresponding to a key-value pair corresponding to a target module to be tested, and the value corresponding to each JSON key can be a parameter value in the key-value pair corresponding to the target module to be tested. If the test data corresponding to a target module to be tested includes multiple key-value pairs, then the target module to be tested can correspond to multiple JSON keys.

[0082] For example, the target application includes three target modules to be tested configured by the server, and the test data corresponding to each target module to be tested includes a key-value pair. Then, after data conversion is performed on the acquired three target modules to be tested, the first JSON data obtained can be as follows.

[0083] {

[0084] According to ab":

[0085] "req": Object{...},

[0086] "res": Array[3]

[0087] },

[0088] "vlocal-to-city": "0",

[0089] "vlocal-to-other-city": "0",

[0090] "city-to-city": "1",

[0091] }

[0092] Among them, vlocal-to-city represents the parameter key in the test data corresponding to the target module to be tested A, and the parameter key indicates whether the switch of the county-to-city switching guidance pop-up window is turned on, and 0 indicates not turned on. vlocal-to-other-city represents the parameter key in the test data corresponding to the target module to be tested B, and the parameter key indicates whether the switch of the county-to-other-city switching guidance pop-up window is turned on, and 0 indicates not turned on. city-to-city represents the parameter key in the test data corresponding to the target module to be tested C, and the parameter key represents city_to_city, which can indicate whether the switch of the city-to-city switching guidance pop-up window is turned on, 0 indicates not turned on, and 1 indicates turned on.

[0093] The above method can be used to integrate the test data corresponding to all target modules to be tested configured by the server into a JSON data structure, which is then distributed to the corresponding client. That is, when the JSON data structure is used for data transmission, after the client receives the first JSON data, it cannot directly use the data to run the target application. The first JSON data can be parsed to obtain the test data corresponding to the multiple modules to be tested in the target application, and then the AB test experiment can be performed on the multiple modules to be tested in the target application based on the test data.

[0094] Specifically, after receiving the first JSON data, the client obtains multiple modules to be tested registered by the client, parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, obtains the key-value pairs corresponding to the multiple modules to be tested, and then performs AB test experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested, so that it can be ensured that the AB test experiments are performed on all the modules to be tested corresponding to the client. Among them, in order to ensure that the AB test experiments of all the modules to be tested corresponding to the client can be completed when the client triggers a request once, then when parsing the first JOSN data, the multiple modules to be tested registered by the client can be obtained, and the target parameter keys corresponding to the multiple modules to be tested can be obtained according to the multiple modules to be tested registered by the client, and the first JSON data can be parsed according to the target parameter keys corresponding to the multiple modules to be tested, so that even if the first JSON data does not contain the test data corresponding to some modules to be tested configured by the client, the key-value pairs corresponding to these modules to be tested can be obtained, that is, the test data corresponding to all the modules to be tested registered by the client can be obtained, and then the AB test experiments can be performed on all the modules to be tested in the client according to the test data corresponding to all the modules to be tested registered by the client.

[0095] In the embodiment of the present invention, the server can centrally manage the test data corresponding to all target modules to be tested configured by the server. When the server receives the data acquisition request triggered by the client, the test data corresponding to all target modules to be tested can be centrally sent to the client. In this way, AB testing can be performed on multiple modules to be tested in the application corresponding to the client at the same time, which can not only improve the efficiency of experimental testing, but also improve the real-time and consistency of experimental statistical data. In addition, before sending the test data, the server converts all the test data into JSON data, and transmits all the test data using the JSON data structure, thereby improving the data transmission efficiency and the operating performance of the client application. At the same time, the consistency of the data can also be improved to avoid the situation in which the test data corresponding to a module to be tested is updated during the test, while the data of other modules to be tested is not updated due to the failure to trigger the request.

[0096] In addition, in actual applications, in order to conduct AB testing experiments on multiple functions in an application, when developing an application, it is usually necessary to add multiple experimental versions of code corresponding to multiple functions to the application at the same time. As the experimental version code accumulates, the application's installation package and runtime memory usage will increase accordingly, which may cause users to spend more time and data traffic when downloading and installing the application, and will also affect the application's running efficiency on the device, thereby affecting the overall performance of the application. In addition, retaining useless experimental version code will increase the difficulty of later maintenance and updates. Therefore, the server can monitor the AB test experiments corresponding to multiple modules to be tested in the target application in real time, and promptly take redundant AB test experiments offline, and promptly take redundant test experimental version codes offline from the application.

[0097] In order to monitor the AB test experiments corresponding to multiple modules to be tested in the target application in real time, after the client parses the first JSON data sent by the server, it obtains the key-value pairs corresponding to the multiple modules to be tested. While performing AB test experiments on the multiple modules to be tested in the target application, it can also integrate the key-value pairs corresponding to the multiple modules to be tested into one JSON data and report it to the server, so that the server can promptly know the experimental test status corresponding to the multiple modules to be tested in each client.

[0098] In actual applications, AB testing experiments are usually performed on multiple modules to be tested in a target application in multiple clients. In order to facilitate understanding of the experimental test results corresponding to each module to be tested, when the server receives the second JSON data sent by the client, the second JSON data can be stored in a database. The database stores the second JSON data corresponding to multiple clients, and the multiple clients all have the target application installed.

[0099] The second JSON data is obtained after the client performs data conversion processing on the key-value pairs corresponding to the multiple modules to be tested. That is, after parsing and obtaining the key-value pairs corresponding to the multiple modules to be tested, the multiple key-value pairs are integrated into a JSON data structure, each JSON key corresponds to a parameter key in a key-value pair, and each JSON key value corresponds to a parameter value corresponding to the parameter key. Specifically, the second JSON data can be obtained by converting the data with the server as described above to obtain JSON data, which will not be repeated here.

[0100] In addition, after receiving the second JSON data reported by each client, the second JSON data corresponding to each client can be stored in the table item corresponding to each client in the database. That is, each client reports data once, and a table item corresponding to the client is added to the database, and the table item records the client identifier and the corresponding second JSON data, so that it is convenient to perform statistical analysis on the AB test experiments corresponding to each client in the future, and timely offline the redundant test experiments or the test experiments with a long test cycle, and at the same time offline the test code corresponding to the test experiment as a whole.

[0101] In conjunction with the following embodiments, the specific implementation process of managing the AB test experiments corresponding to multiple modules to be tested in the target application program by the server according to the second JSON data corresponding to each client is described below.

[0102] Figure 3 A flowchart for managing AB test experiments corresponding to multiple modules to be tested in a target application program is provided in an embodiment of the present invention, such as Figure 3 As shown, the method comprises the following steps:

[0103] 301. According to a preset period, second JSON data reported by multiple clients in the current period are read from the database in sequence.

[0104] 302. Parse the second JSON data reported by multiple clients in the current period to obtain multiple key-value pairs corresponding to the multiple clients in the current period.

[0105] 303. According to the multiple key-value pairs corresponding to the multiple clients in the current cycle, determine the number of experimental groups corresponding to the multiple modules to be tested in the current cycle and the experimental diversion ratio corresponding to the multiple modules to be tested in the current cycle.

[0106] 304. Determine whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current cycle according to the number of experimental groups corresponding to the multiple modules to be tested and the experimental diversion ratios corresponding to the multiple modules to be tested in the current cycle.

[0107] In the embodiment of the present invention, after each client parses and obtains the key-value pairs corresponding to the modules to be tested, the multiple key-value pairs can be integrated into a JSON data structure to obtain the second JSON data, and the second JSON data can be uploaded to the database. That is, the database stores the key-value pairs of the multiple modules to be tested corresponding to each client. In order to timely obtain the test experiment conditions corresponding to each module to be tested in the target application, the second JSON data reported by multiple clients in the current period can be read from the database in sequence according to the preset period.

[0108] Next, the second JSON data reported by multiple clients in the current cycle is parsed to obtain multiple key-value pairs corresponding to each of the multiple clients in the current cycle. Among them, the multiple key-value pairs are key-value pairs corresponding to multiple modules to be tested. It should be noted that the modules to be tested corresponding to each client may be different, so the number of multiple key-value pairs corresponding to each of the multiple clients obtained after parsing may also be different.

[0109] In an optional embodiment, the second JSON data reported by multiple clients in the current period is parsed to obtain multiple key-value pairs corresponding to each of the multiple clients in the current period. The specific implementation process can be: using a user-defined function to parse the second JSON data corresponding to each of the multiple clients to obtain multiple key-value pairs corresponding to each client.

[0110] After obtaining multiple key-value pairs corresponding to each client, the number of experimental groups corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current cycle are determined according to the multiple key-value pairs corresponding to each of the multiple clients in the current cycle. In an optional embodiment, the database explode function can be used to display multiple key-value pairs in columns, and finally the general SQL syntax is used to count the multiple key-value pairs according to the time information and version information.

[0111] Among them, the key-value pair includes a parameter key and a parameter value corresponding to the parameter key. For different test experiment versions, the parameter value corresponding to the same parameter key after parsing is different. Therefore, by determining the number of reported different parameter values ​​corresponding to each parameter key, the number of experimental groups corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current cycle can be determined.

[0112] Specifically, in an optional embodiment, all different parameter values ​​corresponding to each parameter key in the multiple key-value pairs in the current period and the number of different parameter values ​​reported corresponding to each parameter key can be determined based on the multiple key-value pairs corresponding to each of the multiple clients in the current period. Then, based on the number of different parameter values ​​reported corresponding to each parameter key in the current period, the number of experimental groupings corresponding to each of the multiple modules to be tested in the current period and the experimental diversion ratio corresponding to each of the multiple modules to be tested can be determined.

[0113] For example, suppose that there are 3 clients reporting the second JSON data to the server in the current cycle, the target parameter key corresponding to the module to be tested 1 is key1, the target parameter key corresponding to the module to be tested 2 is key2, and the target parameter key corresponding to the module to be tested 3 is key3. After parsing the second JSON data corresponding to client A, 3 key-value pairs are obtained, namely key1=V1, key2=V2, key3=V3. After parsing the second JSON data corresponding to client B, 3 key-value pairs are obtained, namely key1=V1, key2=V2, key3=V4. After parsing the second JSON data corresponding to client C, 3 key-value pairs are obtained, namely key1=V5, key2=V2, key3=V4.

[0114] According to the key-value pairs corresponding to each client, it can be determined that all different parameter values ​​corresponding to the parameter key key1 are V1 and V5, respectively. Among them, the reported number of parameter values ​​corresponding to key1 is V1 is 2, and the reported number of parameter values ​​corresponding to key1 is V5 is 1. Then, there are 2 experimental groups corresponding to the module 1 to be tested corresponding to the parameter key key1, namely, the diversion ratio corresponding to experimental group A (the experimental test version with the parameter key key1 corresponding to the parameter value V1) is 66.5%, and the experimental diversion ratio corresponding to experimental group B (the experimental test version with the parameter key key1 corresponding to the parameter value V5) is 33.5%.

[0115] According to the key-value pairs corresponding to each client, it can be determined that the parameter key key2 corresponds to a parameter value V2, that is, the reported number of parameter values ​​corresponding to key2 is V2 is 3, then the module 2 to be tested corresponding to the parameter key key2 corresponds to 1 experimental group, and the diversion ratio corresponding to experimental group A is 100%.

[0116] According to the key-value pairs corresponding to each client, it can be determined that all different parameter values ​​corresponding to the parameter key key3 are V3 and V4, respectively. Among them, the reported number of the parameter value corresponding to key3 is V3 is 1, and the reported number of the parameter value corresponding to key3 is V4 is 2. Then, there are 2 experimental groups corresponding to the module 3 to be tested corresponding to the parameter key key3, namely, the diversion ratio corresponding to the experimental group A (the experimental test version with the parameter value of parameter key key3 corresponding to V3) is 33.5%, and the experimental diversion ratio corresponding to the experimental group B (the experimental test version with the parameter value of parameter key key3 corresponding to V4) is 66.5%.

[0117] Next, according to the number of experimental groups corresponding to the multiple modules to be tested in the current cycle and the diversion ratios corresponding to the multiple modules to be tested, it is determined whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current cycle. After obtaining the number of experimental groups corresponding to the multiple modules to be tested in the current cycle and the experimental diversion ratios corresponding to the multiple modules to be tested, the test experiment conditions corresponding to the multiple modules to be tested in the current cycle can be determined according to the number of experimental groups and the experimental diversion ratios corresponding to each experimental group, and it can be determined whether to perform offline processing on the AB test experiments corresponding to each module to be tested or adjust the experimental diversion ratios of the AB test experiments corresponding to the modules to be tested according to the test experiment conditions corresponding to the multiple modules to be tested.

[0118] For any module to be tested, if the number of experimental groups corresponding to any module to be tested in the current cycle is greater than or equal to the first preset value, and the experimental diversion ratio corresponding to any module to be tested is greater than or equal to the second preset value, then the AB test experiment corresponding to any module to be tested is offline. Among them, the first preset value and the second preset value can be set according to the actual test requirements. For example, the first preset value is 1, and the second preset value is 100%. Continuing with the above example, for module to be tested 2, its number of experimental groups is equal to the first preset value, and the experimental diversion ratio is equal to the second preset value, then the AB test experiment corresponding to module to be tested 2 can be offline, and the test experiment version code corresponding to the AB test experiment of the module to be tested can be offline, so that only the test experiment version code corresponding to the experimental group A is retained in each application, which can reduce the work of developers and improve the overall performance of the target application in a timely manner.

[0119] In addition, in order to enable developers to monitor the test experiments corresponding to multiple modules to be tested in the target application in real time, when all different parameter values ​​corresponding to each parameter key in multiple modules to be tested in the current cycle and the number of different parameter values ​​corresponding to each parameter key are determined, the experimental results are displayed in the form of a chart. It is also possible to determine the version cycle of the test experiments corresponding to each module to be tested based on the experimental results corresponding to multiple modules to be tested in each cycle, and view the entire experimental process of the test experiment by screening the parameter keys corresponding to different test experiments.

[0120] In addition, a data information table corresponding to each test experiment can also be provided to developers. The data information table may include information such as the number of experimental groups corresponding to each test experiment in different cycles, the experimental diversion ratio corresponding to each experimental group, the number of experiments corresponding to each experimental group, and the total number of experiments corresponding to each test experiment.

[0121] In an embodiment of the present invention, according to a preset cycle, the second JSON data reported by multiple clients in the current cycle is read from the database in sequence, and the second JSON data reported by multiple clients in the current cycle is parsed to obtain multiple key-value pairs corresponding to each of the multiple clients in the current cycle. According to the multiple key-value pairs corresponding to each of the multiple clients in the current cycle, the number of experimental groups corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current cycle are determined. According to the number of experimental groups corresponding to each of the multiple modules to be tested in the current cycle and the experimental diversion ratio corresponding to each of the multiple modules to be tested, it is determined whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current cycle, so that redundant test experiments or test experiments with longer test cycles can be offline processed in time, and the test code corresponding to the test experiment can be offline as a whole, reducing the time and data traffic consumed by users when downloading and installing the target application, and at the same time, the operating efficiency of the target application on the device will be improved, thereby improving the overall performance of the target application. In addition, the useless experimental version code can be offline in time through this method, which can reduce the difficulty of later maintenance and updating.

[0122] Figure 4 A flowchart of another AB testing method provided by an embodiment of the present invention, such as Figure 4 As shown, the execution subject of the method can be an AB test device, which can be located at the client. It can be understood that the device can be implemented as software, or a combination of software and hardware. Specifically, the method includes the following steps:

[0123] 401. When the target application is started, a data acquisition request is sent to the server, so that the server obtains the test data corresponding to multiple target modules to be tested in the target application configured by the server based on the data acquisition request, and performs data conversion processing on the multiple test data to generate first JSON data.

[0124] 402. Receive the first JSON data sent by the server.

[0125] 403. Obtain multiple modules to be tested in the target application configured by the client and target parameter keys corresponding to the multiple modules to be tested.

[0126] 404. Parse the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested to obtain key-value pairs corresponding to the multiple modules to be tested.

[0127] 405. Perform an AB test experiment on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0128] In an embodiment of the present invention, when a user starts a target application through a client, a data acquisition request is sent to a server, and the data acquisition request is used to request the server to obtain test data corresponding to the target application. The server receives the data acquisition request sent by the client, and based on the data acquisition request, obtains test data corresponding to multiple target modules to be tested in the target application configured by the server, performs data conversion processing on the multiple test data, and generates first JSON data.

[0129] Receive first JSON data sent by the server, where the first JSON data includes test data corresponding to multiple target modules to be tested configured by the server.

[0130] Before testing multiple test modules in the target application, the server will pre-configure multiple target modules to be tested in the target application, and write the codes corresponding to the multiple experimental versions corresponding to each target module to be tested into the target application at the same time, so that after the client receives the test data sent by the server, it can call the corresponding experimental version according to the grouping information to perform AB testing experiments. However, in actual applications, the execution process corresponding to certain functional modules in the target application is determined by the client, so in order to perform AB testing experiments on these functional modules in the target application, it is necessary to obtain the test data corresponding to these functional modules at the same time. In order to solve this technical problem, in an embodiment of the present invention, before performing AB testing on multiple modules to be tested in the target application, the client can pre-register multiple modules to be tested and configure corresponding target parameter keys for multiple modules to be tested.

[0131] That is to say, for the client, it is possible that the client decides whether to trigger the function module in the target application corresponding to the client and also needs to conduct an AB test experiment. Then, after receiving the first JSON data, multiple modules to be tested in the target application configured by the client and the target parameter keys corresponding to the multiple modules to be tested can be obtained. Among them, the multiple modules to be tested may include other test modules other than the target module to be tested. In this way, when parsing the first JSON data, the test data corresponding to all the modules to be tested that can conduct AB test experiments corresponding to the client can be obtained, so that the client can trigger a request once and conduct AB test experiments on all the modules to be tested in the client at the same time, thereby improving the test efficiency of the experiment.

[0132] In specific implementation, each client can pre-register multiple modules to be tested corresponding to each client. In order to obtain the test data corresponding to these modules to be tested, the client can also pre-configure the target parameter key corresponding to each module to be tested, so that when parsing the first JSON data, the first JSON data can be directly parsed according to the target parameter keys corresponding to the multiple modules to be tested to obtain the key-value pairs corresponding to the multiple modules to be tested.

[0133] In an optional embodiment, the client can determine multiple modules to be tested and target parameter keys corresponding to the multiple modules to be tested according to the configuration information corresponding to the target application, and write the target parameter keys corresponding to the multiple modules to be tested into the registration table.

[0134] Specifically, the configuration information of the target application can be obtained, and multiple modules to be tested in the target application and target parameter keys corresponding to the multiple modules to be tested can be determined according to the configuration information. According to the target parameter keys corresponding to the multiple modules to be tested, implementation classes corresponding to the multiple modules to be tested are created, and the names of the implementation classes corresponding to the multiple modules to be tested are written into a registry, which stores the corresponding relationship between the identifiers of different modules to be tested and the corresponding implementation class names.

[0135] In an optional embodiment, the specific implementation method of creating implementation classes corresponding to multiple modules to be tested according to the target parameter keys corresponding to multiple modules to be tested can be: first, an abstract class corresponding to multiple modules to be tested in the target application is created, and the abstract class is used to define the construction methods and abstract methods corresponding to multiple modules to be tested. Then, according to the configuration information, the target parameter keys corresponding to the multiple modules to be tested are obtained, the target parameter keys are passed into the construction method defined in the abstract class, and the abstract method defined in the abstract class is rewritten to create implementation classes corresponding to multiple modules to be tested. Subsequently, by instantiating the implementation class, the target parameter keys configured by each module to be tested can be obtained. Then, the names of the implementation classes corresponding to the multiple modules to be tested are written into the registry of the target application.

[0136] Among them, the constructor stipulates that the parameters passed into the constructor can be variable strings, the abstract method stipulates that the parameter key passed into the abstract method is of string type, the key value passed into the abstract method is of wildcard type, and the data structure corresponding to the output result after calling the abstract method is a key-value pair.

[0137] Then, after receiving the first JSON data sent by the server, the registry of the target application and the identifiers corresponding to the multiple modules to be tested can be obtained. The registry stores the correspondence between the identifiers corresponding to the different modules to be tested and the corresponding implementation class names. Then, according to the identifiers corresponding to the multiple modules to be tested, the registry is traversed to obtain the implementation class names corresponding to the multiple modules to be tested. Then, according to the implementation class names corresponding to the multiple modules to be tested, the implementation classes corresponding to the multiple modules to be tested are respectively instantiated to obtain the target parameter keys corresponding to the multiple modules to be tested.

[0138] Since the parameter keys corresponding to each functional module are configured in the implementation class corresponding to each module to be tested, the target parameter key corresponding to the module to be tested can be obtained when the implementation class corresponding to the module to be tested is instantiated. Among them, when the implementation classes corresponding to multiple modules to be tested are instantiated, a reflection mechanism can be used. The reflection mechanism allows the target application to dynamically obtain information about the implementation class, call class methods, access class properties, etc. at runtime to obtain target parameter keys corresponding to multiple modules to be tested.

[0139] After obtaining the target parameter keys corresponding to the multiple modules to be tested, the first JSON data is parsed according to the target parameter keys corresponding to the multiple modules to be tested to obtain key-value pairs corresponding to the multiple modules to be tested.

[0140] In an optional embodiment, parameter values ​​corresponding to target parameter keys corresponding to multiple modules to be tested may be extracted from the first JSON data, and key-value pairs corresponding to multiple modules to be tested may be determined according to the target parameter keys corresponding to the multiple modules to be tested and the parameter values ​​corresponding to the target parameter keys corresponding to the multiple modules to be tested. In other words, the first JSON data may be traversed according to the target parameter keys corresponding to the multiple modules to be tested, and the parameter values ​​corresponding to the target parameter keys corresponding to the multiple modules to be tested may be sequentially retrieved.

[0141] Then, according to the key-value pairs corresponding to the multiple modules to be tested, an AB test experiment is performed on the multiple modules to be tested in the target application. In particular, when the server sends the test data corresponding to the target application to the client, it can also send the experimental grouping information corresponding to the client to the client. After the client obtains the test data corresponding to the multiple modules to be tested after parsing, it performs the corresponding test experiment according to the test data corresponding to the multiple modules to be tested and the experimental grouping information corresponding to the client.

[0142] In addition, in order to monitor the AB test experiments corresponding to multiple modules to be tested in the target application in real time, after the client parses and obtains the key-value pairs corresponding to the multiple modules to be tested, the key-value pairs corresponding to the multiple modules to be tested obtained after parsing can be integrated into a JSON data structure while conducting the test experiment, and reported to the server, so that the server can promptly obtain the test experiment information corresponding to each module to be tested.

[0143] Specifically, data conversion processing is performed on the key-value pairs corresponding to the multiple modules to be tested to obtain second JSON data, and the second JSON data is sent to the server so that the server stores the second JSON data in the database, and the second JSON data corresponding to the multiple clients is obtained by traversing the database, and according to the second JSON data corresponding to the multiple clients, it is determined whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested.

[0144] Among them, each JSON key in the second JSON data corresponds to a parameter key in a key-value pair, and each JSON key value corresponds to a parameter value in a key-value pair. In this way, the key-value pairs corresponding to multiple modules to be tested obtained by parsing can be integrated into one JSON data and reported to the server.

[0145] The embodiment of the present invention parses the JSON data according to the target parameter key corresponding to the module to be tested registered by the client to obtain the key-value pairs corresponding to multiple modules to be tested, so as to simultaneously complete the AB test experiments corresponding to all target modules to be tested configured by the server, and can also complete the AB test experiments corresponding to the test modules configured by the client, thereby improving the test experiment efficiency. After the client parses the key-value pairs corresponding to multiple modules to be tested, the key-value pairs corresponding to the multiple modules to be tested can be integrated into one JSON data and centrally reported to the server, so as to realize real-time monitoring of the test experiments corresponding to each module to be tested and statistics of the experimental results, and timely offline processing of redundant test experiments or test experiments with longer test cycles.

[0146] The specific execution process and technical effects of the above steps in this embodiment are similar to the specific execution process and technical effects in the above embodiments. Please refer to the above statements for details, and no further details will be given here.

[0147] In order to facilitate understanding of the AB testing method in each of the above embodiments, the AB testing method is described by example in combination with a specific application scenario. Figure 5 The AB testing system shown includes a server and a client, wherein the server includes a service platform, a tracking point platform, and a monitoring platform.

[0148] The service platform is mainly used to centrally manage the test data corresponding to all target modules to be tested. After receiving the data acquisition request triggered by the client, the test data corresponding to all target modules to be tested can be integrated into a JSON data structure and sent to the corresponding client. Specifically, the test data corresponding to multiple target modules to be tested in the target application configured by the server are obtained, and the test data corresponding to the multiple target modules to be tested are processed by data conversion to generate the first JSON data corresponding to the multiple target modules to be tested. Each JSON key in the first JOSN data corresponds to a module to be tested.

[0149] The client is mainly used to request the test data corresponding to the target application from the server, and perform AB test experiments based on the test data sent by the server. At the same time, the test data is reported to the server so that the server can promptly learn the AB test experiment status corresponding to multiple modules to be tested in each client.

[0150] Specifically, when the user starts the target application, a data acquisition request is sent to the server, so that the server obtains the test data corresponding to the multiple target modules to be tested in the target application configured by the server, and performs data conversion processing on the multiple test data to generate the first JSON data. The first JSON data is received, and the multiple modules to be tested registered by the client and the target parameter keys configured by the multiple modules to be tested are read. According to the target parameter keys corresponding to the multiple modules to be tested, the first JSON data is parsed to obtain the key-value pairs corresponding to the multiple modules to be tested, wherein the key-value pairs include the parameter key and the parameter value corresponding to the parameter key. According to the key-value pairs corresponding to the multiple modules to be tested, an AB test experiment is performed on the multiple modules to be tested in the target application.

[0151] While parsing to obtain the key-value pairs corresponding to the multiple modules to be tested, data conversion processing can also be performed on the key-value pairs corresponding to the multiple modules to be tested to obtain second JSON data, and the second JSON data is sent to the tracking point platform. Among them, in the embodiment of the present invention, the execution order of performing AB test experiments on multiple modules to be tested in the target application and performing data conversion processing on the key-value pairs corresponding to the multiple modules to be tested is not limited. The two processes can be executed simultaneously, or different execution orders can be set according to actual needs.

[0152] The tracking point platform is mainly used to collect the second JSON data and tracking point data reported by each client, and store the collected data reported by each client in the database. Among them, the tracking point data may include performance data of the target application, user experience data, experimental group information corresponding to the client, and other data.

[0153] The monitoring platform is mainly used to monitor the AB test experiments corresponding to multiple modules to be tested in the target application in each client, and to count the test experiment data corresponding to each module to be tested, so as to manage the AB test experiments corresponding to each module to be tested.

[0154] Specifically, the monitoring platform reads the second JSON data reported by each client in the database, and uses the user-defined function json_to_map_recursive to parse the second JSON data to list all the key-value pairs of the second JSON data corresponding to each client, obtain all parameter keys and parameter values, and then count the number of reported parameter values ​​corresponding to each parameter key to obtain the number of experimental groups and experimental diversion ratios corresponding to each test experiment. Among them, the database explode function can be used to display all parameter keys and parameter values ​​separately, and finally the general SQL syntax will completely count the key-value according to time information and version information, and display the experimental situation corresponding to each test experiment in the form of a chart, such as Figure 6 shown.

[0155] According to this chart, the test experiments corresponding to each module to be tested can be detected. In this chart, the left side of the vertical axis represents the total number of hits in a single test experiment, and the broken line on the right side of the vertical axis represents the number of experimental groups in a single test experiment. In addition, the version cycle corresponding to each test experiment can also be counted, and the entire experimental process of each test experiment can be determined by screening the parameter keys corresponding to different test experiments.

[0156] In addition, in order to facilitate developers to understand the experimental conditions corresponding to each test experiment in a timely manner, a data information table corresponding to each test experiment can also be provided to developers. The data information table includes information such as the number of experimental groups corresponding to each test experiment, the experimental diversion ratio corresponding to each test experiment, and the traffic distribution result achievement rate corresponding to each test experiment.

[0157] In an embodiment of the present invention, all AB test data are uniformly stored and transmitted by using the JSON format, which improves the centralization and consistency of the data. At the same time, the monitoring platform uses UDF (user-defined function) to count JSON data, making the originally complex SQL statistical work simple and efficient. The client only needs to report the test data corresponding to the registered module to be tested, so that it can cleverly exclude test experiments that are not its own. In addition, once the client is started, the monitoring platform can perceive the achievement rate of traffic distribution results of all AB test experiments. Compared with traditional decentralized statistics, this method has obvious advantages in timeliness, consistency and integrity.

[0158] The specific execution process and technical effects of the above steps in this embodiment are similar to the specific execution process and technical effects in the above embodiments. Please refer to the above statements for details, and no further details will be given here.

[0159] The AB testing device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0160] Figure 7 A schematic diagram of the structure of an AB test device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device includes: a receiving module 11, an acquiring module 12, a generating module 13, and a sending module 14.

[0161] The receiving module 11 is used to receive a data acquisition request sent by a client, wherein the data acquisition request is used to request the server for test data corresponding to a target application.

[0162] The acquisition module 12 is used to acquire the test data corresponding to the multiple target modules to be tested in the target application configured by the server based on the data acquisition request.

[0163] The generating module 13 is used to perform data conversion processing on the test data corresponding to the multiple target modules to be tested, and generate first JSON data corresponding to the multiple target modules to be tested.

[0164] The sending module 14 is used to send the first JSON data to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, obtains the key-value pairs corresponding to the multiple modules to be tested, and performs AB testing experiments on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested, wherein the multiple modules to be tested include the multiple target modules to be tested.

[0165] Optionally, the device may also include a storage module, which is specifically used to: receive second JSON data sent by the client, the second JSON data is obtained after data conversion processing of key-value pairs corresponding to the multiple modules to be tested; store the second JSON data in a database, the database storing second JSON data corresponding to multiple clients, and the target application is installed in the multiple clients.

[0166] Optionally, the storage module is also used to: read the second JSON data reported by multiple clients in the current period from the database in sequence according to a preset period; parse the second JSON data reported by the multiple clients in the current period to obtain multiple key-value pairs corresponding to each of the multiple clients in the current period, and the multiple key-value pairs are key-value pairs corresponding to the multiple modules to be tested; determine the number of experimental groupings corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current period according to the multiple key-value pairs corresponding to each of the multiple clients in the current period; determine whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current period according to the number of experimental groupings corresponding to each of the multiple modules to be tested in the current period and the experimental diversion ratio corresponding to each of the multiple modules to be tested.

[0167] Optionally, the key-value pair includes a parameter key and a parameter value corresponding to the parameter key, and the storage module is specifically used to: determine all different parameter values ​​corresponding to each parameter key in the multiple key-value pairs in the current period and the reported number of different parameter values ​​corresponding to each parameter key according to the multiple key-value pairs corresponding to each of the multiple clients in the current period; determine the number of experimental groupings corresponding to each of the multiple modules to be tested in the current period and the experimental diversion ratio corresponding to each of the multiple modules to be tested according to the reported number of different parameter values ​​corresponding to each parameter key in the current period.

[0168] Optionally, the key-value pair includes a parameter key and a parameter value corresponding to the parameter key, and the storage module is specifically used to: for any module to be tested, if the number of experimental groups corresponding to any module to be tested in the current cycle is greater than or equal to a first preset value, and the experimental diversion ratio corresponding to any module to be tested is greater than or equal to a second preset value, then offline processing is performed on the AB test experiment corresponding to any module to be tested.

[0169] Figure 7 The device shown can execute the steps in the AB testing method in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description in the aforementioned embodiment, which will not be repeated here.

[0170] The embodiment of the present invention also provides an electronic device, such as Figure 8 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 implements the AB test method in the above-mentioned embodiment.

[0171] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the AB testing method provided in the aforementioned embodiment.

[0172] Fig. 9 A schematic diagram of the structure of another AB testing device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the device includes: a request module 31, a receiving module 32, an acquisition module 33, a parsing module 34, and a testing module 35.

[0173] The request module 31 is used to send a data acquisition request to the server when the target application is started, so that the server can obtain the test data corresponding to multiple target modules to be tested in the target application configured by the server based on the data acquisition request, and perform data conversion processing on the multiple test data to generate the first JSON data.

[0174] The receiving module 32 is used to receive the first JSON data sent by the server.

[0175] The acquisition module 33 is used to acquire a plurality of modules to be tested in a target application configured by the client and target parameter keys corresponding to the plurality of modules to be tested, wherein the plurality of modules to be tested include the plurality of target modules to be tested.

[0176] The parsing module 34 is used to parse the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested, so as to obtain key-value pairs corresponding to the multiple modules to be tested.

[0177] The testing module 35 is used to perform an AB testing experiment on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

[0178] Optionally, the device also includes a configuration module, which is specifically used to: obtain configuration information of the target application; determine multiple modules to be tested in the target application and target parameter keys corresponding to the multiple modules to be tested based on the configuration information; create implementation classes corresponding to the multiple modules to be tested based on the target parameter keys corresponding to the multiple modules to be tested; write the implementation class names corresponding to the multiple modules to be tested into a registry, in which the correspondence between the identifiers of different modules to be tested and the corresponding implementation class names is stored.

[0179] Optionally, the configuration module is specifically used to: obtain the identifiers corresponding to the multiple modules to be tested and the registry of the target application; traverse the registry according to the identifiers corresponding to the multiple modules to be tested to obtain the implementation class names corresponding to the multiple modules to be tested; and instantiate the implementation classes corresponding to the multiple modules to be tested respectively according to the implementation class names corresponding to the multiple modules to be tested to obtain the target parameter keys corresponding to the multiple modules to be tested.

[0180] Optionally, the parsing module 34 is specifically used to: extract parameter values ​​corresponding to target parameter keys corresponding to multiple modules to be tested from the first JSON data; determine the key-value pairs corresponding to the multiple modules to be tested according to the target parameter keys corresponding to the multiple modules to be tested and the parameter values ​​corresponding to the target parameter keys corresponding to the multiple modules to be tested.

[0181] Optionally, the parsing module 34 is also used to: perform data conversion processing on the key-value pairs corresponding to the multiple modules to be tested to obtain second JSON data; send the second JSON data to the server so that the server stores the second JSON data in a database, and obtain the second JSON data corresponding to multiple clients by traversing the database, and determine whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested based on the second JSON data corresponding to the multiple clients, and the database stores the second JSON data corresponding to the multiple clients.

[0182] Fig. 9 The device shown can execute the steps in the AB testing method in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description in the aforementioned embodiment, which will not be repeated here.

[0183] The embodiment of the present invention also provides an electronic device, such as Fig.10 As shown, the electronic device may include: a processor 41, a memory 42, and a communication interface 43. The memory 22 stores executable codes, and when the executable codes are executed by the processor 41, the processor 41 implements the AB test method in the above-mentioned embodiment.

[0184] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the AB testing method provided in the aforementioned embodiment.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AB testing method, characterized in that: Applied to the server, including: Receive a data acquisition request sent by a client, wherein the data acquisition request is used to request test data corresponding to a target application from the server; Based on the data acquisition request, acquiring test data corresponding to a plurality of target modules to be tested in the target application configured by the server; Performing data conversion processing on the test data corresponding to the multiple target modules to be tested to generate first JSON data corresponding to the multiple target modules to be tested; The first JSON data is sent down to the client, so that the client parses the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested configured by the client, and obtains the key-value pairs corresponding to the multiple modules to be tested; and according to the key-value pairs corresponding to the multiple modules to be tested, an AB testing experiment is performed on the multiple modules to be tested in the target application, where the multiple modules to be tested include the multiple target modules to be tested.

2. The method according to claim 1, characterized in that The method further comprises: Receiving second JSON data sent by the client, where the second JSON data is obtained after data conversion processing is performed on key-value pairs corresponding to the multiple modules to be tested; The second JSON data is stored in a database, wherein the database stores second JSON data corresponding to a plurality of clients, and the target application is installed in each of the plurality of clients.

3. The method according to claim 2, characterized in that The method further comprises: According to a preset period, the second JSON data reported by multiple clients in the current period are sequentially read from the database; Parsing the second JSON data reported by the multiple clients in the current cycle to obtain multiple key-value pairs corresponding to the multiple clients in the current cycle, where the multiple key-value pairs are key-value pairs corresponding to the multiple modules to be tested; Determine, according to the multiple key-value pairs corresponding to the multiple clients in the current cycle, the number of experimental groups corresponding to the multiple modules to be tested and the experimental diversion ratio corresponding to the multiple modules to be tested in the current cycle; According to the number of experimental groups corresponding to the multiple modules to be tested and the experimental diversion ratios corresponding to the multiple modules to be tested in the current cycle, it is determined whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current cycle.

4. The method according to claim 3, characterized in that The key-value pair includes a parameter key and a parameter value corresponding to the parameter key, and determining the number of experimental groups corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current period according to the multiple key-value pairs corresponding to each of the multiple clients in the current period, includes: Determine, according to the multiple key-value pairs corresponding to the multiple clients in the current period, all different parameter values ​​corresponding to each parameter key in the multiple key-value pairs in the current period and the number of reported different parameter values ​​corresponding to each parameter key; According to the reported quantity of different parameter values ​​corresponding to each parameter key in the current cycle, the number of experimental groups corresponding to each of the multiple modules to be tested and the experimental diversion ratio corresponding to each of the multiple modules to be tested in the current cycle are determined.

5. The method according to claim 3, characterized in that: The determining whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested in the current cycle according to the number of experimental groups corresponding to the multiple modules to be tested and the experimental diversion ratios corresponding to the multiple modules to be tested in the current cycle includes: For any module to be tested, if the number of experimental groups corresponding to any module to be tested in the current cycle is greater than or equal to the first preset value, and the experimental diversion ratio corresponding to any module to be tested is greater than or equal to the second preset value, then the AB test experiment corresponding to any module to be tested is offline processed.

6. An AB testing method, characterized in that: Applied to the client, including: When the target application is started, a data acquisition request is sent to the server, so that the server acquires test data corresponding to multiple target modules to be tested in the target application configured by the server based on the data acquisition request, and performs data conversion processing on the multiple test data to generate first JSON data; Receive first JSON data sent by the server; Acquire a plurality of modules to be tested in a target application configured by the client and target parameter keys corresponding to the plurality of modules to be tested, wherein the plurality of modules to be tested include the plurality of target modules to be tested; Parsing the first JSON data according to the target parameter keys corresponding to the multiple modules to be tested to obtain key-value pairs corresponding to the multiple modules to be tested; An AB testing experiment is performed on the multiple modules to be tested in the target application according to the key-value pairs corresponding to the multiple modules to be tested.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining configuration information of the target application; Determine, according to the configuration information, a plurality of modules to be tested in the target application and target parameter keys corresponding to the plurality of modules to be tested; Creating implementation classes corresponding to the multiple modules to be tested according to the target parameter keys corresponding to the multiple modules to be tested; The implementation class names corresponding to the multiple modules to be tested are written into a registration table, in which the corresponding relationship between the identifiers of different modules to be tested and the corresponding implementation class names is stored.

8. The method according to claim 7, characterized in that The obtaining target parameter keys corresponding to the plurality of modules to be tested includes: Obtaining identifiers corresponding to the plurality of modules to be tested and a registration table of the target application; According to the identifiers corresponding to the multiple modules to be tested, traverse the registration table to obtain the implementation class names corresponding to the multiple modules to be tested; According to the implementation class names corresponding to the multiple modules to be tested, the implementation classes corresponding to the multiple modules to be tested are respectively instantiated to obtain the target parameter keys corresponding to the multiple modules to be tested.

9. The method according to claim 7, characterized in that: The first JSON data is parsed according to the target parameter keys corresponding to the multiple modules to be tested to obtain key-value pairs corresponding to the multiple modules to be tested, including: Extracting parameter values ​​corresponding to target parameter keys corresponding to the multiple modules to be tested from the first JSON data; According to the target parameter keys corresponding to the multiple modules to be tested and the parameter values ​​corresponding to the target parameter keys of the multiple modules to be tested, the key-value pairs corresponding to the multiple modules to be tested are determined.

10. The method according to claim 6, characterized in that The method further comprises: Performing data conversion processing on the key-value pairs corresponding to the multiple modules to be tested to obtain second JSON data; The second JSON data is sent to the server so that the server stores the second JSON data in a database, and the second JSON data corresponding to multiple clients is obtained by traversing the database, and whether to perform offline processing on the AB test experiments corresponding to the multiple modules to be tested is determined according to the second JSON data corresponding to the multiple clients, and the database stores the second JSON data corresponding to the multiple clients.

11. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the AB testing method as described in any one of claims 1 to 5 or 6 to 10.

Images