ZooKeeper-based AB testing method, system, device and storage medium

Abstract

The present application provides an AB test method, system, device and storage medium based on ZooKeeper, and relates to the technical field of AB test. The method includes: obtaining the configuration information of the AB test, and encapsulating it into an SDK toolkit and storing it in the ZooKeeper framework; performing diversion based on the diversion algorithm, assigning multiple user IDs to the control version and the test version, loading the SDK toolkit in the ZooKeeper framework for testing, obtaining preliminary analysis results, adjusting the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework, so as to adjust the user diversion information and continue the test. The present application directly exchanges the user ID information by adjusting the storage information of some nodes in the ZooKeeper framework, thereby directly adjusting the control version or test version assigned to some user IDs, without the need to replace the user IDs assigned to each test version and the control version as a whole, and the speed of re-diverting users is fast, and basically no memory is consumed.

Description

Technical Field

[0001] This application relates to the technical field of AB testing, and particularly relates to an AB testing method, system, device, and storage medium based on ZooKeeper. Background Art

[0002] As an experimental design and analysis method, AB testing aims to compare two or more versions of products, services, or strategies to determine which version performs better in terms of user behavior, conversion rate, or other key metrics. AB testing is commonly used in fields such as website optimization, advertising effectiveness evaluation, and product feature improvement. AB testing assigns participants to different experimental groups and control groups to eliminate the influence of external factors on the experimental results.

[0003] AB testing relies on statistical analysis to evaluate the significance of experimental results, determining whether there are significant differences between the experimental group and the control group and the degree of such differences through statistical analysis. In addition, the interpretation of AB testing results requires comprehensive consideration of factors such as experimental design, statistical analysis, and business background. In practice, AB testing is usually an iterative process that helps organizations better understand user behavior and improve products or services through continuous testing and optimization.

[0004] In related technologies, trial configuration information is usually stored in the form of local cache, and then multiple stages of testing are carried out. During the testing process, the corresponding trial configuration information is directly loaded from the local cache. This trial configuration information includes the shunt information between users and the experimental group and the control group. When it is necessary to adjust the shunt information, the original cache content needs to be deleted, and the user ID information corresponding to the experimental group and the control group is replaced as a whole. The processing process is cumbersome, and the entire AB testing requires a large amount of time and memory. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides an AB testing method, system, device, and storage medium based on ZooKeeper, which solves the problems of cumbersome processing procedures and high time and memory consumption in AB testing.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a ZooKeeper-based AB testing method, and the AB testing method includes: obtaining configuration information of the AB testing, where the configuration information includes a control version and at least one experimental version to be tested; encapsulating the configuration information into an SDK toolkit and storing it in the ZooKeeper framework; when receiving test request information, determining whether multiple user IDs corresponding to the test request information are in a preset white list; when the multiple user IDs corresponding to the test request information are not in the white list, performing traffic splitting based on a traffic splitting algorithm including an MD5 hash function and modular arithmetic, allocating the multiple user IDs to the control version and the experimental version, and determining user traffic splitting information.

[0008] Based on the user traffic splitting information, loading the SDK toolkit in the ZooKeeper framework for testing; collecting test data, analyzing the test data to obtain a preliminary analysis result, and visually displaying it on a background page; when the preliminary analysis result meets a preset target termination condition, determining the termination of the AB testing; when the preliminary analysis result does not meet the preset target termination condition, adjusting the storage information of some nodes corresponding to the control version and the experimental version in the ZooKeeper framework to adjust the user traffic splitting information and continue the testing.

[0009] According to the first aspect of the embodiment of the present application, when the preliminary analysis result does not meet the preset target termination condition, the foregoing adjusting the storage information of some nodes corresponding to the control version and the experimental version in the ZooKeeper framework to adjust the user traffic splitting information and continue the testing includes: based on the preliminary analysis result, determining that the user ID that needs to be adjusted among the multiple user IDs is the target ID; receiving input user adjustment instruction information, where the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; based on the first mapping relationship, adjusting the target ID to a node corresponding to the corresponding experimental version or control version in the ZooKeeper framework.

[0010] According to the first aspect of the embodiment of the present application, after determining that the user ID that needs to be adjusted among the multiple user IDs is the target ID based on the preliminary analysis result, the foregoing ZooKeeper-based AB testing method further includes: when there is no user adjustment instruction information, based on the target ID, re-performing traffic splitting through a traffic splitting algorithm including an MD5 hash function and modular arithmetic to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; based on the second mapping relationship, re-allocating the target ID to the corresponding node to adjust the user traffic splitting information.

[0011] According to the first aspect of the embodiments of the present application, when the multiple user IDs corresponding to the test request information are not in the whitelist, the foregoing traffic splitting algorithm based on the MD5 hash function and modular arithmetic is used to split the traffic, and the multiple user IDs are assigned to the control version and the experimental version to determine the user traffic splitting information, including: concatenating the user ID and the geographical information to which the user ID belongs into a string; performing MD5 encryption on the string to generate a unique hash value; performing a modular operation on the hash value, and mapping the result of the modular operation to the ranges of the predefined experimental version and the control version to determine the user traffic splitting information.

[0012] According to the first aspect of the embodiments of the present application, the whitelist includes multiple target IDs and a configuration information list, and the configuration information list is used to represent the third mapping relationship between the target ID and the experimental version; before collecting test data, analyzing the test data to obtain a preliminary analysis result, and performing a visual display on the background page, the foregoing AB testing method based on ZooKeeper further includes: when the user ID is the same as the target ID in the whitelist, based on the third mapping relationship, the user ID is assigned to the corresponding experimental version for testing.

[0013] According to the first aspect of the embodiments of the present application, the foregoing collecting test data, analyzing the test data to obtain a preliminary analysis result, and performing a visual display on the background page includes: collecting the test data of the AB test, and sending the test data to a preset Kafka module for data transmission and real-time processing; transmitting the test data to a preset Spark module through the Kafka module; performing cleaning, transformation, and aggregation processing on the test data through the Spark module to obtain first data information; storing the first data information in a preset Hive warehouse for storage and query; sorting out the first data information to obtain a preliminary analysis result and performing a visual display on the background page.

[0014] According to the first aspect of the embodiments of the present application, when the test data corresponds to multiple metrics of the AB test, the foregoing sorting out the first data information to obtain a preliminary analysis result and performing a visual display on the background page includes: classifying the first data information based on multiple metrics to obtain multiple second data information; determining the weight ratio corresponding to the multiple metrics; performing weighted processing on the multiple second data information based on the weight ratio to obtain third data information; performing an alarm prompt when the third data information exceeds a preset target threshold; sorting the third data information into the form of a report or a chart to obtain a preliminary analysis result for visual display on the background page.

[0015] Second aspect, an embodiment of the present application provides a ZooKeeper-based AB testing system, which includes: an acquisition module, a packaging module, a judgment module, a traffic splitting module, a loading module, an analysis module, a termination module, and an adjustment module; the acquisition module is used to acquire the configuration information of the AB test, and the configuration information includes a control version and at least one experimental version to be tested; the packaging module is used to package the configuration information into an SDK toolkit and store it in the ZooKeeper framework; the judgment module is used to judge whether the multiple user IDs corresponding to the test request information are in a preset white list when receiving the test request information; the traffic splitting module is used to perform traffic splitting based on a traffic splitting algorithm including an MD5 hash function and modular arithmetic when the multiple user IDs corresponding to the test request information are not in the white list, allocate the multiple user IDs to the control version and the experimental version, and determine user traffic splitting information; the loading module is used to load the SDK toolkit in the ZooKeeper framework for testing based on the user traffic splitting information; the analysis module is used to collect test data, analyze the test data to obtain a preliminary analysis result, and perform visual display on the background page; the termination module is used to determine the termination of the AB test when the preliminary analysis result meets a preset target termination condition; the adjustment module is used to adjust the storage information of some nodes corresponding to the control version and the experimental version in the ZooKeeper framework when the preliminary analysis result does not meet the preset target termination condition, so as to adjust the user traffic splitting information and continue the test.

[0016] Third aspect, an embodiment of the present application provides an electronic device, which includes: a processor, a memory, and a program stored on the memory and executable on the processor, and when the program is executed by the processor, it implements the ZooKeeper-based AB testing method in the foregoing first aspect.

[0017] Fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or an instruction is stored, and when the program or the instruction is executed by the processor, it implements the ZooKeeper-based AB testing method in the foregoing first aspect.

[0018] The present application provides a ZooKeeper-based AB testing method, system, device, and storage medium. Compared with the prior art, it has the following beneficial effects:

[0019] This application encapsulates configuration information into an SDK and stores it in the ZooKeeper framework, constructs a distributed coordination system through ZooKeeper, and each node can work collaboratively; when diverting users, it can be diverted based on a diversion algorithm that includes an MD5 hash function and modular arithmetic, and specific users can also be bypassed from the diversion algorithm through a preset whitelist for direct user allocation. This application provides two diversion strategies, increasing the flexibility of AB testing; after completing user diversion, this application can directly load the SDK toolkit in the ZooKeeper framework for testing, and by adjusting the storage information of some nodes in the ZooKeeper framework, directly exchange user ID information, so as to directly adjust the control version or experimental version assigned to some user IDs, without having to replace the user IDs assigned to each experimental version and control version as a whole, with a fast speed of re-diverting users and basically no memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a flowchart showing a method for AB testing based on ZooKeeper provided by an embodiment of this application;

[0022] Figure 2 is Figure 1 An exemplary flowchart of S160 in

[0023] Figure 3 It is a structural diagram of a system for AB testing based on ZooKeeper provided by an embodiment of this application;

[0024] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0026] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0027] By providing a ZooKeeper-based AB testing method, system, device and storage medium, the embodiments of the present application solve the problems of cumbersome processing procedures and the need to consume a large amount of time and memory in AB testing.

[0028] The overall idea of the technical solution in the embodiments of the present application to solve the above technical problems is as follows:

[0029] As an experimental design and analysis method, AB testing aims to compare two or more versions of products, services or strategies to determine which version performs better in terms of user behavior, conversion rate or other key metrics. AB testing is commonly used in fields such as website optimization, advertising effectiveness evaluation, and product function improvement. One of the key concepts behind AB testing is random grouping, that is, randomly assigning participants to different experimental groups and control groups to eliminate the influence of external factors on the experimental results. Random grouping requires careful design, including determining the experimental objectives, selecting appropriate variables and metrics, and designing the experimental plan.

[0030] AB testing relies on statistical analysis to evaluate the significance of experimental results, and determines whether there are significant differences between the experimental group and the control group through statistical analysis, as well as the degree of such differences. In addition, the interpretation of the results of AB testing needs to comprehensively consider factors such as experimental design, statistical analysis, and business background. In addition to evaluating the differences between the experimental group and the control group, factors such as the reliability of the experimental results, implementation costs, and user experience also need to be considered to determine the final decision-making recommendations. This comprehensive interpretation is crucial for formulating effective business strategies. In practice, AB testing is usually an iterative process, and by continuously testing and optimizing, it helps organizations better understand user behavior and improve products or services.

[0031] In the related art, the test configuration information is usually stored in the form of local cache, and then multiple stages of testing are carried out. During the testing process, the corresponding test configuration information is directly loaded from the local cache to reduce network latency. The test configuration information includes the shunt information between users and the experimental group and the control group. When the shunt information needs to be adjusted, the original cache content needs to be deleted, and the user ID information corresponding to the experimental group and the control group is replaced as a whole. The processing process is cumbersome, and the entire AB test requires a lot of time and memory.

[0032] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0033] First, a ZooKeeper-based AB testing method provided by an embodiment of the present application will be introduced below.

[0034] A flowchart of a ZooKeeper-based AB testing method provided by an embodiment of the present application is shown as Figure 1 shown, and the AB testing method may include the following steps S110-S180.

[0035] S110. Obtain the configuration information of the AB test. The configuration information includes the control version and at least one test version to be tested.

[0036] S120. Package the configuration information into an SDK toolkit and store it in the ZooKeeper framework.

[0037] S130. When receiving the test request information, determine whether the multiple user IDs corresponding to the test request information are in a preset white list.

[0038] S140. When the multiple user IDs corresponding to the test request information are not in the white list, perform shunting based on a shunting algorithm including an MD5 hash function and modulo operation, allocate the multiple user IDs to the control version and the test version, and determine the user shunting information.

[0039] S150. Based on the user shunting information, load the SDK toolkit in the ZooKeeper framework for testing.

[0040] S160. Collect test data, analyze the test data to obtain a preliminary analysis result, and perform visual display on the background page.

[0041] S170. When the preliminary analysis result meets the preset target termination condition, determine that the AB test terminates.

[0042] S180. When the preliminary analysis result does not meet the preset target termination condition, adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user shunt information and continue the test.

[0043] In the embodiments of the present application, it can be understood that ZooKeeper uses a directory tree structure similar to a file system to store data. Each node can store data and specify other nodes as child nodes. In this way, ZooKeeper constructs a distributed coordination system. The core of ZooKeeper is a distributed coordination service with consistency, durability, and availability, which ensures data consistency and coordination in the distributed system and enables each service to work together.

[0044] It should be noted that before conducting the AB test, an AB test task can be created in the background management system, and multiple target projects can be created in the child nodes of the AB test task. The multiple target projects include a control version and at least one test version. After determining the specific control version and test version, the multiple user IDs participating in the test can be shunted to the lower-level nodes of the control version and the test version to form a tree structure in the ZooKeeper framework. It can be understood that in an AB test corresponding to a test ID, multiple test versions can be set, and user shunting can be performed for the multiple test versions and the control version. A user usually only participates in one test within a period of time.

[0045] The above is the specific implementation method of an AB test method based on ZooKeeper provided by the embodiments of the present application. The present application encapsulates the configuration information into an SDK and stores it in the ZooKeeper framework, and constructs a distributed coordination system through ZooKeeper, enabling each node to work together. When shunting users, shunting can be performed based on a shunting algorithm that includes an MD5 hash function and modular arithmetic. Additionally, specific users can be bypassed from the shunting algorithm through a preset whitelist for direct user assignment. The present application provides two shunting strategies, increasing the flexibility of the AB test. After completing user shunting, the present application can directly load the SDK toolkit in the ZooKeeper framework for testing. By adjusting the storage information of some nodes in the ZooKeeper framework, the user ID information can be directly exchanged, thereby directly adjusting the control version or test version assigned to some user IDs without replacing the user IDs assigned to each test version and control version as a whole. The speed of re-shunting users is fast and basically does not consume memory.

[0046] It should also be noted that the present application encapsulates the configuration information of the control version and at least one test version to be tested into an SDK toolkit, and stores the SDK toolkit in the ZooKeeper framework. It can be understood that encapsulating the configuration information into the SDK toolkit in the present application has the following advantages:

[0047] (1) Easy integration: An SDK is a software development toolkit that encapsulates a series of functions and services and can be easily integrated into existing applications. For developers, they only need to introduce and correctly configure the SDK in the application, and then they can call various APIs and services provided by the SDK. This integration method greatly reduces the workload of developers and eliminates the need to understand and implement the specific logic and functions inside the SDK. For the SDK toolkit for AB testing in the present application, developers only need to call the tool to achieve the main functions of AB testing without having to worry about details such as how to obtain experiment configurations, how to perform version splitting, and how to handle exception situations.

[0048] (2) Code reuse: During the process of software development, code reuse is a common method to improve development efficiency and code quality. Encapsulating the SDK toolkit in AB testing enables code reuse for AB testing functions. This is because if multiple projects or applications need to conduct AB testing, they can directly use the same SDK toolkit for AB testing instead of rewriting the same AB testing code in each project. This can avoid code duplication, improve development efficiency, and at the same time reduce the complexity and cost of code maintenance.

[0049] (3) Standardization: The SDK toolkit can provide a set of standard interfaces and implementation methods for application development. In AB testing, by using the SDK, it can be ensured that the methods and processes for conducting AB testing in different applications are consistent without having to worry about the special implementation methods or differences that may exist in each application. The benefit of this standardization is that it can avoid potential problems caused by inconsistent implementation methods, such as data inconsistency and incomparable test results. At the same time, the standardized interfaces and implementation methods also make the SDK toolkit easier to understand and use.

[0050] (4) Easy update and maintenance: If the logic or implementation method of AB testing needs to be updated, only the SDK needs to be updated, and then the new version of the SDK can be introduced in the application. This can avoid the need to modify the code in each application that references the AB testing function, reducing the workload and complexity of update and maintenance.

[0051] (5) Provide more functions: In addition to the basic AB testing function, the SDK can also provide more functions and services. For example, in addition to version splitting, the SDK can also provide additional functions such as data analysis, result reporting, and exception handling. In this way, the applications using the SDK can not only achieve the main functions of AB testing but also enjoy other additional services provided by the SDK. This method can more comprehensively meet the needs of the applications and provide better services and experiences.

[0052] In some embodiments, when the preliminary analysis result does not meet the preset target termination condition, adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user splitting information and continue the test. That is, the foregoing S180 may specifically include the following steps:

[0053] S210. Based on the preliminary analysis result, determine that the user ID that needs to be adjusted among multiple user IDs is the target ID.

[0054] S220. Receive the input user adjustment instruction information, where the user adjustment instruction information includes the first mapping relationship between the target ID and the configuration information.

[0055] S230. Based on the first mapping relationship, adjust the target ID to the node corresponding to the corresponding test version or control version in the ZooKeeper framework.

[0056] In the embodiments of the present application, it can be understood that when the preliminary analysis result does not meet the preset target termination condition, that is, when the result of the AB test does not meet the expectation, replace the user IDs assigned to the test version and the control version to re-perform user splitting, and then re-perform the test; in this process, if the developer of the AB test has other clear user splitting record strategies based on the preliminary analysis result and inputs the user adjustment instruction information, where the user adjustment instruction information includes the first mapping relationship between the target ID and the configuration information; based on this, directly re-allocate some user IDs.

[0057] In the process of adjusting the target ID to the corresponding test version or control version in the ZooKeeper framework, the corresponding node information can be directly exchanged, which basically does not consume memory and does not require an overall replacement of the user IDs assigned to each test version and control version. When there are problems with the preliminary analysis result, by receiving the user adjustment instruction information input by the developer, the user ID in the test version can be quickly adjusted to the control version to complete the error correction of the AB test.

[0058] In some embodiments, after determining, based on the preliminary analysis result, that the user IDs to be adjusted among the multiple user IDs are target IDs, that is, after the foregoing S210, the AB testing method based on ZooKeeper may specifically further include the following steps:

[0059] S211. In the case where there is no user adjustment instruction information, based on the target ID, re-perform shunting through a shunting algorithm including an MD5 hash function and modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework.

[0060] S212. Based on the second mapping relationship, re-assign the target ID to the corresponding node to adjust the user shunting information.

[0061] In the embodiments of the present application, it can be understood that when the result of the AB test does not meet the expectation, during the process of re-performing user shunting by replacing the user IDs assigned to the test version and the control version, if there is no clear user adjustment instruction information to guide the user re-shunting work, the multiple user IDs can be directly re-shunted again based on the shunting algorithm.

[0062] In some embodiments, in the case where the multiple user IDs corresponding to the test request information are not in the white list, the foregoing shunting based on the shunting algorithm including the MD5 hash function and modulo operation, and assigning the multiple user IDs to the control version and the test version to determine the user shunting information, that is, the foregoing S140 may specifically include the following steps:

[0063] S310. Concatenate the user ID and the geographical information to which the user ID belongs into a string.

[0064] S320. Encrypt the string by MD5 to generate a unique hash value.

[0065] S330. Perform a modulo operation on the hash value, and map the modulo operation result to the ranges of the predefined test version and the control version to determine the user shunting information.

[0066] In the embodiments of the present application, it can be understood that through the shunt algorithm of the MD5 hash function and modular arithmetic, it can be ensured that the same user ID is always assigned to the same test version or control version under the same experimental conditions, thus ensuring the reliability and repeatability of the experimental results. Another advantage of this algorithm is that its shunt process is completely random and uniform, avoiding any human intervention or bias, and ensuring that each user ID has an equal opportunity to participate in different experimental versions. In addition, the shunt algorithm based on the MD5 hash function and modular arithmetic also has the characteristics of high efficiency and scalability, and can operate stably in scenarios with a large number of users and high concurrent requests. Through this algorithm, developers can flexibly perform various experimental configurations and shunt strategies, providing reliable data support for business optimization and decision-making.

[0067] In some embodiments, the foregoing whitelist includes a plurality of target IDs and a configuration information list, and the configuration information list is used to represent a third mapping relationship between the target ID and the test version. Before collecting the test data, analyzing the test data to obtain a preliminary analysis result, and visually displaying it on the background page, that is, before S160, the AB testing method based on ZooKeeper may specifically further include the following steps:

[0068] S151. When the user ID is consistent with the target ID in the whitelist, based on the third mapping relationship, assign the user ID to the corresponding test version for testing.

[0069] In the embodiments of the present application, it can be understood that when a user ID is consistent with the target ID in the whitelist, it is assigned to the specified test version in the whitelist, and no regular shunt algorithm calculation is performed; in this way, developers can ensure that specific user IDs are always in the specified test version, thus facilitating targeted testing and verification.

[0070] It should be noted that in actual applications, there are many scenarios. For example, before a new function is launched, it is necessary to conduct a gray-scale test on internal employees or specific user groups to ensure the stability of the function and the user experience. In this case, developers can add the target IDs corresponding to these specific users to the whitelist and use the specified test version for testing and feedback. In addition, the whitelist function can also be used for the personalized experience of special customers or VIP users, providing better services and experiences by assigning specific test versions to these users. In short, the whitelist function greatly enhances the flexibility and practicality of AB testing, enabling it to better meet diverse business needs and test scenarios.

[0071] In some embodiments, please refer to Figure 2, collecting the foregoing test data, analyzing the test data to obtain a preliminary analysis result, and visually displaying the result on the background page. That is, the foregoing S160 may specifically include the following steps:

[0072] S410. Collect the test data of the A / B test and send the test data to a preset Kafka module for data transmission and real-time processing.

[0073] S420. Transmit the test data to a preset Spark module through the Kafka module.

[0074] S430. Clean, transform, and aggregate the test data through the Spark module to obtain the first data information.

[0075] S440. Store the first data information in a preset Hive warehouse for storage and query.

[0076] S450. Organize the first data information to obtain a preliminary analysis result and visually display the result on the background page.

[0077] In the embodiment of the present application, it can be understood that after collecting the test data of the A / B test, the data is sent to the Kafka module. The Kafka module is an efficient message queue system for data transmission and real-time processing. Through the powerful function of the Kafka module, we can stream a large amount of data to the Spark module. Spark is a powerful distributed computing engine that can be used for data processing and analysis. The fast computing ability of Spark and its characteristic of supporting multiple data sources enable developers to quickly process the A / B test data and perform complex data operations, such as data cleaning, transformation, and aggregation.

[0078] Further, the data processed by the Spark module is stored in a Hive warehouse based on Hadoop. The Hive warehouse provides powerful SQL query functions and scalability, enabling easy data analysis and mining. The Hive warehouse can store a large amount of the first data information and retrieve and analyze this data at any time according to needs. Finally, the first data information of the A / B test is displayed in the form of a data report or chart on a specially designed A / B test background page. These visual displays of data enable users to more intuitively understand the comparison results between different variants, thereby better evaluating the effect and value of the A / B test.

[0079] It should be noted that through data-driven decision-making and visual presentation, developers can continuously optimize the AB test strategy, enhance the user experience and business performance, and achieve more effective product optimization and marketing strategies. Through the closed-loop of the entire process, developers can establish a complete AB test ecosystem to provide strong support for the continuous development of the project.

[0080] In some embodiments, when the test data corresponds to multiple metrics of the AB test, the foregoing collation of the first data information to obtain a preliminary analysis result and visual display on the background page, that is, the foregoing S450 may specifically include the following steps:

[0081] S451. Classify the first data information based on multiple metrics to obtain multiple second data information.

[0082] S452. Determine the weight ratios corresponding to the multiple metrics.

[0083] S453. Perform weighted processing on the multiple second data information based on the weight ratios to obtain third data information.

[0084] S454. When the third data information exceeds a preset target threshold, give an alarm prompt.

[0085] S455. Organize the third data information into the form of a report or chart to obtain a preliminary analysis result for visual display on the background page.

[0086] In the embodiments of the present application, it can be understood that in an AB test, multiple metrics can be set. For example, in the e-commerce industry, the AB test can set different metrics such as sales volume and profit margin. The first data information obtained from the AB test will cover multiple metrics. The data corresponding to each metric is considered separately and weighted based on the preset weight ratios to obtain the expected third data information.

[0087] Furthermore, the present application presets a target threshold. When the obtained third data information exceeds this target threshold, an alarm prompt is given in a timely manner to remind the developer to correct the test loopholes in a timely manner and reduce losses.

[0088] In some embodiments, please refer to Figure 3 , the present application provides an AB test system 500 based on ZooKeeper. The AB test system 500 may specifically include:

[0089] An acquisition module 510, configured to acquire the configuration information of the AB test. The configuration information includes a control version and at least one test version to be tested.

[0090] The encapsulation module 520 is used to encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework.

[0091] The judgment module 530 is used to judge whether the multiple user IDs corresponding to the test request information are in a preset white list when receiving the test request information.

[0092] The shunt module 540 is used to perform shunting based on a shunt algorithm including an MD5 hash function and modular arithmetic when the multiple user IDs corresponding to the test request information are not in the white list, allocate the multiple user IDs to the control version and the experimental version, and determine the user shunt information.

[0093] The loading module 550 is used to load the SDK toolkit in the ZooKeeper framework for testing based on the user shunt information.

[0094] The analysis module 560 is used to collect test data, analyze the test data, obtain a preliminary analysis result, and perform visual display on the background page.

[0095] The termination module 570 is used to determine the termination of the AB test when the preliminary analysis result meets the preset target termination condition.

[0096] The adjustment module 580 is used to adjust the stored information of some nodes corresponding to the control version and the experimental version in the ZooKeeper framework to adjust the user shunt information and continue the test when the preliminary analysis result does not meet the preset target termination condition.

[0097] According to the embodiments of the present application, any multiple of the acquisition module 510, the encapsulation module 520, the judgment module 530, the shunt module 540, the loading module 550, the analysis module 560, the termination module 570, and the adjustment module 580 can be combined and implemented in one module, or any one of them can be split into multiple modules. Or, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0098] In some embodiments, the AB test system 500 may further include:

[0099] The allocation test module 551 is used to allocate the user ID to the corresponding experimental version for testing based on the third mapping relationship when the user ID is consistent with the target ID in the white list.

[0100] In some embodiments, the foregoing adjustment module 580 may specifically be used for:

[0101] Based on the preliminary analysis results, determine the user IDs that need to be adjusted among multiple user IDs as target IDs;

[0102] Receive the input user adjustment instruction information, where the user adjustment instruction information includes the first mapping relationship between the target ID and the configuration information;

[0103] Based on the first mapping relationship, adjust the target ID to the corresponding nodes in the test version or control version in the ZooKeeper framework.

[0104] In some embodiments, the aforementioned adjustment module 580 can specifically also be used for:

[0105] In the absence of user adjustment instruction information, based on the target ID, re - perform sharding through a sharding algorithm that includes an MD5 hash function and modulo operation, to obtain the second mapping relationship between the target ID and some nodes in the ZooKeeper framework;

[0106] Based on the second mapping relationship, re - allocate the target ID to the corresponding nodes to adjust the user sharding information.

[0107] In some embodiments, the aforementioned sharding module 540 can specifically be used for:

[0108] Concatenate the user ID and the geographical information to which the user ID belongs into a string;

[0109] Encrypt the string using MD5 to generate a unique hash value;

[0110] Perform modulo operation on the hash value, and map the modulo operation result to the predefined ranges of the test version and the control version to determine the user sharding information.

[0111] In some embodiments, the aforementioned analysis module 560 can specifically include:

[0112] The first data transmission unit 561, which is used to collect the test data of the AB test and send the test data to a preset Kafka module for data transmission and real - time processing.

[0113] The second data transmission unit 562, which is used to transmit the test data to a preset Spark module through the Kafka module.

[0114] The data processing unit 563, which is used to clean, transform, and aggregate the test data through the Spark module to obtain the first data information.

[0115] The data storage unit 564, which is used to store the first data information in a preset Hive warehouse for storage and query.

[0116] The sorting and display unit 565 is used to sort the first data information to obtain a preliminary analysis result and visually display it on the background page.

[0117] In some embodiments, the aforementioned sorting and display unit 565 can specifically be used for:

[0118] Classify the first data information based on multiple metrics to obtain multiple second data information;

[0119] Determine the weight ratios corresponding to the multiple metrics;

[0120] Perform weighted processing on the multiple second data information based on the weight ratios to obtain third data information;

[0121] In the case where the third data information exceeds a preset target threshold, give an alarm prompt;

[0122] Sort the third data information into the form of a report or chart to obtain a preliminary analysis result for visual display on the background page.

[0123] Figure 3 Each module in the shown system has the function of implementing each step in the aforementioned ZooKeeper-based AB testing method and can achieve its corresponding technical effects. For the sake of brevity, it will not be elaborated here.

[0124] In some embodiments, the present application provides an electronic device, and the structural schematic diagram of the electronic device is as Figure 4 shown.

[0125] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0126] Specifically, the aforementioned processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0127] The memory 620 may include a mass storage for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 620 may include removable or non-removable (or fixed) media. Where appropriate, the memory 620 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, the memory 620 is a non-volatile solid-state memory.

[0128] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the above-described ZooKeeper-based AB testing methods of the embodiments.

[0129] The processor 610 reads and executes the computer program instructions stored in the memory 620 to implement any of the above-described ZooKeeper-based AB testing methods of the embodiments.

[0130] In one example, the electronic device may further include a communication interface 630 and a bus 600. Among them, as Figure 4 shown, the processor 610, the memory 620, and the communication interface 630 are connected through the bus 600 and complete communication with each other.

[0131] The communication interface 630 is mainly used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application.

[0132] The bus 600 includes hardware, software, or both, and couples the components of the online data flow metering device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 600 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0133] In addition, in combination with the ZooKeeper-based AB testing method in the above embodiments, the embodiments of the present application can be implemented by providing a computer storage medium. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the ZooKeeper-based AB testing methods in the above embodiments is implemented.

[0134] It should be clear that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, the detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0135] The functional blocks shown in the above block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or communication link. A "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0136] It should also be noted that in the exemplary embodiments mentioned in this application, some methods or systems are described based on a series of steps or devices. However, this application is not limited to the order of the above steps. That is to say, the steps can be executed in the order mentioned in the embodiments, can be different from the order in the embodiments, or several steps can be executed simultaneously.

[0137] As described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowchart and / or block diagram, and the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0138] In summary, compared with the prior art, this application has the following beneficial effects:

[0139] 1. This application constructs a distributed coordination system through ZooKeeper, and each node can work collaboratively; this application provides two shunting strategies. When shunting users, it can be shunted based on a shunting algorithm that includes an MD5 hash function and modular arithmetic, or specific users can be bypassed the shunting algorithm through a preset whitelist and directly assigned, increasing the flexibility of the AB test.

[0140] 2. By adjusting the storage information of some nodes in the ZooKeeper framework, this application can directly exchange user ID information, thereby directly adjusting the control version or experimental version assigned to some user IDs without having to replace the user IDs assigned to each experimental version and control version as a whole. The speed of re-shunting users is fast and it basically does not consume memory. In the case where there are problems in the preliminary analysis results, by receiving the user adjustment instruction information input by the developer, the user IDs in the experimental version can be quickly adjusted to the control version to complete the error correction of the AB test.

[0141] 3. Encapsulate the configuration information into an SDK and store it in the ZooKeeper framework. After user traffic splitting is completed, this application can directly load the SDK toolkit in the ZooKeeper framework for testing. The SDK toolkit can provide a set of standard interfaces and implementation methods for application development, which are easy to update and maintain, achieve code reuse, have a high degree of integration, and can also provide additional functions such as data analysis, result reporting, and exception handling.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A ZooKeeper-based AB testing method, characterized in that: include: Acquire configuration information of the AB test, where the configuration information includes a control version and at least one test version to be tested; Encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework; When receiving the test request information, determining whether the multiple user IDs corresponding to the test request information are in a preset whitelist; In the case that the multiple user IDs corresponding to the test request information are not in the whitelist, performing diversion based on a diversion algorithm including an MD5 hash function and a modulo operation, allocating the multiple user IDs to the control version and the test version, and determining user diversion information; Based on the user diversion information, loading the SDK toolkit in the ZooKeeper framework for testing; Collect test data, analyze the test data, obtain preliminary analysis results, and visualize them on the background page; When the preliminary analysis result meets the preset target termination condition, determining that the AB test is terminated; If the preliminary analysis result does not meet the preset target termination condition, adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user diversion information and continue the test; When the preliminary analysis result does not meet the preset target termination condition, the adjusting the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user diversion information and continue the test includes: Based on the preliminary analysis result, determining the user ID that needs to be adjusted among the multiple user IDs as the target ID; Receiving input user adjustment instruction information, wherein the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; Based on the first mapping relationship, adjusting the target ID to a node corresponding to the test version or the control version in the ZooKeeper framework; After determining that the user ID that needs to be adjusted among the multiple user IDs is the target ID based on the preliminary analysis result, the ZooKeeper-based AB testing method further includes: In the absence of the user adjustment instruction information, based on the target ID, re-dividing the traffic by a diversion algorithm including an MD5 hash function and a modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; Based on the second mapping relationship, the target ID is reallocated to the corresponding node to adjust the user diversion information.

2. The ZooKeeper-based AB testing method according to claim 1, characterized in that: In the case that the multiple user IDs corresponding to the test request information are not in the whitelist, the diversion algorithm based on the MD5 hash function and the modulus operation is used to perform diversion, the multiple user IDs are allocated to the control version and the test version, and the user diversion information is determined, including: Concatenate the user ID and the region information to which the user ID belongs into a character string; Perform MD5 encryption on the string to generate a unique hash value; A modulo operation is performed on the hash value, and the modulo operation result is mapped to a predefined range of the test version and a predefined range of the control version to determine user diversion information.

3. The ZooKeeper-based AB testing method according to claim 1, characterized in that: The whitelist includes a plurality of target IDs and a configuration information list, wherein the configuration information list is used to represent a third mapping relationship between the target ID and the test version; Before collecting the test data, analyzing the test data, obtaining the preliminary analysis results, and visually displaying them on the background page, the ZooKeeper-based AB testing method further includes: When the user ID is consistent with the target ID in the whitelist, the user ID is allocated to the corresponding test version based on the third mapping relationship for testing.

4. The ZooKeeper-based AB testing method according to claim 1, characterized in that: The test data is collected, analyzed, and preliminary analysis results are obtained, and displayed visually on a backend page, including: Collect test data of the AB test, and send the test data to a preset Kafka module for data transmission and real-time processing; The test data is transmitted to a preset Spark module through the Kafka module; Cleaning, converting and aggregating the test data through the Spark module to obtain first data information; Storing the first data information in a preset Hive warehouse for storage and query; The first data information is sorted to obtain preliminary analysis results, and the results are visualized on the background page.

5. The ZooKeeper-based AB testing method according to claim 4, characterized in that: In the case where the test data corresponds to multiple indicators of the AB test, the first data information is sorted to obtain a preliminary analysis result and is visually displayed on a background page, including: Based on the multiple indicators, classify the first data information to obtain multiple second data information; Determine the weight ratios corresponding to the multiple indicators; Based on the weight ratio, weighted processing is performed on the plurality of the second data information to obtain third data information; When the third data information exceeds a preset target threshold, an alarm is issued; The third data information is organized into a report or chart to obtain a preliminary analysis result for visual display on a background page.

6. A ZooKeeper-based AB testing system, characterized in that: include: An acquisition module, used to acquire configuration information of the AB test, wherein the configuration information includes a control version and at least one test version to be tested; An encapsulation module, used to encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework; A judgment module, used for judging whether the multiple user IDs corresponding to the test request information are in a preset whitelist when receiving the test request information; A diversion module is used to perform diversion based on a diversion algorithm including an MD5 hash function and a modular operation when the multiple user IDs corresponding to the test request information are not in the whitelist, allocate the multiple user IDs to the control version and the test version, and determine user diversion information; A loading module, used for loading the SDK toolkit in the ZooKeeper framework for testing based on the user diversion information; The analysis module is used to collect test data, analyze the test data, obtain preliminary analysis results, and visualize them on the background page; A termination module, configured to determine the termination of the AB test when the preliminary analysis result meets a preset target termination condition; An adjustment module, used for adjusting the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework when the preliminary analysis result does not meet the preset target termination condition, so as to adjust the user diversion information and continue the test; When the preliminary analysis result does not meet the preset target termination condition, the adjusting the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user diversion information and continue the test includes: Based on the preliminary analysis result, determining the user ID that needs to be adjusted among the multiple user IDs as the target ID; Receiving input user adjustment instruction information, wherein the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; Based on the first mapping relationship, adjusting the target ID to a node corresponding to the test version or the control version in the ZooKeeper framework; After determining, based on the preliminary analysis result, that the user ID to be adjusted among the multiple user IDs is the target ID, the method further includes: In the absence of the user adjustment instruction information, based on the target ID, re-dividing the traffic by a diversion algorithm including an MD5 hash function and a modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; Based on the second mapping relationship, the target ID is reallocated to the corresponding node to adjust the user diversion information.

7. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the ZooKeeper-based AB testing method as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the ZooKeeper-based AB testing method as described in any one of claims 1 to 5 is implemented.

Images