AB test method, system and equipment based on ZooKeeper and storage medium

Abstract

The invention provides an AB testing method, system and device based on ZooKeeper and a storage medium, and relates to the technical field of AB testing, the method comprises the following steps: acquiring configuration information of AB testing, packaging the configuration information into an SDK toolkit, and storing the SDK toolkit in a ZooKeeper framework; the method comprises the following steps: carrying out shunting based on a shunting algorithm, distributing a plurality of user IDs to a contrast version and a test version, loading an SDK toolkit in a ZooKeeper framework for testing to obtain a preliminary analysis result, and adjusting storage information of partial nodes corresponding to the contrast version and the test version in the ZooKeeper framework so as to adjust user shunting information and continue testing. According to the method and the device, the storage information of part of nodes in the ZooKeeper framework is adjusted, and the user ID information is directly exchanged, so that the allocated contrast version or test version is directly adjusted for part of user IDs, the user IDs allocated to each test version and the contrast version do not need to be integrally replaced, the speed of re-shunting the users is high, and the user experience is improved. And the memory is basically not consumed.

Description

Technical Field

[0001] The present application relates to the technical field of AB testing, and in particular to a ZooKeeper-based AB testing method, system, device, and storage medium. Background Art

[0002] AB testing, as an experimental design and analysis method, aims to compare two or more versions of products, services or strategies to determine which version performs better in terms of user behavior, conversion rate or other key indicators. AB testing is often used in website optimization, advertising effectiveness evaluation, product function improvement and other fields. AB testing assigns participants to different experimental and control groups to eliminate the influence of external factors on the experimental results.

[0003] AB testing relies on statistical analysis to evaluate the significance of experimental results. Statistical analysis is used to determine whether there is a significant difference between the experimental group and the control group, and the extent of the difference. In addition, the interpretation of AB testing results requires comprehensive consideration of factors such as experimental design, statistical analysis, and business background. In practice, AB testing is usually an iterative process that helps organizations better understand user behavior and improve products or services through continuous testing and optimization.

[0004] In the related art, the test configuration information is usually stored in the form of a local cache, and then multiple stages of testing are performed. During the test, the corresponding test configuration information is directly loaded from the local cache, and the test configuration information includes the diversion information between users and the experimental group and the control group; when the diversion information needs to be adjusted, the original cache content needs to be deleted, and the user ID information corresponding to the experimental group and the control group needs to be replaced as a whole. The processing process is cumbersome, and the entire AB test requires a lot of time and memory. Summary of the invention In view of the deficiencies in the prior art, the present invention provides an AB testing method, system, device and storage medium based on ZooKeeper, which solves the problem that the processing procedures of AB testing are complicated and require a lot of time and memory.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides an AB testing method based on ZooKeeper, which includes: obtaining configuration information of the AB test, the configuration information including a control version and at least one test version to be tested; encapsulating the configuration information into an SDK toolkit, and storing it in the ZooKeeper framework; upon receiving test request information, determining whether multiple user IDs corresponding to the test request information are in a preset whitelist; if the multiple user IDs corresponding to the test request information are not in the whitelist, performing diversion based on a diversion algorithm including an MD5 hash function and a modulo operation, allocating multiple user IDs to the control version and the test version, and determining user diversion information.

[0006] Based on the user diversion information, the SDK toolkit is loaded in the ZooKeeper framework for testing; the test data is collected, analyzed, and preliminary analysis results are obtained, which are visualized on the background page; when the preliminary analysis results meet the preset target termination conditions, the AB test is terminated; when the preliminary analysis results do not meet the preset target termination conditions, the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework is adjusted to adjust the user diversion information and continue the test.

[0007] According to the first aspect of the embodiment of the present application, when the preliminary analysis results do not meet the preset target termination conditions, the aforementioned adjustment of the storage information belonging to some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user diversion information and continue testing includes: based on the preliminary analysis results, determining that the user ID that needs to be adjusted among multiple user IDs is the target ID; receiving input user adjustment instruction information, the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; based on the first mapping relationship, adjusting the target ID to the node corresponding to the corresponding test version or control version in the ZooKeeper framework.

[0008] According to the first aspect of the embodiment of the present application, after determining that the user ID that needs to be adjusted among multiple user IDs is the target ID based on the preliminary analysis results, the aforementioned ZooKeeper-based AB testing method also includes: in the absence of user adjustment instruction information, based on the target ID, re-diverting through a diversion algorithm including an MD5 hash function and a modulus operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; based on the second mapping relationship, reallocating the target ID to the corresponding node to adjust the user diversion information.

[0009] According to the first aspect of the embodiment of the present application, when multiple user IDs corresponding to the test request information are not in the whitelist, the aforementioned diversion algorithm based on the MD5 hash function and the modulus operation is used to perform diversion, and multiple user IDs are assigned to the control version and the test version to determine the user diversion information, including: concatenating the user ID and the regional information to which the user ID belongs into a string; performing MD5 encryption on the string to generate a unique hash value; performing a modulus operation on the hash value, and mapping the modulus operation result to a predefined range of the test version and a range of the control version to determine the user diversion information.

[0010] According to the first aspect of the embodiment of the present application, the whitelist includes multiple target IDs and a configuration information list, and the configuration information list is used to characterize a third mapping relationship between the target ID and the test version; before collecting test data, analyzing the test data, obtaining preliminary analysis results, and visually displaying them on the background page, the aforementioned ZooKeeper-based AB testing method also includes: when the user ID is consistent with the target ID in the whitelist, based on the third mapping relationship, assigning the user ID to the corresponding test version for testing.

[0011] According to the first aspect of the embodiment of the present application, the above-mentioned collection of test data, analysis of the test data, obtaining preliminary analysis results, and visual display on the background page include: collecting test data of AB testing, sending the test data to a preset Kafka module for data transmission and real-time processing; transmitting the test data to a preset Spark module through the Kafka module; cleaning, converting and aggregating the test data through the Spark module to obtain first data information; storing the first data information in a preset Hive warehouse for storage and query; organizing the first data information to obtain preliminary analysis results, and visually displaying it on the background page.

[0012] According to the first aspect of the embodiment of the present application, when the test data corresponds to multiple indicators of the AB test, the first data information is sorted out to obtain preliminary analysis results, and visually displayed on the background page, including: based on multiple indicators, the first data information is classified to obtain multiple second data information; the weight ratio corresponding to the multiple indicators is determined; based on the weight ratio, the multiple second data information is weighted to obtain third data information; when the third data information exceeds the preset target threshold, an alarm is issued; the third data information is sorted into a report or chart to obtain preliminary analysis results for visual display on the background page.

[0013] In the second aspect, an embodiment of the present application provides an AB testing system based on ZooKeeper, the AB testing system comprising: an acquisition module, an encapsulation module, a judgment module, a diversion module, a loading module, an analysis module, a termination module and an adjustment module; the acquisition module is used to obtain the configuration information of the AB test, the configuration information comprising a control version and at least one test version to be tested; the encapsulation module is used to encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework; the judgment module is used to determine whether multiple user IDs corresponding to the test request information are in a preset whitelist when the test request information is received; the diversion module is used to determine whether the multiple user IDs corresponding to the test request information are in a preset whitelist when the multiple user IDs corresponding to the test request information are not in the whitelist based on the MD5 The shunting algorithm based on hash function and modular operation is used to shun the traffic, and multiple user IDs are assigned to the control version and the test version to determine the user shunting information; the loading module is used to load the SDK toolkit in the ZooKeeper framework for testing based on the user shunting information; the analysis module is used to collect test data, analyze the test data, obtain preliminary analysis results, and display them visually on the background page; the termination module is used to determine the termination of the AB test when the preliminary analysis results meet the preset target termination conditions; the adjustment module is used to adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework when the preliminary analysis results do not meet the preset target termination conditions, so as to adjust the user shunting information and continue the test.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the ZooKeeper-based AB testing method in the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the ZooKeeper-based AB testing method in the first aspect is implemented.

[0016] This application provides an AB testing method, system, device and storage medium based on ZooKeeper. Compared with the prior art, it has the following beneficial effects: This application encapsulates configuration information into an SDK and stores it in the ZooKeeper framework. A distributed coordination system is built through ZooKeeper, and each node can work together. When users are diverted, diversion can be performed based on a diversion algorithm that includes an MD5 hash function and a modulo operation. Specific users can also bypass the diversion algorithm through a preset whitelist and directly perform user allocation. This application provides two diversion strategies, which increases the flexibility of AB testing. After completing user diversion, this application can directly load the SDK toolkit in the ZooKeeper framework for testing, and directly exchange user ID information by adjusting the storage information belonging to some nodes in the ZooKeeper framework, thereby directly adjusting the assigned control version or test version for some user IDs. There is no need to replace the user IDs assigned to each test version and control version as a whole, and the speed of re-diverting users is fast and basically no memory is consumed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of an AB testing method based on ZooKeeper provided in an embodiment of the present application; Figure 2 yes Figure 1 An exemplary flow chart of S160; Figure 3 It is a structural diagram of an AB testing system based on ZooKeeper provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0021] The embodiments of the present application provide an AB testing method, system, device and storage medium based on ZooKeeper, thereby solving the problem that the processing procedures of AB testing are complicated and require a lot of time and memory.

[0022] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows: As an experimental design and analysis method, AB testing aims to compare two or more versions of products, services or strategies to determine which version performs better in terms of user behavior, conversion rate or other key indicators. AB testing is often used in the fields of website optimization, advertising effectiveness evaluation, product function improvement, etc. One of the key concepts behind AB testing is random grouping, which is to randomly assign participants to different experimental and control groups to eliminate the influence of external factors on the experimental results. Random grouping requires careful design, including determining the experimental objectives, selecting appropriate variables and indicators, and designing experimental plans.

[0023] AB testing relies on statistical analysis to evaluate the significance of experimental results. Statistical analysis is used to determine whether there is a significant difference between the experimental group and the control group, and the extent of the difference. In addition, the interpretation of the results of AB testing requires comprehensive consideration of factors such as experimental design, statistical analysis, and business background. In addition to evaluating the differences between the experimental group and the control group, factors such as the reliability of the experimental results, implementation costs, and user experience need to be considered to determine the final decision recommendations. This comprehensive interpretation is essential for formulating effective business strategies. In practice, AB testing is usually an iterative process that helps organizations better understand user behavior and improve products or services through continuous testing and optimization.

[0024] In the related art, the test configuration information is usually stored in the form of a local cache, and then multiple stages of testing are performed. During the test, the corresponding test configuration information is directly loaded from the local cache to reduce network latency. The test configuration information includes diversion information between users and the experimental group and the control group. When the diversion information needs to be adjusted, the original cache content needs to be deleted and the user ID information corresponding to the experimental group and the control group needs to be replaced as a whole. The processing process is cumbersome, and the entire AB test requires a lot of time and memory. In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The following first introduces an AB testing method based on ZooKeeper provided in an embodiment of the present application.

[0026] The present application embodiment provides a flow chart of an AB testing method based on ZooKeeper, as shown in Figure 1 As shown, the AB testing method may include the following steps S110 - S180 .

[0027] S110: Obtain configuration information of the AB test, where the configuration information includes a control version and at least one test version to be tested.

[0028] S120: Encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework.

[0029] S130: When receiving the test request information, determine whether the multiple user IDs corresponding to the test request information are in a preset whitelist.

[0030] S140. When the multiple user IDs corresponding to the test request information are not in the whitelist, diversion is performed based on a diversion algorithm including an MD5 hash function and a modular operation, and the multiple user IDs are allocated to a control version and a test version to determine user diversion information.

[0031] S150. Based on the user diversion information, load the SDK toolkit in the ZooKeeper framework for testing.

[0032] S160, collecting test data, analyzing the test data, obtaining preliminary analysis results, and visually displaying them on a background page.

[0033] S170: When the preliminary analysis result meets the preset target termination condition, determine that the AB test is terminated.

[0034] S180. When the preliminary analysis result does not meet the preset target termination condition, the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework is adjusted to adjust the user diversion information and continue the test.

[0035] In the embodiments of the present application, it can be understood that ZooKeeper uses a directory tree structure similar to a file system to store data. Each node can store data and designate other nodes as child nodes. In this way, ZooKeeper builds a distributed coordination system. The core of ZooKeeper is a distributed coordination service of consistency, durability and availability, which ensures data consistency and coordination in a distributed system, so that various services can work together.

[0036] It should be noted that before conducting AB testing, you can create an AB testing task in the background management system, and create multiple target projects in the sub-nodes of the AB testing task. The multiple target projects include a control version and at least one test version. After determining the specific control version and test version, the multiple user IDs participating in the test can be diverted to the subordinate nodes of the control version and the test version to form a tree structure in the ZooKeeper framework; it can be understood that in the AB test corresponding to a test ID, multiple test versions can be set, and users can be diverted for multiple test versions and control versions. A user usually only participates in one test in a period of time.

[0037] The above is a specific implementation method of an AB testing method based on ZooKeeper provided in an embodiment of the present application. The present application encapsulates the configuration information into an SDK and stores it in the ZooKeeper framework. A distributed coordination system is constructed through ZooKeeper, and each node can work together. When users are diverted, diversion can be performed based on a diversion algorithm including an MD5 hash function and a modulo operation. Specific users can also bypass the diversion algorithm through a preset whitelist and directly perform user allocation. The present application provides two diversion strategies, which increases the flexibility of AB testing. After completing user diversion, the present application can directly load the SDK toolkit in the ZooKeeper framework for testing, and directly exchange user ID information by adjusting the storage information belonging to some nodes in the ZooKeeper framework, thereby directly adjusting the assigned control version or test version for some user IDs. There is no need to replace the user IDs assigned to each test version and control version as a whole, and the speed of re-diverting users is fast and basically no memory is consumed.

[0038] It should also be noted that the present application encapsulates the configuration information including the control version and at least one test version to be tested into an SDK toolkit, and stores the SDK toolkit in the ZooKeeper framework. It is understandable that the present application encapsulates the configuration information into an SDK toolkit, which has the following advantages: (1) Easy to integrate: SDK is a software development kit that encapsulates a series of functions and services and can be easily integrated into existing applications. For developers, they only need to introduce and correctly configure the SDK in the application, and then they can call the various APIs and services provided in the SDK. This integration method greatly reduces the workload of developers, and there is no need to understand and implement the specific logic and functions within the SDK. For the SDK toolkit for AB testing in this application, developers only need to call the tool to implement the main functions of AB testing, without having to worry about details such as how to obtain experimental configuration, how to perform version diversion, and how to handle abnormal situations.

[0039] (2) Code reuse: In the process of software development, code reuse is a common method to improve development efficiency and code quality. Encapsulating the SDK toolkit in AB testing can achieve code reuse of AB testing functions. This is because if multiple projects or applications need to perform AB testing, they can directly use the same AB testing SDK toolkit without having to rewrite the same AB testing code in each project. This can avoid code duplication, improve development efficiency, and also reduce the complexity and cost of code maintenance.

[0040] (3) Standardization: SDK toolkits can provide a set of standard interfaces and implementation methods for application development. In AB testing, by using SDK, you can ensure that the methods and processes of AB testing in different applications are consistent, without having to worry about the special implementation methods or differences that may exist in each application. The benefit of this standardization is that it can avoid potential problems caused by inconsistent implementation methods, such as inconsistent data and incomparable test results. At the same time, standardized interfaces and implementation methods also make SDK toolkits easier to understand and use.

[0041] (4) Easy to update and maintain: If the logic or implementation of AB testing needs to be updated, you only need to update the SDK and then introduce the new version of the SDK into the application. This avoids the need to modify the code in every application that references the AB testing function, reducing the workload and complexity of updating and maintenance.

[0042] (5) Provide more functions: In addition to the basic AB testing functions, SDK can also provide more functions and services. For example, in addition to version diversion, SDK can also provide additional functions such as data analysis, result reporting, exception handling, etc. In this way, applications using SDK can not only implement the main functions of AB testing, but also enjoy other additional services provided by SDK. This method can more comprehensively meet the needs of applications and provide better services and experience.

[0043] In some embodiments, when the preliminary analysis result does not meet the preset target termination condition, the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework is adjusted to adjust the user diversion information and continue the test, that is, the aforementioned S180 may specifically include the following steps: S210: Based on the preliminary analysis result, determine the user ID that needs to be adjusted among the multiple user IDs as the target ID.

[0044] S220: Receive input user adjustment instruction information, where the user adjustment instruction information includes a first mapping relationship between a target ID and configuration information.

[0045] S230. Based on the first mapping relationship, adjust the target ID to the node corresponding to the corresponding test version or control version in the ZooKeeper framework.

[0046] In the embodiment of the present application, it can be understood that when the preliminary analysis results do not meet the preset target termination conditions, that is, when the results of the AB test do not meet expectations, the user IDs assigned to the test version and the control version are replaced to re-divert users and then re-test; in this process, if the developer of the AB test has other clear user diversion filing strategies based on the preliminary analysis results, and inputs user adjustment instruction information, the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; based on this, some user IDs are directly reallocated.

[0047] In the process of adjusting the target ID to the corresponding test version or control version in the ZooKeeper framework, the corresponding node information can be directly exchanged, which basically consumes no memory and does not require the overall replacement of the user IDs assigned to each test version and control version. In the case of problems with the preliminary analysis results, by receiving the user adjustment instruction information entered by the developer, the user ID in the test version can be quickly adjusted to the control version to complete the AB test error correction.

[0048] In some embodiments, after determining that the user ID that needs to be adjusted among the multiple user IDs is the target ID based on the preliminary analysis result, that is, after the aforementioned S210, the ZooKeeper-based AB testing method may further include the following steps: S211. In the absence of user adjustment instruction information, based on the target ID, re-divert the traffic by using a diversion algorithm including an MD5 hash function and a modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework.

[0049] S212: Based on the second mapping relationship, reallocate the target ID to the corresponding node to adjust the user diversion information.

[0050] In the embodiments of the present application, it can be understood that when the results of the AB test do not meet expectations, in the process of replacing the user IDs assigned to the test version and the control version to re-divert users, if there is no clear user adjustment instruction information to guide the user re-diverting work, multiple user IDs can be re-diverted again directly based on the diversion algorithm.

[0051] In some embodiments, when the multiple user IDs corresponding to the test request information are not in the whitelist, the aforementioned shunting algorithm based on the MD5 hash function and the modulus operation is used to perform shunting, and the multiple user IDs are assigned to the control version and the test version, and the user shunting information is determined. That is, the aforementioned S140 may specifically include the following steps: S310: Concatenate the user ID and the region information to which the user ID belongs into a character string.

[0052] S320, perform MD5 encryption on the string to generate a unique hash value.

[0053] S330 , performing a modulo operation on the hash value, and mapping the modulo operation result to a predefined range of the test version and a range of the control version to determine user diversion information.

[0054] In the embodiment of the present application, it can be understood that the shunting algorithm based on the MD5 hash function and the modular operation can ensure that the same user ID is always assigned to the same test version or control version under the same experimental conditions, thereby ensuring the reliability and repeatability of the experimental results. Another advantage of the algorithm is that its shunting process is completely random and uniform, avoiding any human intervention or bias, ensuring that each user ID has a fair chance to participate in different experimental versions. In addition, the shunting algorithm based on the MD5 hash function and the modular operation is also efficient and scalable, and can run stably in large-scale users and high concurrent request scenarios. Through this algorithm, developers can flexibly perform a variety of experimental configurations and shunting strategies to provide reliable data support for business optimization and decision-making.

[0055] In some embodiments, the whitelist includes multiple target IDs and a configuration information list, and the configuration information list is used to characterize a third mapping relationship between the target ID and the test version. Before collecting test data, analyzing the test data, obtaining preliminary analysis results, and visually displaying them on the background page, that is, before S160, the ZooKeeper-based AB testing method may further include the following steps: S151. When the user ID is consistent with the target ID in the whitelist, the user ID is assigned to the corresponding test version based on the third mapping relationship for testing.

[0056] In the embodiments of the present application, it can be understood that when a user ID is consistent with the target ID in the whitelist, it is assigned to the test version specified in the whitelist, and the conventional diversion algorithm calculation is no longer performed; in this way, the developer can ensure that a specific user ID is always in the specified test version, thereby facilitating targeted testing and verification.

[0057] It should be noted that there are many scenarios in actual applications. For example, before a new function is launched, grayscale testing needs to be performed on internal employees or specific user groups to ensure the stability of the function and user experience. In this case, developers can add the target IDs corresponding to these specific users to the whitelist and use the specified trial version for testing and feedback. In addition, the whitelist function can also be used to personalize the experience of special customers or VIP users, by assigning specific trial versions to these users to provide better services and experience. In short, the whitelist function greatly enhances the flexibility and practicality of AB testing, enabling it to better meet diverse business needs and testing scenarios.

[0058] In some embodiments, please refer to Figure 2, the aforementioned collecting test data, analyzing the test data, obtaining preliminary analysis results, and visually displaying them on the background page, that is, the aforementioned S160 may specifically include the following steps: S410, collecting test data of the AB test, and sending the test data to a preset Kafka module for data transmission and real-time processing.

[0059] S420. Transmit the test data to the preset Spark module through the Kafka module.

[0060] S430: Clean, convert and aggregate the test data through the Spark module to obtain first data information.

[0061] S440: Store the first data information in a preset Hive warehouse for storage and query.

[0062] S450: sort the first data information to obtain a preliminary analysis result, and display it visually on a background page.

[0063] In the embodiment of the present application, it is understood that after collecting the test data of the AB test, the data is sent to the Kafka module, which is an efficient message queue system for data transmission and real-time processing. Through the powerful functions of the Kafka module, we can stream a large amount of data to the Spark module, which is a powerful distributed computing engine that can be used for data processing and analysis. Spark's fast computing power and support for multiple data sources enable developers to quickly process AB test data and perform complex data operations, such as data cleaning, conversion, and aggregation.

[0064] Furthermore, the data processed by the Spark module is stored in the Hive warehouse based on Hadoop. The Hive warehouse provides powerful SQL query functions and scalability, which can easily perform data analysis and mining. The Hive warehouse can store large-scale first-data information and retrieve and analyze this data at any time as needed. Finally, the first-data information of the AB test is displayed in the form of data reports or charts on the specially designed AB test background page. These data visualization displays allow users to more intuitively understand the comparison results between different variants, so as to better evaluate the effect and value of AB testing.

[0065] It should be noted that through data-driven decision-making and visual presentation, developers can continuously optimize AB testing strategies, improve user experience and business performance, and implement more effective product optimization and marketing strategies. By closing the entire process, developers can establish a complete AB testing ecosystem to provide strong support for the continued development of the project.

[0066] In some embodiments, when the test data corresponds to multiple indicators of the AB test, the first data information is sorted to obtain a preliminary analysis result, and a visual display is performed on the background page, that is, the aforementioned S450 may specifically include the following steps: S451. Classify the first data information based on multiple indicators to obtain multiple second data information.

[0067] S452. Determine the weight ratios corresponding to the multiple indicators.

[0068] S453: Perform weighted processing on the plurality of second data information based on the weight ratio to obtain third data information.

[0069] S454: When the third data information exceeds a preset target threshold, an alarm is issued.

[0070] S455. Arrange the third data information into a report or chart to obtain a preliminary analysis result for visual display on a background page.

[0071] In the embodiments of the present application, it can be understood that in an AB test, multiple indicators can be set. For example, in the e-commerce industry, the AB test can set different indicators such as sales volume and profit margin. The first data information obtained by the AB test will cover multiple indicators, and the data corresponding to each indicator will be considered separately, and weighted processing will be performed based on a preset weight ratio to obtain the expected third data information.

[0072] Furthermore, the present application presets a target threshold value, and when the obtained third data information exceeds the target threshold value, an alarm prompt is given in time, thereby reminding the developer to correct the test vulnerability in time and reduce losses.

[0073] In some embodiments, please refer to Figure 3 The present application provides an AB testing system 500 based on ZooKeeper, and the AB testing system 500 may specifically include: The acquisition module 510 is used to acquire configuration information of the AB test, where the configuration information includes a control version and at least one test version to be tested.

[0074] The encapsulation module 520 is used to encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework.

[0075] The determination module 530 is used to determine whether the multiple user IDs corresponding to the test request information are in a preset whitelist when the test request information is received.

[0076] The diversion module 540 is used to perform diversion based on a diversion algorithm including an MD5 hash function and a modular operation when multiple user IDs corresponding to the test request information are not in the whitelist, allocate multiple user IDs to a control version and a test version, and determine user diversion information.

[0077] The loading module 550 is used to load the SDK toolkit in the ZooKeeper framework for testing based on the user diversion information.

[0078] The analysis module 560 is used to collect test data, analyze the test data, obtain preliminary analysis results, and visualize them on the background page.

[0079] The termination module 570 is used to determine the termination of the AB test when the preliminary analysis result meets the preset target termination condition.

[0080] The adjustment module 580 is used to adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework when the preliminary analysis result does not meet the preset target termination condition, so as to adjust the user diversion information and continue the test.

[0081] According to an embodiment of the present application, any multiple modules among the acquisition module 510, the encapsulation module 520, the judgment module 530, the diversion module 540, the loading module 550, the analysis module 560, the termination module 570 and the adjustment module 580 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0082] In some embodiments, the AB testing system 500 may further include: The allocation test module 551 is used to allocate the user ID to the corresponding test version based on the third mapping relationship for testing when the user ID is consistent with the target ID in the whitelist.

[0083] In some embodiments, the aforementioned adjustment module 580 may be specifically used to: Based on the preliminary analysis result, the user ID that needs to be adjusted among the multiple user IDs is determined as the target ID; Receiving input user adjustment instruction information, where the user adjustment instruction information includes a first mapping relationship between a target ID and configuration information; Based on the first mapping relationship, the target ID is adjusted to the node corresponding to the corresponding test version or control version in the ZooKeeper framework.

[0084] In some embodiments, the aforementioned adjustment module 580 may also be used to: In the absence of user adjustment instruction information, based on the target ID, the traffic is re-diverted through a diversion algorithm including an MD5 hash function and a modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; Based on the second mapping relationship, the target ID is reallocated to the corresponding node to adjust the user diversion information.

[0085] In some embodiments, the aforementioned flow dividing module 540 can be specifically used for: Concatenate the user ID and the region information to which the user ID belongs into a string; Perform MD5 encryption on the string to generate a unique hash value; A modulo operation is performed on the hash value, and the modulo operation result is mapped to a predefined range of the test version and a range of the control version to determine user diversion information.

[0086] In some embodiments, the aforementioned analysis module 560 may specifically include: The first data transmission unit 561 is used to collect test data of the AB test and send the test data to a preset Kafka module for data transmission and real-time processing.

[0087] The second data transmission unit 562 is used to transmit the test data to the preset Spark module through the Kafka module.

[0088] The data processing unit 563 is used to clean, transform and aggregate the test data through the Spark module to obtain first data information.

[0089] The data storage unit 564 is used to store the first data information in a preset Hive warehouse for storage and query.

[0090] The sorting and displaying unit 565 is used to sort the first data information, obtain a preliminary analysis result, and display it visually on a background page.

[0091] In some embodiments, the above-mentioned arrangement and display unit 565 can be specifically used for: Based on multiple indicators, the first data information is classified to obtain multiple second data information; Determine the weight ratios corresponding to multiple indicators; Based on the weight ratio, weighted processing is performed on the plurality of second data information to obtain third data information; When the third data information exceeds a preset target threshold, an alarm is issued; The third data information is organized into reports or charts to obtain preliminary analysis results for visual display on the background page.

[0092] Figure 3 Each module in the system shown has the function of implementing each step in the aforementioned ZooKeeper-based AB testing method and can achieve its corresponding technical effect. For the sake of concise description, it will not be repeated here.

[0093] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 4 shown.

[0094] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0095] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0096] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 620 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0097] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any one of the ZooKeeper-based AB testing methods in the above embodiments.

[0098] The processor 610 implements any one of the ZooKeeper-based AB testing methods in the above embodiments by reading and executing computer program instructions stored in the memory 620 .

[0099] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 4 As shown, the processor 610, the memory 620, and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0100] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application. Bus 600 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0101] In addition, in combination with the ZooKeeper-based AB testing method in the above embodiment, the present application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the ZooKeeper-based AB testing methods in the above embodiment is implemented.

[0102] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0103] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0104] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0105] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0106] In summary, compared with the prior art, this application has the following beneficial effects: 1. This application builds a distributed coordination system through ZooKeeper, and each node can work together; this application provides two diversion strategies. When users are diverted, they can be diverted based on a diversion algorithm that includes MD5 hash function and modulus operation. Specific users can also bypass the diversion algorithm through a preset whitelist and directly allocate users, which increases the flexibility of AB testing.

[0107] 2. This application can directly exchange user ID information by adjusting the storage information of some nodes in the ZooKeeper framework, so as to directly adjust the control version or test version assigned to some user IDs, without having to replace the user IDs assigned to each test version and control version as a whole, and can quickly re-divert users without consuming much memory. If there are problems with the preliminary analysis results, by receiving the user adjustment instruction information input by the developer, the user ID in the test version can be quickly adjusted to the control version, and the AB test error correction can be completed.

[0108] 3. Encapsulate the configuration information into an SDK and store it in the ZooKeeper framework. After completing user diversion, this application can directly load the SDK toolkit in the ZooKeeper framework for testing. The SDK toolkit can provide a set of standard interfaces and implementation methods for application development, which is easy to update and maintain, realizes code reuse, has high integration, and can also provide additional functions such as data analysis, result reporting, and exception handling.

[0109] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ZooKeeper-based AB testing method, characterized in that: include: Acquire configuration information of the AB test, where the configuration information includes a control version and at least one test version to be tested; Encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework; When receiving the test request information, determining whether the multiple user IDs corresponding to the test request information are in a preset whitelist; In the case that the multiple user IDs corresponding to the test request information are not in the whitelist, performing diversion based on a diversion algorithm including an MD5 hash function and a modulo operation, allocating the multiple user IDs to the control version and the test version, and determining user diversion information; Based on the user diversion information, loading the SDK toolkit in the ZooKeeper framework for testing; Collect test data, analyze the test data, obtain preliminary analysis results, and visualize them on the background page; When the preliminary analysis result meets the preset target termination condition, determining that the AB test is terminated; When the preliminary analysis result does not meet the preset target termination condition, the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework is adjusted to adjust the user diversion information and continue the test.

2. The ZooKeeper-based AB testing method according to claim 1, characterized in that: When the preliminary analysis result does not meet the preset target termination condition, the adjusting the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework to adjust the user diversion information and continue the test includes: Based on the preliminary analysis result, determining the user ID that needs to be adjusted among the multiple user IDs as the target ID; Receiving input user adjustment instruction information, wherein the user adjustment instruction information includes a first mapping relationship between the target ID and the configuration information; Based on the first mapping relationship, the target ID is adjusted to a node corresponding to the test version or the control version in the ZooKeeper framework.

3. The ZooKeeper-based AB testing method according to claim 2, characterized in that: After determining that the user ID that needs to be adjusted among the multiple user IDs is the target ID based on the preliminary analysis result, the ZooKeeper-based AB testing method further includes: In the absence of the user adjustment instruction information, based on the target ID, re-dividing the traffic by a diversion algorithm including an MD5 hash function and a modulo operation to obtain a second mapping relationship between the target ID and some nodes in the ZooKeeper framework; Based on the second mapping relationship, the target ID is reallocated to the corresponding node to adjust the user diversion information.

4. The ZooKeeper-based AB testing method according to any one of claims 1 to 3, characterized in that: In the case that the multiple user IDs corresponding to the test request information are not in the whitelist, the diversion algorithm based on the MD5 hash function and the modulus operation is used to perform diversion, the multiple user IDs are allocated to the control version and the test version, and the user diversion information is determined, including: Concatenate the user ID and the region information to which the user ID belongs into a character string; Perform MD5 encryption on the string to generate a unique hash value; A modulo operation is performed on the hash value, and the modulo operation result is mapped to a predefined range of the test version and a predefined range of the control version to determine user diversion information.

5. The ZooKeeper-based AB testing method according to any one of claims 1 to 3, characterized in that: The whitelist includes a plurality of target IDs and a configuration information list, wherein the configuration information list is used to represent a third mapping relationship between the target ID and the test version; Before collecting the test data, analyzing the test data, obtaining the preliminary analysis results, and visually displaying them on the background page, the ZooKeeper-based AB testing method further includes: When the user ID is consistent with the target ID in the whitelist, the user ID is allocated to the corresponding test version based on the third mapping relationship for testing.

6. The ZooKeeper-based AB testing method according to any one of claims 1 to 3, characterized in that: The test data is collected, analyzed, and preliminary analysis results are obtained, and displayed visually on a backend page, including: Collect test data of the AB test, and send the test data to a preset Kafka module for data transmission and real-time processing; The test data is transmitted to a preset Spark module through the Kafka module; Cleaning, converting and aggregating the test data through the Spark module to obtain first data information; Storing the first data information in a preset Hive warehouse for storage and query; The first data information is sorted to obtain preliminary analysis results, and the results are visualized on the background page.

7. The ZooKeeper-based AB testing method according to claim 6, characterized in that: In the case where the test data corresponds to multiple indicators of the AB test, the first data information is sorted to obtain a preliminary analysis result and is visually displayed on a background page, including: Based on the multiple indicators, classify the first data information to obtain multiple second data information; Determine the weight ratios corresponding to the multiple indicators; Based on the weight ratio, weighted processing is performed on the plurality of the second data information to obtain third data information; When the third data information exceeds a preset target threshold, an alarm is issued; The third data information is organized into a report or chart to obtain a preliminary analysis result for visual display on a background page.

8. A ZooKeeper-based AB testing system, characterized in that: include: An acquisition module, used to acquire configuration information of the AB test, wherein the configuration information includes a control version and at least one test version to be tested; An encapsulation module, used to encapsulate the configuration information into an SDK toolkit and store it in the ZooKeeper framework; A judgment module, used for judging whether the multiple user IDs corresponding to the test request information are in a preset whitelist when receiving the test request information; A diversion module is used to perform diversion based on a diversion algorithm including an MD5 hash function and a modular operation when the multiple user IDs corresponding to the test request information are not in the whitelist, allocate the multiple user IDs to the control version and the test version, and determine user diversion information; A loading module, used for loading the SDK toolkit in the ZooKeeper framework for testing based on the user diversion information; The analysis module is used to collect test data, analyze the test data, obtain preliminary analysis results, and visualize them on the background page; A termination module, configured to determine the termination of the AB test when the preliminary analysis result meets a preset target termination condition; The adjustment module is used to adjust the storage information of some nodes corresponding to the control version and the test version in the ZooKeeper framework when the preliminary analysis result does not meet the preset target termination condition, so as to adjust the user diversion information and continue the test.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the ZooKeeper-based AB testing method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the ZooKeeper-based AB testing method according to any one of claims 1 to 7 is implemented.

Images