Adaptive grayscale function release method and device based on AB testing

Through the adaptive grayscale function release method based on AB testing, the grayscale function logs are automatically analyzed and filtered, and the automatic release and rollback of grayscale function is achieved, solving the problem of inefficiency caused by manual intervention, and improving product iteration and Flow conversion efficiency.

CN112181782BActive Publication Date: 2024-12-06INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011109194.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-16
Publication Date
2024-12-06
Estimated Expiration
2040-10-16

AI Technical Summary

Technical Problem

In the development model of continuous integration of projects, as the complexity of business scenarios increases, manual intervention in grayscale function release and rollback leads to a significant increase in workload, product iteration efficiency becomes worse, and traffic conversion efficiency is low, especially in multiple scenarios. Data analysis workloads in fine-grained AB tests are huge.

Method used

Adaptive grayscale function release method based on AB test is adopted, and the SDK buried point logs of each business node and front-end node are obtained, and the logs are analyzed based on the preset AB test strategy, and the optimal grayscale is automatically filtered using transaction parameter calculation and U-test method. Strategy to realize automatic publishing and rollback of grayscale functions.

Benefits of technology

It reduces the workload of manual intervention, improves product iteration efficiency and traffic conversion efficiency, realizes automated grayscale function release and rollback, and supports real-time release by scene function level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112181782B_ABST
    Figure CN112181782B_ABST
Patent Text Reader

Abstract

The present invention provides an adaptive grayscale function release method and device based on AB testing, the method comprising: obtaining grayscale function tracking logs sent by SDK tracking of each business node and front-end tracking logs sent by SDK tracking of each front-end node; analyzing the grayscale function tracking logs and the front-end tracking logs based on a preset AB test strategy; and releasing the grayscale function according to the log analysis results. By adopting the above technical solution, it is possible to adaptively select the grayscale function with the optimal strategy for the grayscale function and formally release it. It can support collecting transaction data of link grayscale functions through business tracking according to scene function levels, calculate and match the optimal weight results according to ABTEST data of the same function level in combination with the preset AB test strategy, and push the optimal strategy to each node in real time through the configuration center to achieve the purpose of automatically formally releasing the grayscale function.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of distributed technology, and in particular to an adaptive grayscale function release method and device based on AB testing. Background Art

[0002] In the context of the project's continuous integration development model, especially with the increasing number of distributed services in the industry, it is becoming more and more common to have multiple grayscale function scenarios for the same application and the same function in different nodes, especially fine-grained functions also have grayscale functions in multiple scenarios. As business scenarios become more and more complex, after a period of project release iteration, it is necessary to manually intervene in the release of the optimal grayscale function after a large number of ABTEST grayscale functions are launched, and intervene in the release and rollback of grayscale functions with the help of data analysis, which greatly increases the workload of manual participation in judgment operations, deteriorates product iteration efficiency, and has low traffic conversion efficiency. Summary of the invention

[0003] In view of the problems in the prior art, the present invention provides an adaptive grayscale function release method and device based on AB testing, an electronic device and a computer-readable storage medium, which can at least partially solve the problems in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solution:

[0005] In a first aspect, a method for releasing an adaptive grayscale function based on AB testing is provided, comprising:

[0006] Obtain the grayscale function tracking logs sent by the SDK tracking of each business node and the front-end tracking logs sent by the SDK tracking of each front-end node;

[0007] Analyze the grayscale function tracking log and the front-end tracking log based on a preset AB test strategy;

[0008] Grayscale function release is performed based on log analysis results.

[0009] Furthermore, the preset AB testing strategy includes: transaction parameter calculation method and parameter expected value;

[0010] The analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy includes:

[0011] Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain the transaction parameter value;

[0012] The transaction parameter value and the parameter expected value are compared to obtain a log analysis result.

[0013] Furthermore, the preset AB testing strategy includes: transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements;

[0014] The analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy includes:

[0015] Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain the transaction parameter value;

[0016] Determine whether the grayscale release time node requirements are met according to the grayscale function tracking log and the front-end tracking log;

[0017] If so, based on the U test method, a significance level U value is obtained according to the transaction parameter value and the parameter expected value;

[0018] The significance level U value and the expected U value are compared to obtain a log analysis result.

[0019] Furthermore, the transaction parameters include: mean value, variance and / or sample size.

[0020] Furthermore, the method for releasing an adaptive grayscale function based on AB testing also includes:

[0021] Screen and select the highest value of the significance level U value of the business node corresponding to several grayscale functions for parallel selection;

[0022] The grayscale function corresponding to the highest value is published as the best grayscale strategy.

[0023] Further, the preset AB test strategy includes: starting the business node identification of each grayscale function;

[0024] The method further comprises:

[0025] The corresponding service node is controlled to start the corresponding grayscale function according to the service node identifier.

[0026] Furthermore, before analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy, the method further includes:

[0027] The grayscale function embedding log and the front-end embedding log are preprocessed.

[0028] Furthermore, the preprocessing of the grayscale function embedding log and the front-end embedding log includes:

[0029] Use kafka to collect the grayscale function tracking log and the front-end tracking log;

[0030] The performance data of the grayscale function tracking log and the front-end tracking log are extracted to obtain key transaction measurement data to serve as basic data for log analysis.

[0031] Furthermore, the grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

[0032] Furthermore, the grayscale function release according to the log analysis result includes:

[0033] Determine whether the log analysis result meets the requirements of the preset AB testing strategy;

[0034] If yes, release the grayscale feature;

[0035] If not, control the business node that starts the grayscale function to roll back the function.

[0036] In a second aspect, a device for releasing an adaptive grayscale function based on AB testing is provided, comprising:

[0037] The log acquisition module obtains the grayscale function tracking logs sent by the SDK tracking of each business node and the front-end tracking logs sent by the SDK tracking of each front-end node;

[0038] A log analysis module, which analyzes the grayscale function tracking log and the front-end tracking log based on a preset AB test strategy;

[0039] The grayscale function release control module releases grayscale functions based on the log analysis results.

[0040] Furthermore, the preset AB testing strategy includes: transaction parameter calculation method and parameter expected value;

[0041] The log analysis module includes:

[0042] A parameter calculation unit, which collects and counts the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method to obtain a transaction parameter value;

[0043] The parameter comparison and analysis unit compares the transaction parameter value and the parameter expected value to obtain a log analysis result.

[0044] Furthermore, the preset AB testing strategy includes: transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements;

[0045] The log analysis module includes:

[0046] A parameter calculation unit, which collects and counts the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method to obtain a transaction parameter value;

[0047] A time judgment unit, which judges whether the grayscale release time node requirement is met according to the grayscale function tracking log and the front-end tracking log;

[0048] U test unit, if the grayscale release time node requirements are met, based on the U test method, obtains the significance level U value according to the transaction parameter value and the parameter expected value;

[0049] The U value comparison and analysis unit compares the significance level U value and the expected U value to obtain a log analysis result.

[0050] Furthermore, the transaction parameters include: mean value, variance and / or sample size.

[0051] Furthermore, the adaptive grayscale function release device based on AB testing also includes:

[0052] The highest value screening module screens the highest value of the significance level U value of the business nodes corresponding to several grayscale functions that are listed in parallel for selection;

[0053] The optimal grayscale publishing module publishes the grayscale function corresponding to the highest value as the optimal grayscale strategy.

[0054] Further, the preset AB test strategy includes: starting the business node identification of each grayscale function;

[0055] The adaptive grayscale function release device based on AB testing also includes:

[0056] The start control module controls the corresponding service node to start the corresponding grayscale function according to the service node identifier.

[0057] Furthermore, the adaptive grayscale function release device based on AB testing also includes:

[0058] The log data preprocessing module preprocesses the grayscale function embedding log and the front-end embedding log.

[0059] Furthermore, the log data preprocessing module includes:

[0060] A log collection unit, using kafka to collect the grayscale function embedding log and the front-end embedding log;

[0061] The data extraction unit extracts performance data from the grayscale function tracking log and the front-end tracking log to obtain key transaction metric data as basic data for log analysis.

[0062] Furthermore, the grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

[0063] Furthermore, the grayscale function release control module includes:

[0064] A result judgment unit, judging whether the log analysis result meets the requirements of a preset AB test strategy;

[0065] The grayscale function release unit releases the grayscale function if it meets the requirements of the preset AB testing strategy;

[0066] The function rollback unit controls the business node that starts the grayscale function to roll back the function if the requirements of the preset AB test strategy are not met.

[0067] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned method for releasing an adaptive grayscale function based on AB testing are implemented.

[0068] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for releasing an adaptive grayscale function based on AB testing are implemented.

[0069] The present invention provides an adaptive grayscale function release method and device based on AB testing, the method comprising: obtaining the grayscale function tracking log sent by the SDK tracking of each business node and the front-end tracking log sent by the SDK tracking of each front-end node; analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy; and releasing the grayscale function according to the log analysis result. By adopting the above technical solution, it is possible to adaptively select the grayscale function with the optimal strategy for the grayscale function and formally release it. It can support collecting the transaction data of the link grayscale function through business tracking according to the scene function level, calculate the matching optimal weight result according to the ABTEST data of the same function level, combined with the preset AB test strategy, and push the optimal strategy to each node in real time through the configuration center to achieve the purpose of automatically formally releasing the grayscale function.

[0070] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0072] Figure 1 A schematic diagram of a system architecture for implementing an adaptive grayscale function release method based on AB testing in an embodiment of the present invention;

[0073] Figure 2 The process diagram of the method for releasing an adaptive grayscale function based on AB testing in an embodiment of the present invention is as follows Figure 1 ;

[0074] Figure 3 A specific step of step S200 in an embodiment of the present invention is shown;

[0075] Figure 4 Another specific step of step S200 in the embodiment of the present invention is shown;

[0076] Figure 5 The process diagram of the method for releasing an adaptive grayscale function based on AB testing in an embodiment of the present invention is as follows Figure 2 ;

[0077] Figure 6 The process diagram of the method for releasing an adaptive grayscale function based on AB testing in an embodiment of the present invention is as follows Figure 3 ;

[0078] Figure 7 The specific steps of step S500 in the embodiment of the present invention are shown;

[0079] Figure 8 The specific steps of step S300 in the embodiment of the present invention are shown;

[0080] Fig. 9 The flow of the log data processing process in the embodiment of the present invention is given;

[0081] Fig.10 It is a structural frame of the adaptive grayscale function release device based on AB test in the embodiment of the present invention. Figure 1 ;

[0082] Fig.11 A structural block diagram of a log analysis module in an embodiment of the present invention is shown;

[0083] Fig.12 Another structural block diagram of the log analysis module in the embodiment of the present invention is shown;

[0084] Fig.13 It is a structural frame of the adaptive grayscale function release device based on AB test in the embodiment of the present invention. Figure 2 ;

[0085] Fig.14 It is a structural frame of the adaptive grayscale function release device based on AB test in the embodiment of the present invention. Figure 3 ;

[0086] Fig.15 The structure block diagram of the log data preprocessing module in the embodiment of the present invention is shown;

[0087] Fig.16 The structure block diagram of the gray function release control module in the embodiment of the present invention is shown;

[0088] Fig.17 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.

[0092] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0093] As business scenarios become more and more complex, after a period of project release iteration, it is necessary to manually intervene in the release of the optimal grayscale function after a large number of ABTEST grayscale functions are launched (manually perform report analysis based on data feedback and adjust the release strategy based on the analysis results). With the help of data analysis, the release and rollback of grayscale functions are intervened, resulting in a significant increase in the workload of manual participation in judgment operations and a long R&D cycle. Product iteration efficiency deteriorates, traffic conversion efficiency is low, and the application side needs to pay a lot of attention to data analysis and data effect work when AB grayscale functions are launched, especially when fine-grained AB testing and the amount of data reaches a certain order of magnitude, the analysis workload is greatly increased.

[0094] In order to at least partially solve the above technical problems, an embodiment of the present invention provides an adaptive grayscale function release method based on AB testing, which can adaptively select the grayscale function with the optimal strategy for the grayscale function and formally release it. It can support collecting transaction data of link grayscale functions through business tracking points according to scene function levels, calculate and match the optimal weight results based on ABTEST data of the same function level, combined with the preset AB test strategy, and push the optimal strategy to each node in real time through the configuration center to achieve the purpose of automatically and formally releasing the grayscale function.

[0095] It is worth mentioning that in the context of the project continuous integration development model, APP and other applications need to continuously update their functions. For example, if Plan B is currently enabled, Plan A is developed to replace Plan B, but the market effect of Plan A is unknown. Then, after Plan A is launched, select some users of the APP to enable Plan A for trial use. Then, based on the data of the trial users, analyze whether the functions of Plan A meet the preset requirements. If so, then fully enable Plan A. If not, then roll back the trial users' plan to Plan B and fully use Plan B.

[0096] Figure 1 Schematic diagram of the system architecture for implementing the method for releasing an adaptive grayscale function based on AB testing in an embodiment of the present invention; Figure 1As shown, the adaptive grayscale function publishing server based on AB testing may include a policy platform and a data platform. The configuration administrator configures the AB test policy on the policy platform. The AB test policy may include: the business node identifier or ID that needs to start the grayscale function, the calculation formula and threshold value when processing log data, etc., wherein multiple grayscale functions may be included, and different business nodes may be enabled for each grayscale function in the AB test policy. The policy configuration management module of the policy platform controls each corresponding business system (i.e., business node) to enable the corresponding grayscale function according to the business node identifier or ID that needs to start the grayscale function in the AB test policy; it is worth noting that each business system includes Plan A (corresponding to the grayscale function) and Plan B (corresponding to the original function), and only needs to enable control according to needs.

[0097] In addition, there are two ways to control each corresponding business system (i.e., business node, also called test node) to enable the corresponding grayscale function. One is to send a control command to the corresponding business system to enable the grayscale function. For example, if business node A1 needs to start grayscale functions B1 and B2, and node A3 needs to start grayscale function B3, then an instruction to start grayscale functions B1 and B2 is sent to node A1, and an instruction to start grayscale function B3 is sent to node A3. The other is to send an enablement control command to all business systems. The enablement control command includes an enablement node and its corresponding grayscale function. Each business system controls whether to enable the grayscale function and which grayscale function to enable according to the enablement control command. In the above example, the enablement control command includes information that business node A1 needs to start grayscale functions B1 and B2, and node A3 needs to start grayscale function B3. After the enablement control command is sent to nodes A1 to A3, A1 starts grayscale functions B1 and B2 according to the enablement control command; A2 does not enable the grayscale function according to the enablement control command, and A3 starts grayscale function B3 according to the enablement control command.

[0098] Furthermore, the business system obtains instructions from the strategy platform through the business system SDK tracking point.

[0099] The user on the application side (i.e., person 2) triggers the page effect by clicking on the page request on the front end; the front end request is sent to the business system; the business system executes the grayscale function and returns the data to the front end. The business system SDK tracking point sends the grayscale function tracking log to the data platform's tracking system, and the front-end SDK tracking point sends the front-end tracking log to the data platform's tracking system; the data platform's tracking system collects and pre-processes the data. For example, after regularly collating the data, the data is sent to the policy calculation module of the policy platform. The policy platform summarizes and calculates the log data of the grayscale function in this scenario based on the preset AB test strategy, and sends the analysis results to the policy configuration management module. The policy configuration management module sends the latest optimization strategy to each node, enables the latest strategy or rolls back, forming a closed loop of policy optimization and grayscale function release.

[0100] Among them, the policy platform is the core module, which mainly includes two parts: policy configuration management module and policy calculation module.

[0101] The policy configuration management module is mainly responsible for configuration management and configuration push through Apollo. Administrator users can complete the initial configuration of the grayscale scene on the policy platform, including the initial configuration of applications, scenes, policy access control, scenes and policy relationships, and the configuration after policy optimization is sent to each business node to implement function release. After the policy platform experiment is released, only the business buried point data needs to be sent up and fed back to the data platform, without manual intervention in the policy platform configuration modification or push release strategy.

[0102] The data platform module mainly realizes the collection, cleaning and push of SDK data. It mainly collects data sent by the front-end and business nodes through Kafka, calculates and produces performance data through Flink, stores it in Druid as key transaction measurement data and notifies the strategy calculation module.

[0103] The strategy calculation module is mainly responsible for classifying, analyzing and calculating transaction data (or log data) to obtain the best experimental data.

[0104] It is understandable that the front end may include a smart phone, a tablet electronic device, a portable computer, a desktop computer, etc.

[0105] The server and the front end or business system may communicate using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocol may include, for example, TCP / IP protocol, UDP / IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol) and REST protocol (Representational State Transfer) used on top of the above protocols.

[0106] In order to make those skilled in the art better understand the present invention, Figures 2 to 17 The specific implementation process of the present invention is described in detail:

[0107] Figure 2 The process diagram of the method for releasing an adaptive grayscale function based on AB testing in an embodiment of the present invention is as follows Figure 1 ;like Figure 2 As shown, the adaptive grayscale function release method based on AB testing may include the following contents:

[0108] Step S100: Obtain the grayscale function tracking log sent by the SDK tracking of each business node and the front-end tracking log sent by the SDK tracking of each front-end node;

[0109] Among them, the grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

[0110] For example, after a user clicks on a page, a traceId (tracing identifier) ​​is generated, which is used to identify the ID of a transaction instance in a certain functional scenario.

[0111] The front-end SDK tracking and the SDK tracking of the business node mainly realize the sending of experimental data and the receiving of experimental configuration. The SDK can realize the annotation of experimental user diversion according to the specific experimental scenario, and the strategy platform will dynamically distribute the experimental configuration. In addition, based on the front-end tracking identifier, the SDK generates tracking identifiers including traceId and abRsult (request result data), which are used for the transparent transmission function of the data platform and business system as statistical indicators of the strategy analysis platform.

[0112] Step S200: Analyze the grayscale function tracking log and the front-end tracking log based on a preset AB test strategy;

[0113] Among them, the effect of the grayscale function is obtained by analyzing the grayscale function tracking log and the front-end tracking log.

[0114] The preset AB testing strategy is set in advance by the test administrator based on the testing requirements, specifically including: 1. transaction parameter calculation method and parameter expected value, or, 2. transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements, as well as the business node identification for starting each grayscale function.

[0115] Step S300: Release the grayscale function according to the log analysis results.

[0116] By adopting the above technical solution, it is possible to adaptively select the grayscale function with the optimal strategy for the grayscale function and officially release it. It can support collecting transaction data of the link grayscale function by scene function level through business tracking points, calculate the matching optimal weight results based on ABTEST data of the same function level, and push the optimal strategy to each node in real time through the configuration center to achieve the purpose of automatically and officially releasing the grayscale function.

[0117] It is worth noting that the adaptive grayscale function release method based on AB testing provided by the present invention is suitable for multi-node, multi-scenario AB test function release and rollback, can automatically screen the optimal strategy and put it into publication, avoid mistakenly publishing a strategy with a conversion rate lower than the set expected value, and automatically complete the release or rollback strategy, thereby accelerating the rhythm of business iterative development and enhancing the automation capabilities of operation and maintenance and production.

[0118] In an alternative embodiment, see Figure 3 , the step S200 may include the following contents:

[0119] Step S210: Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain a transaction parameter value;

[0120] The trading parameters include: mean value, variance and / or sample size, etc.

[0121] Step S220: Compare the transaction parameter value and the parameter expected value to obtain a log analysis result.

[0122] Specifically, when the transaction parameter value meets the parameter expected value requirements, it is considered that the grayscale function effect meets the requirements and can be put online, otherwise, it cannot be put online.

[0123] By adopting the above technical solution, the effect of the grayscale function can be automatically evaluated objectively and reasonably, and automatic grayscale strategy screening can be achieved.

[0124] In another alternative embodiment, see Figure 4 , the step S200 may include the following contents:

[0125] Step S210': Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain the transaction parameter value;

[0126] The trading parameters include: mean value, variance and / or sample size, etc.

[0127] Step S220': judging whether the grayscale release time node requirement is met according to the grayscale function tracking log and the front-end tracking log;

[0128] Specifically, the log will include the time when the log occurred, and determine whether the log occurrence time meets the grayscale release time node requirements. If it does not meet the time requirements, the original strategy is maintained and data collection continues.

[0129] If yes, execute step S230'; if no, return to step S210'.

[0130] Step S230': Based on the U test method, obtain the significance level U value according to the transaction parameter value and the parameter expected value;

[0131] Since the principle of AB testing is consistent with that of sampling experiments, the optimal strategy is usually reflected in user performance. When designing expected values, it is necessary not only to use thresholds and log data, but also to use the U-check method to calculate target statistics and use statistical hypothesis testing methods to scientifically judge experimental comparison data in multiple experimental scenarios.

[0132] Step S240 ′: Compare the significance level U value and the expected U value to obtain a log analysis result.

[0133] Among them, the screening and release of the optimal scenario are completed by judging the statistic (i.e., the significance level U value) and the expected U value.

[0134] By adopting the above technical solution, it is possible to use statistical hypothesis testing methods to scientifically judge experimental comparison data in multiple experimental scenarios, improve the accuracy of screening strategies, and obtain the optimal strategy more accurately.

[0135] In an optional embodiment, the method for releasing an adaptive grayscale function based on AB testing may further include:

[0136] Screen and select the highest value of the significance level U value of the business node corresponding to several grayscale functions for parallel selection;

[0137] The grayscale function corresponding to the highest value is published as the best grayscale strategy.

[0138] It is worth mentioning that each time an AB test is conducted, multiple grayscale functions may be tested simultaneously for the same scene in order to screen the optimal grayscale function. At this time, the grayscale function corresponding to the highest value of the significance level U value needs to be selected as the final grayscale function to be launched.

[0139] For example, there is a black button on the existing page. According to user feedback or design requirements, the visibility of the black button is irrational and cannot attract the user's attention. The black button can be changed to red, yellow or green. The buttons of these three colors correspond to the three grayscale functions, but the market effects of these three colors are unknown. At this time, a part of the users are selected for trial testing. Some users of the APP are selected (selection control is performed according to business nodes) to use the red button, some users to use the yellow button, some users to use the green button, and some users do not enable the grayscale function, but continue to use the original black button. Then, by analyzing the user data of the three grayscale functions, the optimal grayscale function is selected for release. If the effects of the three grayscale functions are not ideal, a rollback setting is performed to roll back the buttons of the trial users to the black button.

[0140] In an alternative embodiment, see Figure 5 The method for releasing an adaptive grayscale function based on AB testing may further include:

[0141] Step S400: controlling the corresponding service node to start the corresponding grayscale function according to the service node identifier.

[0142] By adopting the above technical solution, the business nodes are automatically controlled to perform AB testing, further improving the automation level.

[0143] In an alternative embodiment, see Figure 6 The method for releasing an adaptive grayscale function based on AB testing may further include:

[0144] Step S500: pre-processing the grayscale function tracking log and the front-end tracking log.

[0145] Specifically, see Figure 7 , the step S500 may include:

[0146] Step S510: using Kafka to collect the grayscale function tracking log and the front-end tracking log;

[0147] Step S520: extract performance data from the grayscale function tracking log and the front-end tracking log to obtain key transaction metric data as basic data for log analysis.

[0148] In an alternative embodiment, see Figure 8 , the step S300 may include the following contents:

[0149] Step S310: whether the log analysis result meets the requirements of the preset AB testing strategy;

[0150] If yes, execute step S320; if no, execute step S330.

[0151] Step S320: Release the grayscale function;

[0152] Specifically, controlling a node that has not executed the grayscale function to execute the grayscale function may be achieved by sending a control instruction to a service node.

[0153] Step S330: Control the service node that starts the grayscale function to perform function rollback.

[0154] Specifically, function recovery is achieved by sending a rollback instruction to the business node that starts the grayscale function.

[0155] In order to enable those skilled in the art to better understand the present invention, Fig. 9 The flow of the log data processing process in the embodiment of the present invention is given as follows: Fig. 9 As shown, the processing includes:

[0156] Step 1: Obtain transaction data from the data platform;

[0157] Specifically, the strategy platform obtains transaction result data from Druid.

[0158] Step 2: Obtaining metric statistics;

[0159] Specifically, the transaction result data is summarized by calculating statistics such as mean, variance, sample size, etc.

[0160] Step 3: Whether the measurement time condition requirements are met;

[0161] Specifically, determine whether the grayscale function release conditions have met the grayscale release time node requirements.

[0162] If yes, go to step 5; if no, go to step 4.

[0163] Step 4: The strategy remains unchanged;

[0164] Step 5: Calculate all historical data statistics using the U check method;

[0165] Specifically, the U-check method is used to perform the measurement data and the scenario expected value data to produce a significance level U value.

[0166] Step 6: Whether expectations are met significantly;

[0167] Specifically, determine whether the significance level U value meets the expected significance.

[0168] If yes, go to step 8; if no, go to step 7.

[0169] Step 7: Roll back the node policy.

[0170] Step 8: Filter strategies for MAX (statistics) scenarios;

[0171] Step 9: Push the best strategy.

[0172] Based on the same inventive concept, the embodiments of the present application also provide an adaptive grayscale function release device based on AB testing, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of solving the problem by the adaptive grayscale function release device based on AB testing is similar to the above method, the implementation of the adaptive grayscale function release device based on AB testing can refer to the implementation of the above method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0173] Fig.10 It is a structural frame of the adaptive grayscale function release device based on AB test in the embodiment of the present invention. Figure 1 .like Fig.10 As shown, the adaptive grayscale function release device based on AB testing includes: a log acquisition module 10, a log analysis module 20 and a grayscale function release control module 30.

[0174] The log acquisition module 10 acquires the grayscale function tracking log sent by the SDK tracking of each business node and the front-end tracking log sent by the SDK tracking of each front-end node;

[0175] The log analysis module 20 analyzes the grayscale function tracking log and the front-end tracking log based on a preset AB test strategy;

[0176] The grayscale function release control module 30 releases grayscale functions according to the log analysis results.

[0177] By adopting the above technical solution, it is possible to adaptively select the grayscale function with the optimal strategy for the grayscale function and officially release it. It can support collecting transaction data of the link grayscale function by scene function level through business tracking points, calculate the matching optimal weight results based on ABTEST data of the same function level, and push the optimal strategy to each node in real time through the configuration center to achieve the purpose of automatically and officially releasing the grayscale function.

[0178] In an optional embodiment, the preset AB testing strategy includes: a transaction parameter calculation method and a parameter expected value; see Fig.11 The log analysis module may include: a parameter calculation unit 21 and a parameter comparison and analysis unit 22.

[0179] The parameter calculation unit 21 summarizes and counts the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method to obtain a transaction parameter value;

[0180] The parameter comparison and analysis unit 22 compares the transaction parameter value and the parameter expected value to obtain a log analysis result.

[0181] In another optional embodiment, the preset AB testing strategy includes: transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements, see Fig.12 The log analysis module includes: a parameter calculation unit 21', a time judgment unit 22', a U inspection unit 23' and a U value comparison and analysis unit 24'.

[0182] The parameter calculation unit 21' obtains a transaction parameter value by summarizing and counting the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method;

[0183] The time judgment unit 22' judges whether the grayscale release time node requirement is met according to the grayscale function tracking log and the front-end tracking log;

[0184] If the grayscale release time node requirement is met, the U inspection unit 23' obtains a significance level U value based on the transaction parameter value and the parameter expected value based on the U inspection method;

[0185] The U value comparison and analysis unit 24 ′ compares the significance level U value and the expected U value to obtain a log analysis result.

[0186] It is worth mentioning that trading parameters include: mean, variance and / or sample size.

[0187] In an optional embodiment, the adaptive grayscale function publishing device based on AB testing may also include: a highest value screening module and an optimal grayscale publishing module.

[0188] The highest value screening module screens the highest value of the significance level U value of the business nodes corresponding to several grayscale functions that are listed in parallel for selection;

[0189] The optimal grayscale publishing module publishes the grayscale function corresponding to the highest value as the optimal grayscale strategy.

[0190] In an optional embodiment, the preset AB test strategy includes: starting the service node identifier of each grayscale function; Fig.13 The adaptive grayscale function release device based on AB testing also includes: a start control module 40.

[0191] The startup control module 40 controls the corresponding service node to start the corresponding grayscale function according to the service node identifier.

[0192] In an alternative embodiment, see Fig.14 The adaptive grayscale function publishing device based on AB testing also includes: a log data preprocessing module 50.

[0193] The log data preprocessing module 50 preprocesses the grayscale function embedding log and the front-end embedding log.

[0194] In an alternative embodiment, see Fig.15 The log data preprocessing module includes: a log collection unit 51 and a data extraction unit 52.

[0195] The log collection unit 51 uses kafka to collect the grayscale function tracking log and the front-end tracking log;

[0196] The data extraction unit 52 extracts performance data from the grayscale function tracking log and the front-end tracking log to obtain key transaction metric data as basic data for log analysis.

[0197] It is worth noting that the above-mentioned grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

[0198] In an alternative embodiment, see Fig.16 The grayscale function release control module includes: a result judgment unit 31, a grayscale function release unit 32 and a function rollback unit 33.

[0199] The result judgment unit 31 judges whether the log analysis result meets the requirements of the preset AB test strategy;

[0200] The grayscale function release unit 32 releases the grayscale function if the requirements of the preset AB test strategy are met;

[0201] If the requirement of the preset AB test strategy is not met, the function rollback unit 33 controls the service node that starts the grayscale function to perform function rollback.

[0202] The devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is an electronic device, and the electronic device may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0203] In a typical example, the electronic device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the above-mentioned method for releasing an adaptive grayscale function based on AB testing are implemented.

[0204] Reference below Fig.17 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present application.

[0205] like Fig.17 As shown, electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate operations and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of system 600 are also stored. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0206] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed as needed as the storage section 608.

[0207] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method for releasing an adaptive grayscale function based on AB testing when executed by a processor.

[0208] In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 609 , and / or installed from the removable medium 611 .

[0209] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0210] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0211] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0212] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0214] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0215] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0216] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0217] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0218] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An adaptive grayscale function release method based on AB testing, characterized in that: include: Obtain the grayscale function tracking logs sent by the SDK tracking of each business node and the front-end tracking logs sent by the SDK tracking of each front-end node; Analyze the grayscale function tracking log and the front-end tracking log based on the preset AB testing strategy; wherein the preset AB testing strategy includes: transaction parameter calculation method and parameter expected value or transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements; Release the grayscale function of the optimal strategy based on the log analysis results; The analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy includes: Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain the transaction parameter value; Comparing the transaction parameter value and the parameter expected value to obtain a log analysis result; The analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy also includes: Based on the transaction parameter calculation method, the gray function embedding log and the front-end embedding log are aggregated and counted to obtain the transaction parameter value; Determine whether the grayscale release time node requirements are met according to the grayscale function tracking log and the front-end tracking log; If so, based on the U test method, a significance level U value is obtained according to the transaction parameter value and the parameter expected value; The significance level U value and the expected U value are compared to obtain a log analysis result.

2. The method for releasing an adaptive grayscale function based on AB testing according to claim 1, characterized in that: The trading parameters include: mean value, variance and / or sample size.

3. The method for releasing an adaptive grayscale function based on AB testing according to claim 1, characterized in that: Also includes: Screen and select the highest value of the significance level U value of the business node corresponding to several grayscale functions for parallel selection; The grayscale function corresponding to the highest value is published as the best grayscale strategy.

4. The method for releasing an adaptive grayscale function based on AB testing according to claim 1, characterized in that: The preset AB test strategy includes: starting the business node identification of each grayscale function; The method further comprises: The corresponding service node is controlled to start the corresponding grayscale function according to the service node identifier.

5. The method for releasing adaptive grayscale functions based on AB testing according to claim 1, characterized in that: Before analyzing the grayscale function tracking log and the front-end tracking log based on the preset AB test strategy, the method further includes: Preprocessing the grayscale function tracking log and the front-end tracking log includes: Use kafka to collect the grayscale function tracking log and the front-end tracking log; The performance data of the grayscale function tracking log and the front-end tracking log are extracted to obtain key transaction measurement data to serve as basic data for log analysis.

6. The method for releasing an adaptive grayscale function based on AB testing according to any one of claims 1, 3 to 5, characterized in that: The grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

7. The method for releasing adaptive grayscale functions based on AB testing according to claim 1, characterized in that: The grayscale function release according to the log analysis results includes: Determine whether the log analysis result meets the requirements of the preset AB testing strategy; If yes, release the grayscale feature; If not, control the business node that starts the grayscale function to roll back the function.

8. An adaptive grayscale function release device based on AB testing, characterized in that: include: The log acquisition module obtains the grayscale function tracking logs sent by the SDK tracking of each business node and the front-end tracking logs sent by the SDK tracking of each front-end node; The log analysis module analyzes the grayscale function tracking log and the front-end tracking log based on a preset AB test strategy; wherein the preset AB test strategy includes: transaction parameter calculation method and parameter expected value or transaction parameter calculation method, parameter expected value, expected U value and grayscale release time node requirements; The grayscale function release control module releases grayscale functions with the optimal strategy based on log analysis results; The log analysis module includes: A parameter calculation unit, which collects and counts the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method to obtain a transaction parameter value; A parameter comparison and analysis unit, which compares the transaction parameter value and the parameter expected value to obtain a log analysis result; The log analysis module also includes: A parameter calculation unit, which collects and counts the gray function embedding log and the front-end embedding log based on the transaction parameter calculation method to obtain a transaction parameter value; The time judgment unit judges whether the grayscale release time node requirements are met based on the grayscale function tracking log and the front-end tracking log.

9. The device for releasing adaptive grayscale functions based on AB testing according to claim 8, characterized in that: The trading parameters include: mean value, variance and / or sample size.

10. The device for releasing adaptive grayscale functions based on AB testing according to claim 8, characterized in that: Also includes: The highest value screening module screens the highest value of the significance level U value of the business nodes corresponding to several grayscale functions that are listed in parallel for selection; The optimal grayscale publishing module publishes the grayscale function corresponding to the highest value as the optimal grayscale strategy.

11. The device for releasing adaptive grayscale functions based on AB testing according to claim 8, characterized in that: The preset AB test strategy includes: starting the business node identification of each grayscale function; The adaptive grayscale function release device based on AB testing also includes: The start control module controls the corresponding service node to start the corresponding grayscale function according to the service node identifier.

12. The device for releasing adaptive grayscale functions based on AB testing according to claim 8, characterized in that: Also includes: A log data preprocessing module, which preprocesses the grayscale function embedding log and the front-end embedding log; The log data preprocessing module includes: A log collection unit, using kafka to collect the grayscale function embedding log and the front-end embedding log; The data extraction unit extracts performance data from the grayscale function tracking log and the front-end tracking log to obtain key transaction metric data as basic data for log analysis.

13. The device for releasing an adaptive grayscale function based on AB testing according to any one of claims 8, 10 to 12, characterized in that: The grayscale function tracking log includes: background transaction data; the front-end tracking log includes: transaction instance tracking identifier.

14. The device for releasing adaptive grayscale functions based on AB testing according to claim 8, characterized in that: The grayscale function release control module includes: A result judgment unit, judging whether the log analysis result meets the requirements of a preset AB test strategy; The grayscale function release unit releases the grayscale function if it meets the requirements of the preset AB testing strategy; The function rollback unit controls the business node that starts the grayscale function to roll back the function if the requirements of the preset AB test strategy are not met.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for releasing an adaptive grayscale function based on AB testing as described in any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for releasing an adaptive grayscale function based on AB testing as described in any one of claims 1 to 7 are implemented.