Product effect verification method and device, electronic equipment and medium

Abstract

The invention provides a product effect verification method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing. The method comprises the following steps: dividing user traffic into a control group and an experimental group through a hash function; sorting the user traffic of the control group and the experimental group according to the user activeness to obtain corresponding user sequences, and rejecting the user traffic of a preset proportion at the head and the tail in each user sequence to serve as final user traffic; respectively calculating evaluation indexes of the control group and the experimental group; and determining that the new function of the product is superior to the original function of the product in response to that the evaluation index of the experimental group is superior to the evaluation index of the control group. The problem that the accuracy of a product verification result is low due to random sampling deviation existing in an AB experiment in the prior art is at least solved. The method is suitable for data analysis, product verification and other scenes.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a product effect verification method, device, electronic device and computer-readable storage medium. Background Art

[0002] At present, the effectiveness of new functions of client products is usually verified based on AB experiments (Controlled Experiment). AB experiments refer to independently and randomly sampling a control group of users and an experimental group of users in equal proportion from the overall traffic. The control group of users experiences the baseline strategy, while the experimental group of users experiences the new strategy. Finally, the evaluation indicators of the two groups of users are compared on a daily or hourly basis.

[0003] Due to the random sampling bias in AB experiments, the accuracy of product verification results is low. For example, some users use the product for 3 hours a day, while others use it for 1 minute. Therefore, different users contribute differently to the comprehensive evaluation results of the entire user set, and thus have different impacts on the evaluation indicators. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a product effect verification method, device, electronic device and computer-readable storage medium in view of the above-mentioned deficiencies in the prior art, so as to at least solve the problem existing in the related art that the accuracy of the product verification results is low due to the random sampling bias in AB experiments.

[0005] In a first aspect, the present invention provides a method for verifying product effects, comprising: dividing user traffic into a control group and an experimental group through a hash function; sorting the user traffic of the control group and the experimental group according to user activity to obtain corresponding user sequences, and removing a preset proportion of user traffic at the head and tail of each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product; respectively calculating evaluation indicators of the control group and the experimental group; in response to the evaluation indicator of the experimental group being better than the evaluation indicator of the control group, determining that the new function of the product is better than the original function of the product.

[0006] Preferably, the evaluation indicators include a first key indicator, a second key indicator and a comprehensive indicator.

[0007] The calculation of the evaluation indicators of the control group and the experimental group respectively specifically includes: calculating the first key indicator and the second key indicator of the control group and the experimental group respectively; and calculating the comprehensive indicators of the control group and the experimental group respectively according to the following formula:

[0008] S=a×DA+b×UD

[0009] Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

[0010] Preferably, after respectively calculating the evaluation indicators of the control group and the experimental group, and before determining that the new function of the product is better than the original function of the product in response to the evaluation indicator of the experimental group being better than the evaluation indicator of the control group, the method also includes: obtaining the optimal hyperparameters based on negative feedback optimization of ABB sampling traffic.

[0011] Preferably, the negative feedback optimization based on ABB sampling traffic obtains the optimal hyperparameters, specifically including: configuring a control group, an exploration group and an observation group based on ABB sampling traffic; setting the exploration group and the observation group to have the same set of current hyperparameters; optimizing the next set of hyperparameters in the exploration group based on the negative feedback mechanism of reinforcement learning; obtaining the evaluation index of the exploration group under the next set of hyperparameters; in response to the evaluation index of the exploration group under the next set of hyperparameters being better than its evaluation index under the current hyperparameters, applying the next set of hyperparameters to the observation group, otherwise keeping the current hyperparameters of the observation group unchanged; judging whether the evaluation index of the observation group under the next set of hyperparameters meets the target value, if so, the next set of hyperparameters is the optimal hyperparameter, if not, continuing to optimize the new next set of hyperparameters.

[0012] In a second aspect, the present invention further provides a device for verifying product effects, including a grouping module, a sorting module, a calculation module and a determination module.

[0013] The grouping module is used to divide user traffic into the control group and the experimental group through the hash function.

[0014] The sorting module is connected to the grouping module and is used to sort the user traffic of the control group and the experimental group according to user activity to obtain corresponding user sequences, and remove a preset proportion of user traffic at the head and tail of each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product.

[0015] The calculation module is connected with the sorting module and is used to calculate the evaluation indicators of the control group and the experimental group respectively.

[0016] The determination module is connected to the calculation module and is used to determine that the new function of the product is better than the original function of the product in response to the evaluation index of the experimental group being better than the evaluation index of the control group.

[0017] Preferably, the evaluation indicators include a first key indicator, a second key indicator and a comprehensive indicator.

[0018] The calculation module includes a first calculation unit and a second calculation unit.

[0019] The first calculation unit is used to calculate the first key indicator and the second key indicator of the control group and the experimental group respectively.

[0020] The second calculation unit is connected to the first calculation unit and is used to calculate the comprehensive index of the control group and the experimental group according to the following formula:

[0021] a=a×DA+b×UD

[0022] Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

[0023] Preferably, the device further comprises an optimization module.

[0024] The optimization module is used to obtain the optimal hyperparameters based on the negative feedback optimization of ABB sampling traffic.

[0025] Preferably, the optimization module includes a configuration unit, an optimization unit and a judgment unit.

[0026] The configuration unit is used to configure the control group, exploration group and observation group based on the ABB sampling traffic.

[0027] An optimization unit is connected to the configuration unit, and is used to set the exploration group and the observation group to have the same set of current hyperparameters, and to optimize the next set of hyperparameters in the exploration group based on the negative feedback mechanism of reinforcement learning, and to obtain the evaluation index of the exploration group under the next set of hyperparameters, and to apply the next set of hyperparameters to the observation group in response to the evaluation index of the exploration group under the next set of hyperparameters being better than the evaluation index under the current hyperparameters, otherwise the current hyperparameters of the observation group are kept unchanged.

[0028] The judgment unit is connected to the optimization unit and is used to judge whether the evaluation index of the observation group under the action of the next set of hyperparameters meets the target value. If so, the next set of hyperparameters is the optimal hyperparameter. If not, continue to optimize the next set of new hyperparameters.

[0029] In a third aspect, the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the product effect verification method as described in the first aspect.

[0030] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the product effect verification method as described in the first aspect is implemented.

[0031] The product effect verification method, device, electronic device and computer-readable storage medium provided by the present invention obtain corresponding user sequences by sorting the user traffic of the control group and the experimental group, and remove a preset proportion of user traffic at the head and tail of each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product, thereby eliminating traffic noise, reducing the impact of traffic noise on product verification results, and ultimately improving the accuracy of product verification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a flow chart of a method for verifying the effect of a product according to Embodiment 1 of the present invention;

[0033] Figure 2 It is a schematic diagram of a negative feedback optimization mechanism based on ABB sampling flow in embodiment 1 of the present invention;

[0034] Figure 3 A schematic diagram of the structure of a device for verifying the effect of a product according to Embodiment 2 of the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of a device for verifying the effect of a product according to Embodiment 3 of the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0038] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0039] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0040] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0041] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0042] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0043] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0044] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0045] Embodiment 1:

[0046] like Figure 1 As shown, this embodiment provides a method for verifying product effects, including:

[0047] Step 101, divide the user traffic into a control group and an experimental group through a hash function.

[0048] In this embodiment, in the process of extracting user traffic, in order to ensure the randomness and uniformity of the control group and the experimental group traffic, the user traffic is evenly divided into a preset number of buckets through a hash function. Before step 101, the method also includes: completing the configuration of default values ​​and the configuration of evaluation indicators to be compared, and also configuring the proportion of experimental traffic to be extracted.

[0049] Step 102, sorting the user traffic of the control group and the experimental group according to user activity to obtain corresponding user sequences, and removing a preset proportion of user traffic at the head and tail of each user sequence as the final user traffic, wherein the final user traffic of the control group experiences the original functions of the product, and the final user traffic of the experimental group experiences the new functions of the product.

[0050] In this embodiment, after the user traffic is evenly divided into buckets and extracted, the users are sorted from low to high or from high to low according to their activity levels, and a preset proportion (e.g., 3%) of active user traffic at the head and a preset proportion (e.g., 3%) of active user traffic at the tail are removed. After removing the traffic noise, the evaluation index results related to the product verification effect of the control group and the experimental group can be made more stable and accurate.

[0051] Step 103, respectively calculating the evaluation indicators of the control group and the experimental group.

[0052] In this embodiment, the control group and the experimental group report key indicators (or technical indicators) that can measure the product verification effect from the user experience terminal. For example, for short video recommendation products, it is necessary to regularly report the number of video views, video viewing time, whether there are likes, comments, etc. The data reported by the control group and the experimental group are periodically calculated and counted, and finally an evaluation report containing multiple technical indicators is obtained, and the product verification effect is judged according to the evaluation report.

[0053] Specifically, the evaluation index includes a first key index, a second key index and a comprehensive index. The evaluation indexes of the control group and the experimental group are calculated separately, including:

[0054] Calculate the first key indicator and the second key indicator of the control group and the experimental group respectively;

[0055] The comprehensive indicators of the control group and the experimental group were calculated according to the following formula:

[0056] S=a×DA+b×UD

[0057] Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

[0058] In this embodiment, since the evaluation dimensions of the product effect indicators of the existing control group and experimental group are too single and one-sided, taking the user product of short video recommendation as an example, the key indicators for evaluating the effectiveness of personalized recommendation are the number of videos watched per capita and the length of time the videos are watched per capita. When the overall recommendation effect is finally evaluated, the individual indicators such as the number of videos watched per capita and the length of time the videos are watched per capita are evaluated in isolation. However, the contribution of individual indicators to the verification effect is different, and a more comprehensive and unified indicator is needed to measure the product verification effect. Therefore, a comprehensive indicator is constructed based on the key indicators. For example, a comprehensive indicator for measuring the experimental effect is added on the basis of the existing indicators for measuring the experimental effect: assuming that the existing key indicators are "daily active users" (expressed as DA) and "average usage time" (expressed as UD), then an additional comprehensive indicator S is added, which can be used to reflect the contribution of each key indicator to the product verification effect.

[0059] Step 104 , in response to the evaluation index of the experimental group being better than the evaluation index of the control group, determining that the new function of the product is better than the original function of the product.

[0060] In this embodiment, the ultimate goal of the experimental group is to achieve better product effects than the control group, so it is necessary to optimize some hyperparameters of the experimental group. For example, for user products such as short video recommendations, it is necessary to determine the hyperparameter of the number of videos recommended to users at a time (c). The existing method is to manually set multiple groups of hyperparameters for the number of videos sent: c1, c2, ..., cn, where n is a positive integer, and conduct a group of experiments for each group of hyperparameters for the number of videos sent. Finally, the group of experiments with better product recommendation effects uses the corresponding group of hyperparameters. The manual iterative optimization solution based on experience is inefficient, and a more efficient and intelligent solution is needed to complete the optimization of hyperparameters in AB experiments.

[0061] Optionally, after respectively calculating the evaluation indicators of the control group and the experimental group, and before determining that the new function of the product is better than the original function of the product in response to the evaluation indicator of the experimental group being better than the evaluation indicator of the control group, the method further includes: obtaining the optimal hyperparameters based on negative feedback optimization of ABB sampling traffic.

[0062] Specifically, the negative feedback optimization based on ABB sampling flow to obtain the optimal hyperparameters includes:

[0063] Configure control group, exploration group and observation group based on ABB sampling traffic;

[0064] Set the exploration group and observation group to have the same set of current hyperparameters;

[0065] The negative feedback mechanism based on reinforcement learning is used to optimize the next set of hyperparameters in the exploration group;

[0066] Get the evaluation indicators of the exploration group under the next set of hyperparameters;

[0067] In response to the evaluation index of the exploration group under the next set of hyperparameters being better than the evaluation index under the current hyperparameters, applying the next set of hyperparameters to the observation group, otherwise keeping the current hyperparameters of the observation group unchanged;

[0068] Determine whether the evaluation index of the observation group under the action of the next set of hyperparameters meets the target value. If so, the next set of hyperparameters is the optimal hyperparameter. If not, continue to optimize the next set of new hyperparameters.

[0069] In this embodiment, the negative feedback optimization framework based on ABB sampling flow is as follows: Figure 2As shown, three groups of traffic A, B1, and B2 are used as the control group, exploration group, and observation group respectively. First, a set of initial hyperparameter configurations p1 is given to the exploration group and the observation group in the configuration layer. Then, based on the negative feedback mechanism of reinforcement learning, the next set of hyperparameters p2 is optimized in the exploration group, and the p2 hyperparameters are applied to the exploration group traffic. The experimental effect evaluation index of the exploration group traffic is statistically analyzed. If the experimental effect evaluation index is better than the previous evaluation index, p2 is applied to the observation group traffic, otherwise the hyperparameter p1 of the observation group is kept unchanged; then the above process is repeated until the experimental effect evaluation index of the observation group meets the established goals. Obtaining the optimal hyperparameters through negative feedback optimization of ABB sampling traffic can improve product verification efficiency.

[0070] The product effect verification method provided in this embodiment is to obtain the corresponding user sequence by sorting the user traffic of the control group and the experimental group, and to remove the preset proportion of user traffic at the head and tail of each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original function of the product, and the final user traffic of the experimental group is used to experience the new function of the product, thereby eliminating the traffic noise, reducing the impact of traffic noise on the product verification results, and ultimately improving the accuracy of the product verification results. Furthermore, since the evaluation dimensions of the product effect indicators of the existing control group and the experimental group are too single and one-sided, and the contribution of a single indicator to the verification effect is different, a comprehensive indicator is constructed in the evaluation indicator to reflect the contribution of each key indicator to the product verification effect, thereby improving the accuracy of the verification result. In addition, the optimal hyperparameters are obtained through negative feedback optimization of ABB sampling traffic, the optimization efficiency is improved, thereby improving the efficiency of product verification, and avoiding manual iterative optimization that consumes computing resources, time and labor costs for effect verification.

[0071] Embodiment 2:

[0072] like Figure 3 As shown, this embodiment provides a product effect verification device, including a grouping module 31 , a sorting module 32 , a calculation module 33 and a determination module 34 .

[0073] The grouping module 31 is used to divide the user traffic into a control group and an experimental group through a hash function.

[0074] The sorting module 32 is connected to the grouping module 31, and is used to sort the user traffic of the control group and the experimental group according to the user activity to obtain the corresponding user sequence, and remove the user traffic of the head and tail of each user sequence with a preset ratio as the final user traffic, wherein the final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product.

[0075] The calculation module 33 is connected to the sorting module 32 and is used to calculate the evaluation indicators of the control group and the experimental group respectively.

[0076] The determination module 34 is connected to the calculation module 33 and is used to determine that the new function of the product is better than the original function of the product in response to the evaluation index of the experimental group being better than the evaluation index of the control group.

[0077] Optionally, the evaluation indicators include a first key indicator, a second key indicator and a comprehensive indicator.

[0078] The calculation module includes a first calculation unit and a second calculation unit.

[0079] The first calculation unit is used to calculate the first key indicator and the second key indicator of the control group and the experimental group respectively.

[0080] The second calculation unit is connected to the first calculation unit and is used to calculate the comprehensive index of the control group and the experimental group according to the following formula:

[0081] S=a×DA+b×UD

[0082] Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

[0083] Optionally, the device further includes an optimization module.

[0084] The optimization module is used to obtain the optimal hyperparameters based on the negative feedback optimization of ABB sampling traffic.

[0085] Optionally, the optimization module includes a configuration unit, an optimization unit and a judgment unit.

[0086] The configuration unit is used to configure the control group, exploration group and observation group based on the ABB sampling traffic.

[0087] The optimization unit is connected to the configuration unit and is used to set the exploration group and the observation group to have the same set of current hyperparameters, and optimize the next set of hyperparameters in the exploration group based on the negative feedback mechanism of reinforcement learning, so as to obtain the evaluation index of the exploration group under the action of the next set of hyperparameters.

[0088] And, in response to an evaluation index of the exploration group under the next set of hyperparameters being better than its evaluation index under the current hyperparameters, applying the next set of hyperparameters to the observation group, otherwise keeping the current hyperparameters of the observation group unchanged.

[0089] The judgment unit is connected to the optimization unit and is used to judge whether the evaluation index of the observation group under the action of the next set of hyperparameters meets the target value. If so, the next set of hyperparameters is the optimal hyperparameter. If not, continue to optimize the next set of new hyperparameters.

[0090] The product effect verification device provided in this embodiment is used to obtain the corresponding user sequence by sorting the user traffic of the control group and the experimental group, and is also used to remove the user traffic of the head and tail preset proportions in each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original function of the product, and the final user traffic of the experimental group is used to experience the new function of the product, thereby eliminating the traffic noise, reducing the impact of the traffic noise on the product verification result, and finally improving the accuracy of the product verification result. Furthermore, since the evaluation dimensions of the product effect indicators of the existing control group and the experimental group are too single and one-sided, and the contribution of a single indicator to the verification effect is different, it is used to reflect the contribution of each key indicator to the product verification effect by constructing a comprehensive indicator in the evaluation indicator, so as to improve the accuracy of the verification result. In addition, it is used to obtain the optimal hyperparameters through negative feedback optimization of ABB sampling traffic, improve the optimization efficiency, thereby improving the product verification efficiency, and avoiding manual iterative optimization that consumes computing resources, consumes effect verification time and labor costs.

[0091] Embodiment 3:

[0092] like Figure 4 As shown, this embodiment provides a device for verifying product effects, including a configuration layer, a traffic layer, a logic layer, a data reporting layer and an experimental evaluation layer.

[0093] The configuration layer is used to complete the configuration of default values ​​and the experimental indicators that need to be compared. It is also necessary to configure the proportion of experimental traffic that needs to be extracted.

[0094] The traffic layer is connected to the configuration layer and is used to complete the extraction of traffic from the experimental group and the control group. The extraction process needs to ensure the randomness and uniformity of the two groups of traffic. The common method is to evenly divide the user traffic into a certain number of buckets through a hash function. After the traffic is evenly divided into buckets, the users are sorted from low to high according to their activity level, and the top 3% active user traffic and the tail 3% active user traffic are removed. After the noise is removed, the effect indicators of the control group and the experimental group can be more stable and accurate.

[0095] The logic layer is connected to the traffic layer and is used to divert traffic in the business logic: user traffic hitting the control group experiences the original functions, and user traffic hitting the experimental group experiences the new functions.

[0096] The data reporting layer is connected to the logic layer, and is used by the control group and the experimental group to report technical indicators that can measure the product experience effect from the user experience end (generally the front end of the product). For example, in short video recommendation products, it is necessary to regularly report the number of video views, video viewing time, whether there are likes, comments, and other behaviors.

[0097] The experimental evaluation layer is connected to the data reporting layer and is used to perform periodic calculations and statistics on the data reported by the control group and the experimental group, and ultimately produce an evaluation report containing multiple key indicators. The engineering staff responsible for the AB experiment uses the evaluation report to judge the experimental effect.

[0098] Optionally, the experiment evaluation layer is also used to add a comprehensive indicator to measure the experimental effect: assuming that the existing key experimental indicators are "daily active users" (expressed by DA) and "average usage time per person" (expressed by UD), then the additional comprehensive indicator = a*DA+b*UD (a is the weight of DA, b is the weight of UD).

[0099] Optionally, the experimental evaluation layer is also used to obtain hyperparameters based on the negative feedback optimization framework of ABB sampling traffic: three groups of traffic A, B1, and B2 are used as the control group, exploration group, and observation group, respectively. First, a set of initial hyperparameter configurations p1 that are the same as the exploration group and the observation group are given in the configuration layer, and then a set of hyperparameters p2 are optimized in the exploration group based on the negative feedback mechanism of reinforcement learning, and the p2 hyperparameters are applied to the exploration group traffic. The comprehensive experimental effect index of the exploration group traffic is counted. If the comprehensive experimental effect index is better than the previous experimental index, p2 is applied to the observation group traffic, otherwise the hyperparameter p1 of the observation group is kept unchanged; then the above process is repeated until the comprehensive experimental effect index of the observation group meets the established goals.

[0100] Embodiment 4:

[0101] like Figure 5 As shown, this embodiment provides an electronic device, including a memory 51 and a processor 52, wherein the memory 51 stores a computer program, and the processor 52 is configured to run the computer program to implement the product effect verification method as described in Example 1.

[0102] Embodiment 5:

[0103] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the product effect verification method as described in Embodiment 1 is implemented.

[0104] In summary, the product effect verification method, device, electronic device and computer-readable storage medium provided by the embodiment of the present invention are used to obtain the corresponding user sequence by sorting the user traffic of the control group and the experimental group, and are also used to remove the user traffic of the head and tail preset proportions in each user sequence as the final user traffic, wherein the final user traffic of the control group is used to experience the original function of the product, and the final user traffic of the experimental group is used to experience the new function of the product, thereby eliminating the traffic noise, reducing the impact of the traffic noise on the product verification result, and finally improving the accuracy of the product verification result. Further, since the evaluation dimensions of the product effect indicators of the existing control group and the experimental group are too single and one-sided, and the contribution of a single indicator to the verification effect is different, it is used to construct a comprehensive indicator in the evaluation indicator to reflect the contribution of each key indicator to the product verification effect, so as to improve the accuracy of the verification result. In addition, it is used to obtain the optimal hyperparameters through negative feedback optimization of ABB sampling traffic, improve the optimization efficiency, thereby improving the product verification efficiency, and avoiding manual iterative optimization that consumes computing resources, consumes effect verification time and labor costs.

[0105] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for verifying the effect of a product. It is characterized in that include: Divide user traffic into control group and experimental group through hash function; The user traffic of the control group and the experimental group is sorted according to the user activity to obtain the corresponding user sequence, and the user traffic of the head and tail of each user sequence is removed as the final user traffic. The final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product; Calculate the evaluation indicators of the control group and the experimental group respectively; In response to the evaluation index of the experimental group being better than the evaluation index of the control group, it is determined that the new function of the product is better than the original function of the product.

2. The method according to claim 1, It is characterized in that The evaluation indicators include the first key indicator, the second key indicator and the comprehensive indicator. The evaluation indicators of the control group and the experimental group are calculated separately, specifically including: Calculate the first key indicator and the second key indicator of the control group and the experimental group respectively; The comprehensive indicators of the control group and the experimental group were calculated according to the following formula: S=a×DA+b×UD Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

3. The method according to claim 1, It is characterized in that After respectively calculating the evaluation indicators of the control group and the experimental group, and before determining that the new function of the product is better than the original function of the product in response to the evaluation indicator of the experimental group being better than the evaluation indicator of the control group, the method further includes: The optimal hyperparameters are obtained based on negative feedback optimization of ABB sampling traffic.

4. The method according to claim 3, It is characterized in that The negative feedback optimization based on ABB sampling flow to obtain the optimal hyperparameters specifically includes: Configure control group, exploration group and observation group based on ABB sampling traffic; Set the exploration group and observation group to have the same set of current hyperparameters; The negative feedback mechanism based on reinforcement learning is used to optimize the next set of hyperparameters in the exploration group; Get the evaluation indicators of the exploration group under the next set of hyperparameters; In response to the evaluation index of the exploration group under the next set of hyperparameters being better than the evaluation index under the current hyperparameters, applying the next set of hyperparameters to the observation group, otherwise keeping the current hyperparameters of the observation group unchanged; Determine whether the evaluation index of the observation group under the action of the next set of hyperparameters meets the target value. If so, the next set of hyperparameters is the optimal hyperparameter. If not, continue to optimize the next set of new hyperparameters.

5. A device for verifying the effect of a product, It is characterized in that It includes grouping module, sorting module, calculation module and determination module. The grouping module is used to divide user traffic into the control group and the experimental group through the hash function. The sorting module is connected to the grouping module and is used to sort the user traffic of the control group and the experimental group according to the user activity to obtain the corresponding user sequence, and remove the user traffic of the head and tail of each user sequence as the final user traffic. The final user traffic of the control group is used to experience the original functions of the product, and the final user traffic of the experimental group is used to experience the new functions of the product. The calculation module is connected to the sorting module and is used to calculate the evaluation indicators of the control group and the experimental group respectively. The determination module is connected to the calculation module and is used to determine that the new function of the product is better than the original function of the product in response to the evaluation index of the experimental group being better than the evaluation index of the control group.

6. The device according to claim 5, It is characterized in that The evaluation indicators include the first key indicator, the second key indicator and the comprehensive indicator. The calculation module includes a first calculation unit and a second calculation unit. The first calculation unit is used to calculate the first key indicator and the second key indicator of the control group and the experimental group respectively. The second calculation unit is connected to the first calculation unit and is used to calculate the comprehensive index of the control group and the experimental group according to the following formula: S=a×DA+b×UD Among them, DA is the first key indicator, UD is the second key indicator, a is the weight of the first key indicator, and b is the weight of the second key indicator.

7. The device according to claim 6, It is characterized in that It also includes an optimization module. The optimization module is used to obtain the optimal hyperparameters based on the negative feedback optimization of ABB sampling traffic.

8. The device according to claim 7, It is characterized in that The optimization module includes a configuration unit, an optimization unit and a judgment unit. Configuration unit, used to configure the control group, exploration group and observation group based on ABB sampling traffic, The optimization unit is connected to the configuration unit and is used to set the exploration group and the observation group to have the same set of current hyperparameters, and optimize the next set of hyperparameters in the exploration group based on the negative feedback mechanism of reinforcement learning, so as to obtain the evaluation index of the exploration group under the action of the next set of hyperparameters. and, in response to the evaluation index of the exploration group under the next set of hyperparameters being better than the evaluation index under the current hyperparameters, applying the next set of hyperparameters to the observation group, otherwise keeping the current hyperparameters of the observation group unchanged, The judgment unit is connected to the optimization unit and is used to judge whether the evaluation index of the observation group under the action of the next set of hyperparameters meets the target value. If so, the next set of hyperparameters is the optimal hyperparameter. If not, continue to optimize the next set of new hyperparameters.

9. An electronic device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the product effect verification method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the product effect verification method according to any one of claims 1 to 4 is implemented.

Images