Experimental index correction method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an experimental index correction method and device, electronic equipment and storage medium, and relates to the technical field of software testing. The specific implementation scheme is: obtaining the value of the experimental index of the experimental group in the release experiment and the data information of the experimental group; based on the pre-established convergence coefficient table and the data information of the experimental group, the value of the experimental index is corrected. The technology of the present disclosure can effectively shorten the experimental period and improve the experimental efficiency.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to the technical field of software testing, and especially to an experiment index correction method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the prior art, A / B Test can accurately evaluate the effect of a single function or product, and is a commonly used test method after a new version of an application is put into use.

[0003] However, for the effect of a version put into use, there is a user accumulation process after the version is put into use, and the rate of user accumulation affects the test efficiency. In particular, when the user base in the idle period is inconsistent with that in the experiment period, the fluctuation range of the user index will also be inconsistent. From the user upgrade curve, the fluctuation range of the user data in the idle period relative to that in the experiment period will be larger, which will lead to a decrease in sensitivity in the test.

[0004] In order to solve the problem of large index fluctuation range, a mainstream method used in the industry is to avoid the convergence stage of the development version, and to use time to replace measurability. The entire period from the release of a version to the convergence of the version is calculated, a key node at which the user convergence rate slows down on the convergence curve is found, and it is considered that when the convergence rate of the new version exceeds the convergence rate corresponding to the key node, the user convergence rate tends to be flat, and the index fluctuation of the release is relatively small and can be used for the experiment. SUMMARY

[0005] The present disclosure provides an experiment index correction method and device, an electronic device and a storage medium.

[0006] According to an aspect of the present disclosure, an experiment index correction method is provided, comprising:

[0007] obtaining the value of the experiment index of the experiment group in the release experiment and the data information of the experiment group;

[0008] correcting the value of the experiment index based on the pre-established convergence coefficient table and the data information of the experiment group.

[0009] According to another aspect of the present disclosure, an experiment index correction device is provided, comprising:

[0010] an information acquisition module configured to obtain the value of the experiment index of the experiment group in the release experiment and the data information of the experiment group;

[0011] a correction module configured to correct the value of the experiment index based on the pre-established convergence coefficient table and the data information of the experiment group.

[0012] According to still another aspect of the present disclosure, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein

[0015] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of the aspects and any possible implementation manner as described above.

[0016] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause the computer to perform the method of the aspects and any possible implementation manner as described above.

[0017] According to still another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the aspects and any possible implementation manner as described above.

[0018] According to the technology of the present disclosure, the experimental period can be effectively shortened and the experimental efficiency can be improved.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0021] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0026] Figure 6 is a block diagram of an electronic device used to implement the method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Accordingly, it should be understood that various changes and modifications to the embodiments described herein can be made by those having ordinary skill in the art without departing from the scope and spirit of the present disclosure. Also, for the purpose of clarity and a concise description, descriptions of well-known functions and constructions are omitted from the following description.

[0028] It is apparent that the described embodiments are merely a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of the present disclosure.

[0029] It should be noted that the terminal device involved in the embodiments of the present disclosure can include, but is not limited to, a mobile phone, a personal digital assistant (PDA), a wireless handheld device, a tablet computer, and the like. The display device can include, but is not limited to, a personal computer, a television, and the like.

[0030] In addition, the term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone. In addition, the character " / " in this paper generally represents an "or" relationship between the front and rear associated objects.

[0031] For the mainstream development version convergence phase scheme in the industry, the advantage is strong operability, but the disadvantage is also obvious. The whole cycle of waiting for the new version user convergence is too long, and the user convergence process is uncontrollable, and can only be judged by experience. However, the product requires rapid iteration, and with the improvement of the whole application development efficiency, the new version is released more and more frequently, and the long-period version convergence and the requirement of rapid iteration are in conflict, so the development version convergence phase method will lead to a long experimental cycle and become more and more unfeasible.

[0032] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; as Figure 1 shown, the present embodiment provides an experimental index correction method, which can be applied to any experimental index correction device, and can specifically include the following steps:

[0033] S101, obtaining the value of the experimental index of the experimental group in the development experiment and the data information of the experimental group;

[0034] S102, correcting the value of the experimental index based on the pre-established convergence coefficient table and the data information of the experimental group.

[0035] The experimental index correction method of the embodiment is used to correct the experimental index, so that the fluctuation range of the experimental index in the idle period and the experimental period is not too large, and the two are comparable. The experimental effect can be correctly statistically tested.

[0036] The experimental group of the embodiment can refer to the A group or the B group in the A / B test. Specifically, the experimental group can be the experimental group of the idle period or the experimental group of the experimental period.

[0037] In the specific experimental process, the experimental index of the A group and the B group in the idle period and the experimental period can be corrected in the manner of the embodiment, so that all indexes can be more objectively compared under the same standard, so as to effectively detect the experimental effect.

[0038] The experimental index of the embodiment can include various test indexes after the new version of the application is released, such as the average duration of the user using the application, the heat of the user to the application, and the like. The experimental index can test the user's reaction after the new version is released, and then the acceptance degree of the user to the new version can be reflected.

[0039] The data information of the experimental group of the embodiment can include various data information of the experimental group, such as the sample number of the experimental group, the sample number of the experimental group upgraded to the new version, and various detailed experimental group information.

[0040] The sample in the experimental group of the embodiment is the account of the user participating in the experiment. In actual application, after the new version of the application is released, the experimental invitation information can also be pushed to the user's account, and after the user agrees, the user can join the experimental group. The various behaviors of the user using the new version of the application in the experimental group will directly affect the experimental index, and then affect the experimental effect. In addition, the application party can also randomly select part of the authorized users to participate in the experiment under the authorization of the user using the application, so as to test the new version of the application.

[0041] In the embodiment, the convergence coefficient table established in advance is the convergence-related coefficient collected in advance through statistical experiment. The convergence-related coefficient itself represents the share of the sample using the new version, which is only related to the version upgrade behavior in the sample, and is irrelevant to the experimental index and the experimental effect corresponding to the experimental index. Therefore, when the value of the experimental index is corrected based on the pre-established convergence coefficient table and the data information of the experimental group, the experimental effect will not be affected.

[0042] In this embodiment, by using the pre-established convergence coefficient table and the data information of the experimental group, the values of the experimental indexes of the experimental group in the release experiment can be effectively corrected, so that all the corrected experimental indexes are under the same standard and have comparability, and then the test is performed based on the corrected experimental indexes. By using the technical solution of this embodiment, the experiment can be performed without waiting for the user convergence period after the release, the experimental period can be effectively shortened, the experimental efficiency can be improved, and then the release rate and release effect of the application can be promoted.

[0043] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; as shown in Figure 2 The establishment process of the convergence coefficient table is provided to facilitate the correction of the values of the experimental indexes, and can specifically include the following steps:

[0044] S201, based on each traffic in the preset plurality of traffics, sampling of the second sample set is performed multiple times from the preset first sample set;

[0045] The traffic in this embodiment can be equal to the proportion of samples in the second sample set and the first sample set. The samples in the first sample set and the second sample set in this embodiment are all information of simulated users using the application, and each sample corresponds to the information of a user. In this embodiment, a part of the samples in the first sample set are randomly set to correspond to users upgrading the new version of the application. Specifically, the proportion of the set users upgrading the new version can be set according to experience, which is not limited herein.

[0046] In order to cover all experimental situations, in this embodiment, a plurality of different sizes of traffic such as 1%, 2%, 3%, 5%, 10%, 20%, etc. can be preset. If the number of the first sample set is N, the number of samples included in the second sample set is N*1% when the traffic is 1%. The number of samples included in the second sample set is N*20% when the traffic is 20%. Other cases are similar. That is, the traffic is used to identify the proportion of the second sample set to be evaluated in the total first sample set.

[0047] For each traffic, multiple samplings of the second sample set can be performed in this embodiment, for example, 1000 times, 2000 times or other times, which can be set according to actual experience, which is not limited herein. Through multiple different samplings, a more accurate convergence coefficient table can be obtained.

[0048] S202, obtaining the convergence rates corresponding to the second sample sets sampled multiple times under each traffic;

[0049] For each traffic, the second sample set is sampled each time, and the corresponding convergence rate can be calculated. For example, the specific calculation process can include the following steps:

[0050] (1) Count the number of samples upgraded to the new version in the second sample set of each sampling under each traffic;

[0051] (2) Based on the number of samples upgraded to the new version in the second sample set of each sampling under each traffic and the number of samples included in the second sample set, the convergence rate corresponding to each sampling of the second sample set under each traffic is calculated.

[0052] Specifically, the number of samples upgraded to the new version in the second sample set under each traffic is divided by the number of samples included in the second sample set, which is equal to the convergence rate corresponding to the second sample set of the sampling under the traffic.

[0053] S203, based on the convergence rates corresponding to the second version sets of multiple samplings under each traffic, the convergence rate variance and the convergence rate mean of multiple samplings under each traffic are obtained;

[0054] For the same traffic, the corresponding convergence rate can be obtained through multiple samplings. Then, the convergence rate variance and the convergence rate mean of multiple samplings under the traffic are calculated based on the multiple convergence rates corresponding to the traffic.

[0055] S204, based on each traffic, the convergence rate variance and the convergence rate mean corresponding to each traffic, and the sample number of the preset first sample set, a convergence coefficient table is established.

[0056] For different traffic, the corresponding convergence rate variance and the convergence rate mean under each traffic can be obtained, and at this time, the corresponding convergence coefficient table can be established based on each traffic, the convergence rate variance and the convergence rate mean corresponding to each traffic. That is, the convergence coefficient table of the embodiment stores the convergence rate variance and the convergence rate mean corresponding to each traffic.

[0057] Meanwhile, in the embodiment, the sample number of the preset first sample set can also be marked in the convergence coefficient table, so as to facilitate subsequent statistics of the traffic of the experimental group, and then the relevant convergence information can be queried based on the traffic.

[0058] Bootstrap is a mature non-parametric inference algorithm, which relies on resampling the observed sample to approximate the true distribution using the empirical distribution. In the evaluation of the release experiment, the convergence rate fluctuation distribution under the true traffic size cannot be obtained, so in the above technical solution of the embodiment, the Bootstrap repeated sampling method can be used, and for different traffic, the first sample set is repeatedly randomly sampled, a small sample set, i.e., the second sample set, is extracted, and then the convergence rate of each small sample set, i.e., the second sample set, is calculated, and then the fluctuation distribution of the convergence rate under different traffic is obtained, and a corresponding convergence coefficient table is established in the traffic coefficient table.

[0059] In the embodiment, an accurate, reasonable and effective traffic coefficient table can be established by the above method to facilitate subsequent correction of experimental indicators.

[0060] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; as Figure 3 shown in the above Figure 1 and Figure 2 the embodiment, the embodiment provides an experimental index correction method, which can specifically include the following steps:

[0061] S301, obtaining the value of the experimental index of the experimental group in the release experiment and the data information of the experimental group;

[0062] S302, obtaining the traffic of the experimental group based on the data information of the experimental group and the convergence coefficient table;

[0063] For example, the traffic of the experimental group can be obtained in the following way:

[0064] (a1) obtaining the sample number of the experimental group from the data information of the experimental group;

[0065] (b1) obtaining the sample number of the preset first sample set from the convergence coefficient table;

[0066] (c1) obtaining the traffic corresponding to the experimental group based on the sample number of the experimental group and the sample number of the preset first sample set.

[0067] Specifically, dividing the sample number of the experimental group by the sample number in the first sample set is equal to the traffic corresponding to the experimental group.

[0068] S303, obtaining the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group from the traffic convergence coefficient query;

[0069] S304, obtaining the convergence rate of the experimental group based on the data information of the experimental group;

[0070] For example, the convergence rate of the experimental group can be obtained in the following way:

[0071] (a2) obtaining the number of samples in the experimental group from the data information of the experimental group;

[0072] (b2) obtaining the number of samples in the experimental group that upgrade the new version from the data information of the experimental group;

[0073] (c2) obtaining the convergence rate of the experimental group based on the number of samples in the experimental group that upgrade the new version and the number of samples in the experimental group.

[0074] S305, based on the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group, the convergence rate of the experimental group, the value of the experimental index of the experimental group is corrected.

[0075] The controlled-experiment using pre-experiment data (CUPED) is a general method, which mainly uses the unbiased data before the experiment, generally selects the same index before the experiment, corrects the core index of the experiment, so that the new index has lower variance and more sensitive new index, and the influence of the experiment effect is amplified.

[0076] In this embodiment, in the release experiment, in order to flatten the indicators of the experiment in different periods, the idea of using related indicators for correction can be used, which is CUPED. From the principle, because the convergence itself depends on the application version upgrade behavior of the users in the sample, and is completely irrelevant to the experimental effect, and will not be disturbed by the experimental effect. Therefore, the convergence rate is used as a covariate to correct the experimental index.

[0077] For example, in this embodiment, based on the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group, the convergence rate of the experimental group, the value of the experimental index of the experimental group can be corrected by the following formula:

[0078]

[0079] Where, Index ad represents the value of the corrected experimental index; Index represents the value of the experimental index before correction; X represents the convergence rate of the experimental group, cor(Index,X) represents the variance of the value of the experimental index before correction and the convergence rate of the experimental group; var(X) represents the convergence rate variance under the current traffic corresponding to the experimental group; E(X) represents the convergence rate average value under the current traffic corresponding to the experimental group.

[0080] In fact, for the same size traffic of the same version, is a constant value, for each experiment, only the convergence rate X of the experimental group and the index Index are used for correction calculation.

[0081] Since the coefficients related to the convergence rate, such as the convergence rate, the convergence rate variance, the convergence rate average, are only related to the version upgrade behavior in the sample, and are irrelevant to the experimental index, and the experimental effect corresponding to the experimental index is irrelevant, in this embodiment, when the values of the experimental index of the experimental group are modified based on the convergence rate variance and the convergence rate average corresponding to the traffic of the experimental group, the convergence rate of the experimental group, the experimental effect will not be affected. Moreover, all experimental indexes can be modified to a unified standard, which is very convenient for the inspection of experimental indexes.

[0082] For example, in the prior art, the user data fluctuation range in the idle period and the experimental period is large, and the results of the A / B test in the experimental period and the results of the A / B test in the idle period are compared, which may not obtain effective detection results. After using the technical solution of this embodiment, the detection results of all experimental groups can be modified to a unified standard, and then very accurate detection can be performed, which can effectively improve the detection efficiency of the experiment.

[0083] Based on this idea, the technical solution of this embodiment can be applied to experimental detection at any time after the release. Therefore, by using the technical solution of this embodiment, the experiment can be performed at any time after the release without waiting for the user convergence period, which can effectively shorten the experimental period and improve the experimental efficiency, and can also promote the release rate and release effect of the application.

[0084] Using the technical solution of this embodiment for simulation experiment, from the data, 887 experiments are simulated by using random sampling of a certain index, and under the same experimental effect after the release, the number of experiments that detect significant experiments is 778 times after using the experimental index modification method of this embodiment. Compared with using unmodified indexes for monitoring, the number of correct inspections increases by 685 times.

[0085] At the same time, the method of this embodiment is compared with the scheme of not processing but avoiding the release convergence stage, for example, the experimental data used in the release convergence stage is delayed by 5 days, and 943 experiments are simulated by using random sampling simulation. Among them, the number of experiments that detect significant experiments is 828 times after using the experimental index modification method of this embodiment, and the scheme of avoiding the release convergence stage not only increases the total experimental days by 5 days, but also the number of experiments that detect significant experiments is 415 times. Compared with the scheme of avoiding the release convergence stage, the correct inspection number of the scheme using the method of this embodiment increases by 413 times.

[0086] Based on the above, it can be known that in the experimental detection of the release experiment, the experimental index modification method of this embodiment can effectively significantly reduce the experimental period without losing precision.

[0087] Figure 4 is a schematic view according to the fourth embodiment of the present disclosure; as Figure 4As shown, the embodiment provides an experimental index correction device 400, comprising:

[0088] An information acquisition module 401 is configured to acquire a value of an experimental index of an experimental group in a release experiment and data information of the experimental group.

[0089] A correction module 402 is configured to correct the value of the experimental index based on a pre-established convergence coefficient table and the data information of the experimental group.

[0090] The experimental index correction device 400 of the embodiment realizes the implementation principle and technical effect of experimental index correction by using the above modules, which are the same as the implementation of the above related method embodiments. For details, refer to the description of the above related method embodiments, which will not be repeated here.

[0091] Figure 5 is a schematic diagram according to the fifth embodiment of the disclosure; as Figure 5 shown, the embodiment provides an experimental index correction device 500, comprising the same name and function modules of the above Figure 4 shown embodiments, an information acquisition module 501 and a correction module 502.

[0092] Further, as Figure 5 shown, the experimental index correction device 500 of the embodiment further comprises:

[0093] A sampling module 503 is configured to sample a second sample set from a pre-set first sample set based on each flow in a pre-set plurality of flows.

[0094] A convergence rate acquisition module 504 is configured to acquire a convergence rate corresponding to each of the second sample sets sampled multiple times.

[0095] The convergence rate acquisition module 504 is further configured to acquire a convergence rate variance and a convergence rate mean of the multiple samplings based on the convergence rates corresponding to the second sample sets sampled multiple times under each flow.

[0096] An establishment module 505 is configured to establish a convergence coefficient table based on each flow, the convergence rate variance and the convergence rate mean corresponding to each flow, and a sample number of the pre-set first sample set.

[0097] Further, the convergence rate acquisition module 504 is configured to:

[0098] count the number of samples upgraded to a new version in each of the second sample sets sampled multiple times under each flow;

[0099] calculate the convergence rate corresponding to each of the second sample sets sampled multiple times under each flow based on the number of samples upgraded to a new version in each of the second sample sets sampled multiple times under each flow and the number of samples included in the second sample set.

[0100] Further, the correction module 502 is configured to:

[0101] obtain the flow of the experimental group based on the data information of the experimental group and the convergence coefficient table;

[0102] obtain the convergence rate variance and the convergence rate mean corresponding to the flow of the experimental group from the flow-convergence coefficient query;

[0103] obtain the convergence rate of the experimental group based on the data information of the experimental group;

[0104] correct the value of the experimental index of the experimental group based on the convergence rate variance and the convergence rate mean corresponding to the flow of the experimental group and the convergence rate of the experimental group.

[0105] Further, the correction module 502 is configured to:

[0106] obtain the sample number of the experimental group from the data information of the experimental group;

[0107] obtain the sample number of the first preset sample set from the convergence coefficient table;

[0108] obtain the flow corresponding to the experimental group based on the sample number of the experimental group and the sample number of the first preset sample set.

[0109] Further, the correction module 502 is configured to:

[0110] obtain the sample number of the experimental group from the data information of the experimental group;

[0111] obtain the sample number of the experimental group that has upgraded the new version from the data information of the experimental group;

[0112] obtain the convergence rate of the experimental group based on the sample number of the experimental group that has upgraded the new version and the sample number of the experimental group.

[0113] The experimental index correction device 500 of the embodiment implements the implementation principle and technical effects of experimental index correction by using the above modules, which are the same as the implementation of the above related method embodiments. For details, refer to the description of the above related method embodiments, which will not be repeated here.

[0114] In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0115] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0118] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 601 performs various methods and processes described above, such as the above-described methods of the present disclosure. For example, in some embodiments, the above-described methods of the present disclosure can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the above-described methods of the present disclosure described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the above-described methods of the present disclosure by any other suitable means, such as by means of firmware.

[0120] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0121] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0124] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0126] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology described in the present disclosure are achieved.

[0127] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An experimental index correction method, comprising: obtaining a value of an experimental index of an experimental group in a release experiment and data information of the experimental group; the experimental index includes a test index after a new version is released, which is used to test the reaction of users after the new version is released; the test index includes the average length of time that the users use the application or the heat of the users to the application; the data information of the experimental group includes the sample number of the experimental group and the sample number of the experimental group that upgrades the new version of the application; based on a pre-established convergence coefficient table and the data information of the experimental group, the value of the experimental index is corrected; the convergence coefficient table includes pre-collected convergence-related coefficients, which are used to represent the sample occupancy rate of the application using the new version, which is related to the version upgrade behavior in the sample, and is irrelevant to the experimental index and the experimental effect corresponding to the experimental index; wherein, based on the pre-established convergence coefficient table and the data information of the experimental group, the value of the experimental index is corrected, comprising: based on the data information of the experimental group and the convergence coefficient table, obtaining the traffic of the experimental group; from the convergence coefficient table, the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group are obtained; based on the data information of the experimental group, the convergence rate of the experimental group is obtained; based on the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group, the convergence rate of the experimental group, the value of the experimental index of the experimental group is corrected.

2. The method of claim 1, wherein, Before the value of the experimental index is corrected based on the pre-established convergence coefficient table and the data information of the experimental group, the method further comprises: based on each traffic in a plurality of preset traffics, sampling a plurality of times from a preset first sample set to obtain a second sample set; obtaining the convergence rate corresponding to the second sample set sampled a plurality of times under each traffic; based on the convergence rate corresponding to the second sample set sampled a plurality of times under each traffic, obtaining the convergence rate variance and the convergence rate mean of the plurality of times of sampling under each traffic; based on each traffic, the convergence rate variance and the convergence rate mean corresponding to each traffic, and the sample number of the preset first sample set, the convergence coefficient table is established.

3. The method of claim 2, wherein, obtaining the convergence rate corresponding to the second sample set sampled a plurality of times under each traffic, comprising: counting the number of samples that upgrade the new version in the second sample set sampled each time under each traffic; based on the number of samples that upgrade the new version in the second sample set sampled each time under each traffic and the number of samples included in the second sample set, the convergence rate corresponding to the second sample set sampled each time under each traffic is calculated.

4. The method of claim 2, wherein, based on the data information of the experimental group and the convergence coefficient table, obtaining the traffic of the experimental group, comprising: from the data information of the experimental group, the sample number of the experimental group is obtained; from the convergence coefficient table, the sample number of the preset first sample set is obtained; based on the sample number of the experimental group and the sample number of the preset first sample set, the corresponding traffic of the experimental group is obtained.

5. The method of claim 1, wherein, Based on the data information of the experimental group, the convergence rate of the experimental group is obtained, including: From the data information of the experimental group, the sample number of the experimental group is obtained; From the data information of the experimental group, the sample number of the experimental group that upgrades the new version is obtained; Based on the sample number of the experimental group that upgrades the new version and the sample number of the experimental group, the convergence rate of the experimental group is obtained.

6. An experimental index correction device, comprising: an information acquisition module configured to acquire a value of an experimental index of an experimental group in a release experiment and data information of the experimental group; The experimental index includes a test index after the new version is released, which is used to test the reaction of users after the new version is released; The test index includes the average length of time that users use the application or the popularity of the application; The data information of the experimental group includes the sample number of the experimental group and the sample number of the experimental group that upgrades the new version of the application; A correction module is configured to correct the value of the experimental index based on a pre-established convergence coefficient table and the data information of the experimental group; The convergence coefficient table includes pre-collected convergence-related coefficients, which are used to represent the sample occupancy rate of the application using the new version, which is related to the version upgrade behavior in the sample, and is independent of the experimental index and the experimental effect corresponding to the experimental index; The correction module is configured to: Based on the data information of the experimental group and the convergence coefficient table, the traffic of the experimental group is obtained; From the convergence coefficient table, the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group are obtained; Based on the data information of the experimental group, the convergence rate of the experimental group is obtained; Based on the convergence rate variance and the convergence rate mean corresponding to the traffic of the experimental group, the convergence rate of the experimental group, and the value of the experimental index of the experimental group, the value of the experimental index of the experimental group is corrected.

7. The apparatus of claim 6, wherein, The device further comprises: A sampling module is configured to sample a second sample set from a pre-set first sample set based on each traffic in a plurality of pre-set traffics; A convergence rate acquisition module is configured to obtain the convergence rate corresponding to each of the second sample sets sampled multiple times under each of the traffics; The convergence rate acquisition module is further configured to obtain the convergence rate variance and the convergence rate mean of the multiple samplings under each of the traffics based on the convergence rates corresponding to the second sample sets sampled multiple times under each of the traffics; An establishment module is configured to establish the convergence coefficient table based on each of the traffics, the convergence rate variance and the convergence rate mean corresponding to each of the traffics, and the sample number of the first sample set.

8. The apparatus of claim 7, wherein, The convergence rate acquisition module is configured to: Count the sample number of the new version upgraded in each of the second sample sets sampled multiple times under each of the traffics; Calculate the convergence rate corresponding to each of the second sample sets sampled multiple times under each of the traffics based on the sample number of the new version upgraded in each of the second sample sets sampled multiple times under each of the traffics and the sample number included in the second sample set.

9. The apparatus of claim 7, wherein, The correction module is configured to: From the data information of the experimental group, the sample number of the experimental group is obtained; From the convergence coefficient table, a preset sample number of the first sample set is obtained; Based on the sample number of the experimental group and the preset sample number of the first sample set, a flow corresponding to the experimental group is obtained.

10. The apparatus of claim 6, wherein, The correction module is configured to: obtain the sample number of the experimental group from the data information of the experimental group; obtain the sample number of the experimental group that has upgraded to a new version from the data information of the experimental group; obtain the convergence rate of the experimental group based on the sample number of the experimental group that has upgraded to a new version and the sample number of the experimental group. 11.An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method according to any one of claims 1-5. 13.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-5.

Images