Application testing method and device

Abstract

The invention discloses an application testing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining behavior data of a user for a tested application, performing first-order Taylor expansion on non-sample mean value index data, performing normality test to obtain a first intermediate parameter, further calculating a target interestingness index in combination with a second intermediate parameter obtained after sample mean value index data processing, and determining the target interestingness index according to the target interestingness index. And calculating a difference value based on the target interestingness index and the baseline interestingness index, and performing test result analysis based on the difference value. According to the embodiment, the accuracy of data difference value calculation is improved, and then the accuracy of the test result of the application function is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an application testing method and device. Background Art

[0002] At present, in the field of Internet applications, for the development of enterprises, enterprises usually provide a variety of products to realize online business, expand the scale of enterprises, enrich business content, etc. In the process of product optimization, in order to improve the conversion rate and improve product functions, it is often necessary to conduct AB experiments before product optimization to evaluate the product optimization results. Through AB experiments, enterprises can accurately evaluate the actual effects of different strategies and determine the direction and method of product optimization based on the results.

[0003] When evaluating the experimental effect, the z test, t test and other test methods are generally used to test the indicators of user behavior data. However, the existing indicator test methods are mainly used to test the indicator data that meets the normal distribution. The test results for other indicator data are inaccurate, making it difficult to perform accurate analysis, resulting in inaccurate application test results. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an application testing method and device, which can improve the accuracy of data verification results, and further improve the accuracy of application testing results.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, there is provided an application testing method, comprising:

[0006] Obtaining user behavior data on the tested application, the behavior data includes sample mean indicator data and non-sample mean indicator data;

[0007] Perform first-order Taylor expansion on non-sample mean indicator data to obtain random variables, and perform normality test on random variables to obtain the first intermediate parameter;

[0008] Calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data;

[0009] The difference value of the interest index is calculated based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing the user's behavior data on the baseline application. The baseline application is a different version of the same application as the tested application.

[0010] The test results of the tested application are analyzed based on the difference values.

[0011] Optionally, before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data, the method further includes:

[0012] Perform a normality test on the sample mean indicator data to obtain the second intermediate parameter;

[0013] Calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data includes:

[0014] The target interest index is calculated based on the first intermediate parameter and the second intermediate parameter.

[0015] Optionally, the target interest index includes a numerator sample sum, a numerator sample square sum, a denominator sample sum, and a denominator sample square sum and a sum of the products of the numerator and the denominator, wherein the numerator sample and the denominator sample are generated by rewriting non-sample mean indicator data into the form of a quotient of two sample mean indicator data;

[0016] The difference value of the interest index is calculated based on the target interest index and the baseline interest index, including:

[0017] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, and denominator sample sum of squares and the product of the numerator and denominator included in the target interest index, calculate the sample mean and sample variance of the target interest index;

[0018] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares and the product of the numerator and denominator, the sample mean and sample variance of the baseline interest index are calculated;

[0019] Construct statistics based on the sample mean, sample variance and sample size of the target interestingness indicator and the sample mean, sample variance and sample size of the baseline interestingness indicator;

[0020] The difference value of the interestingness index is calculated through the constructed statistic.

[0021] Optionally, the method further comprises:

[0022] In the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, the sample variance calculation formula is reduced and simplified based on the sample size.

[0023] Optionally, the difference value of the interestingness index is calculated by constructing a statistic, including:

[0024] By using the constructed statistics, the probability of deviation between the target interest index and the baseline interest index, the deviation value and the confidence interval of the deviation value are calculated;

[0025] The difference value of the interest index is obtained based on the deviation probability, deviation value and confidence interval of the deviation value.

[0026] According to another aspect of an embodiment of the present invention, there is provided an application testing device, comprising:

[0027] An acquisition module is used to acquire the user's behavior data on the tested application, the behavior data including sample mean index data and non-sample mean index data;

[0028] A test module is used to perform a first-order Taylor expansion on non-sample mean indicator data to obtain a random variable, and perform a normality test on the random variable to obtain a first intermediate parameter;

[0029] A calculation module is used to calculate a target interest index based on the first intermediate parameter and a second intermediate parameter after processing the sample mean index data; a difference value of the interest index is calculated based on the target interest index and the baseline interest index, where the baseline interest index is obtained by processing the user's behavior data on the baseline application, and the baseline application is a different version of the same application as the application under test;

[0030] The analysis module is used for analyzing the test results of the tested application based on the difference value.

[0031] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the application testing method provided by an embodiment of the present invention.

[0032] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the application testing method provided by an embodiment of the present invention is implemented.

[0033] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the application testing method provided by an embodiment of the present invention is implemented.

[0034] An embodiment of the above invention has the following advantages or beneficial effects: it can perform Taylor expansion on the non-sample mean indicator data in the user's behavior data and retain the first-order terms therein, so as to convert it into the sum of multiple sample mean indicator data so that it satisfies the normal distribution, and then subsequent data verification operations can be performed. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated by combining the second intermediate parameter obtained by processing the sample mean indicator data. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby realizing the unified verification of the non-sample mean indicator data and the sample mean indicator data, and at the same time, improving the accuracy of the verification results of the non-sample mean indicator data.

[0035] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0037] Figure 1 is a schematic diagram of main steps of an application testing method according to an embodiment of the present invention;

[0038] Figure 2 It is an overall architecture diagram of an application test;

[0039] Figure 3 is a schematic diagram of main modules of an application testing device according to an embodiment of the present invention;

[0040] Figure 4 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0044] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0045] At present, different methods are used for AB experimental data verification according to different data types. For data of sample mean indicators, the data verification operation can be completed mainly through z test. For data of non-sample mean indicators, the data verification operation can be completed through chi-square test. However, chi-square test can only test the degree of fit, but cannot test the changes of parameters such as mean, and the test results are not accurate enough.

[0046] Figure 1 is a schematic diagram of the main steps of the application test method according to an embodiment of the present invention, such as Figure 1 As shown, the application testing method of the present invention mainly includes steps S101 to S105.

[0047] Step S101, obtaining the user's behavior data on the tested application, the behavior data including sample mean index data and non-sample mean index data.

[0048] The behavior data of users of the tested application is obtained as sample data, wherein the specific type of behavior data can be selected according to the actual test needs, for example, user access behavior data, click-through rate, etc. User behavior data can be divided into sample mean indicator data and non-sample mean indicator data according to the data type. Taking the overall business click-through rate indicator (non-sample mean indicator) as an example, the subsequent implementation process is explained.

[0049] Step S102, performing a first-order Taylor expansion on the non-sample mean indicator data to obtain a random variable, and performing a normality test on the random variable to obtain a first intermediate parameter.

[0050] For non-sample mean indicator data, in order to be able to use the z-test method for testing operations, the indicator data needs to be processed. First, the binary Taylor first-order expansion is performed using the delta method to obtain the expanded random variable, and the normality of the random variable is tested. After confirming that it retains normality, subsequent operations are performed.

[0051] Optionally, after Taylor expansion, only the first-order terms are retained and the remaining terms are deleted, which can effectively reduce the complexity of the calculation.

[0052] Specifically, the overall click-through rate It can be seen as two sample mean indicators constructed by dividing each other, that is, click pv (number of clicks) and exposure pv (number of exposures), and both types of indicators conform to the central limit theorem and meet the requirement that the asymptotic distribution obeys the normal distribution. The random variable is obtained by Taylor expansion The specific process is as follows:

[0053]

[0054] in, is the sample mean of the numerator (random variable), is the sample mean of the denominator (random variable), μ x is the population mean (constant) of the numerator, μ y is the denominator population mean (constant), is the numerator population variance (constant), is the denominator population variance (constant), and n is the sample size (constant). Among them, the constant is the data that can be directly calculated based on the obtained sample data. It is approximately the sum of two sample mean random variables, and its asymptotic normality is preserved, that is,

[0055]

[0056] Substituting the above expansion results into the variance calculation formula, we can get the following formula:

[0057]

[0058] The obtained variable expression and variance calculation formula are used as the first intermediate parameter of the calculation for the subsequent calculation of the test parameters.

[0059] Optionally, the sample size may be determined based on the amount of acquired behavioral data.

[0060] Step S103, calculating the target interest index based on the first intermediate parameter and the second intermediate parameter obtained after processing the sample mean index data.

[0061] After obtaining the first intermediate parameter, the second intermediate parameter obtained after processing the sample mean index data is obtained. The second intermediate parameter is the expression of the random variable and the variance calculation formula under the sample mean index.

[0062] The target interest index is calculated based on the first intermediate parameter and the second intermediate parameter. During the calculation, the target interest index is calculated based on the formula of the first intermediate parameter and the second intermediate parameter and the obtained behavior data, for example, the numerator sample and the like in the formula, which reflects the user's interest level and is used for subsequent calculation of the difference value.

[0063] Step S104, calculating a difference value of the interest index based on the target interest index and the baseline interest index, wherein the baseline interest index is obtained by processing the user's behavior data on the baseline application, and the baseline application is a different version of the same application as the tested application.

[0064] According to the calculated target interest index, the difference value of the AB experiment can be calculated, specifically the difference value between the target interest index and the baseline interest index (experimental group and control group). Among them, the experimental group and the control group can correspond to different versions of the same product. The calculation process of the baseline interest index is the same as that of the target interest index, but the behavioral data used in the calculation is different. For example, the behavioral data corresponding to the target interest index is for product 2.0; the behavioral data corresponding to the baseline interest index can be 1.0.

[0065] Optionally, the target interest index and the baseline interest index may be calculated simultaneously, or the baseline interest index data may be calculated preferentially.

[0066] Step S105 , analyzing the test results of the tested application based on the difference value.

[0067] After the difference value is calculated, an analysis result of the test result is generated according to the difference degree it reflects, for example, analyzing the user click rate of the new version of the application compared with the old version of the application.

[0068] According to the application testing method provided by the embodiment of the present invention, it is possible to perform Taylor expansion on the non-sample mean indicator data in the user's behavior data and retain the first-order terms therein, so as to convert it into the sum of multiple sample mean indicator data so that it satisfies the normal distribution, and then perform subsequent data verification operations. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated by combining the second intermediate parameter obtained by processing the sample mean indicator data. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby realizing the unified verification of the non-sample mean indicator data and the sample mean indicator data, and at the same time, improving the accuracy of the verification results of the non-sample mean indicator data.

[0069] Optionally, before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data, the method also includes: performing a normality test on the sample mean index data to obtain the second intermediate parameter; calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data, including: calculating the target interest index based on the first intermediate parameter and the second intermediate parameter. Before calculating the target interest index, it can also include performing a normality test on the sample mean index data to obtain the second intermediate parameter. Specifically, according to the variance calculation formula of the sample mean Where n is the sample size, that is, the denominator sample sum, is the existing input parameter (known data). Sample variance

[0070] Optionally, the calculation process of the first intermediate parameter is as follows:

[0071] Variance of the sample mean For the covariance Cov(X,Y) of the random variable, we can further bring in samples of user granularity and perform the following expansion transformation:

[0072]

[0073] Because each user is independent of each other, the covariance between users is 0, that is,

[0074]

[0075] The sample variance is therefore given by:

[0076]

[0077] Among them, X sum is the molecular sample and, is the existing input parameter, X square_sum is the sample sum of squares, that is, the numerator sample sum of squares, which is a newly added parameter; Y sum is the denominator sample and, is the existing input parameter, Y square_sum is the denominator sample sum of squares, and is a newly added parameter.

[0078] Optionally, the target interest index includes a numerator sample sum, a numerator sample square sum, a denominator sample sum, a denominator sample square sum and a numerator-denominator product sum, wherein the numerator sample and the denominator sample are generated by rewriting non-sample mean indicator data into the form of a quotient of two sample mean indicator data; the difference value of the interest index is calculated based on the target interest index and the baseline interest index, including: calculating the sample mean and sample variance of the target interest index based on the numerator sample sum, the numerator sample square sum, the denominator sample sum, the denominator sample square sum and the numerator-denominator product sum included in the target interest index; calculating the sample mean and sample variance of the baseline interest index based on the numerator sample sum, the numerator sample square sum, the denominator sample sum, the denominator sample square sum and the numerator-denominator product sum included in the baseline interest index; constructing a statistic based on the sample mean, sample variance and sample size of the target interest index, as well as the sample mean, sample variance and sample size of the baseline interest index; and calculating the difference value of the interest index using the constructed statistic.

[0079] The target interest index includes the numerator sample sum, the numerator sample square sum, the denominator sample sum, and the sum of the denominator sample square sum and the product of the numerator and denominator. When verifying, both the sample mean index and the non-sample mean index can be calculated through these five parameters to obtain the test results and achieve the unification of the hypothesis test framework. Among them, the division of the numerator and denominator is based on rewriting the non-sample mean index data into the form of the quotient of two sample mean index data. In order to achieve the unification of the verification of the sample mean index data and the non-sample mean index data, the sample mean index can be regarded as a term with a denominator of 1, which is used to ensure the unification of the formula form of the non-sample mean index. At this time, in the calculation formula of the sample mean index data, the denominator sample sum is the sample size, and the denominator sample square sum is equal to the denominator sample sum, that is: The sum of the products of the numerator and denominator is equal to the sum of the numerator samples, that is:

[0080] The values ​​of the above parameters are calculated based on the user's behavior data, and according to the values ​​of the above parameters, the sample mean and sample variance of the target interest index and the sample mean and sample variance of the baseline interest index are calculated using the formula of the first intermediate parameter or the second intermediate parameter. The calculation of the difference value is realized by constructing statistics.

[0081] Construct the z statistic of the z test, and then calculate the p value and confidence interval. Below 1 is the experimental group, 2 is the control group, z is the constructed z statistic (random variable), and the difference value is calculated based on z.

[0082]

[0083] The difference value includes the deviation probability p, the deviation value diff and the confidence interval of the deviation value.

[0084] Optionally, the method also includes: in the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, reducing and simplifying the sample variance calculation formula based on the sample size.

[0085] For non-sample mean indicators, in the case of large sample sizes, n can be used instead of n-1 for simplification, that is,

[0086]

[0087] Where n is the sample size, which is negated in the calculation, X sum is the molecular sample and, Y sum is the denominator sample and is the existing input parameter, X square_sum is the sum of squares of the numerator sample, Y square_sum is the denominator sample sum of squares, XY product_sum The sum of the product of the numerator and denominator is the newly added parameter.

[0088] For sample mean indicator data, you can also pass in 5 parameters to calculate the variance without affecting the result. Among them, the denominator sample sum is the sample size, and the denominator sample square sum is equal to the denominator sample sum, that is: The sum of the products of the numerator and denominator is equal to the sum of the numerator samples, that is: The variance is now:

[0089]

[0090] Variance of the sample mean Where n is the sample size, that is, the denominator sample sum, is the input parameter. Sample variance

[0091] When the sample size is large, n is used instead of n-1, and the sample variance is as follows.

[0092]

[0093] According to the application testing method provided by the embodiment of the present invention, it is possible to perform approximate simplification when the sample size is large enough to reduce the amount of calculation. Among them, for non-sample mean indicator data, through approximate simplification, the sample size parameters of the numerator and denominator in the variance calculation formula can be reduced and deleted, further reducing the amount of calculation, reducing the calculation cost, and improving the calculation efficiency.

[0094] Optionally, the difference value of the interest index is calculated by the constructed statistic, including: calculating the deviation probability, deviation value and confidence interval of the deviation value between the target interest index and the baseline interest index by the constructed statistic; and obtaining the difference value of the interest index based on the deviation probability, deviation value and confidence interval of the deviation value.

[0095] Based on the above calculation process, we can get the mean and variance of the indicators of the experimental group and the control group in the AB experiment. At the same time, combined with the corresponding sample size, we can construct the z statistic of the z test, and then calculate the p value and confidence interval.

[0096] Below 1 is the experimental group (the application corresponding to the target interest index), 2 is the control group (the application corresponding to the baseline interest index), and z is the constructed z statistic (random variable)

[0097]

[0098] If the null hypothesis is true, then When the sample size is large, the actual calculated sample variance is used to estimate the population variance, the sample mean is used to estimate the population mean, and the sample covariance is used to estimate the population covariance. The absolute value of the z statistic is substituted into the standard normal cumulative distribution function to calculate the P value.

[0099] pValue=2(1-Φ(|z|))

[0100] Further, calculate the diff and confidence interval of the indicator, and the confidence interval of the absolute diff of the indicator

[0101]

[0102] Among them, α is the probability of making a first-type error, which is generally 0.05, and the confidence interval of the absolute diff is:

[0103]

[0104] Absolute diff confidence interval for sample mean indicators

[0105]

[0106] When the sample size is large, the actual calculated sample variance is used to estimate the population variance, and the sample mean is used to estimate the population mean. For the absolute diff confidence interval of non-sample mean indicators, the variance calculated by the delta method can be used.

[0107] Secondly, the delta method is used to calculate the confidence interval of the relative diff. That is

[0108]

[0109] The experimental group and the control group are independent of each other, and their covariance is 0, that is, there exists:

[0110]

[0111] The confidence interval of the relative diff is:

[0112]

[0113] Relative diff confidence interval of sample mean indicators:

[0114]

[0115] When the sample size is large, the actual calculated sample variance can be used to estimate the population variance, and the sample mean can be used to estimate the population mean; at the same time, the relative diff confidence interval of non-sample mean indicators can be replaced by the approximate variance calculated using the delta method.

[0116] According to the application testing method provided by the embodiment of the present invention, it is possible to implement unified verification of non-sample mean indicator data and sample mean indicator data through a unified architecture. Specifically, for non-sample mean indicator data, it is possible to retain the first-order terms therein through Taylor expansion, so that it can be converted into the sum of multiple sample mean indicator data, and then subsequent data verification operations can be performed. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated in combination with the second intermediate parameter. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby achieving unified verification of different indicator data and improving the accuracy of the verification results. In addition, when calculating the difference value, by performing approximate calculations of the delta method on different indicators, the uniformity of different indicators can be guaranteed, thereby ensuring the uniformity of the overall inspection architecture.

[0117] like Figure 2The figure shows the overall architecture of an application test. The prerequisites for the implementation of this architecture are: each user granularity is independent of each other, and the users in the same experimental group come from the same population independent and identically distributed; the sample size is large enough, and in actual calculation, the sample size n is used instead of n-1 to simplify the calculation; the sample size is large enough, and in actual calculation, the sample variance is considered as the population variance. The variance is known; the index values ​​of various samples are relatively concentrated, and there are no very extreme values, or they have been eliminated. The non-sample mean index is first subjected to Taylor first-order expansion and normality test. After the test is passed, the formula obtained by the normality test of the sample mean index is converted into the same form by the delta approximation method, and the index is calculated uniformly. Calculate the numerator, numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares, and numerator-denominator product and five parameters, and use them as input parameters to calculate the mean, variance, and sample size of the experimental group and the control group, and construct the z statistic, calculate the p value, diff value and confidence interval, and perform the final hypothesis test result analysis based on the calculation results.

[0118] Figure 3 is a schematic diagram of the main modules of the application testing device 300 according to an embodiment of the present invention. Figure 3 As shown, it includes an acquisition module 301 , a verification module 302 , a calculation module 303 and an analysis module 304 .

[0119] An acquisition module 301 is used to acquire user behavior data on the tested application, where the behavior data includes sample mean index data and non-sample mean index data;

[0120] A test module 302 is used to perform a first-order Taylor expansion on the non-sample mean indicator data to obtain a random variable, and perform a normality test on the random variable to obtain a first intermediate parameter;

[0121] A calculation module 303 is used to calculate a target interest index based on the first intermediate parameter and a second intermediate parameter after processing the sample mean index data; a difference value of the interest index is calculated based on the target interest index and the baseline interest index, where the baseline interest index is obtained by processing the user's behavior data on the baseline application, and the baseline application is a different version of the same application as the application under test;

[0122] The analysis module 304 is used to analyze the test results of the tested application based on the difference value.

[0123] According to the application testing device provided by the embodiment of the present invention, it is possible to perform Taylor expansion on the non-sample mean indicator data in the user's behavior data and retain the first-order terms therein, so as to realize conversion into the sum of multiple sample mean indicator data so that it satisfies the normal distribution, and then to perform subsequent data verification operations. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated by combining the second intermediate parameter obtained by processing the sample mean indicator data. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby realizing the unified verification of the non-sample mean indicator data and the sample mean indicator data, and at the same time, improving the accuracy of the verification results of the non-sample mean indicator data.

[0124] Optionally,

[0125] The test module 302 is also used to perform a normality test on the sample mean index data to obtain a second intermediate parameter;

[0126] The calculation module 303 is further used to calculate the target interest index based on the first intermediate parameter and the second intermediate parameter.

[0127] Optionally, the target interest index includes a numerator sample sum, a numerator sample square sum, a denominator sample sum, and a denominator sample square sum and a sum of the products of the numerator and the denominator, wherein the numerator sample and the denominator sample are generated by rewriting non-sample mean indicator data into the form of a quotient of two sample mean indicator data;

[0128] The calculation module 303 is further used for:

[0129] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, and denominator sample sum of squares and the product of the numerator and denominator included in the target interest index, calculate the sample mean and sample variance of the target interest index;

[0130] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares and the product of the numerator and denominator, the sample mean and sample variance of the baseline interest index are calculated;

[0131] Construct statistics based on the sample mean, sample variance and sample size of the target interestingness indicator and the sample mean, sample variance and sample size of the baseline interestingness indicator;

[0132] The difference value of the interestingness index is calculated through the constructed statistic.

[0133] Optionally,

[0134] The calculation module 303 is also used to reduce and simplify the sample variance calculation formula based on the sample size in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold during the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index.

[0135] The calculation module 303 is also used to calculate the deviation probability, deviation value and confidence interval of the deviation value between the target interest index and the baseline interest index through the constructed statistics; and obtain the difference value of the interest index based on the deviation probability, deviation value and confidence interval of the deviation value.

[0136] According to the application testing device provided by the embodiment of the present invention, it is possible to implement unified verification of non-sample mean indicator data and sample mean indicator data through a unified architecture. Specifically, for non-sample mean indicator data, it is possible to retain the first-order terms therein through Taylor expansion, so that it can be converted into the sum of multiple sample mean indicator data, and then subsequent data verification operations can be performed. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated in combination with the second intermediate parameter. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby achieving unified verification of different indicator data and improving the accuracy of the verification results. In addition, when calculating the difference value, by performing approximate calculations of the delta method on different indicators, the uniformity of different indicators can be guaranteed, thereby ensuring the uniformity of the overall inspection architecture.

[0137] Figure 4 An exemplary system architecture 400 is shown to which an application testing method or an application testing device according to an embodiment of the present invention can be applied.

[0138] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, 403, network 404 and server 405. Network 404 is used to provide a medium for communication links between terminal devices 401, 402, 403 and server 405. Network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0139] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0140] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0141] The server 405 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 401, 402, and 403. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0142] It should be noted that the application testing method provided in the embodiment of the present invention is generally executed by the server 405 , and accordingly, the application testing device is generally disposed in the server 405 .

[0143] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0144] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 5 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0145] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0147] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0148] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition module, a verification module, a calculation module, and an analysis module. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves. For example, the acquisition module may also be described as a "module for acquiring user behavior data on the application under test."

[0151] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0152] Obtaining user behavior data on the tested application, the behavior data includes sample mean indicator data and non-sample mean indicator data;

[0153] Perform first-order Taylor expansion on non-sample mean indicator data to obtain random variables, and perform normality test on random variables to obtain the first intermediate parameter;

[0154] Calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data;

[0155] The difference value of the interest index is calculated based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing the user's behavior data on the baseline application. The baseline application is a different version of the same application as the tested application.

[0156] The test results of the tested application are analyzed based on the difference values.

[0157] According to the technical solution of the embodiment of the present invention, it is possible to perform Taylor expansion on the non-sample mean indicator data in the user's behavior data and retain the first-order terms therein, so as to convert it into the sum of multiple sample mean indicator data so that it satisfies the normal distribution, and then to perform subsequent data verification operations. The first intermediate parameter is generated by the random variable obtained after the expansion, and the target interest index is calculated by combining the second intermediate parameter obtained by processing the sample mean indicator data. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used to analyze the test results, thereby realizing the unified verification of the non-sample mean indicator data and the sample mean indicator data, and at the same time, improving the accuracy of the verification results of the non-sample mean indicator data.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An application testing method, characterized in that: include: Obtaining user behavior data on the tested application, wherein the behavior data includes sample mean indicator data and non-sample mean indicator data; Performing a first-order Taylor expansion on the non-sample mean indicator data to obtain a random variable, and performing a normality test on the random variable to obtain a first intermediate parameter; Calculating a target interest index based on the first intermediate parameter and a second intermediate parameter obtained by processing the sample mean index data; Calculating a difference value of the interest index based on the target interest index and a baseline interest index, wherein the baseline interest index is obtained by processing user behavior data of a baseline application, and the baseline application is a different version of the same application as the application under test; A test result analysis of the tested application is performed based on the difference value.

2. The method according to claim 1, characterized in that Before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter processed from the sample mean index data, the method further includes: Performing a normality test on the sample mean index data to obtain a second intermediate parameter; The calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data includes: A target interest index is calculated based on the first intermediate parameter and the second intermediate parameter.

3. The method according to claim 1, characterized in that The target interest index includes a numerator sample sum, a numerator sample square sum, a denominator sample sum, and a denominator sample square sum and a sum of the products of the numerator and the denominator, wherein the numerator sample and the denominator sample are generated by rewriting non-sample mean index data into the form of a quotient of two sample mean index data; The calculating a difference value of the interest index based on the target interest index and the baseline interest index includes: Calculate the sample mean and sample variance of the target interest index based on the numerator sample sum, the numerator sample sum of squares, the denominator sample sum, and the sum of the product of the denominator and the numerator and the denominator included in the target interest index; Calculate the sample mean and sample variance of the baseline interest index based on the numerator sample sum, the numerator sample sum of squares, the denominator sample sum, and the product of the denominator and the denominator sample sum of squares; Constructing statistics based on the sample mean, sample variance and sample size of the target interest index and the sample mean, sample variance and sample size of the baseline interest index; The difference value of the interestingness index is calculated through the constructed statistic.

4. The method according to claim 3, characterized in that: The method further comprises: In the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, the sample variance calculation formula is reduced and simplified based on the sample size.

5. The method according to claim 3, characterized in that: The calculation of the difference value of the interest index by constructing the statistic includes: Calculate the probability of deviation, deviation value and confidence interval of the deviation value between the target interest index and the baseline interest index by using the constructed statistics; A difference value of the interest index is obtained based on the deviation degree probability, the deviation value and the confidence interval of the deviation value.

6. An application testing device, characterized in that: include: An acquisition module is used to acquire the user's behavior data on the tested application, wherein the behavior data includes sample mean index data and non-sample mean index data; A testing module, used for performing a first-order Taylor expansion on the non-sample mean indicator data to obtain a random variable, and performing a normality test on the random variable to obtain a first intermediate parameter; a calculation module, configured to calculate a target interest index based on the first intermediate parameter and a second intermediate parameter processed from the sample mean index data; and to calculate a difference value of the interest index based on the target interest index and a baseline interest index, wherein the baseline interest index is obtained by processing user behavior data of a baseline application, and the baseline application is a different version of the same application as the application under test; An analysis module is used to analyze the test results of the tested application based on the difference value.

7. The device according to claim 6, characterized in that: The device also includes: The test module is further used to perform a normality test on the sample mean index data to obtain a second intermediate parameter; The calculation module is further used to calculate a target interest index based on the first intermediate parameter and the second intermediate parameter.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images