An application testing method and apparatus

Abstract

The application discloses an application testing method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: obtaining behavior data of a user to a tested application, performing first-order Taylor expansion on non-sample mean type index data in the behavior data, and performing normality test to obtain a first intermediate parameter; further, combining a second intermediate parameter obtained by processing sample mean type index data, calculating a target interest degree index, calculating a difference value based on the target interest degree index and a baseline interest degree index, and performing test result analysis based on the difference value. The implementation improves the accuracy of data difference value calculation, and further guarantees the accuracy of the test result of the application function.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an application testing method and apparatus. Background Technology

[0002] Currently, in the field of internet applications, companies typically offer multiple products to facilitate online business operations, thereby expanding their scale and enriching their business offerings. During product optimization, to improve conversion rates and enhance product functionality, A / B testing is often conducted before optimization to evaluate the results. Through A / B testing, companies can accurately assess the actual effectiveness of different strategies and determine the direction and methods of product optimization based on the results.

[0003] When evaluating the effectiveness of experiments, methods such as z-tests and t-tests are generally used to test user behavior data. However, existing indicator testing methods are mainly used to test indicators that follow a normal distribution. For other indicators, the test results are inaccurate, making accurate analysis difficult and resulting in inaccurate application test results. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an application testing method and apparatus that can improve the accuracy of data verification results, thereby improving the accuracy of application testing results.

[0005] To achieve the above objectives, according to one aspect of the present invention, an application testing method is provided, comprising:

[0006] Acquire user behavior data for the tested application. This behavior data includes sample mean-based metrics and non-sample mean-based metrics.

[0007] The first Taylor expansion of non-sample mean index data is used to obtain random variables, and the normality test of the random variables is used to obtain the first intermediate parameter.

[0008] The target interest index is calculated based on the first intermediate parameter and the second intermediate parameter after processing the sample mean class index data.

[0009] The difference in interest metrics is calculated based on the target interest metric and the baseline interest metric. The baseline interest metric is obtained by processing user behavior data on the baseline application. The baseline application is the same application but a different version from the application being tested.

[0010] Analysis of test results for the application under test based on the difference values.

[0011] Optionally, before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean-type index data, the method further includes:

[0012] A normality test is performed on the sample mean-type index data to obtain the second intermediate parameter;

[0013] Based on the first intermediate parameter and the second intermediate parameter after processing the sample mean-type index data, the target interest index is calculated, including:

[0014] The target interest index is calculated based on the first and second intermediate parameters.

[0015] Optionally, the target interest index includes the sum of numerator samples, the sum of squares of numerator samples, the sum of denominator samples, the sum of squares of denominator samples and the product of numerator and denominator, wherein the numerator samples and denominator samples are generated by rewriting the non-sample mean class index data into the form of the quotient of two sample mean class index data;

[0016] The difference in interest indices is calculated based on the target interest index and the baseline interest index, including:

[0017] Based on the sum of numerator samples, sum of squares of numerator samples, sum of denominator samples, sum of squares of denominator samples and sum of products of numerator and denominator included in the target interest index, calculate the sample mean and sample variance of the target interest index.

[0018] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares and the product of numerator and denominator included in the baseline interest index, calculate the sample mean and sample variance of the baseline interest index.

[0019] Based on the sample mean, sample variance, and sample size of the target interest index, as well as the sample mean, sample variance, and sample size of the baseline interest index, a statistic is constructed.

[0020] The difference in interest index is calculated using constructed statistics.

[0021] Optionally, the method further includes:

[0022] In the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, the sample variance calculation formula is simplified based on the sample size.

[0023] Optionally, the difference value of the interest index is calculated using the constructed statistics, including:

[0024] By constructing statistics, we calculate the probability of deviation between the target interest index and the baseline interest index, the deviation value, and the confidence interval of the deviation value.

[0025] The difference value of the interest index is obtained based on the probability of deviation, the deviation value, and the confidence interval of the deviation value.

[0026] According to another aspect of the present invention, an application testing apparatus is provided, comprising:

[0027] The acquisition module is used to acquire user behavior data on the application under test. The behavior data includes sample mean index data and non-sample mean index data.

[0028] The testing module is used to perform a first-order Taylor expansion on non-sample mean index data to obtain random variables, and to perform a normality test on the random variables to obtain the first intermediate parameter.

[0029] The calculation module is used to calculate the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data; and to calculate the difference value of the interest index based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing user behavior data of the baseline application. The baseline application is the same application with a different version than the application being tested.

[0030] The analysis module is used to analyze the test results of the application under test based on the difference values.

[0031] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the application testing method provided in the embodiments of the present invention.

[0032] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the application testing method provided in the embodiments of the present invention.

[0033] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the application testing method provided in the embodiments of the present invention.

[0034] One embodiment of the above invention has the following advantages or beneficial effects: it can transform non-sample mean-based indicator data in user behavior data into a sum of multiple sample mean-based indicator data by performing Taylor expansion and retaining the first-order terms, thus satisfying a normal distribution and enabling subsequent data verification operations. A first intermediate parameter is generated from the expanded random variable, and a second intermediate parameter obtained by processing the sample mean-based indicator data is combined to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for test result analysis. This achieves unified verification of both non-sample mean-based and sample mean-based indicator data, while improving the accuracy of verification results for non-sample mean-based indicator data.

[0035] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0036] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0037] Figure 1 This is a schematic diagram of the main steps of the application testing method according to an embodiment of the present invention;

[0038] Figure 2 This is an overall architecture diagram for application testing;

[0039] Figure 3 This is a schematic diagram of the main modules of the application testing device according to an embodiment of the present invention;

[0040] Figure 4 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0041] Figure 5 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0042] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0043] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0044] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0045] Currently, the methods used for A / B test data validation vary depending on the data type. For data with sample means, the z-test is primarily used for data validation. For data without sample means, the chi-square test is used. However, the chi-square test only examines the goodness of fit and cannot detect changes in parameters such as the mean, resulting in less accurate results.

[0046] Figure 1 This is a schematic diagram of the main steps of the application testing method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the application testing method of the present invention mainly includes steps S101 to S105.

[0047] Step S101: Obtain user behavior data for the tested application. The behavior data includes sample mean index data and non-sample mean index data.

[0048] The user behavior data of the application under test is obtained as sample data. The specific type of behavior data can be selected according to the actual testing needs, such as user access behavior data, click-through rate, etc. User behavior data can be divided into sample mean-based metrics and non-sample mean-based metrics. The following explanation uses the overall business click-through rate metric (a non-sample mean-based metric) as an example to illustrate the subsequent implementation process.

[0049] Step S102: Perform a first-order Taylor expansion on the non-sample mean index data to obtain random variables, and perform a normality test on the random variables to obtain the first intermediate parameter.

[0050] For non-sample mean index data, data processing is required to perform z-tests. First, a bivariate Taylor first-order expansion is performed using the Delta method to obtain the expanded random variable. The normality of this random variable is then tested to confirm that it retains normality before proceeding with subsequent operations.

[0051] Alternatively, after the Taylor expansion, retaining only the first-order terms and deleting the remaining terms can effectively reduce the computational complexity.

[0052] Specifically, overall click-through rate It can be viewed as being constructed by dividing two sample mean-based indicators, specifically by dividing click PV (number of clicks) by impression PV (number of impressions). Both of these indicators conform to the Central Limit Theorem and satisfy the requirement that the asymptotic distribution follows a normal distribution. A random variable is obtained through Taylor expansion. The specific process is as follows:

[0053]

[0054] in, The mean of the molecular sample (random variable), μ is the sample mean (random variable) in the denominator. x μ is the total molecular mean (constant). y The denominator is the population mean (constant). For the total variance of the molecule (constant), Let be the population variance in the denominator (constant), and n be the sample size (constant). Here, the constant is the data that can be directly calculated based on the obtained sample data. This is achieved through the expanded random variable... Approximating the sum of two sample mean random variables, its asymptotic distribution's normality is preserved, i.e.:

[0055]

[0056] Substituting the above expansion results into the variance calculation formula yields the following formula:

[0057]

[0058] The obtained variable expressions and variance calculation formulas are used as the first intermediate parameters for subsequent calculation of test parameters.

[0059] Optionally, the sample size can be determined based on the amount of behavioral data acquired.

[0060] Step S103: Calculate the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data.

[0061] After obtaining the first intermediate parameter, the second intermediate parameter is obtained by processing the sample mean index data. This second intermediate parameter is the expression for the random variable and the formula for calculating the variance under the sample mean index.

[0062] The target interest index is calculated based on the first and second intermediate parameters. During calculation, the target interest index is derived from the formula using the first and second intermediate parameters, along with acquired behavioral data. For example, the molecular samples in the formula, etc., reflect the user's level of interest and are used for subsequent difference value calculations.

[0063] Step S104: Calculate the difference value of the interest index based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing user behavior data of the baseline application. The baseline application is the same application but a different version from the application being tested.

[0064] The calculated target interest index can be used to calculate the difference between A / B experiments, specifically the difference between the target interest index and the baseline interest index (experimental group and control group). The experimental group and control group can correspond to different versions of the same product. The calculation process for the baseline interest index is the same as that for the target interest index, but the behavioral data used is different. For example, the behavioral data corresponding to the target interest index is product 2.0; the behavioral data corresponding to the baseline interest index can be product 1.0.

[0065] Optionally, the target interest index and the baseline interest index can be calculated simultaneously, or the baseline interest index data can be calculated first.

[0066] Step S105: Analyze the test results of the application under test based on the difference values.

[0067] After calculating the difference values, analysis results are generated based on the degree of difference they reflect. For example, analyzing user click-through rates of the new version of the application compared to the old version.

[0068] The application testing method provided by this invention can transform non-sample mean-based indicator data in user behavior data into a sum of multiple sample mean-based indicator data by performing Taylor expansion and retaining the first-order terms, thus ensuring a normal distribution and enabling subsequent data verification. A first intermediate parameter is generated from the expanded random variable, and a second intermediate parameter obtained from processing the sample mean-based indicator data is used to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for analyzing the test results. This achieves unified verification of both non-sample mean-based and sample mean-based indicator data, while improving the accuracy of verification results for non-sample mean-based indicator data.

[0069] Optionally, before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean-type indicator data, the method further includes: performing a normality test on the sample mean-type indicator data to obtain the second intermediate parameter; and calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean-type indicator data, including: calculating the target interest index based on the first intermediate parameter and the second intermediate parameter. Before calculating the target interest index, the method may further include performing a normality test on the sample mean-type indicator data to obtain the second intermediate parameter. Specifically, this is done according to the variance calculation formula for the sample mean. Where n is the sample size, i.e., the denominator is the sample sum, which represents the known input parameters (data). Sample variance

[0070] Optionally, the calculation process for the first intermediate parameter is as follows:

[0071] variance of sample mean For the covariance Cov(X,Y) of the random variables, we can further incorporate user-level samples and perform the following expansion transformation:

[0072]

[0073] Since each user is independent, the covariance between users is 0, i.e.

[0074]

[0075] Therefore, the sample variance is as follows.

[0076]

[0077] Among them, X sum Let X be the molecular sample and X be the existing input parameters. square_sum The sum of squares of the samples, i.e., the sum of squares of the numerator samples, is the newly added input parameter; Y sum Let Y be the denominator sample sum, Y be the existing input parameter. square_sum is the sample sum of squares in the denominator, and is the newly added input parameter.

[0078] Optionally, the target interest index includes the sum of numerator samples, the sum of squares of numerator samples, the sum of denominators samples, and the sum of squares of denominators samples plus the product of numerator and denominator. The numerator and denominator samples are generated by rewriting non-mean index data into the quotient of two mean index data. The difference value of the interest index is calculated based on the target interest index and the baseline interest index, including: calculating the sample mean and sample variance of the target interest index based on the sum of numerator samples, the sum of squares of numerator samples, the sum of denominators samples, the sum of squares of denominators samples, and the product of numerator and denominator; calculating the sample mean and sample variance of the baseline interest index based on the sum of numerator samples, the sum of squares of numerator samples, the sum of denominators samples, the sum of squares of denominators samples, and the product of numerator and denominator; constructing a statistic based on the sample mean, sample variance, and sample size of the target interest index, and the sample mean, sample variance, and sample size of the baseline interest index; and calculating the difference value of the interest index using the constructed statistic.

[0079] The target interest index includes the numerator sample sum, numerator sample sum of squares, denominator sample sum, and the sum of the products of the denominator sample sum of squares and the numerator and denominator. During validation, both sample mean-based and non-sample mean-based indices can be calculated using these five parameters to obtain the test results, achieving a unified hypothesis testing framework. The division of the numerator and denominator is based on rewriting the non-sample mean-based index data into the quotient form of two sample mean-based index data. To achieve uniformity in the validation of sample mean-based and non-sample mean-based index data, the sample mean-based index can be considered as a term with a denominator of 1, ensuring consistency in the formula form between the two indices. In this case, in the calculation formula for the sample mean-based index data, the denominator sample sum represents the sample size, and the denominator sample sum of squares is equal to the denominator sample sum, i.e.: The sum of the products of the numerator and denominator is equal to the sum of the sample numerator, that is:

[0080] The values ​​of the above parameters are calculated based on user behavior data. Then, using the formula for either the first or second intermediate parameter, the sample mean and sample variance of the target interest index, as well as the sample mean and sample variance of the baseline interest index, are calculated. The difference values ​​are calculated by constructing statistical measures.

[0081] Construct the z-statistic for the z-test, and then calculate the p-value and confidence interval. Below, 1 represents the experimental group, 2 represents the control group, and z is the constructed z-statistic (random variable). The difference value is calculated based on z.

[0082]

[0083] The difference values ​​include the probability of deviation p, the deviation value diff, and the confidence interval of the deviation value.

[0084] Optionally, the method further includes: in the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, reducing and simplifying the sample variance calculation formula based on the sample size.

[0085] For indicators that are not based on the sample mean, with a large sample size, we can simplify them by replacing n-1 with n, that is:

[0086]

[0087] Where n is the sample size, which is canceled out in the calculation, X sum For molecular samples and, Y sum Let X be the sample in the denominator and the existing input parameters. square_sum Y is the sum of squares of the molecular samples. square_sum XY is the sample sum of squares in the denominator. product_sum The product of the numerator and denominator is the sum of the products, and the newly added input parameter is the sum of the products of the numerator and denominator.

[0088] For index data such as sample mean, five parameters can be passed to calculate the variance without affecting the result. Here, the denominator is the sample size, and the sum of squares in the denominator is equal to the sum of squares in the denominator, i.e.: The sum of the products of the numerator and denominator is equal to the sum of the sample numerator, that is: The variance at this point is:

[0089]

[0090] variance of sample mean Where n is the sample size, i.e., the denominator is the sample sum, which is the existing input parameter. Sample variance

[0091] With a large sample size, replacing n-1 with n, the sample variance is as follows.

[0092]

[0093] The application testing method provided by the embodiments of the present invention can perform approximate simplification when the sample size is large enough, thereby reducing the amount of computation. Specifically, for non-sample mean index data, approximate simplification can reduce the sample size parameters in the numerator and denominator of the variance calculation formula by eliminating them, further reducing the amount of computation, lowering computational costs, and improving computational efficiency.

[0094] Optionally, the difference value of the interest index is calculated by the constructed statistics, including: calculating the probability of deviation, the deviation value, and the confidence interval of the deviation value between the target interest index and the baseline interest index by the constructed statistics; and obtaining the difference value of the interest index based on the probability of deviation, the deviation value, and the confidence interval of the deviation value.

[0095] Based on the above calculation process, the mean and variance of the indicators of the experimental group and the control group in the AB experiment can be obtained. At the same time, the z-statistic of the z-test can be constructed by combining the corresponding sample size, and then the p-value and confidence interval can be calculated.

[0096] Below, 1 represents the experimental group (the application corresponding to the target interest index), 2 represents the control group (the application corresponding to the baseline interest index), and z represents the constructed z-statistic (random variable).

[0097]

[0098] If the null hypothesis holds, then it is considered that With a large sample size, the population variance is estimated using the actual calculated sample variance, the population mean is estimated using the sample mean, and the population covariance is estimated using the sample covariance. The absolute value of the z-statistic is then substituted into the standard normal cumulative distribution function to calculate the p-value.

[0099] pValue = 2(1-Φ(|z|))

[0100] Furthermore, calculate the diff and confidence interval of the indicator, and the confidence interval of the absolute diff of the indicator.

[0101]

[0102] Where α is the probability of committing a Type I error, typically 0.05, and the confidence interval for the absolute diff is:

[0103]

[0104] Absolute diff confidence intervals for sample mean-based indicators

[0105]

[0106] With large sample sizes, the population variance can be estimated using the actually calculated sample variance, and the population mean can be estimated using the sample mean. For the absolute diff confidence interval of non-sample mean indicators, the variance calculated by the delta method can be substituted into the equation.

[0107] Secondly, the delta method is used to calculate the confidence interval for the relative diff. That is

[0108]

[0109] The experimental group and the control group are independent of each other, and their covariance is 0, meaning that:

[0110]

[0111] The confidence interval relative to diff is:

[0112]

[0113] Confidence intervals for relative diff values ​​of sample mean-based indicators:

[0114]

[0115] With a large sample size, the population variance can be estimated using the actual calculated sample variance, and the population mean can be estimated using the sample mean. Meanwhile, the relative diff confidence intervals of non-sample mean indicators can be obtained by substituting the approximate variance calculated using the delta method.

[0116] The application testing method provided by this invention can achieve unified verification of non-sample mean index data and sample mean index data through a unified architecture. Specifically, for non-sample mean index data, Taylor expansion can be used to retain the first-order terms, transforming it into the sum of multiple sample mean index data, thus enabling subsequent data verification operations. A first intermediate parameter is generated from the expanded random variable, and a second intermediate parameter is combined to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for analyzing the test results. This achieves unified verification of different index data and improves the accuracy of the verification results. Furthermore, when calculating the difference value, the Delta method is used to approximate the calculation of different indicators, ensuring the uniformity of different indicators and thus guaranteeing the uniformity of the overall verification architecture.

[0117] like Figure 2The diagram shows the overall architecture of an application test. The prerequisites for this architecture are: each user granularity is independent, and users in the same experimental group come from the same independent and identically distributed population; the sample size is large enough that, in actual calculations, sample size n is used instead of n-1 for simplified calculations; the sample size is large enough that, in actual calculations, the sample variance is considered as the population variance, given the known variance; and the index values ​​of various samples are relatively concentrated, with no extremely extreme values, or these values ​​have been removed. Non-sample mean indices are first expanded using Taylor series and then tested for normality. After passing the test, the formulas obtained from the normality test of sample mean indices are converted to the same form using the Delta approximation method, and then the indices are calculated uniformly. Five parameters are calculated: the numerator, the numerator sample sum, the numerator sample sum of squares, the denominator sample sum, the denominator sample sum of squares, and the sum of the products of the numerator and denominator. These parameters are used as input parameters to calculate the mean, variance, and sample size of the experimental and control groups, and a z-statistic is constructed. The p-value, diff value, and confidence interval are calculated, and the final hypothesis testing results are analyzed based on the calculation results.

[0118] Figure 3 This is a schematic diagram of the main modules of the application testing device 300 according to an embodiment of the present invention, as shown below. Figure 3 As shown, it includes an acquisition module 301, a verification module 302, a calculation module 303, and an analysis module 304.

[0119] The acquisition module 301 is used to acquire user behavior data on the tested application. The behavior data includes sample mean index data and non-sample mean index data.

[0120] The testing module 302 is used to perform a first-order Taylor expansion on non-sample mean index data to obtain random variables, and to perform a normality test on the random variables to obtain the first intermediate parameter.

[0121] The calculation module 303 is used to calculate the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean index data; and to calculate the difference value of the interest index based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing user behavior data of the baseline application. The baseline application is the same application with a different version than the application being tested.

[0122] Analysis module 304 is used to analyze the test results of the application under test based on the difference values.

[0123] The application testing apparatus provided in this embodiment of the invention can transform non-sample mean index data in user behavior data into a sum of multiple sample mean index data by performing Taylor expansion and retaining the first-order terms, thus ensuring a normal distribution and enabling subsequent data verification operations. A first intermediate parameter is generated from the expanded random variable, and a second intermediate parameter obtained from processing the sample mean index data is used to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for analyzing the test results. This achieves unified verification of both non-sample mean index data and sample mean index data, while improving the accuracy of verification results for non-sample mean index data.

[0124] Optionally,

[0125] The testing module 302 is also used to perform a normality test on the sample mean index data to obtain the second intermediate parameter;

[0126] The calculation module 303 is also used to calculate the target interest index based on the first intermediate parameter and the second intermediate parameter.

[0127] Optionally, the target interest index includes the sum of numerator samples, the sum of squares of numerator samples, the sum of denominator samples, the sum of squares of denominator samples and the product of numerator and denominator, wherein the numerator samples and denominator samples are generated by rewriting the non-sample mean class index data into the form of the quotient of two sample mean class index data;

[0128] Calculation module 303 is also used for:

[0129] Based on the sum of numerator samples, sum of squares of numerator samples, sum of denominator samples, sum of squares of denominator samples and sum of products of numerator and denominator included in the target interest index, calculate the sample mean and sample variance of the target interest index.

[0130] Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares and the product of numerator and denominator included in the baseline interest index, calculate the sample mean and sample variance of the baseline interest index.

[0131] Based on the sample mean, sample variance, and sample size of the target interest index, as well as the sample mean, sample variance, and sample size of the baseline interest index, a statistic is constructed.

[0132] The difference in interest index is calculated using constructed statistics.

[0133] Optionally,

[0134] The calculation module 303 is also used to simplify the sample variance calculation formula based on the sample size in the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold.

[0135] The calculation module 303 is also used to calculate the probability of deviation, the deviation value, and the confidence interval of the deviation value between the target interest index and the baseline interest index by constructing statistics; and to obtain the difference value of the interest index based on the probability of deviation, the deviation value, and the confidence interval of the deviation value.

[0136] The application testing apparatus provided in this embodiment of the invention can achieve unified verification of non-sample mean index data and sample mean index data through a unified architecture. Specifically, for non-sample mean index data, Taylor expansion can be used to retain the first-order terms, transforming it into the sum of multiple sample mean index data, thereby enabling subsequent data verification operations. A first intermediate parameter is generated from the random variable obtained after expansion, and a second intermediate parameter is combined to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for analyzing the test results. This achieves unified verification of different index data and improves the accuracy of the verification results. Furthermore, when calculating the difference value, the Delta method is used to approximate the calculation of different indicators, ensuring the uniformity of different indicators and thus guaranteeing the uniformity of the overall verification architecture.

[0137] Figure 4 An exemplary system architecture 400 is shown that can be applied to the application testing method or application testing apparatus of the present invention.

[0138] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, a network 404, and a server 405. Network 404 serves as the medium for providing communication links between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0139] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, and 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0140] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0141] Server 405 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 401, 402, and 403 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal device.

[0142] It should be noted that the application testing method provided in the embodiments of the present invention is generally executed by server 405, and correspondingly, the application testing device is generally set in server 405.

[0143] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0144] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing terminal devices or servers of the present invention. Figure 5 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0145] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0147] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined above in the system of this invention.

[0148] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a verification module, a calculation module, and an analysis module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, an acquisition module can also be described as "a module for acquiring user behavior data of the application under test."

[0151] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0152] Acquire user behavior data for the tested application. This behavior data includes sample mean-based metrics and non-sample mean-based metrics.

[0153] The first Taylor expansion of non-sample mean index data is used to obtain random variables, and the normality test of the random variables is used to obtain the first intermediate parameter.

[0154] The target interest index is calculated based on the first intermediate parameter and the second intermediate parameter after processing the sample mean class index data.

[0155] The difference in interest metrics is calculated based on the target interest metric and the baseline interest metric. The baseline interest metric is obtained by processing user behavior data on the baseline application. The baseline application is the same application but a different version from the application being tested.

[0156] Analysis of test results for the application under test based on the difference values.

[0157] According to the technical solution of this invention, non-sample mean index data in user behavior data can be transformed into the sum of multiple sample mean index data by performing Taylor expansion and retaining the first-order terms, thus satisfying a normal distribution and enabling subsequent data verification operations. A first intermediate parameter is generated from the expanded random variable, and a second intermediate parameter obtained from processing the sample mean index data is combined to calculate the target interest index. The obtained target interest index is combined with the baseline interest index to calculate the difference value, which is used for test result analysis. This achieves unified verification of both non-sample mean index data and sample mean index data, while improving the accuracy of verification results for non-sample mean index data.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An application testing method, characterized in that, include: Acquire user behavior data on the tested application, including sample mean index data and non-sample mean index data; The non-sample mean index data are subjected to a first-order Taylor expansion to obtain random variables, and the random variables are subjected to a normality test to obtain the first intermediate parameter. Based on the first intermediate parameter and the second intermediate parameter after processing the sample mean class index data, the target interest index is calculated; the target interest index includes the numerator sample sum, the numerator sample sum of squares, the denominator sample sum, the denominator sample sum of squares and the sum of the products of the numerator and denominator, wherein the numerator sample and the denominator sample are generated by rewriting the non-sample mean class index data into the form of the quotient of two sample mean class index data; The difference value of the interest index is calculated based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing user behavior data of the baseline application. The baseline application is the same application but a different version from the application being tested. The test results of the application under test are analyzed based on the aforementioned differences. The step of calculating the difference value of the interest index based on the target interest index and the baseline interest index includes: Based on the sum of numerator samples, sum of squares of numerator samples, sum of denominator samples, sum of squares of denominator samples and sum of products of numerator and denominator included in the target interest index, calculate the sample mean and sample variance of the target interest index. Based on the numerator sample sum, numerator sample sum of squares, denominator sample sum, denominator sample sum of squares and numerator-denominator product sum of the baseline interest index, calculate the sample mean and sample variance of the baseline interest index. Based on the sample mean, sample variance, and sample size of the target interest index, and the sample mean, sample variance, and sample size of the baseline interest index, a statistic is constructed. The probability of deviation, the deviation value, and the confidence interval of the deviation value are calculated using the constructed statistics. The difference value of the interest index is obtained based on the deviation probability, the deviation value, and the confidence interval of the deviation value.

2. The method according to claim 1, characterized in that, Before calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean class index data, the method further includes: A normality test is performed on the sample mean index data to obtain the second intermediate parameter; The step of calculating the target interest index based on the first intermediate parameter and the second intermediate parameter after processing the sample mean class index data includes: The target interest index is calculated based on the first intermediate parameter and the second intermediate parameter.

3. The method according to claim 1, characterized in that, The method further includes: In the process of calculating the sample variance of the target interest index and the sample variance of the baseline interest index, in response to the sample size of the target interest index and the sample size of the baseline interest index being greater than a preset threshold, the sample variance calculation formula is reduced and simplified based on the sample size.

4. An application testing device, characterized in that, include: The acquisition module is used to acquire user behavior data on the tested application, including sample mean index data and non-sample mean index data. The testing module is used to perform a first-order Taylor expansion on the non-sample mean index data to obtain random variables, and to perform a normality test on the random variables to obtain a first intermediate parameter. The calculation module is used to calculate a target interest index based on the first intermediate parameter and a second intermediate parameter after processing the sample mean index data; and to calculate the difference value of the interest index based on the target interest index and the baseline interest index. The baseline interest index is obtained by processing user behavior data of a baseline application, and the baseline application is the same application in a different version from the tested application. The target interest index includes the sum of numerator samples, the sum of squares of numerator samples, the sum of sample denominators, the sum of squares of sample denominators, and the sum of the products of numerator and denominator. The numerator and denominator samples are generated by rewriting the non-sample mean index data into the quotient of two sample mean index data. The calculation module is further configured to: calculate the sample mean and sample variance of the target interest index based on the sum of numerator samples, the sum of squares of numerator samples, the sum of denominator samples, and the sum of squares of denominator samples and the sum of products of numerator and denominator; calculate the sample mean and sample variance of the baseline interest index based on the sum of numerator samples, the sum of squares of numerator samples, the sum of denominator samples, and the sum of squares of denominator samples and the sum of products of numerator and denominator; construct a statistic based on the sample mean, sample variance, and sample size of the target interest index, and the sample mean, sample variance, and sample size of the baseline interest index; calculate the probability of deviation, the deviation value, and the confidence interval of the deviation value between the target interest index and the baseline interest index using the constructed statistic; and obtain the difference value of the interest index based on the probability of deviation, the deviation value, and the confidence interval of the deviation value. The analysis module is used to analyze the test results of the application under test based on the difference values.

5. The apparatus according to claim 4, characterized in that, The device further includes: The testing module is also used to perform a normality test on the sample mean index data to obtain a second intermediate parameter; The calculation module is also used to calculate the target interest index based on the first intermediate parameter and the second intermediate parameter.

6. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images