Method and device for determining abnormality of updated version of application program

Abstract

The present disclosure relates to a method and device for determining whether an updated version of an application is abnormal. The method for determining whether an updated version of an application is abnormal includes: obtaining change trend information of the indicator data of the updated version and the current version of the application, wherein the change trend information represents the difference between the indicator data of the updated version and the current version of the application at each sampling moment; based on the change trend information, performing a difference check between the updated version and the current version of the application; and determining whether the updated version of the application is abnormal based on the result of the difference check. Through the present disclosure, the problem that the related technology cannot accurately detect the abnormality of the gray version is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a method and device for determining an abnormality in an updated version of an application. Background Art

[0002] At present, with the rapid growth of data processing services (such as audio and video services) and demand, in order to provide users with better products and services, engineers will continuously upgrade products and develop new features. Grayscale release is a common means of avoiding release risks in the Internet field. Before making major revisions to product services or launching a new version, a grayscale version will be tried between black and white, and it will only be officially launched after passing the grayscale test.

[0003] However, the methods for determining the anomalies of product update versions in related technologies require that the two indicator data of the input product are independent and identically distributed, ignoring the time dependence between the indicator data (the indicator data has trend changes and periodic changes over time), and do not consider more complex situations of grayscale version monitoring, and cannot accurately detect grayscale version anomalies. Summary of the invention

[0004] The present disclosure provides a method and device for determining an abnormality of an update version of an application program, so as to at least solve the problem that the method for determining an abnormality of an update version in the related art cannot accurately detect an abnormality of a grayscale version.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining whether an updated version of an application is abnormal is provided, comprising: obtaining change trend information of indicator data of the updated version and the current version of the application, wherein the change trend information represents the difference between the indicator data of the updated version and the current version of the application at each sampling moment; based on the change trend information, performing a difference check between the updated version and the current version of the application; and determining whether the updated version of the application is abnormal based on the result of the difference check.

[0006] Optionally, obtaining change trend information of indicator data of updated version and current version of the application includes: obtaining the first number of visits to the updated version of the application within a predetermined time before the current moment; and obtaining change trend information of indicator data of updated version and current version of the application when the first number of visits meets a predetermined condition.

[0007] Optionally, the predetermined condition is: the first number of visits is greater than a preset number, and the ratio of the first number of visits to the second number of visits is greater than a preset ratio, wherein the second number of visits is the sum of the first number of visits and the number of visits of the current version of the application within a predetermined time before the current moment.

[0008] Optionally, obtaining change trend information of indicator data of updated versions and current versions of the application includes: obtaining first indicator data of the updated version of the application; obtaining second indicator data of the current version of the application; obtaining the difference between the first indicator data and the second indicator data; and obtaining change trend information of the indicator data of the updated version and current version of the application based on the ratio of the difference to the second indicator data.

[0009] Optionally, based on the change trend information, a difference check is performed between the updated version and the current version of the application, including: determining the number of predetermined data points whose values ​​on the change trend information satisfy a predetermined relationship with a preset value; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is greater than a predetermined ratio, determining that the updated version of the application is abnormal; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is less than or equal to a predetermined ratio, based on the change trend information, a difference check is performed between the updated version and the current version of the application.

[0010] Optionally, based on the change trend information, performing a difference check between the updated version and the current version of the application, including: determining a relationship between an expected value of the change trend information and a given expected value; and based on the relationship, determining a result of the difference check between the updated version and the current version of the application.

[0011] Optionally, determining whether the updated version of the application is abnormal based on the result of the difference check includes: determining that the updated version of the application is abnormal when the result of the difference check is that the expected value of the change trend information is not equal to a given expected value.

[0012] Optionally, determining the relationship between the expected value of the changing trend information and the given expected value includes: determining the query parameter value of the changing trend information based on the mean value and standard deviation of the changing trend information; obtaining the probability that the expected value of the changing trend information is equal to the given expected value by querying the parameter value and the degree of freedom of the changing trend information in a predetermined boundary value table, wherein the degree of freedom is obtained according to the total number of data points on the changing trend information; determining the relationship between the expected value of the changing trend information and the given expected value based on the probability and the significance level, wherein the significance level is the probability of obtaining a result that the expected value of the changing trend information is not equal to the given expected value when the expected value of the changing trend information is originally equal to the given expected value.

[0013] Optionally, based on the mean value and standard deviation of the change trend information, determining the query parameter value of the change trend information includes: obtaining the mean value and standard deviation of the change trend information; determining the standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; determining the query parameter value of the change trend information based on the standard error and the mean value.

[0014] Optionally, after determining that the updated version of the application is abnormal, it also includes: obtaining the confidence level by querying in a predetermined boundary value table through the significance level and the degrees of freedom of the change trend information; determining the upper limit of the confidence interval based on the confidence level and the average value of the change trend information; and determining the degree of difference between the indicator data of the updated version of the application and the current version based on the upper limit of the confidence interval.

[0015] Optionally, after determining that the updated version of the application is abnormal, the method further includes: determining a change trend of indicator data of the updated version and the current version of the application through a Pearson correlation coefficient or a Spear rank correlation coefficient between the updated version and the current version of the application.

[0016] Optionally, after determining that the updated version of the application is abnormal, it also includes: displaying alarm information, wherein the alarm information includes the difference degree of indicator data between the updated version of the application and the current version and the change trend of the indicator data between the updated version of the application and the current version.

[0017] According to a second aspect of an embodiment of the present disclosure, there is provided a device for determining whether an updated version of an application is abnormal, comprising: a change trend information acquisition unit, configured to acquire change trend information of indicator data of an updated version of the application and a current version, wherein the change trend information represents the difference between the indicator data of the updated version of the application and the current version at each sampling moment; a verification unit, configured to perform a difference verification between the updated version of the application and the current version based on the change trend information; and an abnormality determination unit, configured to determine whether the updated version of the application is abnormal based on a result of the difference verification.

[0018] Optionally, the change trend information acquisition unit is also configured to obtain the first number of visits to the updated version of the application within a predetermined time before the current moment before obtaining the change trend information of the indicator data of the updated version and the current version of the application; and obtain the change trend information of the indicator data of the updated version and the current version of the application when the first number of visits meets a predetermined condition.

[0019] Optionally, the predetermined condition is: the first number of visits is greater than a preset number, and the ratio of the first number of visits to the second number of visits is greater than a preset ratio, wherein the second number of visits is the sum of the first number of visits and the number of visits of the current version of the application within a predetermined time before the current moment.

[0020] Optionally, the change trend information acquisition unit is also configured to obtain first indicator data of an updated version of the application; obtain second indicator data of a current version of the application; obtain the difference between the first indicator data and the second indicator data; and obtain change trend information of the indicator data of the updated version and the current version of the application based on the ratio of the difference to the second indicator data.

[0021] Optionally, the verification unit is further configured to determine the number of predetermined data points in the change trend information whose values ​​satisfy a predetermined relationship with a preset value; when the ratio of the number of predetermined data points to the total number of data points in the change trend information is greater than a predetermined ratio, determine that the updated version of the application is abnormal; when the ratio of the number of predetermined data points to the total number of data points in the change trend information is less than or equal to a predetermined ratio, perform a difference check between the updated version of the application and the current version based on the change trend information.

[0022] Optionally, the verification unit is further configured to determine a relationship between an expected value of the change trend information and a given expected value; and based on the relationship, determine a result of a difference verification between the updated version and the current version of the application.

[0023] Optionally, the abnormality determination unit is further configured to determine that the updated version of the application is abnormal when the result of the difference check is that the expected value of the change trend information is not equal to the given expected value.

[0024] Optionally, the inspection unit is also configured to determine the query parameter value of the changing trend information based on the mean value and standard deviation of the changing trend information; query the predetermined boundary value table to obtain the probability that the expected value of the changing trend information is equal to the given expected value by querying the parameter value and the degree of freedom of the changing trend information, wherein the degree of freedom is obtained according to the total number of data points on the changing trend information; determine the relationship between the expected value of the changing trend information and the given expected value based on the probability and the significance level, wherein the significance level is the probability of obtaining the result that the expected value of the changing trend information is not equal to the given expected value when the expected value of the changing trend information is originally equal to the given expected value.

[0025] Optionally, the inspection unit is also configured to obtain the average value and standard deviation of the change trend information; determine the standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; and determine the query parameter value of the change trend information based on the standard error and the average value.

[0026] Optionally, the anomaly determination unit is further configured to, after determining that the updated version of the application is abnormal, query a predetermined boundary value table to obtain a confidence level through the significance level and the degree of freedom of the change trend information; determine an upper limit of the confidence interval based on the confidence level and the average value of the change trend information; and determine the degree of difference between the indicator data of the updated version of the application and the current version based on the upper limit of the confidence interval.

[0027] Optionally, the anomaly determination unit is further configured to determine, after determining that the updated version of the application is abnormal, a change trend of indicator data of the updated version and the current version of the application by using a Pearson correlation coefficient or a Spear rank correlation coefficient between the updated version and the current version of the application.

[0028] Optionally, the abnormality determination unit is further configured to display an alarm message after determining that the updated version of the application is abnormal, wherein the alarm information includes the degree of difference in indicator data between the updated version of the application and the current version and the change trend of the indicator data between the updated version of the application and the current version.

[0029] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement a method for determining an abnormality in an updated version of an application according to the present disclosure.

[0030] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method for determining an abnormality of an updated version of an application according to the present disclosure.

[0031] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement a method for determining an abnormality in an updated version of an application according to the present disclosure.

[0032] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0033] According to the method and device for determining the abnormality of the updated version of the application disclosed in the present invention, the change trend information is obtained through the indicator data of the two versions, and the difference check between the updated version and the current version of the application is performed using the change trend information. The abnormality of the updated version is determined based on the result of the difference check, thereby realizing the monitoring of the abnormality of the updated version based on the time series, that is, the present invention takes into account the time-dependent information of the business indicator data in the abnormality monitoring, and can timely and accurately discover the abnormality of the updated version, reducing the failure caused by the release of abnormal updated versions. Therefore, the present invention solves the problem that the method for determining the abnormality of the updated version in the related art cannot accurately discover the abnormality of the gray version.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0036] Figure 1is a schematic diagram of an implementation scenario showing a method for determining an abnormal update version of an application according to an exemplary embodiment of the present disclosure;

[0037] Figure 2 is a flowchart of a method for determining an abnormal update version of an application program according to an exemplary embodiment;

[0038] Figure 3 is the number of visits to the gray version and the old version at different times according to an exemplary embodiment;

[0039] Figure 4 is a graph showing two types of indicator data according to an exemplary embodiment;

[0040] Figure 5 is a schematic diagram of a workflow of a method for solving an abnormal update version of an application program according to an exemplary embodiment;

[0041] Figure 6 is a block diagram of a device for determining an abnormal update version of an application program according to an exemplary embodiment;

[0042] Figure 7 is a block diagram of an electronic device 700 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0045] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0046] In the related art, the following two methods are often used to test whether the gray version (i.e., updated version) runs abnormally: 1) a two-sample T test method, which assumes that the input index data conforms to a normal distribution or an approximate normal distribution, and determines whether the means of the two input index data are significantly different; 2) a Bayesian two-sample hypothesis test method, which assumes that the data expectations of the two input index data are equal, and when the final result rejects the assumption (i.e., they are not equal), it will be detected as an abnormal situation. However, both of the above methods require that the two input index data are independent and identically distributed, ignoring the time dependence between the index data (the index data has trend changes and periodic changes over time), and do not consider more complex situations of gray version monitoring, and cannot accurately detect the anomalies of the gray version. In response to the above problems, the present disclosure provides a method for determining the anomalies of an updated version of an application, which can accurately detect the anomalies of the gray version. The following is an example of a game application.

[0047] Figure 1 is a schematic diagram of an implementation scenario of a method for determining an abnormal update version of an application according to an exemplary embodiment of the present disclosure, such as Figure 1 The implementation scenario includes a server 100, a user terminal 110 and a user terminal 120, wherein the user terminals are not limited to two, and include but are not limited to mobile phones, personal computers and other devices. The user terminals may be installed with game applications. The server may be a single server, or a server cluster consisting of several servers, or a cloud computing platform or a virtualization center.

[0048] The server 100 obtains change trend information of the indicator data of the updated version and the current version of the game application from the user terminals 110 and 120, wherein the change trend information represents the difference between the indicator data of the updated version and the current version of the game application at each sampling moment; after obtaining the change trend information, the server 100 performs a difference check between the updated version and the current version of the application based on the change trend information, and then determines whether the updated version of the application is abnormal according to the result of the difference check.

[0049] Next, we will refer to Figures 2 to 7 A method and apparatus for determining an abnormality of an update version of an application program according to an exemplary embodiment of the present disclosure are described in detail.

[0050] Figure 2 is a flowchart of a method for determining an abnormal update version of an application according to an exemplary embodiment. Figure 2 As shown, the method for determining whether the updated version of the application is abnormal includes the following steps:

[0051] In step S201 , change trend information of indicator data of the updated version and the current version of the application is obtained, wherein the change trend information indicates the difference between the indicator data of the updated version and the current version of the application at each sampling moment.

[0052] According to an exemplary embodiment of the present disclosure, before obtaining the change trend information of the indicator data of the updated version and the current version of the application, it also includes: obtaining the first number of visits to the updated version of the application within a predetermined time before the current moment; when the first number of visits meets a predetermined condition, obtaining the change trend information of the indicator data of the updated version and the current version of the application. According to this embodiment, before calculating the change trend information, it can be determined whether the number of visits is sufficient, so that an accurate inspection result can be obtained based on a sufficient number of visits.

[0053] According to an exemplary embodiment of the present disclosure, the above-mentioned predetermined condition may be: the first number of visits is greater than a preset number, and the ratio of the first number of visits to the second number of visits is greater than a preset ratio, wherein the second number of visits is the sum of the first number of visits and the number of visits of the current version of the application within a predetermined time before the current moment. According to this embodiment, it is possible to conveniently and quickly determine that the number of visits is sufficient.

[0054] For example, in the gray version (i.e., the above-mentioned updated version) abnormal alarm monitoring scenario, the number of gray version releases will gradually increase, and the number of visits to the released gray version will also vary greatly at different times of the day, such as Figure 3As shown. In general, when the number of visits to the released gray version reaches a certain requirement, the subsequent analysis is meaningful. Therefore, the confidence judgment of the number of visits can be made in advance, which can be as follows: the number of visits to the gray version (i.e. the updated version mentioned above) and the old version (i.e. the current version mentioned above) in different hours is used to judge whether the number of visits to the gray version in each hour meets the confidence requirement. Only when the requirement is met will the subsequent analysis be carried out. The confidence judgment of the number of visits can be as follows: 1. The number of visits to the gray version in a period of time before time t> the minimum number of visits (i.e. the preset number above); 2. The number of visits to the gray version in a period of time before time t / the total number of visits to the gray version and the old version in a period of time before time t> the minimum number of visits (i.e. the preset ratio above). The values ​​of the minimum number of visits and the minimum number of visits can be configured according to the actual business scenario and requirements (for example, the minimum number of visits can be 100, and the minimum number of visits can be 1%). The confidence requirement will be met only when the above two standards are met at the same time.

[0055] According to an exemplary embodiment of the present disclosure, obtaining change trend information of indicator data of an updated version and a current version of an application includes: obtaining first indicator data of the updated version of the application; obtaining second indicator data of the current version of the application; obtaining the difference between the first indicator data and the second indicator data; and obtaining change trend information of the indicator data of the updated version and the current version of the application based on the ratio of the difference to the second indicator data. According to this embodiment, change trend information can be obtained quickly and conveniently.

[0056] For example, Figure 4 As shown, the indicator data of the application has normal periodicity and trend changes, which will also affect the final analysis results, where 4(a) represents the failure rate of broadcasting, and 4(b) represents the recording frame rate. As introduced in the background technology, the two-sample hypothesis test can compare whether the difference in the average values ​​of the two input indicator data is significant. However, the two-sample hypothesis test ignores the dependency between the indicator data and does not consider the time information. The present disclosure uses the indicator data of the grayscale version and the old version to calculate the change trend information. For example, the change trend information can be represented by a change rate curve. The change rate curve represents the difference between the grayscale version and the old version at each moment, so the time information of the indicator data is taken into account. Specifically, it can be as follows: change rate curve = (indicator data of the grayscale version - indicator data of the old version) / indicator data of the old version.

[0057] return Figure 2 In step S202, based on the change trend information, a difference check is performed between the updated version and the current version of the application.

[0058] According to an exemplary embodiment of the present disclosure, based on the change trend information, a difference check between the updated version of the application and the current version is performed, including: determining the number of predetermined data points whose values ​​on the change trend information satisfy a predetermined relationship with the preset value; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is greater than the predetermined ratio, determining that the updated version of the application is abnormal; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is less than or equal to the predetermined ratio, based on the change trend information, a difference check between the updated version of the application and the current version is performed. The above-mentioned predetermined relationship is determined according to the indicator data. If the indicator data is the start-up failure rate, the smaller the start-up failure rate, the better. At this time, the predetermined relationship is a value greater than the preset value; and if the indicator data is the start-up success rate, the larger the start-up success rate, the better. At this time, the predetermined relationship is a value less than or equal to the preset value. According to this embodiment, before executing the difference check, it is possible to first roughly determine whether the updated version is abnormal based on the data point values. If it is determined to be abnormal at this time, the abnormality can be directly determined. If it is not abnormal, the determination of whether it is abnormal can be further refined based on the expected value. In this way, the complexity of determining the abnormality can be reduced.

[0059] For example, for indicators such as failure rate, the expectation is that the smaller the value, the better. Here, the change rate curve is still used to represent the change trend information. Therefore, if more than half of the data points in the change rate curve have values ​​> 0, the gray version is abnormal. For indicators such as preload ratio, the expectation is that the larger the value, the better. Therefore, if more than half of the data points in the change rate curve have values ​​< 0, the gray version is abnormal. At this time, an indicator knowledge base can be maintained to determine whether the expected value of the indicator is as large as possible or as small as possible. In addition, in order to tolerate the impact of a small number of extreme values ​​in the change rate curve on subsequent analysis, the values ​​before TopN on the change rate curve (for example, N = 1) can also be filtered out.

[0060] According to an exemplary embodiment of the present disclosure, performing a difference check between an updated version of an application and a current version based on change trend information includes: determining a relationship between an expected value of the change trend information and a given expected value; and determining a result of the difference check between the updated version of the application and the current version based on the relationship. According to this embodiment, a difference check can be conveniently and quickly performed based on the expected value.

[0061] According to an exemplary embodiment of the present disclosure, determining the relationship between the expected value of the change trend information and the given expected value includes: determining the query parameter value of the change trend information based on the mean value and standard deviation of the change trend information; querying the probability that the expected value of the change trend information is equal to the given expected value in a predetermined boundary value table by querying the parameter value and the degree of freedom of the change trend information, wherein the degree of freedom is obtained according to the total number of data points on the change trend information; determining the relationship between the expected value of the change trend information and the given expected value based on the probability and the significance level, wherein the significance level is the probability that when the expected value of the change trend information is originally equal to the given expected value, the expected value of the change trend information is not equal to the given expected value. According to this embodiment, the relationship between the expected value and the given expected value can be determined conveniently and quickly.

[0062] According to an exemplary embodiment of the present disclosure, determining the query parameter value of the change trend information based on the average value and standard deviation of the change trend information includes: obtaining the average value and standard deviation of the change trend information; determining the standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; and determining the query parameter value of the change trend information based on the standard error and the average value. According to this embodiment, the query parameter value can be obtained conveniently and quickly.

[0063] Specifically, after obtaining the change trend information, the change rate curve is still used as an example to represent the change trend information, and the two-sample hypothesis test problem is transformed into a single-sample hypothesis test problem. At this time, the T test or the Z test can be used, or other test methods can be used, which are not limited by the present disclosure. The T test is a common method in hypothesis testing. Compared with the Z test, the T test can be applied to situations where the number of samples is small.

[0064] The following is an explanation using the T test as an example.

[0065] The purpose of the one-sample T test is to determine whether the expected value of the change rate curve is equal to the given expected value 0 (that is, there is no difference between the grayscale version and the old version). The process of the one-sample T test can be as follows:

[0066] 1) Formulate a hypothesis

[0067] Null hypothesis: The expected value of the rate of change curve is equal to 0, that is, there is no significant difference between the grayscale version and the old version.

[0068] Alternative hypothesis: The expected value of the rate of change curve is not equal to 0, that is, there is a significant difference between the grayscale version and the old version.

[0069] 2) Calculation of t value

[0070] t = (mean of the rate of change curve - given expected value) / standard error

[0071]

[0072] 3) p-value calculation

[0073] Based on the t value and the degree of freedom (i.e. the total number of data points on the rate of change curve minus 1), the p value can be obtained by looking up the threshold table. Here, the p value represents the probability of obtaining the expected value of the rate of change curve equal to the given expected value when the null hypothesis is true.

[0074] 5) T test result judgment

[0075] If p / 2>significance level, then accept the null hypothesis, that is, the expected value of the rate of change curve is equal to 0, and there is no significant difference between the gray version and the old version, that is, the gray version is normal. Otherwise, reject the null hypothesis and the gray version is abnormal. The above significance level is generally 0.1%, 1%, 10%. If the gray version is always better or worse than the old version, the significance level can be 10%, otherwise it is 1%. For example, the above significance level of 10% means that the expected value of the rate of change curve is actually equal to 0, but there is a 10% risk of mistakenly believing that the expected value of the rate of change curve is not equal to 0.

[0076] return Figure 2 In step S203, it is determined whether the updated version of the application is abnormal based on the result of the difference check.

[0077] According to an exemplary embodiment of the present disclosure, determining whether the updated version of the application is abnormal based on the result of the difference check includes: determining that the updated version of the application is abnormal when the result of the difference check is that the expected value of the change trend information is not equal to the given expected value. According to this embodiment, a judgment standard for determining the abnormality is given.

[0078] According to an exemplary embodiment of the present disclosure, after determining that the updated version of the application is abnormal, it also includes: querying the confidence level in the predetermined boundary value table through the significance level and the degree of freedom of the change trend information; determining the upper limit of the confidence interval based on the confidence level and the average value of the change trend information; and determining the degree of difference between the indicator data of the updated version and the current version of the application based on the upper limit of the confidence interval. According to this embodiment, the degree of difference can be obtained conveniently and quickly for subsequent warning display.

[0079] For example, at this time, we still take the change rate curve to represent the change trend information as an example. We can find the T test confidence level under the degree of freedom of the change rate curve in the boundary value by (1-2*significant level), and then obtain the upper limit of the T test confidence interval by the following method: T test confidence interval upper limit = change rate curve mean + confidence level * standard error. We can also obtain the lower limit of the T test confidence interval: T test confidence interval lower limit = change rate curve mean - confidence level * standard error. By comparing the upper limit of the confidence interval and the confidence change degree threshold, we can rank the degree of the gap, such as:

[0080] The upper limit of the T-test confidence interval is greater than 50%: the degree of difference can be rated as 0;

[0081] The upper limit of the T-test confidence interval is greater than 10% and less than 50%: the degree of difference can be rated as 1;

[0082] The upper limit of the T-test confidence interval is less than 10%: the degree of gap can be rated as 2.

[0083] It should be noted that the smaller the significance level, the larger the "1-2*significance level", the greater the confidence level of the T test, and the stricter the judgment when rejecting the null hypothesis.

[0084] According to an exemplary embodiment of the present disclosure, after determining that the updated version of the application is abnormal, it also includes: determining the change trend of the indicator data of the updated version and the current version of the application through the Pearson correlation coefficient or the Spear rank correlation coefficient of the updated version and the current version of the application. The above-mentioned Pearson correlation coefficient evaluates the linear relationship between two variables, while the Spear rank correlation coefficient evaluates the monotonic relationship between two variables. According to this embodiment, the change trend can be obtained conveniently and quickly for subsequent warning display.

[0085] For example, if the gray version is abnormal in the T test, we can further determine whether the gray version and the old version have the same change trend: the same trend change indicates that the gray version and the old version have similar user behaviors (for example, Figure 4 (a)), the inconsistent trend changes indicate that the user behaviors of the gray version and the old version are inconsistent (e.g. Figure 4 (b)). Specifically, the trend consistency between the grayscale version and the old version can be measured by calculating the Pearson correlation coefficient r between the grayscale version and the old version:

[0086] If |r|<0.4, the gray version and the old version have the same trend change;

[0087] If |r|>=0.4, the trend changes of the gray version and the old version are inconsistent.

[0088] According to an exemplary embodiment of the present disclosure, after determining that the updated version of the application is abnormal, it also includes: displaying alarm information, wherein the alarm information includes the degree of difference between the indicator data of the updated version of the application and the current version and the change trend of the indicator data of the updated version of the application and the current version. According to this embodiment, not only can an alarm be issued for abnormalities, but the abnormality level and the change trend of the two can also be displayed to help make subsequent adjustments to the abnormalities.

[0089] For example, combining the above T test results and linear correlation results, the final alarm result can be:

[0090] 1) The grayscale version is normal;

[0091] 2) The gray version is abnormal, the specific results are:

[0092] The trend changes of the gray version and the old version are consistent, and the gap is graded as 0 / 1 / 2;

[0093] The trend changes of the gray version and the old version are inconsistent, and the degree of difference is rated as 0 / 1 / 2.

[0094] In order to facilitate the understanding of the above embodiments, Figure 5 To explain, Figure 5 FIG. 1 is a flowchart of a method for solving an abnormal update version of an application program according to an exemplary embodiment. Figure 5 As shown, first, a confidence judgment of the access volume is performed based on the access volume of the grayscale version and the old version of the application. If the confidence judgment result meets the preset standard, the change rate curve is calculated based on the indicator data of the grayscale version and the old version. After obtaining the change rate curve, a difference test can be performed based on the single-sample T test to determine whether the grayscale version is abnormal. If it is abnormal, a linear correlation calculation can be further performed to obtain the change trend of the indicator data of the grayscale version and the old version. Then, the change trend is displayed in the grayscale alarm result. It should be noted that the abnormality level can also be judged during the single-sample T test and displayed in the grayscale alarm result. However, the change trend and abnormality level are not essential parts of the alarm.

[0095] In summary, the present disclosure provides a method for determining abnormalities of grayscale versions based on time series, which can automatically and accurately monitor the release of grayscale versions and issue alarms for abnormal grayscale versions. Compared with the original manual inspection and manual writing of rule strategies for each grayscale version, it greatly reduces manpower costs. Moreover, compared with the independent and identically distributed indicator data in the relevant technology, the present disclosure can focus on the time-dependent information of business indicators, and also considers various needs and situations for grayscale version monitoring. It will make confidence judgments on the access volume of grayscale versions, and distinguish and grade different types of abnormal grayscale versions. In addition, through experimental verification on the grayscale launch of audio and video related products, the method disclosed in the present disclosure has an accuracy rate of monitoring grayscale versions greater than 95%, and can timely and accurately detect abnormal grayscale versions, reducing the failures caused by abnormal grayscale releases.

[0096] Figure 6 1 is a block diagram of a device for determining an abnormal update version of an application program according to an exemplary embodiment. Figure 6 The device includes a change trend information acquisition unit 602, a verification unit 604 and an abnormality determination unit 606.

[0097] The change trend information acquisition unit 602 is configured to obtain the change trend information of the indicator data of the updated version and the current version of the application, wherein the change trend information represents the difference between the indicator data of the updated version and the current version of the application at each sampling moment; the verification unit 604 is configured to perform a difference verification between the updated version and the current version of the application based on the change trend information; the abnormality determination unit 606 is configured to determine whether the updated version of the application is abnormal based on the result of the difference verification.

[0098] According to an embodiment of the present disclosure, the change trend information acquisition unit 602 is also configured to obtain the first number of visits to the updated version of the application within a predetermined time before the current moment before obtaining the change trend information of the indicator data of the updated version and the current version of the application; and obtain the change trend information of the indicator data of the updated version and the current version of the application when the first number of visits meets a predetermined condition.

[0099] According to an embodiment of the present disclosure, the above-mentioned predetermined condition is: the first number of visits is greater than a preset number, and the ratio of the first number of visits to the second number of visits is greater than a preset ratio, wherein the second number of visits is the sum of the first number of visits and the number of visits of the current version of the application within a predetermined time before the current moment.

[0100] According to an embodiment of the present disclosure, the change trend information acquisition unit 602 is also configured to obtain first indicator data of an updated version of the application; obtain second indicator data of a current version of the application; obtain the difference between the first indicator data and the second indicator data; and obtain change trend information of the indicator data of the updated version and the current version of the application based on the ratio of the difference to the second indicator data.

[0101] According to an embodiment of the present disclosure, the verification unit 604 is also configured to determine the number of predetermined data points in the change trend information whose values ​​satisfy a predetermined relationship with a preset value; when the ratio of the number of predetermined data points to the total number of data points in the change trend information is greater than the predetermined ratio, determine that the updated version of the application is abnormal; when the ratio of the number of predetermined data points to the total number of data points in the change trend information is less than or equal to the predetermined ratio, perform a difference check between the updated version of the application and the current version based on the change trend information.

[0102] According to an embodiment of the present disclosure, the verification unit 604 is further configured to determine the relationship between the expected value of the change trend information and the given expected value; based on the relationship, determine the result of the difference verification between the updated version and the current version of the application.

[0103] According to an embodiment of the present disclosure, the abnormality determination unit 606 is further configured to determine that the updated version of the application is abnormal when the result of the difference check is that the expected value of the change trend information is not equal to the given expected value.

[0104] According to an embodiment of the present disclosure, the inspection unit 604 is also configured to determine a query parameter value of the change trend information based on the mean value and standard deviation of the change trend information; obtain the probability that the expected value of the change trend information is equal to a given expected value by querying the parameter value and the degree of freedom of the change trend information in a predetermined boundary value table, wherein the degree of freedom is obtained according to the total number of data points on the change trend information; determine the relationship between the expected value of the change trend information and the given expected value based on the probability and the significance level, wherein the significance level is the probability of obtaining a result that the expected value of the change trend information is not equal to the given expected value when the expected value of the change trend information is originally equal to the given expected value.

[0105] According to an embodiment of the present disclosure, the inspection unit 604 is also configured to obtain the average value and standard deviation of the change trend information; determine the standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; and determine the query parameter value of the change trend information based on the standard error and the average value.

[0106] According to an embodiment of the present disclosure, the anomaly determination unit 606 is also configured to, after determining that the updated version of the application is abnormal, query in a predetermined boundary value table to obtain a confidence level through the significance level and the degree of freedom of the change trend information; determine the upper limit of the confidence interval based on the confidence level and the average value of the change trend information; and determine the degree of difference between the indicator data of the updated version of the application and the current version based on the upper limit of the confidence interval.

[0107] According to an embodiment of the present disclosure, the anomaly determination unit 606 is also configured to determine the change trend of the indicator data of the updated version and the current version of the application through the Pearson correlation coefficient or the Spear rank correlation coefficient between the updated version and the current version of the application after determining that the updated version of the application is abnormal.

[0108] According to an embodiment of the present disclosure, the anomaly determination unit 606 is also configured to display alarm information after determining that the updated version of the application is abnormal, wherein the alarm information includes the degree of difference in indicator data between the updated version of the application and the current version and the change trend of the indicator data between the updated version of the application and the current version.

[0109] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 7It is a block diagram of an electronic device 700 according to an embodiment of the present disclosure, and the electronic device includes at least one memory 701 and at least one processor 702, wherein a set of computer executable instructions is stored in the at least one memory, and when the computer executable instruction set is executed by the at least one processor, a method for determining an abnormality of an updated version of an application according to an embodiment of the present disclosure is executed.

[0110] As an example, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 1000 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0111] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 702 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0112] The processor 702 may execute instructions or codes stored in the memory, wherein the memory 701 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0113] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory 701.

[0114] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0115] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method for determining an abnormal update version of the application program of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0116] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implements a method for determining an abnormal update version of an application program according to an embodiment of the present disclosure.

[0117] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0118] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining an abnormality in an updated version of an application, characterized in that: include: Obtaining change trend information of indicator data of an updated version and a current version of an application, wherein the change trend information indicates the difference between the indicator data of the updated version and the current version of the application at each sampling moment; Based on the change trend information, performing a difference check between the updated version and the current version of the application; Determining whether the updated version of the application is abnormal based on the result of the difference check; The step of performing a difference check between the updated version and the current version of the application based on the change trend information includes: Determine the number of predetermined data points whose values ​​on the change trend information satisfy a predetermined relationship with a preset value; In the case where the ratio of the number of the predetermined data points to the total number of data points on the change trend information is greater than a predetermined ratio, determining that the updated version of the application is abnormal; When the ratio of the number of the predetermined data points to the total number of data points on the change trend information is less than or equal to the predetermined ratio, a further difference test is performed between the updated version and the current version of the application based on the change trend information, wherein the further difference test is a single sample hypothesis test.

2. The determination method according to claim 1, characterized in that: The obtaining of the change trend information of the indicator data of the updated version and the current version of the application includes: Obtaining the first number of page views of the updated version of the application within a predetermined time before the current moment; When the first page views satisfy a predetermined condition, change trend information of the indicator data of the updated version and the current version of the application is obtained.

3. The determination method according to claim 2, characterized in that: The predetermined conditions are: The first access amount is greater than a preset amount, and The ratio of the first page views to the second page views is greater than a preset ratio, wherein the second page views are the sum of the first page views and the page views of the current version of the application within a predetermined time before the current moment.

4. The determination method according to claim 1, characterized in that: The obtaining of the change trend information of the indicator data of the updated version and the current version of the application includes: Obtaining first indicator data of an updated version of the application; Obtaining second indicator data of the current version of the application; Obtaining a difference between the first indicator data and the second indicator data; Based on the ratio of the difference value to the second indicator data, change trend information of the indicator data of the updated version and the current version of the application is obtained.

5. The determination method according to claim 1, characterized in that: The further performing of the difference check between the updated version and the current version of the application based on the change trend information includes: Determining the relationship between the expected value of the change trend information and the given expected value; Based on the relationship, a result of a difference check between the updated version and the current version of the application is determined.

6. The determination method according to claim 5, characterized in that: The step of determining whether the updated version of the application is abnormal according to the result of the difference check includes: When the result of the difference check is that the expected value of the change trend information is not equal to the given expected value, it is determined that the updated version of the application is abnormal.

7. The determination method according to claim 5, characterized in that: The determining the relationship between the expected value of the change trend information and the given expected value includes: Determining a query parameter value of the change trend information based on an average value and a standard deviation of the change trend information; By querying the query parameter value and the degree of freedom of the change trend information, a probability that the expected value of the change trend information is equal to the given expected value is obtained in a predetermined boundary value table, wherein the degree of freedom is obtained according to the total number of data points on the change trend information; Based on the probability and significance level, the relationship between the expected value of the changing trend information and the given expected value is determined, wherein the significance level is the probability of obtaining a result that the expected value of the changing trend information is not equal to the given expected value when the expected value of the changing trend information is originally equal to the given expected value.

8. The determination method according to claim 7, characterized in that: The determining, based on the average value and the standard deviation of the change trend information, the query parameter value of the change trend information includes: Obtaining the average value and standard deviation of the change trend information; Determining a standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; Based on the standard error and the average value, a query parameter value of the change trend information is determined.

9. The determination method according to claim 7, characterized in that: After determining that the updated version of the application is abnormal, the method further includes: Obtaining a confidence level by searching the predetermined threshold table according to the significance level and the degree of freedom of the change trend information; Determining an upper limit of a confidence interval based on the confidence level and an average value of the change trend information; Based on the upper limit of the confidence interval, the degree of difference between the indicator data of the updated version and the current version of the application is determined.

10. The determination method according to claim 9, characterized in that: After determining that the updated version of the application is abnormal, the method further includes: The change trend of the indicator data of the updated version and the current version of the application is determined by the Pearson correlation coefficient or the Spear rank correlation coefficient between the updated version and the current version of the application.

11. The determination method according to claim 10, characterized in that: After determining that the updated version of the application is abnormal, the method further includes: The alarm information is displayed, wherein the alarm information includes the difference degree of the indicator data between the updated version and the current version of the application and the change trend of the indicator data between the updated version and the current version of the application.

12. A device for determining an abnormality in an updated version of an application, characterized in that: include: a change trend information acquisition unit, configured to acquire change trend information of the indicator data of the updated version and the current version of the application, wherein the change trend information represents the difference between the indicator data of the updated version and the current version of the application at each sampling moment; a verification unit configured to perform a difference verification between the updated version and the current version of the application based on the change trend information; an abnormality determination unit, configured to determine whether the updated version of the application is abnormal based on the result of the difference check; Wherein, the verification unit is further configured to determine the number of predetermined data points on the change trend information whose values ​​satisfy a predetermined relationship with a preset value; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is greater than a predetermined ratio, determine that the updated version of the application is abnormal; when the ratio of the number of predetermined data points to the total number of data points on the change trend information is less than or equal to the predetermined ratio, perform further difference verification between the updated version of the application and the current version based on the change trend information, wherein the further difference verification is a single sample hypothesis test.

13. The determination device according to claim 12, characterized in that: The change trend information acquisition unit is also configured to obtain the first number of visits to the updated version of the application within a predetermined time before the current moment before obtaining the change trend information of the indicator data of the updated version and the current version of the application; and obtain the change trend information of the indicator data of the updated version and the current version of the application when the first number of visits meets a predetermined condition.

14. The determination device according to claim 13, characterized in that: The predetermined condition is: the first number of visits is greater than a preset number, and the ratio of the first number of visits to the second number of visits is greater than a preset ratio, wherein the second number of visits is the sum of the first number of visits and the number of visits of the current version of the application within a predetermined time before the current moment.

15. The determination device according to claim 12, characterized in that: The change trend information acquisition unit is further configured to acquire first indicator data of an updated version of the application; acquire second indicator data of a current version of the application; and acquire a difference between the first indicator data and the second indicator data; Based on the ratio of the difference value to the second indicator data, change trend information of the indicator data of the updated version and the current version of the application is obtained.

16. The determination device according to claim 12, characterized in that: The verification unit is further configured to determine a relationship between an expected value of the change trend information and a given expected value; and based on the relationship, determine a result of a difference verification between the updated version and the current version of the application.

17. The determination device according to claim 16, characterized in that: The abnormality determination unit is further configured to determine that the updated version of the application is abnormal when a result of the difference check is that the expected value of the change trend information is not equal to the given expected value.

18. The determination device according to claim 16, characterized in that: The verification unit is further configured to determine a query parameter value of the change trend information based on an average value and a standard deviation of the change trend information; The probability that the expected value of the changing trend information is equal to the given expected value is obtained by querying in a predetermined boundary value table through the query parameter value and the degree of freedom of the changing trend information, wherein the degree of freedom is obtained according to the total number of data points on the changing trend information; based on the probability and the significance level, the relationship between the expected value of the changing trend information and the given expected value is determined, wherein the significance level is the probability of obtaining a result that the expected value of the changing trend information is not equal to the given expected value when the expected value of the changing trend information is originally equal to the given expected value.

19. The determination device according to claim 18, characterized in that: The inspection unit is further configured to obtain an average value and a standard deviation of the change trend information; and determine a standard error of the change trend information based on the standard deviation and the total number of data points on the change trend information; Based on the standard error and the average value, a query parameter value of the change trend information is determined.

20. The determination device according to claim 18, characterized in that: The abnormality determination unit is further configured to, after determining that the updated version of the application is abnormal, query the predetermined boundary value table to obtain a confidence level through the significance level and the degree of freedom of the change trend information; and determine an upper limit of the confidence interval based on the confidence level and the average value of the change trend information; Based on the upper limit of the confidence interval, the degree of difference between the indicator data of the updated version and the current version of the application is determined.

21. The determination device according to claim 20, characterized in that: The abnormality determination unit is further configured to determine the change trend of the indicator data of the updated version and the current version of the application through the Pearson correlation coefficient or the Spear rank correlation coefficient of the updated version and the current version of the application after determining that the updated version of the application is abnormal.

22. The determination device according to claim 21, characterized in that The abnormality determination unit is also configured to display an alarm message after determining that the updated version of the application is abnormal, wherein the alarm message includes the degree of difference in the indicator data between the updated version and the current version of the application and the change trend of the indicator data between the updated version and the current version of the application.

23. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for determining whether an updated version of an application is abnormal according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the method for determining an abnormality in an updated version of an application program according to any one of claims 1 to 11.

25. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for determining whether an updated version of an application program is abnormal is implemented according to any one of claims 1 to 11.

Images