Application performance testing method, device, equipment and storage medium

Abstract

The present disclosure relates to a performance testing method for an application. After the running logic of the application is changed, test indicator information is collected from the data generated by the application running within a preset test cycle. The test indicator information is used to characterize the state change of the application due to the change in the running logic. Describe parameters corresponding to the test indicator information are determined. The descriptive parameters are determined based on the state of the application within a first time range. The start time of the first time range is earlier than the start time of the test cycle. Based on the test indicator information and the descriptive parameters, the performance test result of the application is determined. The performance test result indicates whether the performance of the application after the running logic is changed within a second time range meets the requirements. Because the start time of the second time range is later than the end time of the test cycle, and the test indicator information corresponds to multiple businesses of the application, the efficiency of obtaining performance test results can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a performance testing method, device, equipment and storage medium for an application program. Background Art

[0002] With the development of smart terminals, there are more and more types of applications suitable for smart terminals. As a product that provides services to users, applications will cause changes in product performance when the functions provided by the product change. In order to determine the changes in product performance, it is necessary to test the performance of the application.

[0003] The existing performance testing methods have low testing efficiency. Summary of the invention

[0004] The present disclosure provides a performance testing method, device, equipment and storage medium for an application program, so as to at least solve the problem of low efficiency of the performance testing method for an application program. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a performance testing method for an application is provided, comprising:

[0006] After the running logic of the application is changed, test indicator information is collected from the data generated by the running of the application in a preset test cycle, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application;

[0007] Determine a description parameter corresponding to the test indicator information, wherein the description parameter is determined based on a state of the application within a first time range, and a start time of the first time range is earlier than a start time of the test cycle;

[0008] Based on the test indicator information and the description parameters, a performance test result of the application is determined, and the performance test result is used to indicate whether the performance of the application after the running logic is changed within a second time range meets the requirements, and the start time of the second time range is later than the end time of the test cycle.

[0009] Optionally, determining the performance test result of the application based on the test indicator information and the description parameter includes:

[0010] Determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business;

[0011] Normalizing the service test results of each service to obtain a normalized result;

[0012] The normalized results are fused to obtain the performance test results.

[0013] Optionally, determining the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business includes:

[0014] A weighted operation is performed on the description parameter and the test index information of each service to obtain a service test result of each service, wherein the description parameter is a weighted coefficient of the weighted operation.

[0015] Optionally, the test indicator information includes: direct test indicator information and indirect test indicator information, the direct test indicator information indicates the running status of the service, and the indirect test indicator information indicates the running result of the service;

[0016] Wherein, collecting the test index information includes:

[0017] When the operating frequency of the service is lower than a preset threshold, the indirect test indicator information is collected as the test indicator information.

[0018] Optionally, after determining the performance test result of the application, the method further includes:

[0019] Obtaining a comparison result, where the comparison result is used to indicate the performance of the application program when the running logic is unchanged;

[0020] Based on the difference between the comparison result and the performance test result, determining whether a positive change occurs in the state of the application after the running logic of the application is changed;

[0021] If it is determined that the state of the application program changes in a positive direction after the running logic of the application program is changed, the test is determined to be successful.

[0022] Optionally, after determining the performance test result of the application, the method further includes:

[0023] Obtaining the online result after the running logic of the application is changed;

[0024] Comparing the difference between the online result and the performance test result;

[0025] If the difference between the online result and the performance test result is greater than a predetermined threshold, the test indicator information and / or the description parameter are adjusted.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a performance testing device for an application program, comprising:

[0027] A collection module, used to collect test indicator information from data generated by the application running in a preset test cycle after the running logic of the application is changed, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application;

[0028] A description parameter determination module, used to determine the description parameter corresponding to the test indicator information, wherein the description parameter is determined based on the state of the application within a first time range, and the start time of the first time range is earlier than the start time of the test cycle;

[0029] A performance test result determination module is used to determine the performance test result of the application based on the test indicator information and the description parameters, and the performance test result is used to indicate whether the performance of the application after the running logic is changed within a second time range meets the requirements, and the start time of the second time range is later than the end time of the test cycle.

[0030] Optionally, the performance test result determination module is used to determine the performance test result of the application based on the test indicator information and the description parameter, including:

[0031] The performance test result determination module is specifically used to determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business; normalize the business test results of each business to obtain a normalized result; and fuse the normalized result to obtain the performance test result.

[0032] Optionally, the performance test result determination module is used to determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business, including:

[0033] The performance test result determination module is specifically used to perform a weighted operation on the description parameters and the test index information of each service to obtain a service test result of each service, wherein the description parameter is a weighted coefficient of the weighted operation.

[0034] Optionally, the test indicator information includes: direct test indicator information and indirect test indicator information, the direct test indicator information indicates the running status of the service, and the indirect test indicator information indicates the running result of the service;

[0035] The acquisition module is used to acquire the test index information, including:

[0036] The collection module is specifically used to collect the indirect test indicator information as the test indicator information when the operating frequency of the service is lower than a preset threshold.

[0037] Optionally, also include:

[0038] The adjustment module is used to obtain a control result after the performance test result determination module determines the performance test result of the application, and the control result is used to indicate the performance of the application when the running logic does not change; based on the difference between the control result and the performance test result, determine whether the state of the application changes positively after the running logic of the application is changed; if it is determined that the state of the application changes positively after the running logic of the application is changed, then determine that the test is successful.

[0039] Optionally, the adjustment module is further used for:

[0040] After the performance test result determination module determines the performance test result of the application, obtaining the online result of the application after the running logic is changed;

[0041] Comparing the difference between the online result and the performance test result;

[0042] If the difference between the online result and the performance test result is greater than a predetermined threshold, the test indicator information and / or the description parameter are adjusted.

[0043] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0044] processor;

[0045] a memory for storing instructions executable by the processor;

[0046] The processor is configured to execute the instructions to implement the performance testing method of the application disclosed in the first aspect.

[0047] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the performance testing method of the application disclosed in the first aspect.

[0048] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided. When the computer program product is executed by a processor of an electronic device, the electronic device is able to perform the performance testing method of the application as described above.

[0049] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0050] After the running logic of the application is changed, test indicator information is collected from the data generated by the application running in a preset test cycle, wherein the test indicator information is used to characterize the state change of the application due to the change in the running logic, and the test indicator information corresponds to multiple services of the application. Determine the description parameters corresponding to the test indicator information, wherein the description parameters are determined based on the state of the application within the first time range, and the start time of the first time range is earlier than the start time of the test cycle. Based on the test indicator information and the description parameters, determine the performance test results of the application, and the performance test results are used to indicate whether the performance of the application after the running logic is changed meets the requirements within the second time range. Because the start time of the second time range is later than the end time of the test cycle, the application with the changed logic does not need to run within the second time range, but only needs to run before the second time range to obtain the performance test results within the second time range, so the efficiency of obtaining the performance test results can be improved. And because the test indicator information corresponds to multiple services of the application, there is no need to obtain the performance test results for each service separately, which can also improve the efficiency of the test.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0053] Figure 1 is a flow chart showing a method for performance testing of an application program according to an exemplary embodiment;

[0054] Figure 2 is a flow chart showing, according to an exemplary embodiment, evaluating the accuracy of a performance test result and performing corresponding operations based on the accuracy;

[0055] Figure 3 is a schematic diagram of a performance testing method for an application program according to an exemplary embodiment;

[0056] Figure 4 is a flow chart of obtaining description parameters by machine learning according to an exemplary embodiment;

[0057] Figure 5 is a block diagram of a performance testing device for an application program according to an exemplary embodiment;

[0058] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] The performance testing method for an application disclosed in the present invention can be applied to an apparatus or device connected to a client and a server on which the application is running, with the purpose of obtaining information generated during the operation of the application through interaction with the client and the server, and based on the obtained information, predicting the long-term impact of the change in operating logic on the performance of the application for an application whose operating logic has changed.

[0062] In the following embodiments, the operation logic changes of the application may include, but are not limited to: launching new services, changing the logic of existing services, deactivating existing services, and adding new services.

[0063] The following will take the launch of a new strategy in a video playback application as an example to explain in detail the performance testing method of the application disclosed in the present invention.

[0064] Assuming that the implementation of the new policy will affect the video service, live broadcast service and social service of the application, in this embodiment, the user viewing time of the video and live broadcast service (including the short video viewing time and the live broadcast viewing time) and the number of new user followers of the social service are measured as test indicator information.

[0065] The purpose of the performance testing method of the application described in this embodiment is to obtain the impact of the new policy on the performance of the application in the week from M+T day to M+T+7 day after the new policy is implemented on the Mth day, that is, to obtain the performance test results of the application in the week from M+T day to M+T+7 day. In this embodiment, the performance of the application is described by taking the "used state" of the application as an example.

[0066] In this embodiment, it is assumed that T=28. That is, after the new policy is implemented, the impact of the new policy on the application usage status within one week after 28 days is obtained.

[0067] like Figure 1 As shown, the performance testing method of the application described in this embodiment includes the following steps:

[0068] S101: Build a logistic regression model.

[0069] The logistic regression model is based on the principle that:

[0070] The probability of an application being used = constant term + short video duration term * short video duration term weighting coefficient + live broadcast duration term * live broadcast duration term weighting coefficient + attention term * attention term weighting coefficient + two-way attention term * two-way attention term weighting coefficient.

[0071] As mentioned above, the implementation of the new strategy will affect the video business, live broadcast business, and social business of the application. Therefore, the principle on which the logistic regression model is based includes the weighted sum of the test indicator information corresponding to these businesses.

[0072] It is understandable that the test indicator information (such as identification) of the business affected by the implementation of the new policy can be pre-configured by the R&D personnel. In this embodiment, after the test indicator information of the business is obtained from the configuration information, a logistic regression model is established.

[0073] In the above principle, the probability of an application being used is the probability that a user uses the application, which is used to indicate the user's retention probability. If the user's login information is collected, the probability that the application is used is 1, otherwise, the probability that the application is used is 0.

[0074] The constant term is a pre-configured constant value. The constant value is configured based on experience, and it is understandable that the constant term can also be pre-configured by a developer.

[0075] The short video duration item is the duration of the user watching the short video, the live broadcast duration item is the duration of the user watching the live broadcast, the follow item is the number of other users that the user has newly followed, and the two-way follow item is the number of other users that the user has newly followed. It should be noted that the user in the above rules refers to any user.

[0076] The weighted coefficient of the multiplication of each item is the weighted coefficient corresponding to each item.

[0077] S102: Collect the time the user spends watching short videos, the time the user spends watching live broadcasts, the number of other users the user follows, and the number of other users the user follows as values ​​of test indicator information on the Nth day.

[0078] Among them, N = M-28. That is, the values ​​of the evaluation indicators are collected 28 days before the implementation of the new strategy. The reason for choosing N = M-28 is that it can cover a part of the monthly active users with low activity (such as users who are only active for less than 10 days a month), so that the performance test results obtained can also be better used to optimize the experience of low-activity users.

[0079] S103: Collect the user's login information during the week from N+T day to N+T+7 day. If the user's login information exists in the application information during the week from N+T day to N+T+7 day, the probability that the application is used is 1; otherwise, the probability that the application is used is 0.

[0080] It is understandable that, within a week, as long as the user logs in once, that is, one day's login information is collected, the probability that the application is used is 1.

[0081] In this embodiment, as mentioned above, the purpose of this embodiment is to obtain the impact of the new strategy implemented on the Mth day on the performance of the application after T days, so the generation time of the numerical value collected in S102 and the interval time of the user usage probability collected in S103 are also T.

[0082] It is understandable that the execution order of S102 and S103 is not limited.

[0083] S104: Input the information collected in S102 and S103 into the logistic regression model to obtain various weighted coefficients output by the logistic regression model.

[0084] Specifically, the probability of the application being used and the values ​​of each test indicator information are known, so each weighting coefficient can be obtained.

[0085] It can be understood that the user in the above formula is any user, and the data of multiple users can be collected and input into the model as training data to obtain various weighting coefficients.

[0086] It can be seen that the above S101-S104 is a process of obtaining weighting coefficients corresponding to each test indicator information. It can be understood that as long as the test indicator information remains unchanged, the weighting coefficients can be used repeatedly without repeatedly executing S101-S104.

[0087] S105: After the new strategy is implemented on the Mth day, the time the user spends watching short videos, the time the user spends watching live broadcasts, the number of other users the user has newly followed, and the number of other users the user has newly followed as the numerical value of the test indicator information.

[0088] It is understandable that, because the method provided in the present application can use short-term data to predict long-term impact, the data collected in this embodiment can be data within one week after the implementation of the new strategy, which is used to predict the performance of the application after T=28 days.

[0089] It should be noted that, because the data collected in this step is generated after the data collected in S102, although they are all values ​​of test indicator information, for the sake of distinction, the values ​​in S102 can be called historical values.

[0090] S106: Substitute the values ​​of the test indicator information (collected in S105 instead of historical values) and the corresponding weighting coefficients into the performance test rules:

[0091] Comprehensive score = short video length item * short video length item weighted coefficient + live broadcast length item * live broadcast length item weighted coefficient + one-way attention item * one-way attention item weighted coefficient + two-way attention item * two-way attention item weighted coefficient, and the comprehensive score is obtained.

[0092] In the above performance test rules, the definitions of short video duration item, live broadcast duration item, one-way attention item and two-way attention item are as mentioned above and will not be repeated here. The weighting coefficient corresponding to each item is the weighting coefficient output by the logistic regression model in the previous step. The comprehensive score indicates the performance of the application within one week after 28 days after the running logic is changed. From the performance test rules, it can be seen that the higher the score, the better the performance. You can set a score threshold to determine whether the performance of the application within one week after 28 days after the running logic is changed meets the requirements.

[0093] It can be seen from the performance testing rules that the specific manifestations of the application's performance meeting the requirements can be: the application is used for a long time (represented by the short video duration item and the live broadcast duration item) and / or the number of users using the application is large (represented by the one-way attention item and the two-way attention item).

[0094] It can be seen from the above process that the method described in this embodiment has the following advantages:

[0095] 1. The test indicator information of video business, live broadcast business and social business is different, that is, the test indicator information for measuring video and live broadcast business is the user's viewing time, and the test indicator information for measuring social business is the number of new followers of the user. In this embodiment, different test indicator information of different businesses can be included in the performance test, achieving the purpose of comprehensive measurement. And the comprehensive score provides a unified evaluation system in the product and company dimensions, ensuring the consistency of various business decisions.

[0096] 2. In non-search engine businesses, test indicators such as conversion rate and click-through rate can only describe short-term user behavior, and may not be strongly correlated with long-term user behavior and retention. For example, on short video platforms, using click-through rate as a traction indicator in a double-column display format can easily lead to the "cover party" problem. The author deceives users into clicking through with an excellent cover, but the video quality itself is poor, which will cause user disgust and a decline in long-term retention.

[0097] In this embodiment, conversion rate and click-through rate are no longer used. Instead, the contribution of each test indicator information to the application performance in 28 days is learned through the historical values ​​of the test indicator information, and the weighted coefficient corresponding to each test indicator information is obtained (which can be regarded as a long-term indicator). In addition, the short-term values ​​of the test indicator information after the implementation of the new strategy and the corresponding weighted coefficient are used to predict the impact after 28 days. Therefore, compared with the existing method of only using short-term indicators to obtain short-term impacts and using them as long-term impacts, this method has higher accuracy.

[0098] Through experiments, it was found that the comprehensive score predicts the long-term retention trend and helps the product to iterate better when there may be inconsistencies between short-term and long-term benefits. For example, a product experiment was conducted on a short video application. This experiment forced users to use the discovery page after opening the application. The next-day retention of the experimental group (i.e., the group that used the product) in the first two weeks increased by 0.05% compared with the control group (i.e., the group that did not use the product), but the comprehensive score of the experimental group decreased by 7.28% compared with the control group. After long-term tracking of this experiment, the final long-term retention decreased by 0.02%, which is consistent with the change in the comprehensive score, indicating that this experiment optimized short-term benefits without bringing about an increase in long-term retention, verifying the advantage of using comprehensive scores in long-term benefit evaluation compared to using short-term indicators.

[0099] 3. Long-term experiments have defects, such as not taking into account the "survivor bias". That is, the user composition of the experimental group and the control group will change, and the behavior of users retained in the later period will be different from that of users in the early period. If the period is too long, then what is actually compared is the difference between users retained in the later period, which cannot represent all users. The method described in this embodiment does not require long-term experiments. For example, as mentioned above, it is only necessary to collect values ​​within one week of the launch of the new strategy, so the accuracy of the evaluation results can be improved. And, as mentioned above, it is necessary to obtain the performance of the application after the running logic is changed for one month. There is no need to wait for the application with the running logic changed to run for one month, but only one week, so the test efficiency can be improved.

[0100] 4. By statistically analyzing the performance test results under various changes, the results show that when the confidence interval width of the user usage probability is 70%, the comprehensive score prediction accuracy can reach 93%. The accuracy acquisition process will be described in the following embodiments.

[0101] 5. The probability of the application being used the next day after the running logic is changed is used as a control indicator for the performance test results. Statistics show that: when the confidence interval width of the probability of the application being used is 70%, the accuracy of the probability of the application being used the next day in predicting the probability of the application being used 28 days later is 4pp lower than the accuracy of using the comprehensive score described in this embodiment to predict the probability of the application being used 28 days later.

[0102] At the same time, we also compared the probability of the application being used the next day and the variation range of the comprehensive score of the experimental group (the group with changed application running logic) with that of the control group (the group with unchanged application running logic). We found that the 90% variation range of the comprehensive score is about 3 times the 90% variation range of the probability of the application being used the next day, that is, the variation range of the comprehensive score is larger, indicating that the comprehensive score is more sensitive than short-term indicators such as the next-day indicators.

[0103] Furthermore, the present application also discloses a process of testing the accuracy of the performance test results and performing corresponding operations based on the accuracy test results, such as Figure 2 As shown, the following steps are included:

[0104] S201: Obtain comparison results.

[0105] The control result is used to indicate the performance of the application without changing the running logic. Specifically, the control result is the performance test result of the control group, that is, the performance of the application without the new strategy (following Figure 1 Compared with the process shown in the figure, the only difference is whether the new strategy is launched, and the value and coefficient of the test indicator information are obtained. Figure 1 Compared with the process shown in the figure, the only difference is that the application does not launch a new strategy. After the application runs for M days, the time users watch short videos, the time users watch live broadcasts, the number of other users that users newly follow, and the number of other users that users newly follow each other are collected as the values ​​of the test indicator information, and used Figure 1 The various weighted coefficients obtained in S104 are used to obtain a comparison result using the performance test rule described in S104.

[0106] S202: Based on the difference between the comparison result and the performance test result, determine whether the state of the application program changes positively after the running logic of the application program is changed. If yes, execute S203; if not, execute S204.

[0107] The performance test result is the performance test result obtained by the above method, that is, the comprehensive score obtained in S104.

[0108] The difference between the control result and the performance test result includes a first difference and a second difference.

[0109] Specifically, the first difference is the difference between the performance test result and the control result.

[0110] The second difference is the difference between the first probability and the second probability. Figure 1 The probability of the application being used in the embodiment shown. The second probability is the probability of the application in the control group being used. That is, the application is used when there is no new strategy (following Figure 1 Compared with the process shown in FIG. 1 , the only difference is the probability of the application being used in the week from day N+T to day N+T+7 when a new policy is launched.

[0111] If the first difference is positive, the change trend of the performance test result is positive; if the first difference is negative, the change trend of the performance test result is negative.

[0112] If the second difference is positive, the probability trend is positive; if the second difference is negative, the probability trend is negative.

[0113] A positive change in the state of the application means that the change trend of the performance test result is consistent with the change trend of the probability (both are positive or both are negative), and / or the first probability and the second probability are within a preset confidence interval. A positive change in the state of the application indicates that the accuracy of the performance test result meets the requirements. Otherwise, it indicates that the accuracy of the performance test result does not meet the requirements.

[0114] The preset confidence interval width may be 70%.

[0115] S203: Determine whether the test is successful.

[0116] If the state of the application program changes positively, it means that the accuracy of the performance test result meets the requirement, and the test can be considered successful. After S203, the process continues with S205.

[0117] S204: Adjust test indicator information and / or weighting coefficient.

[0118] Specifically, the weighting coefficient can be adjusted by increasing the number of samples used to train the logistic regression model, for example Figure 1 In the process shown, the number of users used is 1000, which can be expanded to 5000, and calculation and iteration are performed again to obtain new weighting coefficients.

[0119] S205: Deploy the model for obtaining performance test results online.

[0120] The model for obtaining the performance test results is constructed based on weighting coefficients and performance test rules, such as the calculation rules for the above-mentioned comprehensive score.

[0121] After S205, you can continue to perform the following steps:

[0122] S206: Obtain the online result after the running logic of the application is changed.

[0123] The online result refers to the actual performance of the application after 28 days of operation after the application is officially launched after the running logic is changed, for example, the probability of the application being used after 28 days of operation.

[0124] S207: Compare the difference between the online result and the performance test result. If the difference between the online result and the performance test result is greater than a preset threshold, return to execute S204.

[0125] The difference between the online results and the performance test results is greater than the preset threshold, indicating that the actual performance of the application after running the logic change for 28 days is different from that of the application using the Figure 1 The performance predicted by the method shown is inconsistent, so the prediction method needs to be adjusted. In this embodiment, the performance test method is adjusted by adjusting the test indicator information and / or the weighting coefficient.

[0126] Figure 2 The process shown uses the experimental group data and the control group data to test the accuracy of the performance test results from the perspective of the performance test results and the probability of the application being used. If the accuracy condition is met, it means that the performance test results are relatively accurate. Therefore, the performance test method model of the application can be deployed online. Otherwise, it means that the performance test method of the application needs to be improved, and then it can be improved to further improve the accuracy of the performance test results. Furthermore, after the performance test method model of the application is deployed online, the actual performance can be compared with the predicted performance. If the difference is large, the performance test method can be improved to further improve the accuracy of the performance test method model.

[0127] That is to say, Figure 2 The method shown provides ways to improve the performance testing method model from two different approaches. In actual applications, you can choose one as needed.

[0128] It should be noted that the above description is based on the example of launching a new strategy in a video playback application, but the method provided by the present invention is not limited to the above scenarios. For example, it is not limited to video playback applications, is not limited to launching a new strategy, and is therefore not limited to the above-mentioned services, the above-mentioned evaluation indicators, the above-mentioned T values ​​and N values, etc.

[0129] Therefore, the method described in the above embodiment can be summarized as follows: Figure 3 The process shown includes the following steps:

[0130] S301: After the running logic of the application is changed, test indicator information is collected from the data generated when the application runs within a preset test cycle.

[0131] The test indicator information is the test indicator information of the business affected by the change of the running logic of the application program indicated by the pre-acquired information. The pre-acquired information can be information pre-configured by the R&D personnel, as described above, and will not be repeated here.

[0132] In this embodiment, the services affected by the change in the running logic of the application program may be multiple (as in the above embodiment) or one, and may be pre-configured by the R&D personnel according to the logic of the services provided by the application.

[0133] Optionally, the test indicator information of the service includes direct test indicator information and indirect test indicator information. Direct test indicator information is information of an indicator indicating the operating status of the service. Taking the live broadcast service as an example, the live broadcast viewing time of the user indicates that the live broadcast service is used by the user, so the live broadcast viewing time of the user is the direct test indicator information of the live broadcast service.

[0134] Indirect test indicator information is information about indicators that indicate the operating results of a business. Taking the short video publishing business as an example, users publishing short videos is a low-frequency behavior. If the weighting coefficient of this business is directly trained in the logistic regression model, the weighting coefficient obtained is often not credible enough due to the small amount of data. Therefore, for the short video publishing business, indirect test indicator information is specified based on the effect it produces.

[0135] Specifically, the upload of a short video may lead to an increase in the number of the author's pun friends, and because more friends have seen the work, their willingness to log in to the application increases. Therefore, the indirect test indicator information corresponding to the short video publishing business can be:

[0136] Number of pun friends * probability of a pun friend watching * viewing time * value of unit viewing time (this item can be obtained according to preset rules).

[0137] That is, starting from the perspective of the effect produced after the service is used (i.e. the operating results of the service), the test indicator information is designed to improve the accuracy of the performance test results.

[0138] Optionally, when the frequency of use of the service is lower than a preset threshold, the test indicator information of the service is configured as indirect test indicator information, that is, the indirect test indicator information of the service is collected as the test indicator information to improve the accuracy of the performance test result. Otherwise, the test indicator information of the service is direct test indicator information.

[0139] S302: Determine description parameters corresponding to the test indicator information.

[0140] The description parameter is a parameter that identifies the weight of the test indicator information, that is, the contribution to the performance test result. The description parameter includes but is not limited to the above-mentioned weighting coefficient. The description parameter is determined based on the state of the application within the first time range, and the start time of the first time range is earlier than the start time of the test cycle.

[0141] Specifically, the description parameters are obtained by machine learning the historical values ​​of the test indicator information of the business, the probability of the application being used within the first time range, and the rules between the probability, the description parameters and the historical values. Optionally, the specific implementation of machine learning includes but is not limited to the above-mentioned logistic regression model, and other types of models can also be used as long as the purpose of learning the description can be achieved.

[0142] In this embodiment, optionally, the start time of the first time range is a preset time distance from the generation time of the historical value. In the above embodiment, the example of the first time range is the N+Tth day to the N+T+7th day, that is, at least T days away from the generation time of the historical value.

[0143] In the above embodiment, the value of T is a value determined after multiple attempts in the video playback application scenario. For other applications, the value of T can be determined from the following two aspects: 1. The relative size of each description parameter obtained is in line with expectations (i.e., the situation considered normal by the R&D personnel), and there will be no situation where the description parameter of a certain important test indicator information is negative. 2. The accuracy of the performance test result obtained is higher than other values.

[0144] Similarly, the generation time of the historical value is earlier than the time when the running logic of the application program changes, that is, the value of N can be other than 28. In addition to the aforementioned embodiment, the optional value basis can also be other basis, such as the number of video or live broadcasts. Specifically, the specific method of determining based on the user's active time can be: N = X times the minimum user activity + Y. The values ​​of X and Y can be set according to requirements.

[0145] Combined with the expansion of each parameter value, the way machine learning obtains description parameters can be summarized as follows: Figure 4 shown.

[0146] Another way to obtain the description parameters is to read the description parameters. The description parameters can be obtained and stored in advance by machine learning. In this step, they can be directly read.

[0147] It is understandable that, in addition to "day", the value units of all time in this embodiment can also be other units, such as "hour".

[0148] S303: Determine the performance test result of the application based on the test indicator information and the description parameters.

[0149] The performance test result is used to indicate whether the performance of the application program after the logic change meets the requirements within the second time range, wherein the start time of the second time range is later than the end time of the test cycle.

[0150] Optionally, the business test results of each business of the application can be determined based on the test indicator information and description parameters corresponding to each business. The business test results of each business are normalized to obtain a normalized result. The normalized results are fused to obtain a performance test result. Among them, the method of multiplying the numerical value of the above-mentioned test indicator information with the weighting coefficient is only a specific implementation method of determining the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business, and the above-mentioned weighted sum is only an implementation method of fusion processing, which is not limited in this embodiment.

[0151] Compared with the method of obtaining multiple performance test results and then fusing the multiple performance test results by testing indicator information for each business separately, in this embodiment, the business test results of each business (that is, the product of the test indicator information and the weighting coefficient) are obtained and then the test results of each business are merged. This method has higher efficiency because the values ​​of the test indicator information of each business can be collected at the same time.

[0152] The normalized result is the ratio of the service test result of each service to the standard value, and the standard value is any service test result. The standard value can be specified by R&D personnel according to requirements.

[0153] It can be seen that the normalized result can show the difference between the business test results of each business, so, optionally, the normalized result can also be displayed to facilitate R&D personnel to observe and adjust the description coefficient.

[0154] Figure 3 The embodiment shown has the following advantages:

[0155] 1. By combining short-term business indicators with long-term user retention through machine learning methods, it not only solves the evaluation problem that short-term indicators cannot represent long-term benefits, but also avoids long-term experiments, improves the efficiency of performance testing, and saves decision-making time. Furthermore, the test indicator information of multiple businesses is integrated to obtain performance test results, which can also improve test efficiency.

[0156] 2. Through machine learning methods, descriptive parameters of multiple business test indicator information are found. The obtained performance test results can not only be used to judge a single business, but also to make trade-offs when multiple businesses conflict and obtain the final benefit judgment.

[0157] 3. The display of each normalized result is helpful for R&D personnel to correct and adjust the description, making the performance test results more accurate.

[0158] Figure 4 The process of obtaining description parameters for machine learning disclosed in the present invention includes the following steps:

[0159] S401: Collect historical values ​​of test indicator information at target time.

[0160] The target time is earlier than the time when the change occurs. In this embodiment, the example of the time when the change occurs is the above M, and the example of the target time is the above N. It is understandable that the unit of measurement of time is not limited to "day". The interval between N and M can be set as required and is not limited to the examples in the above embodiment.

[0161] S402: Collect state parameters of the application within a first time range (such as the probability of the application being used as described in the above embodiment).

[0162] The target time is earlier than the start time of the first time range by a preset time length. An example of the preset time length is the above T, but the value is not limited to the example in the above embodiment.

[0163] S403: Input historical values ​​and probabilities into a preset model to obtain descriptive parameters.

[0164] The model is established based on the test index information and probability. The model example may be the above-mentioned logistic regression model, or other models, such as a neural network model.

[0165] In this embodiment, the model is trained through historical numerical values ​​to obtain descriptive parameters of each test indicator information, thereby laying a foundation for predicting the long-term impact of changes in the operating logic of the application.

[0166] Figure 5 FIG. 1 is a block diagram of a performance testing device for an application program according to an exemplary embodiment. Figure 5 , the device comprises:

[0167] A collection module, used to collect test indicator information from data generated by the application running in a preset test cycle after the running logic of the application is changed, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application;

[0168] A description parameter determination module, used to determine the description parameter corresponding to the test indicator information, wherein the description parameter is determined based on the state of the application within a first time range, and the start time of the first time range is earlier than the start time of the test cycle;

[0169] A performance test result determination module is used to determine the performance test result of the application based on the test indicator information and the description parameters, and the performance test result is used to indicate whether the performance of the application after the running logic is changed within a second time range meets the requirements, and the start time of the second time range is later than the end time of the test cycle.

[0170] Regarding the performance testing device for the application in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0171] The performance testing device for the application disclosed in the above embodiment can be applied to electronic devices, such as mobile phones, computers, servers, etc. Optionally, Figure 6 The hardware structure diagram of the electronic device is shown in FIG. Figure 6 , the hardware structure of the electronic device may include: a processor 1, a communication interface 2, a memory 3 and a communication bus 4;

[0172] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0173] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0174] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0175] The memory stores executable instructions, and the processor can call the instructions stored in the memory, wherein the instructions are used to:

[0176] A performance testing method for an application is provided, including:

[0177] After the running logic of the application is changed, test indicator information is collected from the data generated by the running of the application in a preset test cycle, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application;

[0178] Determine a description parameter corresponding to the test indicator information, wherein the description parameter is determined based on a state of the application within a first time range, and a start time of the first time range is earlier than a start time of the test cycle;

[0179] Based on the test indicator information and the description parameters, a performance test result of the application is determined, and the performance test result is used to indicate whether the performance of the application after the running logic is changed within a second time range meets the requirements, and the start time of the second time range is later than the end time of the test cycle.

[0180] Optionally, determining the performance test result of the application based on the test indicator information and the description parameter includes:

[0181] Determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business;

[0182] Normalizing the service test results of each service to obtain a normalized result;

[0183] The normalized results are fused to obtain the performance test results.

[0184] Optionally, determining the business test results of each business applied to the application based on the test indicator information and description parameters corresponding to each business includes:

[0185] A weighted operation is performed on the description parameter and the test index information of each service to obtain a service test result of each service, wherein the description parameter is a weighted coefficient of the weighted operation.

[0186] Optionally, the test indicator information includes: direct test indicator information and indirect test indicator information, the direct test indicator information indicates the running status of the service, and the indirect test indicator information indicates the running result of the service;

[0187] Wherein, collecting the test index information includes:

[0188] When the operating frequency of the service is lower than a preset threshold, the indirect test indicator information is collected as the test indicator information.

[0189] Optionally, after determining the performance test result of the application, the method further includes:

[0190] Obtaining a comparison result, where the comparison result is used to indicate the performance of the application program when the running logic is unchanged;

[0191] Based on the difference between the comparison result and the performance test result, determining whether a positive change occurs in the state of the application after the running logic of the application is changed;

[0192] If it is determined that the state of the application program changes in a positive direction after the running logic of the application program is changed, the test is determined to be successful.

[0193] Optionally, after determining the performance test result of the application, the method further includes:

[0194] Obtaining the online result after the running logic of the application is changed;

[0195] Comparing the difference between the online result and the performance test result;

[0196] If the difference between the online result and the performance test result is greater than a predetermined threshold, the test indicator information and / or the description parameter are adjusted.

[0197] Optionally, the detailed functions and extended functions of the instruction may refer to the above description.

[0198] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0199] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0200] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A performance testing method for an application, characterized in that: include: After the running logic of the application is changed, test indicator information is collected from the data generated by the running of the application in a preset test cycle, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application; Determine a description parameter corresponding to the test indicator information, wherein the description parameter is determined based on a state of the application within a first time range, and a start time of the first time range is earlier than a start time of the test cycle; Based on the test indicator information and the description parameters, a performance test result of the application is determined, and the performance test result is used to indicate whether the performance of the application after the running logic is changed meets the requirements within a second time range, and the start time of the second time range is later than the end time of the test cycle.

2. The method according to claim 1, characterized in that The step of determining the performance test result of the application based on the test indicator information and the description parameter includes: Determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business; Normalizing the service test results of each service to obtain a normalized result; The normalized results are fused to obtain the performance test results.

3. The method according to claim 2, characterized in that Determining the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business includes: A weighted operation is performed on the description parameter and the test index information of each service to obtain a service test result of each service, wherein the description parameter is a weighted coefficient of the weighted operation.

4. The method according to claim 1 or 2, characterized in that: The test indicator information includes: Direct test indicator information and indirect test indicator information, wherein the direct test indicator information indicates the operation status of the service, and the indirect test indicator information indicates the operation result of the service; Wherein, collecting the test index information includes: When the operating frequency of the service is lower than a preset threshold, the indirect test indicator information is collected as the test indicator information.

5. The method according to claim 1, characterized in that After determining the performance test result of the application, the method further includes: Obtaining a comparison result, where the comparison result is used to indicate the performance of the application program when the running logic is unchanged; Based on the difference between the comparison result and the performance test result, determining whether a positive change occurs in the state of the application after the running logic of the application is changed; If it is determined that the state of the application program changes in a positive direction after the running logic of the application program is changed, the test is determined to be successful.

6. The method according to claim 5, characterized in that After determining the performance test result of the application, the method further includes: Obtaining the online result after the running logic of the application is changed; Comparing the difference between the online result and the performance test result; If the difference between the online result and the performance test result is greater than a predetermined threshold, the test indicator information and / or the description parameter are adjusted.

7. A performance testing device for an application, characterized in that: include: A collection module, used to collect test indicator information from data generated by the application running in a preset test cycle after the running logic of the application is changed, wherein the test indicator information is used to characterize the state change of the application due to the running logic change, and the test indicator information corresponds to multiple services of the application; A description parameter determination module, used to determine the description parameter corresponding to the test indicator information, wherein the description parameter is determined based on the state of the application within a first time range, and the start time of the first time range is earlier than the start time of the test cycle; A performance test result determination module is used to determine the performance test result of the application based on the test indicator information and the description parameters, and the performance test result is used to indicate whether the performance of the application after the running logic is changed within a second time range meets the requirements, and the start time of the second time range is later than the end time of the test cycle.

8. The device according to claim 7, characterized in that The performance test result determination module is used to determine the performance test result of the application based on the test indicator information and the description parameter, including: The performance test result determination module is specifically used to determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business; normalize the business test results of each business to obtain a normalized result; and fuse the normalized result to obtain the performance test result.

9. The device according to claim 8, characterized in that The performance test result determination module is used to determine the business test results of each business of the application based on the test indicator information and description parameters corresponding to each business, including: The performance test result determination module is specifically used to perform a weighted operation on the description parameters and the test index information of each service to obtain a service test result of each service, wherein the description parameter is a weighted coefficient of the weighted operation.

10. The device according to claim 7 or 8, characterized in that The test indicator information includes: direct test indicator information and indirect test indicator information, wherein the direct test indicator information indicates the operation status of the service, and the indirect test indicator information indicates the operation result of the service; The acquisition module is used to acquire the test index information, including: The collection module is specifically used to collect the indirect test indicator information as the test indicator information when the operating frequency of the service is lower than a preset threshold.

11. The device according to claim 7, characterized in that Also includes: The adjustment module is used to obtain a control result after the performance test result determination module determines the performance test result of the application, and the control result is used to indicate the performance of the application when the running logic does not change; based on the difference between the control result and the performance test result, determine whether the state of the application changes positively after the running logic of the application is changed; if it is determined that the state of the application changes positively after the running logic of the application is changed, then determine that the test is successful.

12. The device according to claim 11, characterized in that The adjustment module is also used for: After the performance test result determination module determines the performance test result of the application, obtaining the online result of the application after the running logic is changed; Comparing the difference between the online result and the performance test result; If the difference between the online result and the performance test result is greater than a predetermined threshold, the test indicator information and / or the description parameter are adjusted.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the performance testing method of the application program as described in any one of claims 1-6.

14. A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the performance testing method of an application as described in any one of claims 1-6.

Images