A method for performance evaluation based on autonomous adjustment of virtual user number

Abstract

The present application relates to the field of intelligent performance evaluation, and more particularly to a method for self-regulating virtual user number for performance evaluation, comprising: for each collapse point analysis, determining the collapse paragraph extraction method as extracting according to the number of peak points to obtain the collapse paragraph or recording the relevant paragraph corresponding to the target collapse point as the collapse paragraph; determining the cache data state according to the cache amplitude coefficient and the cache difference coefficient of the cache tracing duration corresponding to the collapse paragraph, and determining the effective collapse paragraph according to the cache data state; clustering according to the cache strategy alternation similarity and the cache topology structure similarity corresponding to each effective collapse paragraph to obtain a plurality of paragraph sets; calculating the user characteristic uniformity according to the operation parameters of each user in each paragraph set, and determining the parameter setting strategy according to the user characteristic uniformity; performing a preset number of virtual tests and recording the results. The present application optimizes the virtual user number regulation method, effectively improves the precision and efficiency of performance evaluation.

Description

Technical Field

[0001] This invention relates to the field of intelligent performance evaluation, and in particular to a method for performance evaluation that autonomously adjusts the number of virtual users. Background Technology

[0002] Current performance testing tools on the market have limitations in intelligently, automatically, and efficiently detecting the optimal and maximum performance indicators of a system. They often require continuous manual intervention to adjust the number of virtual users and their parameters until the expected goals are achieved. Therefore, how to achieve intelligent, efficient, and accurate performance testing has become an urgent problem to be solved.

[0003] Chinese Patent Publication No. CN117056145A discloses a method for loading a specified number of virtual users based on automated test virtual machine technology. This technical solution includes a stress test task, a main automated test virtual machine, a virtual user allocation module, slave automated test virtual machines, and a server under test. This method, based on automated test virtual machine technology, automatically allocates virtual users to each automated test virtual machine for different stress test tasks. However, this technical solution has the following problems: although it can automatically allocate virtual users, the initial total number of virtual users is manually set, and it does not consider the proportion of virtual users for different operation types, nor can it adaptively adjust virtual test parameters based on the historical data of the application under test. This results in the test's realism failing to meet the needs of actual scenarios, leading to poor test results. Summary of the Invention

[0004] To address this, the present invention provides a method for performance evaluation by autonomously adjusting the number of virtual users, thereby overcoming the problems in the prior art where the setting of the proportion of virtual users of different operation types is not considered and the virtual test parameters cannot be adaptively adjusted according to the historical records of the application under test, resulting in poor test results.

[0005] To achieve the above objectives, a method for performance evaluation that autonomously adjusts the number of virtual users includes:

[0006] For each crash point, relevant analysis is conducted to determine the corresponding crash segment extraction method. This method can be based on the peak number of people to obtain the crash segment, or the relevant segments corresponding to the target crash point can be recorded as the crash segment.

[0007] The cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache trace duration of the crash segment, and the cache data status is used to determine whether the crash segment is a valid crash segment.

[0008] Clustering is performed based on the similarity of cache strategy iterations and cache topology of each valid crash segment to obtain several segment sets;

[0009] Calculate the user characteristic uniformity based on the operation parameters of each user within each paragraph set, and determine the parameter setting strategy based on the user characteristic uniformity.

[0010] Perform a preset number of virtual tests and record the results of each virtual test.

[0011] Furthermore, correlation analysis is performed on each crash point to determine the corresponding crash segment extraction method. This correlation analysis for a single crash point includes:

[0012] The crash point that is subjected to relevant analysis is recorded as the target crash point;

[0013] The method for extracting collapse segments is determined based on the density of relevant collapse points corresponding to the target collapse point;

[0014] If the density of relevant collapse points is less than the preset density of relevant collapse points, the collapse paragraph extraction method is to extract based on the peak number of people; if the density of relevant collapse points is greater than or equal to the preset density of relevant collapse points, the collapse paragraph extraction method is to record the relevant paragraphs corresponding to the target collapse point as collapse paragraphs.

[0015] Furthermore, when extracting the collapse segment based on the peak number of people, the peak number of people in the relevant segment corresponding to the target collapse point is detected, and the time segment between the peak number of people and the target collapse point is recorded as the collapse segment.

[0016] Furthermore, the cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache traceability time of the crash segment. When the cache data status is that the cache amplitude coefficient is greater than the preset cache amplitude coefficient or the cache difference coefficient is greater than the preset cache difference coefficient, the crash segment is determined to be a valid crash segment.

[0017] Furthermore, the cache tracing duration is determined based on the number of errors corresponding to the crash point;

[0018] The number of reported errors is positively correlated with the cache traceability time.

[0019] Furthermore, clustering is performed based on the similarity of cache strategy iterations and cache topology similarity corresponding to each effective crash segment to obtain several segment sets;

[0020] For a single paragraph set, which includes several valid crash paragraphs, the similarity of cache strategy iterations between any two valid crash paragraphs is greater than the similarity of preset cache strategy iterations or the similarity of cache topology is greater than the similarity of preset cache topology.

[0021] Furthermore, the similarity of cache strategy iterations is determined based on the frequency of dynamic cache adjustment and the richness of cache strategy types;

[0022] The difference between the similarity of cache strategy iterations and the frequency of dynamic cache adjustments is negatively correlated.

[0023] The similarity of cache strategy iterations is negatively correlated with the difference in cache strategy types.

[0024] The similarity of the cache topology is determined based on the number of different servers;

[0025] The similarity of the cache topology is negatively correlated with the number of differences between the nodes.

[0026] Furthermore, the user characteristic uniformity is calculated based on the operation parameters of each user within each paragraph set, and the parameter setting strategy is determined based on the user characteristic uniformity.

[0027] If the uniformity of user characteristics is greater than or equal to the preset uniformity of user characteristics, the parameter setting strategy is to set the virtual parameters based on the average value of each operation parameter.

[0028] If the uniformity of user features is less than the preset uniformity of user features, the parameter setting strategy is to set the setting range and virtual number of virtual parameters according to the uniformity difference.

[0029] The operating parameters include the user's thinking time coefficient and the user's operation complexity coefficient.

[0030] Furthermore, a single virtual test includes:

[0031] S1, perform relevant data settings and virtual parameter settings;

[0032] S2, run the virtual user count of the preset scenario in a preset order, and store the running data in the database during the process;

[0033] S3, obtain the comparison result between the error rate and the preset error rate. If the error rate is greater than the preset error rate, proceed to step S4. If the error rate is less than or equal to the preset error rate, proceed to step S5.

[0034] S4, the number of virtual users is reduced and adjusted, and step S6 is executed;

[0035] S5, the number of virtual users is increased and adjusted, and step S6 is executed;

[0036] S6, jump to step S2, until the absolute value of the difference between the adjusted number of virtual users and the number of virtual users before adjustment is less than the preset difference in the number of users or the number of loops is less than the preset number of loops, then the single virtual test ends.

[0037] Furthermore, the preset number of times is the number of parameter setting strategies.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: In the technical solution of the present invention, relevant analysis is performed on each crash point to determine the corresponding crash segment extraction method. The density of relevant crash points corresponding to the crash point reflects the degree of crash of the relevant segment corresponding to the crash point, and different crash segment extraction methods are selected accordingly. This makes the selection of crash segment extraction methods more consistent with the actual distribution of crash points, accurately locates the key segments of the system performance load limit, provides accurate data basis for subsequent analysis, avoids the time loss caused by a large amount of data analysis, and further improves the efficiency of performance evaluation.

[0039] Furthermore, in the technical solution of the present invention, the cache data status is determined based on the cache amplitude coefficient and cache difference coefficient of the cache traceability time corresponding to the crash segment, and the valid crash segment is determined based on the cache data status. The cache amplitude coefficient and cache difference coefficient reflect whether there is an anomaly in the cache status of the server corresponding to the crash segment, and the valid crash segment is selected, which reduces the amount of data for subsequent analysis and further improves the efficiency of performance evaluation.

[0040] Furthermore, in the technical solution of this invention, clustering is performed based on the similarity of cache strategy iterations and cache topology structure corresponding to each effective crash segment to obtain several segment sets. Taking into account the similarity of the cache logic of the server corresponding to the effective crash segment and the similarity of the server connection topology results, comprehensive clustering is performed to ensure accurate clustering of effective crash segments, thereby improving the accuracy of subsequent analysis of user feature coefficients, thus improving the accuracy of parameter setting strategies, and further improving the accuracy of performance evaluation.

[0041] Furthermore, in the technical solution of the present invention, the parameter setting strategy is determined based on the uniformity of user characteristics. The uniformity of user characteristics effectively represents the degree of similarity of operations between users, and different parameter setting strategies are selected accordingly, so that the parameter setting strategy is more in line with the actual user operation, thereby effectively improving the test accuracy of performance evaluation. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the method for performance evaluation based on autonomously adjusting the number of virtual users according to the present invention;

[0043] Figure 2 This is a flowchart for determining valid crash segments in this invention;

[0044] Figure 3 This is a flowchart illustrating the parameter setting strategy determined based on user characteristic uniformity in this invention.

[0045] Figure 4 This is a flowchart of the virtual test of the present invention;

[0046] Figure 5 This is a schematic diagram illustrating the adjustment of the virtual user number reduction method according to the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the adjustment of the number of virtual users in this invention. Detailed Implementation

[0048] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0049] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0050] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0051] Please see Figures 1 to 3 As shown, the present invention provides a method for performance evaluation based on autonomously adjusting the number of virtual users, characterized in that it includes:

[0052] For each crash point, relevant analysis is conducted to determine the corresponding crash segment extraction method. This method can be based on the peak number of people to obtain the crash segment, or the relevant segments corresponding to the target crash point can be recorded as the crash segment.

[0053] The cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache trace duration of the crash segment, and the cache data status is used to determine whether the crash segment is a valid crash segment.

[0054] Clustering is performed based on the similarity of cache strategy iterations and cache topology of each valid crash segment to obtain several segment sets;

[0055] Calculate the user characteristic uniformity based on the operation parameters of each user within each paragraph set, and determine the parameter setting strategy based on the user characteristic uniformity.

[0056] Perform a preset number of virtual tests and record the results of each virtual test.

[0057] This invention is applied to software performance development testing. Based on the application of older software versions, it performs targeted virtual testing on the new version of the software to be tested to improve the accuracy of the test. The software is a knowledge base software system. Users can send data requests to the server, and the server connects to different distributed databases to obtain data according to the requests, integrates and processes the data, and then sends it to the user.

[0058] This invention utilizes historical records, each of which includes at least the crash point, number of errors, density of related crash points, number of users, crash segment, peak number of users, cache traceability duration, cache amplitude coefficient, cache difference coefficient, effective crash segment, strategy iteration similarity, cache topology similarity, difference, difference in cache dynamic adjustment frequency, difference in cache strategy variety richness, difference in operation complexity coefficient, difference in thinking time coefficient, user thinking time coefficient, user operation complexity coefficient, user characteristic uniformity, and the number of server database connections. Furthermore, each historical record is assigned a qualification marker, indicating whether the historical record meets the operator's requirements. Whether a historical record meets the operator's requirements can be determined, but is not limited to, based on the number of software errors within several monitoring periods. It is understood that determining whether a historical record meets the operator's requirements based on server performance indicators (e.g., number of errors, user ratings) set by the administrator is content already known to those skilled in the art and is not limited here.

[0059] Specifically, correlation analysis is performed on each crash point to determine the corresponding crash segment extraction method. This includes correlation analysis on a single crash point, which includes:

[0060] The crash point that is subjected to relevant analysis is recorded as the target crash point;

[0061] The method for extracting collapse segments is determined based on the density of relevant collapse points corresponding to the target collapse point;

[0062] If the density of relevant collapse points is less than the preset density of relevant collapse points, the collapse paragraph extraction method is to extract based on the peak number of people; if the density of relevant collapse points is greater than or equal to the preset density of relevant collapse points, the collapse paragraph extraction method is to record the relevant paragraphs corresponding to the target collapse point as collapse paragraphs.

[0063] The crash point is the point in time when the number of errors reported in the old version of the software during historical use exceeds the preset error count. It is understandable that software application crashes usually trigger error reports, so the number of errors can be used as an important indicator to measure the crash frequency. The higher the number of errors, the greater the probability of a crash. The preset error count can be set by the operator. The greater the operator's need for the stability of the software application, the smaller the value of the preset error count. A preset error count value is provided. The error count corresponding to the historical records that meet the operator's needs is extracted, and outliers are removed. The average value of the error count after removing outliers is recorded as the preset error count. The methods for removing outliers include, but are not limited to, the 3σ criterion method or the IQR method.

[0064] The relevant crash point is the crash point within the relevant segment. The relevant segment is a time segment centered on the target crash point. The duration of the relevant segment can be set by the operator according to actual needs. It is understood that the longer the duration of the relevant segment, the more likely the number of crash points within the relevant segment will increase, making it easier to achieve a relevant crash point density greater than or equal to the preset relevant crash point density. Therefore, the lower the operator's tolerance for the density of crash points, the longer the duration of the relevant segment. One possible value for the duration of the relevant segment is 10 minutes.

[0065] The operator can set the preset value of the relevant crash point density according to the actual scenario. It can be understood that the present invention reflects the software crash probability in the relevant paragraph corresponding to the target crash point through the relevant crash point density. The higher the relevant crash point density, the higher the software crash probability. Therefore, the greater the operator's requirement for the accuracy of the crash paragraph determined based on the relevant paragraph, the higher the preset relevant crash point density. A value setting method is provided to extract the relevant crash point density corresponding to the historical record that meets the operator's needs, remove the outliers of the relevant crash point density, and record the average value of the relevant crash point density after removing outliers as the relevant crash point density.

[0066] Specifically, when extracting the collapse segment based on the peak number of people, the peak number of people in the relevant segment corresponding to the target collapse point is detected, and the time segment between the peak number of people and the target collapse point is recorded as the collapse segment.

[0067] The method for identifying the peak number of users is to extract the moment when the number of users of the software is the largest in the relevant paragraph corresponding to the target crash point. If the number of users at that moment is greater than the preset number of users, then that moment is recorded as the peak number of users.

[0068] If the number of users at that moment is less than or equal to the preset number of users, then that moment will not be recorded as the peak number of users, and it will be determined that there is no corresponding collapse segment at the target collapse point;

[0069] The preset number of users can be set by the operator according to the actual scenario. Understandably, the higher the operator's requirement for the accuracy of the crash segment judgment, the larger the preset number of users will be. A value selection method is provided to extract the corresponding number of users in the historical records that meet the needs of the management personnel, remove outliers in the number of users, and record the number of users after removing outliers as the preset number of users.

[0070] Specifically, the cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache traceback time of the crash segment. When the cache data status is that the cache amplitude coefficient is greater than the preset cache amplitude coefficient or the cache difference coefficient is greater than the preset cache difference coefficient, the crash segment is determined to be a valid crash segment. If the cache data status is that the cache amplitude coefficient is less than or equal to the preset cache amplitude coefficient and the cache difference coefficient is less than or equal to the preset cache difference coefficient, the crash segment is determined to be an invalid crash segment.

[0071] The cache amplitude coefficient is determined by uniformly extracting several time points within the crash segment, detecting the average amount of cached data corresponding to each server at each time point, which is recorded as the sub-cache amount, and recording the average amount of sub-cache amount corresponding to each time point as the cache amplitude coefficient.

[0072] The cache difference coefficient is determined by detecting the amount of cached data on each server at each time point, calculating the sub-difference quantity S at each time point, and the cache difference coefficient is the average of the sub-difference quantities. The formula for calculating the sub-difference quantity is as follows:

[0073]

[0074] Where Si is the amount of cached data corresponding to the i-th server, S0 is the average amount of cached data corresponding to each server, i = 1, 2, 3, ..., n, where n is the total number of servers. It can be understood that the value of i corresponding to the server will not affect the calculation result.

[0075] The preset cache amplitude coefficient and preset cache difference coefficient can be set by the operator according to the actual scenario. It can be understood that the larger the value of the cache amplitude coefficient, the greater the server's cache load, the higher the probability of crash, and the more representative the crash segment reflected. The larger the value of the cache difference coefficient, the greater the imbalance of cache load between servers, and the more representative the crash segment reflected. Therefore, the greater the operator's requirement for the accuracy of the determination of effective crash segments, the larger the preset cache amplitude coefficient and preset cache difference coefficient should be. A value setting method is provided to extract the corresponding cache amplitude coefficient and cache difference coefficient from the historical records that meet the operator's needs, remove outliers from the cache amplitude coefficient and cache difference coefficient respectively, and record the average value of the cache amplitude coefficient and cache difference coefficient after removing outliers as the preset cache amplitude coefficient and preset cache difference coefficient respectively.

[0076] Specifically, the cache lookup duration is determined based on the number of errors corresponding to the crash point;

[0077] The number of reported errors is positively correlated with the cache traceability time.

[0078] The cache trace duration is calculated as follows: Cache trace duration = Base trace duration + K × Number of errors, where K is the conversion factor.

[0079] The operator can set the baseline traceability duration according to actual needs. It is understood that the larger the baseline traceability duration, the greater the accuracy of the cached data status determination. One baseline traceability duration is provided, which is 15 minutes. The operator can also set the value of K according to actual needs. It is understood that the greater the importance the operator attaches to the number of errors, the larger the value of K. One value of K is provided, where the current number of errors is 103, and K is 0.6. The unit of the value obtained by K × the number of errors is fixed to seconds.

[0080] Specifically, clustering is performed based on the similarity of cache strategy iterations and cache topology similarity corresponding to each valid crash segment to obtain several segment sets;

[0081] For a single paragraph set, which includes several valid crash paragraphs, the similarity of cache strategy iterations between any two valid crash paragraphs is greater than the similarity of preset cache strategy iterations or the similarity of cache topology is greater than the similarity of preset cache topology.

[0082] The specific clustering process includes: taking any valid crash segment that is not included in the segment set, detecting all other valid crash segments that are not included in the segment set whose cache strategy change similarity is greater than the preset cache strategy change similarity or cache topology similarity is greater than the preset cache topology similarity, and recording them as a single segment set.

[0083] The operators can set the values ​​of preset cache strategy iteration similarity and preset cache topology similarity according to the actual scenario. It can be understood that a higher value for cache strategy iteration similarity indicates a more similar frequency of cache strategy iterations among the corresponding valid crash segments, reflecting a greater degree of similarity among the corresponding valid crash segments. Similarly, a higher value for cache topology similarity indicates a more similar server architecture and transmission efficiency among the corresponding valid crash segments, further reflecting a greater degree of similarity among the corresponding valid crash segments. Therefore, the higher the accuracy requirement of the clustered segment sets obtained by the operators, the higher the preset cache strategy iteration similarity and preset cache topology similarity values ​​should be. A value setting method is provided to extract the corresponding cache strategy iteration similarity and cache topology similarity from the historical records that meet the operator's needs. Outliers in both cache strategy iteration similarity and cache topology similarity are then removed, and the average values ​​of the cache strategy iteration similarity and cache topology similarity after outlier removal are recorded as the preset cache strategy iteration similarity and preset cache topology similarity, respectively.

[0084] Specifically, the similarity of cache strategy iterations is determined based on the frequency of dynamic cache adjustments and the richness of cache strategy types;

[0085] The difference between the similarity of cache strategy iterations and the frequency of dynamic cache adjustments is negatively correlated.

[0086] The similarity of cache strategy iterations is negatively correlated with the difference in cache strategy types.

[0087] The similarity of the cache topology is determined based on the number of different servers;

[0088] The similarity of the cache topology is negatively correlated with the number of differences between the nodes.

[0089] The calculation method for the similarity of cache strategy iterations for any two valid crash segments is as follows: Cache iteration similarity = difference of preset dynamic cache adjustment frequency / difference of dynamic cache adjustment frequency + difference of preset cache strategy richness / difference of cache strategy richness.

[0090] For any two valid crash segments, the corresponding cache dynamic adjustment frequency difference is equal to the absolute value of the difference between the cache dynamic adjustment frequencies of the two valid crash segments. For a single valid crash segment, the cache dynamic adjustment frequency is determined by uniformly extracting several time points within the valid crash segment, detecting the cache logic of each server at each time point, and detecting the difference in cache logic between adjacent time points. The dynamic adjustment frequency is equal to the number of differences greater than the preset difference / the total number of differences. The difference is the number of cache logics applied only at one of the two adjacent time points.

[0091] The operator can set the value of the preset difference amount according to the actual scenario. It is understandable that the greater the operator's requirement for the calculation accuracy of the similarity of cache strategy iteration, the larger the value of the preset difference amount. A value setting method is provided to extract the corresponding difference amount in the historical records that meet the operator's needs, remove outliers in the difference amount, and record the average value of the difference amount after removing outliers as the preset difference amount.

[0092] For any two valid crash segments, the difference in cache strategy richness is the number of cache logics that are applied only in one of the two valid crash segments. For example, if there are valid crash segments A and B, and a cache logic is applied only in valid crash segment A, then the above-mentioned criteria for the difference in cache strategy richness are met.

[0093] The preset cache dynamic adjustment frequency difference and the preset cache strategy richness difference can be set by operators according to the actual scenario. It can be understood that the larger the cache dynamic adjustment frequency difference, the smaller the cache iteration similarity, reflecting that the similarity between any two valid crash segments is smaller. Similarly, the larger the value of the cache strategy richness difference, the smaller the cache iteration similarity, reflecting that the similarity between any two valid crash segments is smaller. Therefore, the greater the operator's need for accurate determination of the similarity between the corresponding valid crash segments, the smaller the preset cache dynamic adjustment frequency difference and the preset cache strategy richness difference should be. One method is to extract the corresponding cache dynamic adjustment frequency difference and cache strategy richness difference from the historical records that meet the operator's needs, remove outliers from the cache dynamic adjustment frequency difference and cache strategy richness difference, and record the average values ​​of the cache dynamic adjustment frequency difference and cache strategy richness difference after removing outliers as the preset cache dynamic adjustment frequency difference and the preset cache strategy richness difference, respectively.

[0094] For any two valid crash segments, the method for determining the corresponding node difference is as follows: calculate the first node representation value and the second node representation value corresponding to each of the two valid crash segments respectively, and record the absolute value of the difference between the first node representation value and the sum of the absolute values ​​of the difference between the second node representation value as the node difference number. Extract a number of time points evenly within the valid crash segments. The first node representation value is the average number of databases corresponding to each server at each time point within the valid crash segment, and the second node representation value is the average number of users corresponding to each server at each time point within the valid crash segment.

[0095] Specifically, the user characteristic uniformity is calculated based on the operation parameters of each user within each paragraph set, and the parameter setting strategy is determined based on the user characteristic uniformity.

[0096] If the uniformity of user characteristics is greater than or equal to the preset uniformity of user characteristics, the parameter setting strategy is to set the virtual parameters based on the average value of each operation parameter.

[0097] If the uniformity of user features is less than the preset uniformity of user features, the parameter setting strategy is to set the setting range and virtual number of virtual parameters according to the uniformity difference.

[0098] The operating parameters include the user's thinking time coefficient and the user's operation complexity coefficient.

[0099] The value of user feature uniformity is determined as follows: User feature uniformity = User thinking time feature + User operation complexity feature.

[0100] The method for confirming the user's thinking time feature is to detect the similarity of the thinking time coefficients for each user and record the maximum value of the similarity of the thinking time coefficients as the user's thinking time feature. Specifically, the method for confirming the similarity of the thinking time coefficients for a single user is to record the user as the target user and record the number of other users whose absolute value of the difference between their thinking time coefficients and those of the target user is less than the preset difference of the thinking time coefficients as the similarity of the thinking time coefficients of the target user.

[0101] The method for confirming the user operation complexity feature is to detect the similarity of the operation complexity coefficients for each user, and record the maximum value of the similarity of the operation complexity coefficients as the user operation complexity feature. The method for confirming the similarity of the operation complexity coefficients for a single user is to record the user as the target user, and record the number of other users whose absolute value of the difference between their operation complexity coefficients and those of the target user is less than the preset difference of the operation complexity coefficients as the similarity of the operation complexity coefficients of the target user.

[0102] The preset operation complexity coefficient difference and preset thinking time coefficient difference can be set by operators according to the actual scenario. It can be understood that the smaller the operation complexity coefficient difference, the more similar the difficulty of the system in processing user operations during the use of the system. The smaller the thinking time coefficient difference, the more similar the thinking time intervals of users during the use of the system. Therefore, the higher the operator's need for the similarity of the user characteristics to be judged, the smaller the preset operation complexity coefficient difference and preset thinking time coefficient difference. One way to obtain the values ​​is to extract the corresponding operation complexity coefficient difference and thinking time coefficient difference from the historical records that meet the operator's needs, remove the outliers in the operation complexity coefficient difference and thinking time coefficient difference respectively, and record the average values ​​of the operation complexity coefficient difference and thinking time coefficient difference after removing outliers as the preset operation complexity coefficient difference and preset thinking time coefficient difference respectively.

[0103] The preset value of user feature uniformity can be set by the operator according to the actual scenario. It can be understood that the higher the user feature uniformity, the closer the user's operation behavior is. The corresponding parameter setting strategy is to set the virtual parameters based on the average value of each operation parameter. Therefore, the higher the operator's demand for user feature uniformity, the higher the preset value of user feature uniformity. A value setting method is provided, which extracts the user feature uniformity corresponding to the historical records that meet the operator's needs, removes outliers in the user feature uniformity, and records the average value of the user feature uniformity after removing outliers as the preset user feature uniformity.

[0104] For a single user, the corresponding user thinking time coefficient is the average time interval between two consecutive data requests of that user in the historical records. The corresponding user operation complexity coefficient is determined by extracting the maximum number of server database connections corresponding to each data request of that user in the historical records, and recording it as the user operation complexity coefficient.

[0105] Virtual parameters include virtual thinking time and virtual operational complexity;

[0106] When setting virtual parameters using the average value of each operation parameter, the virtual thinking time is the average value of the user thinking time coefficient for each user, and the virtual operation complexity is the average value of the user operation complexity coefficient for each user.

[0107] When setting the parameter range based on the uniformity difference, the parameter setting strategy involves setting a virtual thinking time range. The minimum value of the virtual thinking time range is the minimum value of the user thinking time coefficient corresponding to each user, and the maximum value is the maximum value of the user thinking time coefficient corresponding to each user. The virtual thinking time for each virtual user is randomly selected, but it should be ensured that the virtual thinking time for each virtual user is within the virtual thinking time range and that the virtual thinking time for each virtual user is different. Similarly, the parameter setting strategy involves setting a virtual operation complexity range. The minimum value of the virtual operation complexity range is the minimum value of the user operation complexity coefficient corresponding to each user, and the maximum value is the maximum value of the user operation complexity coefficient corresponding to each user. The user operation complexity coefficient for each user is randomly selected, but it should be ensured that the virtual operation complexity for each virtual user is within the virtual operation complexity range and that the number of database connections for each server corresponding to each virtual user is different.

[0108] Specifically, a single virtual test includes:

[0109] S1, perform relevant data settings and virtual parameter settings;

[0110] S2, runs the number of virtual users that have not been set according to the preset scene in the preset order, and stores the running data in the process to the database;

[0111] S3, obtain the comparison result between the error rate and the preset error rate. If the error rate is greater than the preset error rate, proceed to step S4. If the error rate is less than or equal to the preset error rate, proceed to step S5.

[0112] S4, the number of virtual users is reduced and adjusted, and step S6 is executed;

[0113] S5. Check whether the error rate is greater than the preset error rate in previous comparison results. If not, proceed to step S2. If so, increase the number of virtual users and execute step S6.

[0114] S6, jump to step S2, until the absolute value of the difference between the adjusted number of virtual users and the number of virtual users before adjustment is less than the preset difference in the number of users or the number of loops is less than the preset number of loops, then the single virtual test ends.

[0115] The relevant data includes: a list of scenario names to be tested, a list of virtual users, a preset error rate, and a preset user count difference;

[0116] The scenario name list stores the names of the performance test scenarios to be executed this time in a preset order, such as concurrent user test. The data type can be list or dict.

[0117] The list of virtual users stores the number of virtual users to be verified in this performance test in a preset order. The data type is list. The length of this list will affect the number of times the performance test is executed in each performance test scenario. The longer the list is, that is, the larger the number of virtual users is set, the more times the performance test will be executed.

[0118] The preset error rate is a value less than the overall error rate requirement for each scenario in this performance test. This is set by the operator. The greater the user's performance requirements for the software to be tested, the smaller the preset error rate will be. This will not be elaborated on here.

[0119] The preset user number difference is the value that the difference between the virtual user number corresponding to the automatic search for the optimal and maximum performance indicators in each performance test scenario and the virtual user number in the previous round is required to be less than or equal to. This value is set by the operator. The greater the user's requirement for the stability of the final determined virtual user number, the smaller the preset user number difference will be.

[0120] Virtual thinking time is the time interval between each virtual data request made by a virtual user.

[0121] The complexity of virtual operations is the number of database connections the server initially establishes.

[0122] The preset order of the test scenario names and virtual user numbers stored in the scenario name list and virtual user number list is set by the operator and has no impact on the test results. The order is in the form of Arabic numerals from smallest to largest.

[0123] The virtual user count of the preset scenario is run in a preset order, and the running data is stored in the database. This includes reading the scenario name from the scenario name list and the virtual user count from the virtual user count list in a preset order, loading them into the performance testing tool JMeter, and running the performance test scenario. During the performance test, the request and response information of each virtual user in the performance test scenario is stored in the time series database InfluxDB. The request and response record information records the time of each request, the corresponding virtual user, the specific request, whether the request was successful, and the time of the request response.

[0124] Error rate = Total number of failed virtual user requests in the current virtual test process / Total number of requests;

[0125] Reduce the number of virtual users V after adjustment ui+1 = Current number of virtual users Vui - Adjustment amount, where the adjustment amount is the absolute value of the baseline adjustment amount, and the baseline adjustment amount = the difference V between the current number of virtual users and the number of virtual users in the previous test. ui-1 2 / 2, the adjustment amount is rounded up to the nearest integer;

[0126] Increase the number of virtual users V after adjustment ui+1 = Current number of virtual users (Vui) + adjustment amount;

[0127] It is worth noting that if the current number of virtual users corresponds to the first virtual test, the adjustment amount is fixed at the current number of virtual users / 2.

[0128] The preset number of loops is the length of the list, which is the set number of virtual users.

[0129] Specifically, the preset number of times refers to the number of parameter setting strategies.

[0130] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for performance evaluation based on autonomously adjustable virtual user numbers, characterized in that, include: For each crash point, relevant analysis is conducted to determine the corresponding crash segment extraction method. This method can be based on the peak number of people to obtain the crash segment, or the relevant segments corresponding to the target crash point can be recorded as the crash segment. The cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache trace duration of the crash segment, and the cache data status is used to determine whether the crash segment is a valid crash segment. Clustering is performed based on the similarity of cache strategy iterations and cache topology of each valid crash segment to obtain several segment sets; The user characteristic uniformity is calculated based on the operation parameters of each user in each paragraph set, and the parameter setting strategy is determined based on the user characteristic uniformity: either setting the virtual parameter based on the average value of each operation parameter or setting the virtual parameter setting range based on the uniformity difference. Perform a preset number of virtual tests and record the results of each virtual test.

2. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 1, characterized in that, Correlation analysis is performed on each crash point to determine the corresponding crash segment extraction method. This includes correlation analysis on a single crash point, which includes: The crash point that is subjected to relevant analysis is recorded as the target crash point; The method for extracting collapse segments is determined based on the density of relevant collapse points corresponding to the target collapse point; If the density of relevant collapse points is less than the preset density of relevant collapse points, the collapse paragraph extraction method is to extract based on the peak number of people; if the density of relevant collapse points is greater than or equal to the preset density of relevant collapse points, the collapse paragraph extraction method is to record the relevant paragraphs corresponding to the target collapse point as collapse paragraphs.

3. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 2, characterized in that, When extracting the collapse segment based on the peak number of people, the peak number of people in the relevant segment corresponding to the target collapse point is detected, and the time segment between the peak number of people and the target collapse point is recorded as the collapse segment.

4. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 3, characterized in that, The cache data status is determined based on the cache amplitude coefficient and cache difference coefficient corresponding to the cache trace duration of the crash segment. When the cache data status is that the cache amplitude coefficient is greater than the preset cache amplitude coefficient or the cache difference coefficient is greater than the preset cache difference coefficient, the crash segment is determined to be a valid crash segment.

5. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 4, characterized in that, The cache traceback duration is determined based on the number of errors corresponding to the crash point; The number of reported errors is positively correlated with the cache traceability time.

6. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 5, characterized in that, Clustering is performed based on the similarity of cache strategy iterations and cache topology of each valid crash segment to obtain several segment sets; For a single paragraph set, which includes several valid crash paragraphs, the similarity of cache strategy iterations between any two valid crash paragraphs is greater than the similarity of preset cache strategy iterations or the similarity of cache topology structure is greater than the similarity of preset cache topology structure.

7. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 6, characterized in that, The similarity of cache strategy iterations is determined based on the frequency of dynamic cache adjustments and the richness of cache strategy types. The difference between the similarity of cache strategy iterations and the frequency of dynamic cache adjustments is negatively correlated. The similarity of cache strategy iterations is negatively correlated with the difference in cache strategy types. The similarity of the cache topology is determined based on the number of different servers; The similarity of the cache topology is negatively correlated with the number of differences between nodes.

8. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 7, characterized in that, Calculate the user characteristic uniformity based on the operation parameters of each user within each paragraph set, and determine the parameter setting strategy based on the user characteristic uniformity. If the uniformity of user characteristics is greater than or equal to the preset uniformity of user characteristics, the parameter setting strategy is to set the virtual parameters based on the average value of each operation parameter. If the uniformity of user features is less than the preset uniformity of user features, the parameter setting strategy is to set the range of virtual parameters based on the uniformity difference. The operating parameters include the user's thinking time coefficient and the user's operation complexity coefficient.

9. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 1, characterized in that, A single virtual test includes: S1, perform relevant data settings and virtual parameter settings; S2, run the virtual user count of the preset scenario in a preset order, and store the running data in the database during the process; S3, obtain the comparison result between the error rate and the preset error rate. If the error rate is greater than the preset error rate, proceed to step S4. If the error rate is less than or equal to the preset error rate, proceed to step S5. S4, the number of virtual users is reduced and adjusted, and step S6 is executed; S5, the number of virtual users is increased and adjusted, and step S6 is executed; S6. If the absolute value of the difference between the adjusted number of virtual users and the number of virtual users before adjustment is less than the preset difference in the number of users or the number of loops is less than the preset number of loops, then the single virtual test ends; if the absolute value of the difference between the adjusted number of virtual users and the number of virtual users before adjustment is greater than or equal to the preset difference in the number of users, or the number of loops is greater than or equal to the preset number of loops, then proceed to step S2.

10. The method for performance evaluation based on autonomously adjustable virtual user numbers according to claim 1, characterized in that, The preset number of times is the number of parameter setting strategies.

Images