Code Performance Evaluation Method and Apparatus, Electronic Device, and Storage Medium

Abstract

The present disclosure relates to a code performance evaluation method, apparatus, electronic device, and storage medium, and relates to the technical field of software testing, and can be applied to an application scenario of testing code performance to determine code performance. The code performance evaluation method includes selecting a judgment index from multiple judgment indexes for analyzing code performance as a first index; obtaining the running performance data of the first code as first data, screening out the index value of the first index from the first data, and determining its data distribution as the first data distribution of the first index; obtaining the running performance data of the second code as second data, screening out the index value of the first index from the second data, and determining its data distribution as the second data distribution of the first index; determining a code performance evaluation result according to the similarity between the first data distribution and the second data distribution. The present disclosure can effectively avoid the problem of being unable to accurately judge code performance due to factors such as unstable downstream basic services.

Description

Background Art

[0002] The performance test of code is an important part of software testing. Before the software goes online, the code needs to be tested to determine whether the software can go online normally; after the software goes online, due to the increase in business requirements, it may be necessary to frequently iterate the online software. At this time, it is necessary to conduct multiple performance tests on the code again.

[0003] Currently, the test tool Jmeter is usually used to perform performance tests on a single machine. For the test results, a method of judging by hard indicators is adopted, that is, if the indicator is outside the threshold range, the test fails. The results obtained by this method are relatively reliable when the upstream and downstream services are stable.

[0004] However, when the upstream and downstream services are unstable, it may lead to performance fluctuations and exceed the threshold range. It is impossible for humans to accurately judge whether there are problems. Therefore, it is necessary to retest the code, resulting in a long test time, affecting the online efficiency, and consuming a large amount of human and material resources.

[0005] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a code performance evaluation method, a code performance evaluation device, an electronic device, and a computer-readable storage medium, so as to at least overcome to some extent the problem that the code performance cannot be judged due to various factors such as unstable downstream basic services.

[0007] According to the first aspect of the present disclosure, a code performance evaluation method is provided, including: determining a plurality of judgment indicators for analyzing code performance, and selecting one judgment indicator from the plurality of judgment indicators as the first indicator; obtaining the running performance data of the first code as the first data, screening the indicator value of the first indicator from the first data as the first indicator value, and determining the data distribution of the first indicator value as the first data distribution of the first indicator; obtaining the running performance data of the second code as the second data, screening the indicator value of the first indicator from the second data as the second indicator value, and determining the data distribution of the second indicator value as the second data distribution of the first indicator; determining the similarity between the first data distribution of the first indicator and the second data distribution of the first indicator as the first similarity, and determining the code performance evaluation result in combination with the first similarity.

[0008] Optionally, obtaining the running performance data of the first code as the first data includes: obtaining the running performance data of the first code as intermediate data at preset time intervals; screening the intermediate data with the number of data items greater than the first preset threshold from the intermediate data as the first data.

[0009] Optionally, determine the index value of the first index from the first data as the first sample data, and determine the index value of the first index from the second data as the second sample data; wherein, determining the similarity between the first data distribution of the first index and the second data distribution of the first index as the first similarity includes: determining the numbered level sum of the first sample data as the first data level sum, and determining the numbered level sum of the second sample data as the second data level sum; combining the first data level sum and the second data level sum to determine an intermediate variable and the standard score of the intermediate variable; performing an integral operation on the standard score to determine the first similarity.

[0010] Optionally, determining the numbered level sum of the first sample data as the first data level sum includes: performing a mixing process on the first sample data and the second sample data to form mixed data, sorting the mixed data in ascending order of data values, and determining the numbered level for each of the sorted mixed data one by one to form the third sample data; adding up the numbered levels of the first sample data in the third sample data to obtain the first data level sum.

[0011] Optionally, determining the code performance evaluation result according to the first similarity includes: determining the distribution result of the first data distribution and the second data distribution according to the relationship between the first similarity and the first preset threshold interval; determining the code performance evaluation result according to the distribution result.

[0012] Optionally, determining the code performance evaluation result according to the distribution result includes: if the first similarity is within the first preset threshold interval, the distribution result is the target distribution result, and it is determined that the code performance meets the preset requirements; if the first similarity is not within the first preset threshold interval, determine the index type of the first index, and determine the code performance evaluation result based on the index type.

[0013] Optionally, the type of the first index includes a first type and a second type, wherein, determining the code performance evaluation result based on the index type includes: if the type of the first index is the first type and the first similarity is greater than the second preset threshold, it is determined that the code performance meets the preset requirements; if the type of the first index is the second type and the first similarity is less than the third preset threshold, it is determined that the code performance meets the preset requirements.

[0014] Optionally, determining the code performance evaluation result according to the distribution result further includes: if the first similarity is not within the first preset threshold interval, determine the mean value of the first sample data as the first mean value, and determine the mean value of the second sample data as the second mean value; determine the mean ratio of the first mean value to the second mean value; if the mean ratio is within the second preset threshold interval, it is determined that the code performance meets the preset requirements.

[0015] Optionally, determining the code performance evaluation result in combination with the first similarity includes: determining another determination index other than the first index among the various determination indexes as the second index; screening the index value of the second index from the first data as the third index value, and determining the data distribution of the third index value as the first data distribution of the second index; screening the index value of the second index from the second data as the fourth index value, and determining the data distribution of the fourth index value as the second data distribution of the second index; determining the similarity between the first data distribution of the second index and the second data distribution of the second index as the second similarity; determining the code performance evaluation result according to the first similarity and the second similarity.

[0016] Optionally, determining the code performance evaluation result in combination with the first similarity includes: determining at least two determination indexes other than the first index among the multiple determination indexes as the third index; screening the index value of the third index from the first data as the fifth index value, and determining the data distribution of the fifth index value as the second data distribution of each third index; screening the index value of the third index from the second data as the sixth index value, and determining the data distribution of the sixth index value as the second data distribution of each third index; determining the similarity between the first data distribution of each third index and the second data distribution of each third index; determining the code performance evaluation result according to the determined similarity between the first data distribution of each third index and the second data distribution of each third index and the first similarity.

[0017] According to a second aspect of the present disclosure, there is provided a code performance evaluation device, including: a first index determination module, configured to determine multiple determination indexes for analyzing code performance, and select one determination index from the multiple determination indexes as the first index; a first distribution determination module, configured to obtain the running performance data of the first code as the first data, screen the index value of the first index from the first data as the first index value, and determine the data distribution of the first index value as the first data distribution of the first index; a second distribution determination module, configured to obtain the running performance data of the second code as the second data, screen the index value of the first index from the second data as the second index value, and determine the data distribution of the second index value as the second data distribution of the first index; a first result determination module, configured to determine the similarity between the first data distribution of the first index and the second data distribution of the first index as the first similarity, and determine the code performance evaluation result in combination with the first similarity.

[0018] Optionally, the first distribution determination module includes a data acquisition unit, configured to acquire the running performance data of the first code as intermediate data at each preset time period; screen out the intermediate data with the number of data items greater than the first preset threshold from the intermediate data as the first data.

[0019] Optionally, the first result determination module includes a similarity determination unit, which is configured to determine the metric value of the first metric from the first data as the first sample data, and determine the metric value of the first metric from the second data as the second sample data; determine the serial number level and the sum of levels of the first data of the first sample data, and determine the serial number level and the sum of levels of the second data of the second sample data; combine the sum of levels of the first data and the sum of levels of the second data to determine an intermediate variable and the standard score of the intermediate variable; perform an integration operation on the standard score to determine the first similarity.

[0020] Optionally, the similarity determination unit includes a sum-of-levels determination subunit, which is configured to perform a mixing process on the first sample data and the second sample data to form mixed data, sort the mixed data in ascending order of data values, and determine the serial number levels for each of the sorted mixed data one by one to form the third sample data; add up the serial number levels of the first sample data in the third sample data to obtain the sum of levels of the first data.

[0021] Optionally, the first result determination module further includes a result determination unit, which is configured to determine the distribution result of the first data distribution and the second data distribution according to the relationship between the first similarity and the first preset threshold interval; determine the code performance evaluation result according to the distribution result.

[0022] Optionally, the result determination unit includes a first judgment subunit, and determining the code performance evaluation result according to the distribution result includes: if the first similarity is within the first preset threshold interval, the distribution result is the target distribution result, and it is determined that the code performance meets the preset requirements; if the first similarity is not within the first preset threshold interval, determine the metric type of the first metric, and determine the code performance evaluation result based on the metric type.

[0023] Optionally, the result determination unit further includes a second judgment subunit, which is configured to, when the type of the first metric includes a first type and a second type, if the type of the first metric is the first type and the first similarity is greater than a second preset threshold, determine that the code performance meets the preset requirements; if the type of the first metric is the second type and the first similarity is less than a third preset threshold, determine that the code performance meets the preset requirements.

[0024] Optionally, the result determination unit further includes a third judgment subunit, and determining the code performance evaluation result according to the distribution result further includes: if the first similarity is not within the first preset threshold interval, determine the mean value of the first sample data as the first mean value, and determine the mean value of the second sample data as the second mean value; determine the mean ratio of the first mean value and the second mean value; if the mean ratio is within the second preset threshold interval, determine that the code performance meets the preset requirements.

[0025] Optionally, the code performance evaluation device further includes a second result determination module, configured to determine another determination index other than the first index among the various determination indexes as the second index; screen the index values of the second index from the first data as the third index value, and determine the data distribution of the third index value as the first data distribution of the second index; screen the index values of the second index from the second data as the fourth index value, and determine the data distribution of the fourth index value as the second data distribution of the second index; determine the similarity between the first data distribution of the second index and the second data distribution of the second index as the second similarity; determine the code performance evaluation result according to the first similarity and the second similarity.

[0026] Optionally, the code performance evaluation device further includes a third result determination module, configured to determine at least two determination indexes other than the first index among the multiple determination indexes as the third index; screen the index values of the third index from the first data as the fifth index value, and determine the data distribution of the fifth index value as the second data distribution of each third index; screen the index values of the third index from the second data as the sixth index value, and determine the data distribution of the sixth index value as the second data distribution of each third index; determine the similarity between the first data distribution of each third index and the second data distribution of each third index; determine the code performance evaluation result according to the determined similarity between the first data distribution of each third index and the second data distribution of each third index and the first similarity.

[0027] According to a third aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory, where a computer-readable instruction is stored on the memory, and when the computer-readable instruction is executed by the processor, the code performance evaluation method according to any one of the above is implemented.

[0028] According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the code performance evaluation method according to any one of the above is implemented.

[0029] The code performance evaluation method in the exemplary embodiments of the present disclosure first determines multiple judgment metrics for analyzing code performance, and selects one judgment metric from the multiple judgment metrics as the first metric. Secondly, obtain the running performance data of the first code as the first data, screen the metric value of the first metric from the first data as the first metric value, and determine the data distribution of the first metric value as the first data distribution of the first metric; obtain the running performance data of the second code as the second data, screen the metric value of the first metric from the second data as the second metric value, and determine the data distribution of the second metric value as the second data distribution of the first metric. Thirdly, determine the similarity between the first data distribution of the first metric and the second data distribution of the first metric as the first similarity, and determine the code performance evaluation result in combination with the first similarity. Through the code performance evaluation method of the present disclosure, on the one hand, when testing two groups of codes in the same environment, it can effectively avoid the influence on the test results of code performance caused by factors such as unstable basic services or unstable machine performance. On the other hand, it can automatically calculate and judge the performance test data of the two groups of codes captured, reduce the manual intervention in the performance test process, and reduce the workload of testers. On the other hand, by analyzing the metric values to judge the code performance test, the test results can be made more real and reliable, reducing the number of tests and saving material resources such as servers.

[0030] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In the drawings:

[0032] Figure 1 Schematically shows a flowchart of the code performance evaluation method according to an exemplary embodiment of the present disclosure;

[0033] Figure 2 Schematically shows a detailed flowchart of the code performance evaluation process according to an exemplary embodiment of the present disclosure;

[0034] Figure 3 Schematically shows a flowchart of result logic judgment according to metric values according to an exemplary embodiment of the present disclosure;

[0035] Figure 4Schematically shows a result comparison graph of the metric avg under normal code performance according to an exemplary embodiment of the present disclosure;

[0036] Figure 5 Schematically shows a result comparison graph of the metric tp99 under normal code performance according to an exemplary embodiment of the present disclosure;

[0037] Figure 6 Schematically shows a result comparison graph of the metric avg under abnormal code performance according to an exemplary embodiment of the present disclosure;

[0038] Figure 7 Schematically shows a result comparison graph of the metric tp99 under abnormal code performance according to an exemplary embodiment of the present disclosure;

[0039] Figure 8 Schematically shows a first block diagram of a code performance evaluation device according to an exemplary embodiment of the present disclosure;

[0040] Figure 9 Schematically shows a block diagram of a first distribution determination module according to an exemplary embodiment of the present disclosure;

[0041] Figure 10 Schematically shows a first block diagram of a first result determination module according to an exemplary embodiment of the present disclosure;

[0042] Figure 11 Schematically shows a block diagram of a similarity determination unit according to an exemplary embodiment of the present disclosure;

[0043] Figure 12 Schematically shows a second block diagram of a first result determination module according to an exemplary embodiment of the present disclosure;

[0044] Figure 13 Schematically shows a first block diagram of a result determination unit according to an exemplary embodiment of the present disclosure;

[0045] Figure 14 Schematically shows a second block diagram of a result determination unit according to an exemplary embodiment of the present disclosure;

[0046] Figure 15 Schematically shows a third block diagram of a result determination unit according to an exemplary embodiment of the present disclosure;

[0047] Figure 16 Schematically shows a second block diagram of a code performance evaluation device according to an exemplary embodiment of the present disclosure;

[0048] Figure 17Schematically shows a second block diagram of a code performance evaluation device according to an exemplary embodiment of the present disclosure;

[0049] Figure 18 Schematically shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure; and

[0050] Figure 19 Schematically shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed implementation manners

[0051] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repetitive description will be omitted.

[0052] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0053] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more software-hardened modules, or in different networks and / or processor devices and / or microcontroller devices.

[0054] Currently, for the results of software performance testing, a method of judging by hard indicators is usually adopted, that is, the test fails when the indicator is outside the threshold range. However, when using the method of judging by hard indicators, when the downstream service is unstable, it may cause performance fluctuations and then make the indicator exceed the threshold range, and it is impossible for humans to accurately judge whether there is a problem. When it is impossible to accurately judge, the code needs to be retested, which takes a long time and affects the online efficiency. In addition, if new functions are added to the code and the online performance changes, multiple tests are required to obtain a relatively reliable threshold range, consuming human and material resources.

[0055] Based on this, in the exemplary embodiment of the present disclosure, a method for evaluating code performance is first provided. The method of evaluating code performance of the present disclosure can be implemented by a server or by a terminal device. The terminal device can be various electronic devices such as a mobile phone or a computer. Refer to Figure 1 , the method for evaluating code performance may include the following steps:

[0056] Step S110. Determine multiple judgment indicators for analyzing code performance, and select one judgment indicator from the multiple judgment indicators as the first indicator.

[0057] In some exemplary embodiments of the present disclosure, it can be determined whether the software meets the expected design value by analyzing the code performance. The code performance analysis can be implemented by analyzing the specific numerical distribution of multiple judgment indicators during the software operation. For example, the judgment indicators may include, but are not limited to, availability, execution speed, memory allocation, startup time, load-bearing capacity, execution time, Central Processing Unit (CPU) time, disk throughput, and response time, etc. For example, in the present disclosure, three judgment indicators, namely, availability, average delay, and the delay at the 99th percentile (TopPercentile99, tp99), are selected as examples to analyze the process of code performance evaluation. The availability can be the ratio of the number of available data returned during the code operation to all data. For example, if one of the 100 data returns is empty and the return value of one data is an error value, the availability is 98%. The average delay can be represented by avg, which represents the average delay of response requests, that is, the average of the delays of multiple requests. The tp99 can be obtained by sorting the delays of multiple response requests in ascending order of the time used and selecting the request delay at the 99th position after sorting. Select one indicator from the determined multiple judgment indicators as the first indicator for analyzing the performance during code operation.

[0058] Those skilled in the art can easily understand that in other exemplary embodiments of the present disclosure, other code performance analysis judgment indicators can be selected to analyze the performance of the code to obtain the code evaluation result. No special limitation is made in this exemplary embodiment.

[0059] Step S120. Obtain the running performance data of the first code as the first data, filter the indicator value of the first indicator from the first data as the first indicator value, and determine the data distribution of the first indicator value as the first data distribution of the first indicator.

[0060] In some exemplary embodiments of the present disclosure, before starting to test the code performance, it is necessary to first set up two sets of test environments with similar performance. For example, a base environment can be set up for deploying the code running online, and another compare environment for deploying the test code. The two environments are identical except for the code, and the test data used for testing is exactly the same, and other aspects are also exactly the same, including the versions of upstream and downstream services. After the environment setup is completed, restart the services, perform tests, etc. on the two environments simultaneously, and capture the monitoring data according to the start time of the test respectively.

[0061] The running performance data of the code can be obtained through a data capture platform. When performing code tests, a continuous integration platform can be used to complete the code testing (i.e., stress testing) work. For example, using the Jenkins platform to monitor the software development process can quickly locate and handle problems. And a Unified Monitoring Platform (UMP) and an Observer platform can be used to capture the running performance data when the code is running. Depending on the different business processes, for example, the UMP platform can be used to obtain all recommendation position data, and data is captured at a granularity of 1 minute, and the Observer platform can be used to obtain single recommendation position data, and data is captured at a granularity of 30 seconds. The code running in the base environment can be used as the first code, and the running performance data of the first code can be used as the first data. The running data of the first metric is obtained from the captured first data as the metric value of the first metric, and the distribution of the metric value of the first metric is determined.

[0062] According to some exemplary embodiments of the present disclosure, the running performance data of the first code is obtained as intermediate data at preset time intervals; the intermediate data with the number of data items greater than the first preset threshold is selected from the intermediate data as the first data. The preset time interval can be the time interval between two adjacent captures of the code running performance data. For example, according to different data, it can be set as: all recommendation position data is captured every 1 minute, and single recommendation position data is captured every 30 seconds. For the captured monitoring data, there are cases where the Query Per Second (QPS) of some data is low, resulting in too little captured data. If the amount of data is too small, it may cause inaccurate subsequent result calculations and lack credibility.

[0063] Therefore, when selecting data, if the sample size of the base group or the compare group is less than 10, the data will be discarded and no subsequent calculations will be performed. Additionally, at the start and end of the test, the service may be unstable and the performance may fluctuate significantly. To ensure the authenticity and credibility of the sample data, the data for a period at the beginning of the test and the data for a preset period before the end of the test can be removed. For example, during a code test, the data for the first 4 minutes after the start of the test and the data for 1 minute before the end of the test were removed, and the remaining code running performance data was used for the calculations.

[0064] The first preset threshold can be a value pre-configured before the code test. The first preset threshold specifies the minimum amount of data that can be included in the first data. For example, the first preset threshold can be set to 10. For the captured running data, the test data with more than 10 data items is retained as the first data.

[0065] Step S130. Obtain the running performance data of the second code as the second data, filter the metric values of the first metric from the second data as the second metric values, and determine the data distribution of the second metric values as the second data distribution of the first metric.

[0066] In some exemplary embodiments of the present disclosure, the second code can be a test code deployed in the compare environment. The first metric of the second code is the same as the first metric of the first code. For example, avg can be selected as the first metric.

[0067] It is easy to understand that the method for obtaining the second data and the method for determining the second data distribution are the same as the method for obtaining the first data and the method for determining the first data distribution respectively, and the present disclosure will not elaborate on this.

[0068] Step S140. Determine the similarity between the first data distribution of the first metric and the second data distribution of the first metric as the first similarity, and determine the code performance evaluation result in combination with the first similarity.

[0069] In some exemplary embodiments of the present disclosure, the first similarity can be the distribution similarity between the first data distribution and the second data distribution determined based on the first metric. After determining the first similarity, the code performance evaluation result is determined in combination with the first similarity. Calculating the first similarity can be completed using the Mann–Whitney U test method. The Mann–Whitney U test method uses the obtained code running performance sample data to infer the overall distribution pattern of the code running performance data, and can fully reflect the differences between two independent samples.

[0070] According to some exemplary embodiments of the present disclosure, determine the metric value of the first metric from the first data as the first sample data, and determine the metric value of the first metric from the second data as the second sample data; determine the rank level sum of the first sample data as the first data level sum, and determine the rank level sum of the second sample data as the second data level sum; combine the first data level sum and the second data level sum to determine an intermediate variable and the standard score of the intermediate variable; perform an integration operation on the standard score to determine the first similarity.

[0071] The first data may be the metric value data including all metrics of the first code running performance, and the second data may be the metric value data including all metrics of the second code running performance. Determine the metric value of the first metric from the first data as the first sample data, and determine the metric value of the first metric from the second data as the second sample data. The first sample data may be the running performance data generated when the code deployed in the base environment runs, and the first sample data may be denoted as the base group sample data; the second sample data may be the running performance data generated when the test code deployed in the compare environment runs, and the second sample data may be denoted as the compare group sample data.

[0072] According to another exemplary embodiment of the present disclosure, perform a mixing process on the first sample data and the second sample data to form mixed data, sort the mixed data in ascending order of data values, and determine the rank level for each of the sorted mixed data one by one to form the third sample data; add up the rank levels of the first sample data in the third sample data to obtain the first data level sum. The third sample data may be formed by performing a mixing process on the determined first sample data and the second sample data, sorting the mixed data in ascending order of data size, and determining the rank level. That is, the smallest data rank level is 1, the second smallest data rank level is 2, and so on, until the rank levels of all data in the mixed data are determined one by one. Additionally, if there are equal cases in the mixed data, then the rank level values of the same data should be the same, and the average value in the unranked array should be taken. For example, for a set of data {4, 7, 7, 9, 13}, then the rank levels of this set of data should be {1, 2.5, 2.5, 4, 5}, and the rank level sum of this set of data is 15.

[0073] According to the above method, the sample data of the base group and the compare group can be mixed, and the number levels can be arranged in ascending order according to the data size. The mixed data set is denoted as A, where the data from the base group forms set B, and the data from the compare group forms set C. Determine the number level sum of the sample data of the base group as the first data level sum, and determine the number level sum of the sample data of the compare group as the second data level sum. Calculate the rank sums R base and R compare for the two sample data groups respectively, and denote them as R b and R c respectively. Formulas 1 and 2 respectively define the calculation methods of R b and R c .

[0074]

[0075]

[0076] Among them, A i refers to the data from the base group in set A, and A j refers to the data from the compare group in set A.

[0077] When the sample data volume is greater than 10, it is considered that the random variable approximately follows a normal distribution. Calculate the U values U base and U compare for the two groups of data according to Formulas 3 and 4 respectively, and denote them as U b and U c respectively. Among them, n b refers to the number of data in set B, and n c refers to the number of data in set C.

[0078]

[0079]

[0080] Calculate the standard score Z compare of U compare according to Formula 5, and denote it as Z c . Among them, m U refers to the mean of the random variable U, and σ U is the standard deviation obtained by approximating U to follow a standard normal distribution and considering it. a refers to the adjustment factor, which takes 0.5 when U c is higher than m U , and takes -0.5 when U c is lower than m U . Formula 6 defines the calculation method of m U and Formula 7 defines the calculation method of σU Calculation method.

[0081]

[0082]

[0083]

[0084] According to the cumulative distribution function (CDF) curve of the (0, 1) normal distribution, the standard score Z compare can be mapped into a value within the range of (0, 1), and this value is used as the distribution similarity of the two sets of data, denoted as p. Formula 8 determines the calculation method of p.

[0085]

[0086] According to Formula 5 and Formula 8, it is easy to obtain that when the standard scores calculated by the base group and the compare group are close, U c is approximately equal to the mean m U , Z c takes the value of 0, and the p-value is calculated as 0.5. At this time, it can be considered that the distribution difference value between the two groups is the smallest. When the p-value is closer to 1, it can be known that the higher the frequency that the result of the compare group is numerically greater than that of the base group. When the p-value is closer to 0, the higher the frequency that the result of the compare group is numerically less than that of the base group. Generally speaking, the p-value can be used as the distribution difference value between the two sets of data. When the p-value is close to 0.5, it is considered that the distributions of the two groups are consistent.

[0087] According to another exemplary embodiment of the present disclosure, the distribution result of the first data distribution and the second data distribution is determined according to the relationship between the first similarity and the first preset threshold interval; the code performance evaluation result is determined according to the distribution result. The first preset threshold interval can be a pre-configured threshold interval. According to whether the value of the first similarity is within the first preset threshold interval, the distribution result of the first data distribution and the second data distribution is determined. For example, the first preset threshold interval can be configured as (0.4, 0.6).

[0088] According to some exemplary embodiments of the present disclosure, if the first similarity is within the first preset threshold interval, the distribution result is the target distribution result, and it is determined that the code performance meets the preset requirements; if the first similarity is not within the first preset threshold interval, the index type of the first index is determined, and the code performance evaluation result is determined based on the index type. The target distribution result can be the result that the data distributions of the two sets of sample data are similar. Refer to Figure 2, in steps S201 to S202, it is determined whether the p-value is within the threshold range. For example, when the p-value is within (0.4, 0.6), it can be considered that the distributions of the two sets of sample data are similar, that is, the code performance meets the preset requirements. Otherwise, if the p-value is not within (0.4, 0.6), it is necessary to further determine the index type of the first index, and then determine the code performance evaluation result based on the index type. The index type can be an index that distinguishes different types of code performance.

[0089] According to another exemplary embodiment of the present disclosure, the types of the first index include a first type and a second type. If the type of the first index is the first type and the first similarity is greater than the second preset threshold, it is determined that the code performance meets the preset requirements; if the type of the first index is the second type and the first similarity is less than the third preset threshold, it is determined that the code performance meets the preset requirements. Generally, judgment indexes can be divided into two types. One is a high index, that is, the larger the index value, the better the performance. For example, the availability rate, and the high index can be determined as the index of the first type; the other is a low index, that is, the smaller the index value, the better the performance. For example, avg, and the low index can be determined as the index of the second type. The second preset threshold and the third preset threshold are respectively pre-configured values for comparing with the index values of the first type and the second type.

[0090] Reference Figure 2 , in steps S203 to S209, if the p-value is not within (0.4, 0.6), it can be considered that the distributions of the two sets of sample data do not reach the result of the target distribution. Therefore, it is necessary to further determine the index type of the first index, and mark the tag corresponding to the first type index (i.e., tag) as "High"; mark the tag corresponding to the second type index as "Low". When judging the index with the tag of "High", if the p-value is greater than the upper threshold, it is considered that the performance of the test code is better than that of the online version code, and it is considered that the code performance meets the preset requirements, that is, the stress test passes. Similarly, for the index with the tag of "Low", if the p-value is less than the lower threshold, it is also considered that the performance of the test code is better than that of the online version code, and it is considered that the code performance meets the preset requirements, that is, the stress test passes.

[0091] For example, configure the second preset threshold to 95%. When the first index is the availability rate, if the calculated availability rate value is 98%, then 98% > 95%, and the availability rate value is greater than the second preset threshold. It is considered that the performance of the test code is better than that of the online version code, and the code performance meets the preset requirements, that is, the stress test passes; if the calculated availability rate value is 90%, then 90% < 95%, and the availability rate value is less than the second preset threshold. It is considered that the performance of the test code is relatively poor compared to the online version code performance, and the code performance does not meet the preset requirements, that is, the stress test fails.

[0092] In addition, configure the third preset threshold to 100. When the first indicator is avg, if the calculated value of avg is 98, then 98 < 100, and the value of avg is less than the third preset threshold. It is considered that the performance of the test code is better than that of the online version code, so the code performance meets the preset requirements, that is, the stress test passes. If the calculated availability value is 105, then 105 > 100, and the value of avg is greater than the third preset threshold. It is considered that the performance of the test code is relatively poor compared to the performance of the online version code, so the code performance does not meet the preset requirements, that is, the stress test fails.

[0093] According to another exemplary embodiment of the present disclosure, if the first similarity is not within the first preset threshold range, then determine the mean of the first sample data as the first mean value, and determine the mean of the second sample data as the second mean value; determine the mean ratio of the first mean value and the second mean value; if the mean ratio is within the second preset threshold range, then determine that the code performance meets the preset requirements. The first mean value can be the mean of all data of the first sample data, and the second mean value can be the mean of all data of the second sample data. The mean ratio can be the ratio of the second mean value to the first mean value, and the second preset threshold range can be a threshold range used for judging the mean ratio. After determining the mean ratio, compare the mean ratio with the second preset threshold range. If the mean ratio is within the second preset threshold range, even if the p-value is judged to fail, it can still be considered that the stress test passes. Only when the p-value is judged to fail and the mean ratio is also outside the threshold range, is it considered that the stress test fails, and there may be a problem with the test code this time, and the previous version of the code can be rolled back for further inspection.

[0094] Two thresholds are involved in the above result judgment process. One is the threshold of the distribution difference value used for comparing with the distribution similarity, and the other is the mean ratio threshold used for comparing with the mean ratio. For the threshold of the distribution difference value, since the standard score is calculated and the final result is normalized, there is no need to set different thresholds for different types of indicators. Also, because the two sets of environments are similar, when there are fluctuations in the upstream and downstream services, the stress test results of this section will fluctuate simultaneously, and there is no need to make additional adjustments to this threshold. For the mean ratio threshold, since the allowable change range and degree of change of each indicator are different, each indicator needs to be adjusted separately; after the initial threshold is set, there is no need to make additional adjustments either.

[0095] In addition to the adjustment of the threshold parameters, the selection of the machine is also a crucial factor. In the actual process, it is recommended to select physical machines with similar memory, computing speed, etc. as the test machines, and try to reduce the interference caused by inconsistent performance of the upstream and downstream services. In this way, the final result will be more real and reliable.

[0096] According to some exemplary embodiments of the present disclosure, another determination index among the determination indexes except the first index is determined as the second index; the index values of the second index are screened from the first data as the third index values, and the data distribution of the third index values is determined as the first data distribution of the second index; the index values of the second index are screened from the second data as the fourth index values, and the data distribution of the fourth index values is determined as the second data distribution of the second index; the similarity between the first data distribution of the second index and the second data distribution of the second index is determined as the second similarity; the code performance evaluation result is determined according to the first similarity and the second similarity. The second index may be one of the other determination indexes except the first index among the multiple determination indexes. After the second index is determined, the data distribution of the index values of the second index of the two groups of codes during the code testing process is determined by using the determination method of the first index to determine the second similarity. After the second similarity is calculated, the evaluation result of the code performance is comprehensively determined in combination with the result of the first similarity.

[0097] According to another exemplary embodiment of the present disclosure, at least two determination indexes among the multiple determination indexes except the first index are determined as the third index; the index values of the third index are screened from the first data as the fifth index values, and the data distribution of the fifth index values is determined as the second data distribution of each third index; the index values of the third index are screened from the second data as the sixth index values, and the data distribution of the sixth index values is determined as the second data distribution of each third index; the similarity between the first data distribution of each third index and the second data distribution of each third index is determined; the code performance evaluation result is determined according to the determined similarity between the first data distribution of each third index and the second data distribution of each third index and the first similarity.

[0098] The third index may be multiple determination indexes among the other determination indexes except the first index among the multiple determination indexes. After the third index is determined, by using the data distribution determination method of the index values of the first index, the distribution similarity between the data distribution of the index values of the third index of the first code and the data distribution of the index values of the third index of the second code is determined. After the distribution similarity is calculated, the evaluation result of the code performance is comprehensively determined in combination with the result of the first similarity. Judging the code performance comprehensively by using the data distribution similarities of multiple determination indexes can make the determined code performance evaluation more accurate.

[0099] It is easy for those skilled in the art to understand that the methods for determining the first data distribution of the second index, the second data distribution of the second index, the first data distribution of each third index, and the second data distribution of each third index are the same as the method for determining the first data distribution of the first index, and the present disclosure will not repeat it here.

[0100] Reference Figure 3, in steps S301 to S303, two sets of codes are respectively pushed to the deployed base environment and compare environment for stress testing, and various index item data values during stress testing are captured. The captured data values are determined as two sets of sample data, so as to determine the distribution similarity of the two sets of sample data based on the two sets of sample data. In steps S304 to S308, the loop starts to test various indexes, and judges the data distribution situation corresponding to each index. When the data distribution difference value is within the threshold range, it is considered that the test of this index passes. When the data distribution difference values corresponding to all index items are within the threshold range, it is considered that the code performance meets the preset requirements, that is, the stress test passes. In steps S309 to S314, if the data distribution difference value of a certain index is not within the preset threshold range, the mean ratio of this index will continue to be judged, and whether the code performance meets the preset requirements is judged according to whether the mean ratio is within the preset threshold range.

[0101] According to the code performance evaluation method of the present disclosure, the code performance evaluation results of the data distribution of the judgment indexes are given respectively under normal conditions and abnormal conditions.

[0102] (1) Under normal conditions, the calculated p-value corresponding to the judgment index is within the threshold range. When the difference between the two sets of results is very small or there is no difference, the p-value should be around 0.5. Table 1 shows the analysis and judgment results obtained from a certain stress test. Table 1 gives the index values corresponding to different judgment indexes, where metric represents the judgment index, mean_compare represents the mean result of the compare group, mean_base represents the mean result of the base group, deviation represents the mean ratio, statistic represents the rank sum of the compare group, pvalue represents the p-value, and result represents the judgment result given. Refer to Figure 4 and Figure 5 , the monitoring index situation during this stress test period was intercepted on the UMP platform. Among them, the broken line corresponding to the port "172.28.78.40" represents the avg / tp99 of the base group, and the broken line corresponding to the port "172.28.78.44" represents the avg / tp99 of the compare group. It can be seen that the two sets of stress test results are indeed very close, and the curve trend is stable and can pass. This is consistent with the judgment result in the analysis table.

[0103] Table 1

[0104]

[0105] (2) When an abnormal situation occurs, there is at least one judgment index, and the corresponding p-value calculated for this judgment index is outside the threshold range. Table 2 shows the calculation results of an abnormal situation. It can be seen that both the avg and tp99 indicators are judged as failed. Similarly, referring to Figure 6 and Figure 7 as shown, among which, the broken line corresponding to the port "172.28.78.40" represents the avg / tp99 of the base group, and the broken line corresponding to the port "172.28.78.44" represents the avg / tp99 of the compare group. On the UMP platform, it can be seen that during this period (09 / 12 16:23-09 / 12 16:43), among which, the broken line corresponding to the compare group is stably above the broken line corresponding to the base group, and the avg of the compare group is always 7-8 ms higher than that of the base group. In order to determine whether the performance test result is a code problem, the compare group can be rolled back to the previous version of the code and the stress test can be performed again. The stress test results show that the two curves are nearly coincident (09 / 12 16:50-09 / 12 17:03). Finally, perform the deployment stress test on the newly submitted code again. It can be seen that the abnormal situation of 7-8 ms higher is reproduced (09 / 12 17:08-09 / 12 17:21). Therefore, it can be considered that there is a problem with the code submitted this time and further inspection is required.

[0106] Table 2

[0107]

[0108] It should be noted that the terms "first", "second", "third", "fourth", "fifth", "sixth", etc. used in this disclosure are only used to distinguish different similarities, different preset thresholds, different indicators, different indicator values, different indicator types, different preset threshold intervals, different means, different data distributions, etc., and should not cause any limitation to this disclosure.

[0109] In summary, for the code performance evaluation method of the present disclosure, first, multiple judgment indicators for analyzing code performance are determined, and one judgment indicator is selected from the multiple judgment indicators as the first indicator; second, the running performance data of the first code is obtained as the first data, the indicator value of the first indicator is screened from the first data as the first indicator value, and the data distribution of the first indicator value is determined as the first data distribution of the first indicator; the running performance data of the second code is obtained as the second data, the indicator value of the first indicator is screened from the second data as the second indicator value, and the data distribution of the second indicator value is determined as the second data distribution of the first indicator; third, the similarity between the first data distribution of the first indicator and the second data distribution of the first indicator is determined as the first similarity, and the code performance evaluation result is determined in combination with the first similarity. For the code performance evaluation method of the present disclosure, on the one hand, two groups of different codes are subjected to performance tests using two groups of exactly the same environments and services, and the evaluation result of the code performance is determined by analyzing the distribution of the code running performance data, which can effectively avoid the influence on the code test result caused by performance fluctuations that may be caused by factors such as instability of upstream and downstream basic services or differences in machine performance. On the other hand, based on the monitoring platform, the code running performance data during the code test process can be obtained. After screening and processing the obtained data, judgment logic processing is performed, which can reduce the manual intervention in the code test evaluation process and reduce the work burden of testers. On the other hand, the present disclosure can avoid problems such as long test time, low test efficiency, and impact on software upper limit caused by the need to retest the code due to inaccurate judgment of the code performance result, and can shorten the code development and test cycle. On the other hand, using factors such as distribution similarity and mean ratio to comprehensively judge the code performance evaluation result can reflect the state of the code performance change caused by the new functions added to the code, making the test result more real and reliable.

[0110] In addition, in the present exemplary embodiment, a code performance evaluation device is also provided. Referring to Figure 8 , the code performance evaluation device 800 may include a first indicator determination module 810, a first distribution determination module 820, a second distribution determination module 830, and a first result determination module 840.

[0111] Specifically, the first index determination module 810 can be used to determine multiple judgment indexes for analyzing code performance, and select one judgment index from the multiple judgment indexes as the first index; the first distribution determination module 820 can be used to obtain the running performance data of the first code as the first data, filter the index values of the first index from the first data as the first index values, and determine the data distribution of the first index values as the first data distribution of the first index; the second distribution determination module 830 can be used to obtain the running performance data of the second code as the second data, filter the index values of the first index from the second data as the second index values, and determine the data distribution of the second index values as the second data distribution of the first index; the first result determination module 840 can be used to determine the similarity between the first data distribution of the first index and the second data distribution of the first index as the first similarity, and determine the code performance evaluation result in combination with the first similarity.

[0112] The code performance evaluation device 800 can test the running performance of two groups of codes in the same environment based on the determined multiple judgment indexes for analyzing code performance, and determine the code performance evaluation result based on the distribution of the performance running data of the two groups of codes, which can effectively avoid the influence of unstable basic services on the test results and ensure the accuracy of the test results. It is an effective code performance evaluation device.

[0113] According to some exemplary embodiments of the present disclosure, referring to Figure 9 , the first distribution determination module 820 may include a data acquisition unit 910.

[0114] Specifically, the data acquisition unit 910 can be used to obtain the running performance data of the first code as intermediate data at preset time intervals; filter out the intermediate data with the number of data items greater than the first preset threshold from the intermediate data as the first data.

[0115] The data acquisition unit 910 can capture the performance data during code running according to preset time intervals, and filter out the data that meets the conditions as analysis data to ensure the availability of sample data and provide data support for subsequent calculation of distribution similarity.

[0116] According to another exemplary embodiment of the present disclosure, referring to Figure 10 , the first result determination module 840 may include a similarity determination unit 1010.

[0117] Specifically, the similarity determination unit 1010 is configured to determine the index value of the first index from the first data as the first sample data, and determine the index value of the first index from the second data as the second sample data; determine the number rank sum of the first sample data as the first data rank sum, and determine the number rank sum of the second sample data as the second data rank sum; combine the first data rank sum and the second data rank sum to determine an intermediate variable and the standard score of the intermediate variable; perform an integration operation on the standard score to determine the first similarity.

[0118] The similarity determination unit 1010 applies the principle of the Mann-Whitney rank sum test method to the similarity calculation process of data distributions. By using this method, more abundant sample data information can be used in the calculation process, and the difference between the two sample data can be calculated better.

[0119] According to another exemplary embodiment of the present disclosure, refer to Figure 11 , the similarity determination unit 1010 may include a rank sum determination subunit 1110.

[0120] Specifically, the rank sum determination subunit 1110 is configured to mix the first sample data and the second sample data to form mixed data, sort the mixed data in ascending order of data values, and determine the number ranks one by one for the sorted mixed data to form the third sample data; add up the number ranks of the first sample data in the third sample data to obtain the first data rank sum.

[0121] The rank sum determination subunit 1110 can mix the two sets of sample data and calculate the number rank sums of each set of sample data respectively, so as to calculate the distribution similarity of the two sets of sample data based on the number rank sums.

[0122] According to still another exemplary embodiment of the present disclosure, refer to Figure 12 , compared with the first result determination module 840, the first result determination module 1210 may further include a result determination unit 1220 in addition to the similarity determination unit 1010.

[0123] Specifically, the result determination unit 1220 may be configured to determine the distribution result of the first data distribution and the second data distribution according to the relationship between the first similarity and the first preset threshold interval; determine the code performance evaluation result according to the distribution result.

[0124] The result determination unit 1220 can determine the evaluation result of the code performance according to the distribution situation of the first data distribution and the second data distribution.

[0125] According to some exemplary embodiments of the present disclosure, refer to Figure 13 , the result determination unit 1220 may include a first judgment subunit 1310.

[0126] Specifically, the first determination subunit 1310 is configured to determine the code performance evaluation result according to the distribution result, including: if the first similarity is within the first preset threshold range, the distribution result is the target distribution result, and it is determined that the code performance meets the preset requirements; if the first similarity is not within the first preset threshold range, the index type of the first index is determined, and the code performance evaluation result is determined based on the index type.

[0127] The first determination subunit 1310 determines that when the first similarity is within the preset threshold range, the test result of the code performance meets the preset requirements; when the first similarity is not within the preset threshold range, the code performance evaluation result is further determined in combination with the index type of the first index.

[0128] According to another exemplary embodiment of the present disclosure, referring to Figure 14 , compared with the result determination unit 1220, the result determination unit 1410 may further include a second determination subunit 1420 in addition to the first determination subunit 1310.

[0129] Specifically, when the types of the first index include a first type and a second type, the second determination subunit 1420 may be configured to: if the type of the first index is the first type and the first similarity is greater than a second preset threshold, determine that the code performance meets the preset requirements; if the type of the first index is the second type and the first similarity is less than a third preset threshold, determine that the code performance meets the preset requirements.

[0130] The second determination subunit 1420 may determine the code performance evaluation result according to the relationship between the first similarity and the preset threshold after determining the specific type to which the first index belongs.

[0131] According to still another exemplary embodiment of the present disclosure, referring to Figure 15 , compared with the result determination unit 1410, the result determination unit 1510 may further include a third determination subunit 1520 in addition to the first determination subunit 1310 and the second determination subunit 1420.

[0132] Specifically, the third determination subunit 1520 is configured to determine the code performance evaluation result according to the distribution result, further including: if the first similarity is not within the first preset threshold range, determine the mean of the first sample data as the first mean, and determine the mean of the second sample data as the second mean; determine the mean ratio of the first mean to the second mean; if the mean ratio is within the second preset threshold range, determine that the code performance meets the preset requirements.

[0133] The third determination subunit 1520 may, when the first similarity is not within the first preset threshold range, determine the evaluation result of the code performance by using the relationship between the mean ratio of the two sets of sample data and the preset threshold.

[0134] According to some exemplary embodiments of the present disclosure, with reference to Figure 16 , compared with the code performance evaluation device 800, the code performance evaluation device 1600 may further include a second result determination module 1610 in addition to the first index determination module 810, the first distribution determination module 820, the second distribution determination module 830, and the first result determination module 840.

[0135] Specifically, the second result determination module 1610 may be configured to determine another determination index other than the first index among the determination indexes as the second index; screen the index values of the second index from the first data as the third index value, and determine the data distribution of the third index value as the first data distribution of the second index; screen the index values of the second index from the second data as the fourth index value, and determine the data distribution of the fourth index value as the second data distribution of the second index; determine the similarity between the first data distribution of the second index and the second data distribution of the second index as the second similarity; and determine the code performance evaluation result according to the first similarity and the second similarity.

[0136] The second result determination module 1610 may determine another index other than the first index from the determination indexes as the second index, determine the second similarity according to the data distribution of the index values of the second index, and combine the second similarity with the first similarity to determine the code performance evaluation result.

[0137] According to some exemplary embodiments of the present disclosure, with reference to Figure 17 , compared with the code performance evaluation device 1600, the code performance evaluation device 1700 may further include a third result determination module 1710 in addition to the first index determination module 810, the first distribution determination module 820, the second distribution determination module 830, the first result determination module 840, and the second result determination module 1610.

[0138] Specifically, the third result determination module 1710 may be configured to determine at least two determination indexes other than the first index among the multiple determination indexes as the third index; screen the index values of the third index from the first data as the fifth index value, and determine the data distribution of the fifth index value as the second data distribution of each third index; screen the index values of the third index from the second data as the sixth index value, and determine the data distribution of the sixth index value as the second data distribution of each third index; determine the similarity between the first data distribution of each third index and the second data distribution of each third index; and determine the code performance evaluation result according to the determined similarity between the first data distribution of each third index and the second data distribution of each third index and the first similarity.

[0139] The third result determination module 1710 can determine multiple other indicators except the first indicator from the judgment indicators as the third indicators, determine the distribution similarity of the two sets of sample data according to the data distribution of the indicator values of each third indicator, and combine the determined distribution similarity with the first similarity to determine the code performance evaluation result. When determining the code performance evaluation result, this module synthesizes the common results of multiple indicators, making the evaluation result more accurate.

[0140] The specific details of each module of the virtual code performance evaluation device described above have been described in detail in the corresponding code performance evaluation method, so they will not be elaborated here.

[0141] It should be noted that although several modules or units of the code performance evaluation device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0142] In addition, in the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0143] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, method, or program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0144] Next, refer to Figure 18 to describe the electronic device 1800 according to this embodiment of the present invention. Figure 18 The displayed electronic device 1800 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0145] As Figure 18 shown, the electronic device 1800 is presented in the form of a general-purpose computing device. The components of the electronic device 1800 may include, but are not limited to: at least one of the above processing units 1810, at least one of the above storage units 1820, a bus 1830 connecting different system components (including the storage unit 1820 and the processing unit 1810), and a display unit 1840.

[0146] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1810, so that the processing unit 1810 executes the steps according to various exemplary embodiments of the present invention described in the "Exemplary Method" section above of this specification.

[0147] The storage unit 1820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 1821 and / or a cache storage unit 1822, and may further include a read-only storage unit (ROM) 1823.

[0148] The storage unit 1820 may include a program / utility 1824 having a set (at least one) of program modules 1825. Such program modules 1825 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0149] The bus 1830 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0150] The electronic device 1800 can also communicate with one or more external devices 1870 (such as a keyboard, a pointing device, a Bluetooth device, etc.), can also communicate with one or more devices that enable a user to interact with the electronic device 1800, and / or communicate with any device that enables the electronic device 1800 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 1850. And, the electronic device 1800 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 1860. As shown in the figure, the network adapter 1860 communicates with other modules of the electronic device 1800 through the bus 1830. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 1800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0151] From the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0152] In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above method of this specification. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0153] Reference Figure 19 As shown, a program product 1900 for implementing the above method according to an embodiment of the present invention is described. It can be a portable compact disc read-only memory (CD-ROM) and includes program code and can run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] The program product can adopt any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0156] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0157] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0158] In addition, the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present invention, and are not for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0159] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0160] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for evaluating code performance, characterized in that, Including: Determine multiple judgment metrics for analyzing code performance, and select one of the multiple judgment metrics as the first metric; Obtain the running performance data of the first code as the first data, screen the metric value of the first metric from the first data as the first metric value, and determine the data distribution of the first metric value as the first data distribution of the first metric. The first code is the running code deployed and running in the first test environment; Obtain the running performance data of the second code as the second data, screen the metric value of the first metric from the second data as the second metric value, and determine the data distribution of the second metric value as the second data distribution of the first metric. The second code is the test code deployed and running in the second test environment; Determine the similarity between the first data distribution of the first metric and the second data distribution of the first metric as the first similarity, and determine the code performance evaluation result in combination with the first similarity; The determining the similarity between the first data distribution of the first metric and the second data distribution of the first metric as the first similarity includes: Determine the metric value of the first metric from the first data as the first sample data, and determine the metric value of the first metric from the second data as the second sample data; Determine the sum of the serial numbers and levels of the first sample data as the sum of the first data levels, and determine the sum of the serial numbers and levels of the second sample data as the sum of the second data levels; Determine an intermediate variable and the standard score of the intermediate variable in combination with the sum of the first data levels and the sum of the second data levels; Perform an integral operation on the standard score to determine the first similarity.

2. The method for evaluating code performance according to claim 1, characterized in that, Obtaining the running performance data of the first code as the first data includes: Obtain the running performance data of the first code as intermediate data at preset time intervals; Screen out the intermediate data with the number of data items greater than the first preset threshold from the intermediate data as the first data.

3. The method for evaluating code performance according to claim 1, characterized in that, Determining the sum of the serial numbers and levels of the first sample data as the sum of the first data levels includes: Perform a mixing process on the first sample data and the second sample data to form mixed data, sort the mixed data in ascending order of data values, and determine the serial numbers and levels one by one for the sorted mixed data to form the third sample data; Add up the serial numbers and levels of the first sample data in the third sample data to obtain the sum of the first data levels.

4. The method for evaluating code performance according to claim 1, characterized in that, Determining the code performance evaluation result according to the first similarity includes: Determine the distribution result of the first data distribution and the second data distribution according to the relationship between the first similarity and the first preset threshold interval; Determine the code performance evaluation result according to the distribution result.

5. The method for evaluating code performance according to claim 4, characterized in that, Determining the code performance evaluation result according to the distribution result includes: If the first similarity is within the first preset threshold interval, the distribution result is the target distribution result, and it is determined that the code performance meets the preset requirements; If the first similarity is not within the first preset threshold interval, determine the metric type of the first metric, and determine the code performance evaluation result based on the metric type.

6. The method for evaluating code performance according to claim 5, characterized in that, The types of the first index include a first type and a second type. Among them, determining the code performance evaluation result based on the index type includes: If the type of the first index is the first type and the first similarity is greater than a second preset threshold, it is determined that the code performance meets the preset requirements; If the type of the first index is the second type and the first similarity is less than a third preset threshold, it is determined that the code performance meets the preset requirements.

7. The method for evaluating code performance according to claim 5, characterized in that, Determining the code performance evaluation result according to the distribution result further includes: If the first similarity is not within the first preset threshold range, the mean of the first sample data is determined as the first mean, and the mean of the second sample data is determined as the second mean; Determine the mean ratio of the first mean to the second mean; If the mean ratio is within the second preset threshold range, it is determined that the code performance meets the preset requirements.

8. The method for evaluating code performance according to claim 1, characterized in that, Determining the code performance evaluation result in combination with the first similarity includes: Determine another judgment index among the judgment indexes except the first index as the second index; Screen the index value of the second index from the first data as the third index value, and determine the data distribution of the third index value as the first data distribution of the second index; Screen the index value of the second index from the second data as the fourth index value, and determine the data distribution of the fourth index value as the second data distribution of the second index; Determine the similarity between the first data distribution of the second index and the second data distribution of the second index as the second similarity; Determine the code performance evaluation result according to the first similarity and the second similarity.

9. The code performance evaluation method according to claim 1, wherein, Determining the code performance evaluation result in combination with the first similarity includes: Determine at least two judgment indexes among the multiple judgment indexes except the first index as the third index; Screen the index value of the third index from the first data as the fifth index value, and determine the data distribution of the fifth index value as the first data distribution of each third index; Screen the index value of the third index from the second data as the sixth index value, and determine the data distribution of the sixth index value as the data distribution of each third index; Determine the similarity between the first data distribution of each third index and the second data distribution of each third index; Determine the code performance evaluation result according to the determined similarity between the first data distribution of each third index and the second data distribution of each third index and the first similarity.

10. A code performance evaluation device, wherein, Including: A first index determination module, configured to determine multiple judgment indexes for analyzing code performance, and select one of the multiple judgment indexes as the first index; A first distribution determination module, configured to obtain the running performance data of the first code as the first data, screen the index value of the first index from the first data as the first index value, and determine the data distribution of the first index value as the first data distribution of the first index, where the first code is the running code deployed and running in the first test environment; A second distribution determination module, configured to obtain the running performance data of a second code as second data, screen the metric values of the first metric from the second data as second metric values, and determine the data distribution of the second metric values as the second data distribution of the first metric, where the second code is a test code deployed and run in a second test environment; A first result determination module, configured to determine the similarity between the first data distribution of the first metric and the second data distribution of the first metric as a first similarity, and determine a code performance evaluation result in combination with the first similarity; The first result determination module is further configured to determine the metric values of the first metric from the first data as first sample data, and determine the metric values of the first metric from the second data as second sample data; Determine the serial number sum of the first sample data as the first data sum, and determine the serial number sum of the second sample data as the second data sum; Determine an intermediate variable and the standard score of the intermediate variable in combination with the first data sum and the second data sum; Perform an integral operation on the standard score to determine the first similarity.

11. An electronic device, wherein, Comprising: A processor; And A memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the code performance evaluation method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, on which a computer program is stored, wherein, When the computer program is executed by the processor, the code performance evaluation method according to any one of claims 1 to 9 is implemented.

Images