A method to improve the accuracy of software system delivery quality detection based on traffic playback

Abstract

The present invention discloses a method for improving the accuracy of software system delivery quality detection based on traffic playback, including: S1) collecting software system program code, configuration files, log files and user operation records; S2) cleaning and processing the collected user operation data, removing irrelevant or redundant information and adopting a standardized data format; S3) using the cleaned data to train a machine learning model and construct a baseline standard set B; S4) constructing a retest task, repeatedly testing the updated software system, and obtaining a set of data sets X to be tested; S5) calculating the degree of difference between the data set X to be tested and the baseline standard set B; S6) calculating the quality fluctuation factor, and automatically judging whether the quality of the software system meets the requirements according to the set threshold. The present invention can avoid the leakage of customer sensitive information, and improve the test coverage, efficiency and accuracy of defect detection of the software system.

Description

Technical Field

[0001] The present invention relates to a software system delivery quality detection method, and in particular to a method for improving the accuracy of software system delivery quality detection based on traffic playback. Background Art

[0002] Most existing software system defect detection methods rely on manual review and semi-automated testing tools, which have problems such as low detection efficiency, high omission rate, and inability to detect complex system defects in a timely manner. The main problems in existing automated testing technologies include: high writing and maintenance costs, insufficient test coverage, and delayed defect discovery.

[0003] Traffic replay is a process of recording online data, obtaining real traffic, and replaying it in a test environment to verify the correctness of code logic. Recording online data not only involves customer privacy information and causes the risk of customer information leakage, but also generates billions of data that need to be compared. Comparing and analyzing the differences in this data is very time-consuming and labor-intensive. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for improving the accuracy of software system delivery quality detection based on traffic playback, which can improve the degree of automation of software testing and the accuracy of defect detection and reduce the cost of manual analysis.

[0005] The technical solution adopted by the present invention to solve the above technical problems is to provide a method for improving the accuracy of software system delivery quality detection based on traffic playback, comprising the following steps: S1) Traffic recording stage: collecting software system program code, configuration files, log files and user operation records; S2) Data preprocessing: cleaning and processing the collected user operation data, removing irrelevant or redundant information and adopting a standardized data format; S3) Building a baseline standard set: using the cleaned data to train a machine learning model to build a baseline standard set B; S4) Traffic playback: building a retest task, repeatedly testing the updated software system, and obtaining a set of data sets X to be tested; S5) Data difference comparison: calculating the degree of difference between the data set X to be tested and the baseline standard set B; S6) Outputting the judgment result: calculating the quality fluctuation factor, and automatically judging whether the quality of the software system meets the requirements according to the set threshold.

[0006] Furthermore, the user operation record in step S1 includes recording the sequence and operation object of the user operation.

[0007] Furthermore, the step S2 presets a first time threshold. If the same character is input into the same input box for more than a certain number of times within the first time threshold, this part of the record is regarded as an erroneous operation record and deleted, while the record of the last correct input is retained.

[0008] Furthermore, the step S2 presets a second time threshold. If multiple operations on the same object have the same result within the second time threshold, only the first operation record is retained, and subsequent repeated operation records are deleted.

[0009] Furthermore, step S3 provides the preprocessed data as input to the decision tree machine learning model. The model learns the patterns and rules of the data during the training process, extracts the user operation frequency, operation sequence, and operating system response results, and constructs a baseline standard set B.

[0010] Furthermore, before performing the data difference comparison, step S5 also includes filtering and removing the data that does not meet the conditions according to preset rules.

[0011] Furthermore, the step S5 uses the following formula to calculate the difference between the to-be-detected data set X and the baseline standard set B as the quality fluctuation factor:

[0012]

[0013] Xi represents the i-th data point in the data set to be tested, Bi represents the i-th data point in the baseline standard set, i represents the i-th corresponding data point in the baseline standard set, n is the number of data points, wi is the weight of the i-th data point, and dif is the final calculated software system quality fluctuation factor;

[0014] wi=ai*Importance_i+bi*Frequency_i

[0015] Importance_i is the importance of the business, and its value range is usually from 0 to 1; Frequency_i is the frequency of data occurrence; a and b are adjustment coefficients.

[0016] Furthermore, the number of data points in the data set X to be tested is the same as that in the baseline standard set B. If they are not the same and the difference is less than a preset threshold, the data interpolation method is used to increase the data points in the smaller set until the number of data points in the two sets is the same; when the quality fluctuation factor dif is lower than the preset threshold, it is determined that the software system meets the quality requirements.

[0017] Furthermore, the step S6 also includes displaying the overall calculation of the quality fluctuation factor, and automatically judging whether the quality of the software system meets the requirements according to the set threshold.

[0018] Furthermore, the step S6 sets the threshold range of the quality fluctuation factor to

[0019]

[0020] Where k is the adjustment coefficient, dif j It represents the mass fluctuation factor obtained from the jth historical calculation, and N is the number of historical calculations.

[0021] Compared with the prior art, the present invention has the following beneficial effects: the method provided by the present invention for improving the accuracy of software system delivery quality detection based on traffic playback can avoid leakage of customer sensitive information, and improve the test coverage, efficiency and accuracy of defect detection of the software system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the automatic detection of the quality accuracy of software system delivery based on traffic playback of the present invention;

[0023] Figure 2 The figure is a flow chart of data difference detection and comparison of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] Figure 1 The present invention is a flow chart of automatically detecting the accuracy of software system delivery quality based on traffic playback.

[0026] See also Figure 1 The method for improving the accuracy of software system delivery quality detection based on traffic playback provided by the present invention can intelligently output analysis results based on classification comparison of operating data and baseline standard data sets, discover differences before and after system changes, and output software system detection reports more accurately.

[0027] Combining machine learning technology with difference detection algorithms does have certain difficulties in automatically detecting the accuracy of software system delivery quality. First, data collection and preprocessing are difficult. In order to train an effective machine learning model, it is necessary to collect and process diverse software system defect data from different sources, but this process is costly and time-consuming, and the data quality is difficult to guarantee. Secondly, feature extraction and selection are difficult. The complexity of the software system makes feature extraction difficult, and there may be redundancy and correlation between features. How to select the most useful features for defect detection from many features is a key issue and an important factor affecting model performance. Furthermore, the difference detection algorithm is difficult to apply. The difference detection algorithm needs to adapt to different types of software systems and accurately identify changes or differences in the software system. However, due to the complexity and diversity of software systems, the algorithm may produce false positives or false negatives, and may face efficiency problems when processing large amounts of data.

[0028] The method for improving the accuracy of software system delivery quality detection based on traffic playback provided by the present invention comprises the following specific steps:

[0029] S1. Data set collection: collected software system program code, configuration files, log files, and user operation records, etc.

[0030] S2. Data preprocessing: Clean the data and remove irrelevant or redundant information. Standardize the data format to ensure data consistency.

[0031] S3. Model training: Use the cleaned data to train the machine learning model and build a baseline standard set to support subsequent difference detection.

[0032] S4, traffic replay: build a retest task to repeatedly test the updated software system and obtain a set of data to be tested X.

[0033] S5, difference detection algorithm: construct a baseline standard set B, which contains the data characteristics when the software system is running in an ideal state. At the same time, the present invention has a software system operation data set X to be detected. For the data set X, data that does not meet the conditions can be removed according to the preset rules. For example, a specific system does not meet the user data within the working hours (Monday to Friday, 9:00-17:00), and the user data outside this range will be filtered out.

[0034] Please continue to see Figure 2 , the goal of the present invention is to calculate the difference between X and B; the difference detection algorithm formula is:

[0035]

[0036] in:

[0037] Xi represents the i-th data point in the data set X to be detected.

[0038] Bi represents the i-th data point in the baseline standard set B.

[0039] i represents the i-th corresponding data point in the baseline standard set.

[0040] ●n is the number of data points.

[0041] wi is the weight of the ith data point, which is used to adjust the importance of different data points (explained later).

[0042] ·Indicates the absolute difference between Xi and Bi.

[0043] dif is the final calculated software system quality fluctuation factor.

[0044] Quality fluctuation factor algorithm description:

[0045] ① First, make sure that the number of data points in X and B is the same, that is, n is equal;

[0046] ②For each data point i in X and B, calculate the absolute difference |Xi-Bi| and adjust the importance of different data points according to the weight wi;

[0047] ③ Add the absolute differences of different weights to get the total difference;

[0048] ④ Divide the total difference by the number of data points n to obtain the final quality fluctuation factor dif.

[0049] Calculation instructions for weight wi:

[0050] wi=ai*Importance_i+bi*Frequency_i

[0051] in:

[0052] Importance_i is the importance of the business, reflecting the criticality of a specific data point in the business carried by the entire software system. The value range is usually from 0 to 1, where 0 means completely unimportant and 1 means extremely important.

[0053] Frequency_i is the frequency of data occurrence, which reflects the proportion of the number of occurrences of a specific data point in a period of time to the total number of data points, and is classified into high-frequency data points and low-frequency data points. When calculating the final weight, business importance and data frequency complement each other.

[0054] a and b are adjustment coefficients. For data points with high business importance and high frequency, their weights will be relatively large; for data points with low business importance but high frequency, the weights can be adjusted according to the actual situation to avoid excessive attention to unimportant data points. For example, the frequency of system login data points is high and the business importance is also high, so its weight can be set higher; while the frequency of order payment data points is relatively low, but its business importance is high, so its weight also needs to be set higher.

[0055] For the case where missing data may cause the number of n in the algorithm to be inconsistent, if the difference in n is small, data interpolation can be used to increase the data points in the smaller set. If the difference in n is large, data resampling can be performed. When a false positive in difference detection occurs, it will be manually treated as an ignored item, and the false positive difference will not be detected again in subsequent tests.

[0056] Preferably, the quality fluctuation factor sets a threshold interval, and the threshold interval range setting formula is explained as follows: It represents the mass fluctuation factor obtained from the jth historical calculation, and N is the number of historical calculations.

[0057]

[0058] Set the threshold range of the quality fluctuation factor to Where k is the adjustment coefficient, which can be adjusted according to the actual situation: if the stability of the software system is relatively high, that is, the values ​​of the historical quality fluctuation factors are relatively concentrated and the standard deviation is small, a relatively small value can be taken, and k is considered to be between [1,2]. If the system stability is poor, the quality fluctuation factor changes greatly, and the standard deviation is large, in order to avoid too many false alarms and improve the accuracy of detection, a larger value should be taken, and k is considered to be between [2,5]. For scenarios with high business importance and low tolerance for defects, such as transaction data, a smaller value should be taken to improve the sensitivity of detection. k is selected between [1,2] to ensure that potential serious problems are discovered in time and reduce risks. For some non-critical businesses or scenarios with low real-time requirements and relatively strong risk tolerance, k is between [2,4] to reduce manual analysis costs without causing a significant impact on the normal operation of the system and user experience.

[0059] Calculate the average

[0060] This step is to obtain the central trend of the historical quality fluctuation factor. By averaging the results of multiple historical calculations, we can understand the approximate level of the quality fluctuation factor of the software system under normal operation.

[0061] Calculate the standard deviation σ dif

[0062] The standard deviation reflects the degree of dispersion of the historical quality fluctuation factor. If the standard deviation is small, it means that the value of the historical quality fluctuation factor is relatively concentrated, and the stability of the system is relatively high; if the standard deviation is large, it means that the quality fluctuation factor changes greatly, and there may be problems with the stability of the system.

[0063] Calculate threshold range

[0064] By adding or subtracting a certain multiple of the standard deviation from the average value, a reasonable threshold range can be determined. When the quality fluctuation factor is within this range, it means that the system's operating status is close to the historical normal state; when the quality fluctuation factor exceeds this range, it means that the system may have large differences and further analysis is needed.

[0065] The adjustment coefficient k is used to adjust the width of the threshold range. If the k value is small, the threshold range is narrow, the system is more sensitive to differences, and may result in more out-of-range data, resulting in excessive manual analysis costs; if the k value is large, the threshold range is wide, the system is less sensitive to differences, and some important issues may be missed. Therefore, it is necessary to select an appropriate value based on the actual situation and requirements of the software system.

[0066] S6. Output test report: Output the test results in the form of a retest report, which specifically includes [Report Overview] and [Report Details]. The [Report Overview] shows the overall calculation of the quality fluctuation factor, and automatically analyzes whether the quality of the software system meets the requirements according to the set threshold; the report details show in detail the comparison of each data point in the data set X to be tested and the baseline standard set B. When dif is lower than this threshold, the present invention believes that the software system meets the quality requirements; when dif is higher than this threshold, the present invention believes that the software system has quality problems and potential defects. Using pre-set rules, such as ignoring suspected problems, and then analyzing and confirming potential defects, defects confirmed as problems will generate detailed defect reports.

[0067] In summary, the present invention can improve the degree of automation of software testing and the accuracy of defect detection, reduce the cost of manual analysis, and has the following technical effects: 1. Significantly improve the detection efficiency and accuracy of software defects, and reduce omissions and false alarms; 2. Reduce testing costs and improve the economy of software testing; 3. Reduce errors introduced by human factors and improve the reliability of software testing; 4. It has strong adaptability and can detect different types of software systems, with good versatility.

[0068] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A method for improving the accuracy of software system delivery quality detection based on traffic playback, characterized in that: The steps include: S1) Traffic recording stage: collecting software system program code, configuration files, log files and user operation records; S2) Data preprocessing: Clean and process the collected user operation data, remove irrelevant or redundant information and use a standardized data format; S3) Building a baseline standard set: Use the cleaned data to train the machine learning model and build a baseline standard set B; S4) Traffic replay: Construct a retest task to repeatedly test the updated software system and obtain a set of data sets X to be tested; S5) Data difference comparison: Calculate the difference between the data set X to be tested and the baseline standard set B; S6) Output judgment result: calculate the quality fluctuation factor and automatically judge whether the quality of the software system meets the requirements according to the set threshold.

2. The method for improving the accuracy of software system delivery quality detection based on traffic playback according to claim 1 is characterized in that: The user operation record in step S1 includes recording the sequence and operation object of the user operation.

3. The method for improving the accuracy of software system delivery quality detection based on traffic playback as claimed in claim 1 is characterized in that: The step S2 presets a first time threshold. If the same character is input into the same input box more than a certain number of times within the first time threshold, this part of the record is regarded as an erroneous operation record and deleted, while the record of the last correct input is retained.

4. The method for improving the accuracy of software system delivery quality detection based on traffic playback according to claim 1, characterized in that: The step S2 presets a second time threshold. If multiple operations on the same object have the same result within the second time threshold, only the first operation record is retained and subsequent repeated operation records are deleted.

5. The method for improving the accuracy of software system delivery quality detection based on traffic playback as claimed in claim 1, characterized in that: The step S3 provides the preprocessed data as input to the decision tree machine learning model. The model learns the patterns and rules of the data during the training process, extracts the user operation frequency, operation sequence, and operating system response results, and constructs a baseline standard set B.

6. The method for improving the accuracy of software system delivery quality detection based on traffic playback according to claim 1, characterized in that: The step S5 also includes filtering and removing data that does not meet the conditions according to preset rules before performing data difference comparison.

7. The method for improving the accuracy of software system delivery quality detection based on traffic playback as claimed in claim 1, characterized in that: The step S5 uses the following formula to calculate the difference between the to-be-detected data set X and the baseline standard set B as the quality fluctuation factor: Xi represents the i-th data point in the data set to be tested, Bi represents the i-th data point in the baseline standard set, i represents the i-th corresponding data point in the baseline standard set, n is the number of data points, wi is the weight of the i-th data point, and dif is the final calculated software system quality fluctuation factor; wi=ai*Importance_i+bi*Frequency_i Importance_i is the importance of the business, and its value range is usually from 0 to 1; Frequency_i is the frequency of data occurrence; a and b are adjustment coefficients.

8. The method for improving the accuracy of software system delivery quality detection based on traffic playback as claimed in claim 7, characterized in that: The number of data points in the data set X to be tested is the same as that in the baseline standard set B. If they are not the same and the difference is less than a preset threshold, the data interpolation method is used to increase the data points in the smaller set until the number of data points in the two sets is the same; when the quality fluctuation factor dif is lower than the preset threshold, it is determined that the software system meets the quality requirements.

9. The method for improving the accuracy of software system delivery quality detection based on traffic playback as claimed in claim 7, characterized in that: The step S6 also includes displaying the overall calculation of the quality fluctuation factor and displaying the comparison of each data point in the to-be-detected data set X and the baseline standard set B.

10. The method for improving the accuracy of software system delivery quality detection based on traffic playback according to claim 1, characterized in that: The step S6 sets the threshold range of the quality fluctuation factor to Where k is the adjustment coefficient, dif j It represents the mass fluctuation factor obtained from the jth historical calculation, and N is the number of historical calculations.

Images