Simulation regeneration method for software reliability test failure data

Abstract

The invention discloses a software reliability test failure data simulation regeneration method, and relates to the technical field of software testing, the method comprises the following steps: extracting an operation profile from test cases of system testing and confirmation testing, and determining a main path and the occurrence probability of each operation; performing cleaning and time and quantity conversion on the failure data, and generating continuous base data by adopting polynomial fitting; simulation regeneration failure data meeting Poisson process characteristic requirements are generated through a standardized sampling method; according to the method, the existing test data is fully utilized, the problems of insufficient data and low data quality due to limited test time are solved, and the accuracy and efficiency of software reliability evaluation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a simulation regeneration method for software reliability testing failure data. Background Art

[0002] With the rapid development of information technology and the continuous expansion of software application fields, software quality has become an important factor affecting the overall performance of the system and user experience. Among them, software reliability is an important indicator for measuring software quality, and its evaluation results are directly related to the stability and security of the software in the actual operating environment. In order to verify whether the software can run stably under the specified conditions of use, it is necessary to conduct comprehensive reliability tests on the software, and systematically count and analyze the failure data generated during the test, so as to provide a scientific basis for subsequent reliability prediction, risk assessment and quality improvement.

[0003] However, the existing software reliability testing methods have obvious shortcomings, mainly in terms of the quantity and quality of failure data. First, since testing work is usually limited by project schedule and resource investment, software reliability testing is often implemented after system testing and confirmation testing. At this time, the software has been preliminarily verified and has high stability, so it is difficult to naturally generate a large amount of failure data in a short period of time. This lack of data directly affects the reliability assessment model's accurate capture and assessment of potential software defects. Secondly, the failure data collected during actual testing, experiments and use often have problems such as missing data items, missing items or irregular records, resulting in uneven data quality. Most existing reliability assessment models, such as the Schneidewind model, the Musa-Okumoto model, the Duane model, etc., require that the input data must meet the statistical assumption that the cumulative number of failures conforms to the Poisson process, while the data obtained by traditional testing often cannot meet this key requirement due to its incompleteness and local concentration.

[0004] In addition, some failure data generated in traditional tests are often affected by limit and boundary value effects, and their data distribution cannot truly reflect the failure trend of the software in actual use, further weakening the application value of the data in reliability assessment. Therefore, how to achieve simulation regeneration of failure data through in-depth processing and mathematical processing of existing test data under limited test time and resource conditions has become a key issue that needs to be solved in software reliability engineering. Summary of the invention

[0005] The purpose of the present invention is to provide a method for utilizing the test cases and failure data generated in the system testing and confirmation testing phases to simulate and regenerate failure data with sufficient quantity, qualified quality and in accordance with the requirements of the Poisson process distribution through technical means such as extracting operation profiles, data cleaning, time and quantity conversion, polynomial fitting and standardized sampling, thereby effectively making up for the defects of insufficient data and low data quality in traditional testing, providing reliable data support for software reliability evaluation, and promoting the implementation and promotion of software reliability engineering in practical applications.

[0006] The technical solution of the present invention is to provide a simulation regeneration method for software reliability test failure data, the method comprising:

[0007] S1. Extracting operation profiles from test cases of system testing and confirmation testing, including determining software usage patterns, screening main path test cases, identifying operation initiating roles, building operation sets and calculating the probability of each operation occurring;

[0008] S2. Fit the failure data of system test and confirmation test into base data, including eliminating non-main path failure data, rearranging data according to operation flow, implementing time conversion and quantity conversion processing, and generating base data fitting function through polynomial fitting

[0009] S3. Perform normalized sampling on the base data, and perform sample screening through the recommended function ρ(t) and the acceptance threshold α to obtain simulated regenerated failure data whose quantity and quality meet the requirements of the software reliability model.

[0010] In any of the above technical solutions, further, extracting the operation profile in step S1 specifically includes:

[0011] S11. Obtain all test cases from system testing and confirmation testing, and extract the software usage mode by referring to the software requirement specification;

[0012] S12. Based on the usage pattern extracted in S11, the main path test cases are screened out from the test cases of the system test and the confirmation test. The screening method is to group the test cases according to the operation sequence, and then perform deduplication processing, merge the test cases with exactly the same operation process, parameters and execution results, select the test case that obtains the earliest successful result in each group as the first successful sample, and aggregate all the first successful samples to form the main path test cases;

[0013] S13. According to the software usage mode and the main path test case, determine the initiating role of each operation in the actual operation of the software, that is, identify the user or module that actively initiates each operation in the actual operation of the software;

[0014] S14. Classify and summarize the main path test cases corresponding to each role according to the operation initiation role determined in S13 to form an operation set for each role;

[0015] S15. Count the number of occurrences of each operation in the main path test cases in S14, and calculate the occurrence probability P of each operation based on the ratio of the sample number of each operation to the total sample number of all main path samples. The calculation formula is as follows: i , The calculation formula is as follows:

[0016]

[0017] Among them, n i is the total sample number of the main path test cases corresponding to this operation, and N is the total sample number of the main path test cases corresponding to all operations.

[0018] In any of the above technical solutions, further, the failure data fitting in step S2 specifically includes:

[0019] S21. Eliminate the failure data generated by non-main path test cases in the failure data of system testing and confirmation testing;

[0020] S22. Based on the operation set determined in S1, rearrange the failure data generated by the remaining main path test cases in the actual execution order of each operation, so that the arrangement order of the failure data is consistent with the actual software operation process;

[0021] S23. According to the time unit required for reliability assessment specified by the user and the predetermined conversion ratio, perform time conversion processing on the failure occurrence time recorded in the rearranged failure data in S22, and convert the original time data into time data that meets the assessment requirements;

[0022] S24. According to the occurrence probability of each operation calculated in S15, perform conversion processing on the quantity of the rearranged failure data in S22. The calculation formula is as follows:

[0023]

[0024] Among them, N 折算前 and N 折算后 are the quantities of failure data before and after conversion respectively;

[0025] S25. Use the failure data adjusted by time and quantity conversion in S8 and S9, and adopt the polynomial fitting method to fit the data to obtain a continuous basic data fitting curve, and this curve is used as the basic data fitting function which can reflect the cumulative number of failures at any moment t.

[0026] In any of the above technical solutions, further, the standardized sampling in step S3 includes:

[0027] S31. Determine the number of samples n required for standardized sampling, and provide a proposed function ρ(t) that meets the requirements of the Poisson process characteristics, a calculation constant M, and an acceptance threshold α for sampling determined by the expert judgment method;

[0028] S32. Randomly sample from the proposed function ρ(t) to obtain a sample point s, and calculate the absolute value of the difference between the basic data fitting function

[0029] and the proposed function ρ(t) at the sample point s;

[0030] S33. Determine whether the calculated absolute value of the difference β is less than the acceptance threshold α. If the absolute value of the difference β is less than the acceptance threshold α, accept the sample point as valid sampling data; otherwise, discard the sample point and return to step S32 for re-random sampling;

[0031] S34. Repeat the above steps until the number of sample points that meet the requirements reaches the number of samples n required for standardized sampling.

[0031] In any of the above technical solutions, further, the calculation constant M needs to satisfy that for any t, and the closer the calculation constant M is to 1, the better.

[0032] The beneficial effects of the present invention are as follows:

[0033] By making full use of and mathematically processing the test cases and failure data generated during the system testing and validation testing processes, the present invention realizes the purpose of simulating and "regenerating" software failure data, thereby solving the defects of insufficient quantity of failure data, low data quality, and non-compliance with the requirements of software reliability assessment models in traditional testing.

[0034] Specifically, the prior art mainly relies on long-term and intensive testing to obtain failure data, often unable to meet the quantity and quality of data required for reliability assessment; while the present invention utilizes the existing data in system testing and validation testing, and realizes the simulation and regeneration of failure data by extracting the operation profile and mathematical processing, which not only shortens the testing cycle but also can generate a large number of qualified failure data within a limited testing time.

[0035] By extracting the operation profile from test cases, the present invention accurately reflects the user operation mode and operation probability during the actual operation of the software, enabling the subsequent generated data to truly reproduce the failure situations that may occur when users use the software, fundamentally improving the representativeness and reliability of the data, and the regenerated failure data can be closer to the actual operating environment. In contrast, the prior art usually ignores the analysis of the operation mode and fails to reflect the real user usage characteristics during the data generation process.

[0036] When dealing with failure data, traditional methods fail to effectively eliminate the data that does not conform to the actual usage situation due to reasons such as extreme values and boundary values, resulting in uneven data distribution and non - satisfaction of the Poisson process assumption. The present invention obtains the basic data that can truly reflect the failure accumulation trend through cleaning, re - arranging, time and quantity conversion, and polynomial fitting of the failure data, and then ensures that the generated data meets the requirements of the statistical model through normalized sampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and additional advantages of the present invention will become obvious and easy to understand in conjunction with the description of the embodiments with the following drawings, wherein:

[0038] Figure 1 is a schematic flowchart of the operation profile extraction of the simulation regeneration method for software reliability test failure data according to an embodiment of the present invention;

[0039] Figure 2 is a schematic flowchart of the basic data fitting of the simulation regeneration method for software reliability test failure data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] In order to more clearly understand the above - mentioned objects, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0041] In the following description, many specific details are set forth to facilitate a thorough understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0042] In software development, it is usually necessary to test whether the software is stable and reliable, and an important basis for judging this is the error or failure data that occurs during the test. Ideally, sufficient failure data should be collected during the test, but this is often not the case in reality, because before the formal reliability test, the software has undergone other tests and is relatively stable, so it is difficult to naturally generate a large amount of error data within a limited time, and the collected error information may be missing or non-standard, and many statistical methods require these error data to conform to specific mathematical laws (such as the Poisson process), otherwise they cannot be directly used to evaluate software reliability.

[0043] like Figure 1 and Figure 2 As shown, this embodiment provides a simulation regeneration method for software reliability test failure data, which uses existing test data to supplement or regenerate sufficient and appropriate error data, so as to better perform software reliability assessment. The method includes:

[0044] S1. Extract operation profiles from test cases of system testing and confirmation testing.

[0045] Software reliability testing, like general software functional testing, is a black box test, but software reliability testing is conducted according to the way users actually use the software. Users' actual use is often concentrated on the main path of the software; while system testing and confirmation testing are tests conducted according to software requirements or structure, and will conduct detailed tests on valid domains, invalid domains, and boundary values.

[0046] Operation profile is a technique for describing how software is used, and is also the key to distinguishing software reliability testing from general software functional testing. Operation profile models the usage of software, including operation sets and operation occurrence probabilities. In practical applications, the identification and construction of operation profiles requires a lot of manpower and time. Therefore, the present invention proposes a method for extracting operation profiles from test cases of system testing and confirmation testing, and the specific steps include:

[0047] S11. Obtain all test cases from system testing and confirmation testing, and extract the software usage pattern with reference to the software requirement specification.

[0048] S12. Based on the usage patterns extracted in S11, main path test cases are screened out from the test cases of system testing and confirmation testing. The main path test cases are the most effective operation processes during normal use of the software. The screening method is to group the test cases according to the operation sequence, and then perform deduplication processing. The test cases with exactly the same operation processes, parameters and execution results are merged, and the test case that obtains the earliest successful result in each group is selected as the first successful sample. All the first successful samples are aggregated to form the main path test cases.

[0049] S13. Determine the initiating roles of each operation during the actual operation of the software according to the software usage pattern and the main path test cases, that is, identify the users or modules that actively initiate various operations during the actual operation of the software.

[0050] S14. Classify and summarize the main path test cases corresponding to each role according to the operation initiating roles determined in S13 to form an operation set for each role.

[0051] S15. Count the number of occurrences of each operation in the main path test cases in S14, and calculate the occurrence probability P of each operation based on the ratio of the sample number of each operation to the total sample number of all main paths. i , and the calculation formula is as follows:

[0052]

[0053] where n i is the total sample number of the main path test cases corresponding to this operation, and N is the total sample number of the main path test cases corresponding to all operations.

[0054] S2. Fit the failure data of system testing and validation testing into base data.

[0055] The biggest difference between the failure data generated in the system testing and validation testing phases and the failure data generated in the reliability testing is that the system testing and validation testing conduct intensive testing on the software in a short period of time, and the occurrence time of failures will be relatively concentrated, making it difficult to accurately reflect the trend of the number of failures changing with time during the reliability testing process; moreover, there are many failures generated by extreme values and boundary values in the failure data of system testing and validation testing, which does not conform to the feature of the reliability testing of "conducting testing in accordance with the actual way users use the software". Therefore, the present invention proposes a method for fitting the failure data of system testing and validation testing into base data, and the specific steps include:

[0056] S21. Eliminate the failure data generated by non-main path test cases in the failure data of system testing and validation testing.

[0057] S22. Based on the operation set determined in S1, rearrange the failure data generated by the remaining main path test cases according to the actual execution order of each operation, so that the arrangement order of the failure data is consistent with the actual software operation process.

[0058] S23. Since the failure occurrence times recorded in the failure data may be different from the time units required for the actual reliability test, perform a time conversion process on the failure occurrence times recorded in the rearranged failure data in S22 according to the time unit required for the reliability assessment specified by the user and the predetermined conversion ratio, and convert the original time data into time data that meets the assessment requirements.

[0059] S24. According to the probabilities of each operation occurrence calculated in S15, perform a conversion process on the quantity of the rearranged failure data in S22. The calculation formula is as follows:

[0060]

[0061] where N 折算前 and N 折算后 are the quantities of the failure data before and after conversion, respectively.

[0062] S25. Using the failure data adjusted by time and quantity conversion in S8 and S9, perform polynomial fitting on the data to obtain a continuous basic data fitting curve, and this curve serves as the basic data fitting function that can reflect the cumulative number of failures at any moment t.

[0063] S3. Perform standardized sampling on the basic data to obtain simulated and regenerated failure data that meet the requirements of the software reliability model in terms of both quantity and quality.

[0064] Currently, many widely used reliability assessment models such as the Schneidewind model, the Musa-Okumoto model, the Duane model, etc. require that the failure data must satisfy the assumption of "the cumulative number of failures conforms to the Poisson process". The basic data fitting function obtained through polynomial fitting is continuous, but often has a complex distribution and does not conform to the Poisson process. This results in the generated basic data not being able to be directly used as the input of the reliability assessment model and must be further standardized and sampled to meet the assumption requirements of the model. Therefore, the present invention proposes a method for standardizing and sampling the basic data to perform a normalization process on the basic data fitting function to generate samples that conform to the Poisson process. The specific steps are as follows:

[0065] S31. Determine the number of samples n required for standardized sampling, and provide a proposed function ρ(t) that meets the requirements of the Poisson process characteristics, a calculation constant M, and an acceptance threshold α for sampling determined by the expert judgment method. The calculation constant M needs to satisfy that for any t, and the closer the calculation constant M is to 1, the better.

[0066] S32. Randomly sample from the proposed function ρ(t) to obtain a sample point s, and calculate the absolute value of the difference between the basic data fitting function and the proposed function ρ(t) at the sample point s

[0067] S33. Determine whether the calculated absolute value of the difference β is less than the acceptance threshold α. If the absolute value of the difference β is less than the acceptance threshold α, accept this sample point as valid sampling data; otherwise, discard this sample point and return to step S32 to perform random sampling again.

[0068] S34. Repeat the above steps until the number of sample points that meet the requirements collected reaches the number of samples n required for normalized sampling.

[0069] Through this step, we select a set of samples from the previously smoothed basic data. These samples not only have sufficient quantity, but also their statistical distribution conforms to the Poisson process required in software reliability assessment, so they can be directly used for software reliability testing.

[0070] Generally speaking, this method provides a method to "regenerate" software failure data by using existing test data. It first summarizes how users use the software from test records, and then sorts out and mathematically fits the collected error data to form a curve representing the error accumulation trend. Finally, through a method of normalized sampling, error data that meets the requirements of a specific statistical distribution is selected from this curve, so as to generate a set of sufficient and high-quality data for evaluating whether the software is reliable.

[0071] This method can not only solve the problem of insufficient error data caused by test time limitations, but also ensure that the generated data meets the requirements of various reliability assessment models, thus greatly improving the efficiency and accuracy of software reliability testing.

[0072] In summary, the present invention provides a method for simulating and regenerating software reliability test failure data, and this method includes:

[0073] S1. Extract the operation profile from the test cases of system testing and validation testing, including determining the software usage pattern, screening the main path test cases, identifying the operation initiating roles, constructing the operation set and calculating the occurrence probability of each operation.

[0074] S2. Fit the failure data of system testing and validation testing into basic data, including eliminating non-main path failure data, rearranging the data according to the operation process, performing time conversion and quantity conversion processing, and generating a basic data fitting function through polynomial fitting

[0075] S3. Normalize and sample the base data, and screen the samples through the proposal function ρ(t) and the acceptance threshold α to obtain the simulated regenerated failure data that meets the requirements of the software reliability model in terms of quantity and quality.

[0076] The steps in the present invention can be adjusted, combined, and deleted in sequence according to actual needs.

[0077] The units in the device of the present invention can be combined, divided, and deleted according to actual needs.

[0078] Although the present invention has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present invention. The protection scope of the present invention is defined by the appended claims and may include various variations, modifications, and equivalent solutions made to the invention without departing from the protection scope and spirit of the present invention.

Claims

1. A simulation regeneration method for software reliability test failure data, characterized in that: The method comprises: S1. Extracting operation profiles from test cases of system testing and confirmation testing, including determining software usage patterns, screening main path test cases, identifying operation initiating roles, building operation sets and calculating the probability of each operation occurring; S2. Fit the failure data of system test and confirmation test into base data, including eliminating non-main path failure data, rearranging data according to operation flow, implementing time conversion and quantity conversion processing, and generating base data fitting function through polynomial fitting S3. Perform normalized sampling on the base data, and perform sample screening through the recommended function ρ(t) and the acceptance threshold α to obtain simulated regenerated failure data whose quantity and quality meet the requirements of the software reliability model.

2. The simulation regeneration method for software reliability test failure data according to claim 1, characterized in that: The step S1 of extracting the operation profile specifically includes: S11. Obtain all test cases from system testing and confirmation testing, and extract the software usage mode by referring to the software requirement specification; S12. Based on the usage pattern extracted in S11, the main path test cases are screened out from the test cases of the system test and the confirmation test. The screening method is to group the test cases according to the operation sequence, and then perform deduplication processing, merge the test cases with exactly the same operation process, parameters and execution results, select the test case that obtains the earliest successful result in each group as the first successful sample, and aggregate all the first successful samples to form the main path test cases; S13. According to the software usage mode and the main path test case, determine the initiating role of each operation in the actual operation of the software, that is, identify the user or module that actively initiates each operation in the actual operation of the software; S14, according to the operation initiating roles determined in S13, the main path test cases corresponding to each role are classified and summarized to form an operation set for each role; S15, counting the number of times each operation in S14 appears in the main path test case, and calculating the occurrence probability P of each operation based on the ratio of the number of samples of each operation to the total number of samples of all main paths. i , the calculation formula is as follows: Among them, n i is the total number of samples of the main path test cases corresponding to the operation, and N is the total number of samples of the main path test cases corresponding to all operations.

3. The simulation regeneration method for software reliability test failure data according to claim 1, characterized in that: The failure data fitting in step S2 specifically includes: S21, eliminating the failure data generated by non-main path test cases from the failure data of system testing and confirmation testing; S22, based on the operation set determined in S1, rearrange the failure data generated by the remaining main path test cases according to the actual execution order of each operation, so that the arrangement order of the failure data is consistent with the actual software operation process; S23, according to the time unit required for reliability evaluation specified by the user and the predetermined conversion ratio, the failure occurrence time recorded in the failure data rearranged in S22 is converted into time data that meets the evaluation requirements; S24, according to the occurrence probability of each operation calculated in S15, the number of invalid data rearranged in S22 is converted, and the calculation formula is as follows: Among them, N 折算前 and N 折算后 are the number of failure data before and after conversion, respectively; S25, using the failure data adjusted by time and quantity conversion in S8 and S9, the data is fitted by a polynomial fitting method to obtain a continuous base data fitting curve, which is used as the base data fitting function It can reflect the cumulative number of failures at any time t.

4. The simulation regeneration method for software reliability test failure data according to claim 1, characterized in that: The normalized sampling in step S3 includes: S31. Determine the number of samples n required for standardized sampling, and provide a recommended function ρ(t) that meets the requirements of the Poisson process characteristics, a calculation constant M, and a sampling acceptance threshold α determined by expert judgment; S32, randomly sample from the proposed function ρ(t), obtain sample point s, and calculate the base data fitting function The absolute value of the difference with the proposed function ρ(t) at sample point s S33, judging whether the calculated absolute value β of the difference is less than the acceptance threshold α, if the absolute value β of the difference is less than the acceptance threshold α, accepting the sample point as valid sampling data, otherwise discarding the sample point and returning to step S32 for random sampling again; S34, repeat the above steps until the number of sample points that meet the requirements reaches the number of samples m required for standardized sampling.

5. The simulation regeneration method for software reliability test failure data according to claim 1, characterized in that: The calculation constant M needs to satisfy for any t, And the closer the calculation constant M is to 1, the better.

Images