Software fault-tolerant capability evaluation method and system based on fault injection

Abstract

The invention provides a software fault tolerance evaluation method and system based on fault injection, and the method comprises the steps: determining a fault type needing to be detected and a fault model corresponding to the fault type according to the operation logic of to-be-detected software, and generating a test case containing a fault injection position, an injection time and an injection parameter based on the fault model; executing the test case through a fault injection method, and collecting field data when a fault is triggered; calculating a software fault detection rate gamma SFDR and a software fault isolation rate based on the data of the collection site; and generating a fault-tolerant capability preliminary evaluation report based on the weighted comprehensive value of the software fault detection rate gamma SFDR and the software fault isolation rate gamma SFIR. According to the technical scheme, quantitative evaluation of the software fault detection rate and the fault isolation rate can be achieved by designing the fault model to generate the test case and combining the fault injection and data automatic collection technology, and the reliability verification efficiency of software in a complex environment is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a method and system for evaluating software fault tolerance based on fault injection. Background Art

[0002] To implement fault injection and obtain on-site data when a fault occurs, determine the type of fault to be detected in the program. Combine the fault model for this type of fault with the software fault injection method to generate test cases targeting this fault model. If this type of fault does exist in the program, the test case will likely trigger it. Determine the data collection requirements, data collection methods, and test data storage. Data collection requirements include determining the type and scope of data to be collected, the timing of collection, and the collection configuration plan. The test data collection method extends the test case-driven model to ensure that the model on each simulation node supports the configuration of data collection plans and data acquisition functions to achieve automated data collection.

[0003] Based on the research on the design method of software fault-tolerance confirmation test cases based on fault injection, the test cases were executed and statistical data was collected to obtain the software fault detection rate and software fault isolation rate to evaluate the software's fault tolerance capability. Based on the task scenario, random fault interference was simulated to confirm whether the software's task capability in the task scenario met the task requirements.

[0004] Therefore, how to provide a software fault tolerance evaluation method and system based on fault injection, generate test cases by designing fault models and combine fault injection with automatic data collection technology to achieve quantitative evaluation of software fault detection rate and fault isolation rate, and improve the reliability verification efficiency of software in complex environments has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The embodiment of the present invention provides a software fault tolerance evaluation method and system based on fault injection. By designing a fault model to generate test cases and combining fault injection with automatic data collection technology, it can achieve quantitative evaluation of software fault detection rate and fault isolation rate, and effectively improve the reliability verification efficiency of software in complex environments.

[0006] In one embodiment of the present invention, a method for evaluating software fault tolerance based on fault injection is provided, comprising:

[0007] S101: Determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing, and injection parameters based on the fault model;

[0008] S102, executing the test case using a fault injection method, and collecting field data when the fault is triggered, wherein the field data includes fault injection data, software failure data, and running status data;

[0009] S103: Calculate the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now:

[0010]

[0011] Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected;

[0012] S104, based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

[0013] Furthermore, the fault injection data includes initial state data before fault injection, non-failure state data after fault injection, and statistical data of software failure;

[0014] The software failure statistics are accumulated according to the software failure flag f_flag(i). If the software fails after the i-th round of fault injection, f_flag(i)=1; otherwise, f_flag(i)=0.

[0015] If N rounds of fault injection are performed in the fault injection scenario, the number of failures a_num(i) is obtained based on the collected fault injection data, that is,

[0016]

[0017] Furthermore, the method includes: defining task fault tolerance evaluation indicators in a task scenario, including fault injection intensity γ and mean time to failure MTTF, that is,

[0018] γ=[a_num(N)] / [f_time(N)-s_time(1)]

[0019]

[0020] Where s_time(1) is the start time of fault injection, f_time(N) is the end time of fault injection, and e_time(i) is the interval between the i-th round and the previous round of fault injection.

[0021] Furthermore, based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of γ1,γ2,...,γ i ,...γ N}, the mean time to failure is {MTTF1,MTTF2,...,MTTTF i ,...,MTTF N}, then the comprehensive reliability evaluation index R e for:

[0022]

[0023] Wherein, α and β are the weight coefficients of the injection intensity and mean time before failure indicators, respectively, and α + β = 1; max() and min() represent the maximum and minimum values ​​of the injection intensity or mean time before failure, respectively.

[0024] Furthermore, the fault-tolerance evaluation of software tasks for fault injection calculates task fault-tolerance evaluation indicators based on the fault injection data collected during the execution of the fault injection scenario, and performs evaluation based on the indicator values;

[0025] According to the definitions of fault injection intensity and mean time before failure, the greater the calculated software fault injection intensity, the weaker the fault tolerance of the software environment faults, and vice versa; if the software mean time before failure is longer, the stronger the fault tolerance of the software environment faults, and vice versa.

[0026] Furthermore, the distribution mode of the fault injection intensity γ includes at least one of a uniform distribution mode and a normal distribution description;

[0027] The uniform distribution mode is used to simulate uniform interference in a steady-state environment;

[0028] The normal distribution pattern is used to simulate concentrated interference in sudden high-load scenarios.

[0029] Furthermore, the determination of software failure includes any of the following:

[0030] Abnormal termination failures, including catastrophic crashes, restart crashes, abnormal interruption crashes, silent crashes, and obstructive crashes;

[0031] Invalid output class failure, the deviation between output data and expected value is detected through predefined result comparison rules.

[0032] Furthermore, the method comprises:

[0033] When the program runs to a preset breakpoint, a fault is injected by overwriting the memory address or tampering with the register;

[0034] Use uniform distribution or normal distribution model to control the timing and frequency of fault injection to simulate random interference in real environment.

[0035] In another embodiment of the present invention, a software fault tolerance confirmation system based on fault injection is provided. The system is based on any one of the above software fault tolerance confirmation methods based on fault injection, and the system includes: a fault model configuration module, a dynamic fault injection module, a real-time data acquisition module, and a fault tolerance evaluation module;

[0036] The fault model configuration module is used to determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing and injection parameters based on the fault model;

[0037] The dynamic fault injection module executes the test case through a fault injection method and collects field data when the fault is triggered, including fault injection data, software failure data and running status data;

[0038] The real-time data acquisition module calculates the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now:

[0039]

[0040] Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected;

[0041] The fault tolerance evaluation module is based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

[0042] In another embodiment of the present invention, a software fault tolerance confirmation system based on fault injection includes: a processor and a memory, the memory storing a computer program, and the computer program, when executed by the processor, implements any of the above-mentioned software fault tolerance evaluation methods based on fault injection.

[0043] The beneficial effects brought about by the present invention are as follows:

[0044] As can be seen from the above scheme, an embodiment of the present invention provides a software fault tolerance evaluation method and system based on fault injection. The method includes: determining the fault type to be detected and its corresponding fault model based on the operating logic of the software to be tested, generating a test case including the fault injection location, injection timing and injection parameters based on the fault model; executing the test case through the fault injection method, collecting field data when the fault is triggered, including fault injection data, software failure data and operating status data; calculating the software fault detection rate γ based on the collected field data SFDR and software fault isolation rate; based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of the fault tolerance test is used to generate a preliminary fault tolerance evaluation report. The technical solution of the present invention, by designing a fault model to generate test cases and combining fault injection with automatic data collection technology, can achieve quantitative evaluation of software fault detection and fault isolation rates, effectively improving the efficiency of software reliability verification in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a software fault tolerance evaluation method based on fault injection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0047] like Figure 1 As shown, Figure 1 This is a flow chart of a method for evaluating software fault tolerance based on fault injection according to an embodiment of the present invention.

[0048] Figure 1 In [1], a software fault tolerance evaluation method based on fault injection is proposed, including:

[0049] S101: Determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing, and injection parameters based on the fault model;

[0050] S102, executing the test case using a fault injection method, and collecting field data when the fault is triggered, wherein the field data includes fault injection data, software failure data, and running status data;

[0051] S103: Calculate the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now:

[0052]

[0053] Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected;

[0054] S104, based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

[0055] In the embodiments of the present invention, since the fault injection scenario simulates the impact of the external environment in which the software is running, the injection time and location are generated using a uniform and normal distribution, and the injection intensity is very high during the scenario execution, which has a greater impact on the software than the actual operating environment of the software. Therefore, the index value obtained by the fault injection-based software task fault tolerance evaluation is greater than the actual index value, which can be used to assess the reliability of the software or provide suggestions for software reinforcement. Specifically, by executing multiple typical fault injection scenario tests, the average value of the fault injection intensity and failure time is calculated, and based on the size of the injection intensity and failure time average value, the software system's fault tolerance for memory and register faults is evaluated.

[0056] In one embodiment of the present invention, the fault injection data includes initial state data before fault injection, non-failure state data after fault injection, and statistical data of software failure;

[0057] The software failure statistics are accumulated according to the software failure flag f_flag(i). If the software fails after the i-th round of fault injection, f_flag(i)=1; otherwise, f_flag(i)=0.

[0058] If N rounds of fault injection are performed in the fault injection scenario, the number of failures a_num(i) is obtained based on the collected fault injection data, that is,

[0059]

[0060] In one embodiment of the present invention, the method includes: defining task fault tolerance evaluation indicators in a task scenario, including fault injection intensity γ and mean time to failure MTTF, that is,

[0061] γ=[a_num(N)] / [f_time(N)-s_time(1)]

[0062]

[0063] Where s_time(1) is the start time of fault injection, f_time(N) is the end time of fault injection, and e_time(i) is the interval between the i-th round and the previous round of fault injection.

[0064] In one embodiment of the present invention, based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of γ1,γ2,...,γ i ,...γ N}, the mean time to failure is {MTTF1,MTTF2,...,MTTTF i ,...,MTTF N}, then the comprehensive reliability evaluation index R e for:

[0065]

[0066] Wherein, α and β are the weight coefficients of the injection intensity and mean time before failure indicators, respectively, and α + β = 1; max() and min() represent the maximum and minimum values ​​of the injection intensity or mean time before failure, respectively.

[0067] In one embodiment of the present invention, the fault-tolerance evaluation of a software task for fault injection calculates a task fault-tolerance evaluation index based on the fault injection data collected during the execution of the fault injection scenario, and performs evaluation based on the index value;

[0068] According to the definitions of fault injection intensity and mean time before failure, the greater the calculated software fault injection intensity, the weaker the fault tolerance of the software environment faults, and vice versa; if the software mean time before failure is longer, the stronger the fault tolerance of the software environment faults, and vice versa.

[0069] In one embodiment of the present invention, the distribution mode of the fault injection intensity γ includes at least one of a uniform distribution mode and a normal distribution description;

[0070] The uniform distribution mode is used to simulate uniform interference in a steady-state environment;

[0071] The normal distribution pattern is used to simulate concentrated interference in sudden high-load scenarios.

[0072] In one embodiment of the present invention, the software failure determination is performed in a software fault injection scenario. There are generally two responses to the software after the fault is injected: one is that the software performs fault tolerance processing and provides a corresponding prompt; the other is that the software does not perform fault tolerance processing and may produce abnormal output or software crashes. The second situation mentioned above is that the software fails after the fault injection. Software failures mainly include any of the following categories:

[0073] Abnormal termination failures include catastrophic crashes, restart crashes, abnormal interrupt crashes, silent crashes and obstructive crashes; catastrophic crashes refer to failures that can cause the entire system to stop working and require the machine to be restarted; restart crashes are crashes detected by the monitor timer; abnormal interrupt crashes refer to abnormal interruptions of the test process; silent crashes refer to when illegal input is executed, the corresponding error handling code should be returned, but no indication is given; obstructive crashes refer to when an error handling code is returned, but the error handling code does not match the corresponding illegal data input.

[0074] Invalid output failures detect deviations between output data and expected values ​​using predefined result comparison rules. Invalid output occurs when a faulty application completes normally, without an abnormal termination, but produces invalid or incorrect results. This type of failure cannot be detected by monitoring control flow, but can be detected through duplicate and bounds checking. Result comparison mechanisms can reveal computational failures.

[0075] In one embodiment of the present invention, the method includes:

[0076] When the program runs to a preset breakpoint, a fault is injected by overwriting the memory address or tampering with the register;

[0077] Use uniform distribution or normal distribution model to control the timing and frequency of fault injection to simulate random interference in real environment.

[0078] In another embodiment of the present invention, a software fault tolerance confirmation system based on fault injection is provided. The system is based on any one of the above software fault tolerance confirmation methods based on fault injection, and the system includes: a fault model configuration module, a dynamic fault injection module, a real-time data acquisition module, and a fault tolerance evaluation module;

[0079] The fault model configuration module is used to determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing and injection parameters based on the fault model;

[0080] The dynamic fault injection module executes the test case through a fault injection method and collects field data when the fault is triggered, including fault injection data, software failure data and running status data;

[0081] The real-time data acquisition module calculates the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now:

[0082]

[0083]

[0084] Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected;

[0085] The fault tolerance evaluation module is based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

[0086] In another embodiment of the present invention, a software fault tolerance confirmation system based on fault injection includes: a processor and a memory, the memory storing a computer program, and the computer program, when executed by the processor, implements any of the above-mentioned software fault tolerance evaluation methods based on fault injection.

[0087] An embodiment of the present invention provides a software fault tolerance evaluation method and system based on fault injection. The method comprises: determining the fault type to be detected and its corresponding fault model based on the operating logic of the software to be tested; generating a test case including the fault injection location, injection timing, and injection parameters based on the fault model; executing the test case using a fault injection method to collect field data when the fault is triggered, including fault injection data, software failure data, and operating status data; and calculating the software fault detection rate γ based on the collected field data. SFDR and software fault isolation rate; based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

[0088] The technical solution of the present invention, by designing fault models to generate test cases combined with fault injection and automatic data collection technology, can achieve quantitative evaluation of software fault detection rate and fault isolation rate, and effectively improve the reliability verification efficiency of software in complex environments.

[0089] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for evaluating software fault tolerance based on fault injection, characterized in that: The method comprises: S101: Determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing, and injection parameters based on the fault model; S102, executing the test case using a fault injection method, and collecting field data when the fault is triggered, wherein the field data includes fault injection data, software failure data, and running status data; S103: Calculate the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now: Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected; S104, based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

2. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: The fault injection data includes the initial state data before the fault injection, the non-failure state data after the fault injection, and the statistical data of software failure; The software failure statistics are accumulated according to the software failure flag f_flag(i). If the software fails after the i-th round of fault injection, then f_flag(i)=1; Otherwise, f_flag(i)=0; If N rounds of fault injection are performed in the fault injection scenario, the number of failures a_num(i) is obtained based on the collected fault injection data, that is, 3. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: The method includes: defining task fault tolerance evaluation indicators in a task scenario, including fault injection intensity γ and mean time to failure MTTF, that is, γ=[a_num(N)] / [f_time(N)-s_time(1)] Where s_time(1) is the start time of fault injection, f_time(N) is the end time of fault injection, and e_time(i) is the interval between the i-th round and the previous round of fault injection.

4. The method for evaluating software fault tolerance based on fault injection according to claim 3, characterized in that: Based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of γ1,γ2,...,γ i ,...γ N }, the mean time to failure is {MTTF1,MTTF2,...,MTTTF i ,...,MTTF N }, then the comprehensive reliability evaluation index R e for: Wherein, α and β are the weight coefficients of the injection intensity and mean time before failure indicators, respectively, and α + β = 1; max() and min() represent the maximum and minimum values ​​of the injection intensity or mean time before failure, respectively.

5. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: Evaluation of the fault tolerance of software tasks using fault injection is done by calculating the task fault tolerance evaluation index based on the fault injection data collected during the execution of the fault injection scenario, and performing evaluation based on the index value; According to the definitions of fault injection intensity and mean time before failure, the greater the calculated software fault injection intensity, the weaker the fault tolerance of the software environment faults, and vice versa; if the software mean time before failure is longer, the stronger the fault tolerance of the software environment faults, and vice versa.

6. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: The distribution mode of the fault injection intensity γ includes at least one of a uniform distribution mode and a normal distribution description; The uniform distribution mode is used to simulate uniform interference in a steady-state environment; The normal distribution pattern is used to simulate concentrated interference in sudden high-load scenarios.

7. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: The determination of software failure includes any of the following: Abnormal termination failures, including catastrophic crashes, restart crashes, abnormal interruption crashes, silent crashes, and obstructive crashes; Invalid output class failure, the deviation between output data and expected value is detected through predefined result comparison rules.

8. The method for evaluating software fault tolerance based on fault injection according to claim 1, characterized in that: The method comprises: When the program runs to a preset breakpoint, a fault is injected by overwriting the memory address or tampering with the register; Use uniform distribution or normal distribution model to control the timing and frequency of fault injection to simulate random interference in real environment.

9. A software fault-tolerance confirmation system based on fault injection, characterized in that: The system is based on a software fault tolerance confirmation method based on fault injection according to any one of claims 1 to 8, and the system comprises: a fault model configuration module, a dynamic fault injection module, a real-time data acquisition module and a fault tolerance evaluation module; The fault model configuration module is used to determine the fault type to be detected and its corresponding fault model according to the operating logic of the software to be tested, and generate a test case including the fault injection location, injection timing and injection parameters based on the fault model; The dynamic fault injection module executes the test case through a fault injection method and collects field data when the fault is triggered, wherein the field data includes fault injection data, software failure data and operation status data; The real-time data acquisition module calculates the software fault detection rate γ based on the data collected at the scene SFDR and software fault isolation rate γ SFIR ,Right now: Among them, N D is the number of detected faults, M is the total number of executed use cases, N L is the number of faults isolated to the replaceable unit level, and N is the total number of faults detected; The fault tolerance evaluation module is based on the software fault detection rate γ SFDR and software fault isolation rate γ SFIR The weighted comprehensive value of is used to generate a preliminary evaluation report on fault tolerance.

10. A software fault-tolerance confirmation system based on fault injection, characterized in that: The system includes: a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for evaluating software fault tolerance based on fault injection according to any one of claims 1 to 8 is implemented.

Images