FMEA-based operating system robustness test case construction method and system

Abstract

The invention discloses an FMEA-based operating system robustness test case construction method and system. The method comprises the following steps: constructing a robustness test analysis criterion based on software related standard requirements; determining a use scene based on the to-be-analyzed operating system; determining a fault mode of the use scene based on a robustness test analysis criterion; processing based on the fault mode to obtain fault data; establishing a mapping relation based on the fault data and a software function item of the to-be-analyzed operating system; and obtaining an operating system robustness test case library based on the mapping relationship. The advantages of the FMEA in the field of robustness test are fully exerted, the number of fault discovery is greatly increased, and the quality of the robustness test case is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software system testing, and more particularly to a method and system for constructing operating system robustness test cases based on FMEA. Background Art

[0002] Operating system robustness testing is a type of robustness testing specifically for operating systems. Its purpose is to ensure that the operating system can continue to run stably in the face of various unexpected situations or extreme conditions, or can gracefully recover or handle errors when a failure occurs, without causing serious problems such as crashes and deadlocks.

[0003] The existing operating system robustness test mainly adopts fuzz testing or fault injection method, which generates and executes a large number of random or semi-random test instances to cause system failures. Some specific failure modes are difficult to detect and are easily overlooked. For example, many severe abnormal conditions such as high load and high pressure are difficult to trigger. If these problems are not discovered in time, there will be safety hazards. At the same time, the failures caused by this method are difficult to reproduce and it is not easy to find the cause of the failure. With the increase of system complexity, the potential failure modes are also increasing. Fuzz testing and fault injection may require a large number of test runs and data collection, consume more computing resources and time, and have not formed a priority evaluation mechanism. It is impossible to effectively determine the severity of many failure modes. Therefore, in the face of more complete and accurate robustness testing needs, more stringent requirements are put forward for the entire analysis method.

[0004] Therefore, how to effectively improve the quality of robustness test cases and thus increase the number of fault discoveries is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a method and system for constructing operating system robustness test cases based on FMEA, which gives full play to the advantages of FMEA in the field of robustness testing, greatly increases the number of faults discovered, and effectively improves the quality of robustness test cases.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for constructing operating system robustness test cases based on FMEA, comprising:

[0008] Construct robustness test analysis criteria based on software-related standard requirements;

[0009] Determine the usage scenario based on the operating system to be analyzed;

[0010] Determining a failure mode of the usage scenario based on the robustness test analysis criteria;

[0011] Performing processing based on the fault mode to obtain fault data;

[0012] Establishing a mapping relationship between the fault data and the software function items of the operating system to be analyzed;

[0013] An operating system robustness test case library is obtained based on the mapping relationship.

[0014] Preferably, constructing robustness test analysis criteria specifically includes:

[0015] The software-related standard requirements include: software security design guidelines and software testing guidelines;

[0016] Obtain robustness test specifications and key points based on the software safety design guide and the software testing guide;

[0017] Based on the robustness test specifications and key points, the robustness test analysis criteria are constructed by classifying them from three aspects: erroneous data processing, abnormal situation processing and illegal operation processing, and combining the functional characteristics and running characteristics of the operating system.

[0018] Preferably, the usage scenario is determined, specifically including:

[0019] Selecting requirement documents, design documents and user documents of the operating system to be analyzed as relevant documents;

[0020] Based on the relevant documents, the steps are decomposed to obtain a plurality of complete usage scenarios including one or more functional points.

[0021] Preferably, determining the failure mode of the usage scenario specifically includes:

[0022] Based on the usage scenario, the robustness test analysis criteria are matched to obtain relevant analysis criteria;

[0023] The failure mode of the usage scenario is determined based on the relevant analysis criteria.

[0024] Preferably, the matching process is specifically as follows:

[0025] Based on the usage scenario and the robustness test analysis criteria, the usage scenario vector and the analysis criterion vector are input into a pre-trained semantic similarity model to obtain a usage scenario vector and an analysis criterion vector respectively;

[0026] Obtaining cosine similarity based on the usage scenario vector and the analysis criterion vector;

[0027] The correlation analysis criterion is determined based on the cosine similarity.

[0028] Preferably, obtaining fault data specifically includes:

[0029] Based on the failure mode, analyze according to relevant standards to obtain all possible causes of the failure;

[0030] Analyze the impact of the failure mode on the operational safety of the local and entire system and the task completion and user experience of the operating system user to obtain the failure impact;

[0031] Determine the severity of the impact based on the fault impact to obtain a severity level;

[0032] The fault data is constructed based on the fault mode, the fault cause, the fault impact, the severity level and the usage scenario.

[0033] Preferably, establishing a mapping relationship specifically includes:

[0034] The software function items include: external input interface data, function processing process and external output interface data;

[0035] Establishing a mapping based on the fault cause and the external input interface data or the functional processing process, determining the value space and value equivalence class of the external input interface data, or determining the execution logic and constraint conditions of the functional processing process;

[0036] Establishing a mapping based on the fault mode and the external output interface data, determining the value space and value equivalence class of the external input interface data;

[0037] A mapping is established based on the severity level and the execution priority number of the robustness test case.

[0038] Preferably, an operating system robustness test case library is obtained, specifically including:

[0039] Determining robustness test case content based on the mapping relationship;

[0040] An initial test case is formed based on the robustness test case content and the execution priority number;

[0041] The initial test cases are sorted from high to low based on the execution priority numbers to obtain a logical operating system robustness test case library.

[0042] Preferably, the higher the severity level, the higher the execution priority.

[0043] An operating system robustness test case construction system based on FMEA, comprising: a criterion construction module, a usage scenario determination module, a failure mode determination module, a failure data acquisition module, a mapping relationship determination module and a test case construction module;

[0044] The criterion building module is used to build robustness test analysis criteria based on software-related standard requirements;

[0045] The usage scenario determination module is used to determine the usage scenario based on the operating system to be analyzed;

[0046] The failure mode determination module is used to determine the failure mode of the usage scenario based on the robustness test analysis criterion;

[0047] The fault data acquisition module is used to process based on the fault mode to obtain fault data;

[0048] The mapping relationship determination module is used to establish a mapping relationship between the fault data and the software function items of the operating system to be analyzed;

[0049] The test case construction module is used to obtain an operating system robustness test case library based on the mapping relationship.

[0050] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for constructing robustness test cases for an operating system based on FMEA, which uses relevant standard requirements in the testing field to design robustness test analysis criteria, guides testers to analyze potential failure modes in operating system-related documents according to the criteria, conducts failure mode and impact analysis, and further constructs a robustness test case set, which has the following beneficial effects:

[0051] 1. The present invention can help software developers and testers fully discover the robustness problems in the operating system requirements and design stages, formulate requirements and design strategies to enhance the robustness of the operating system; enhance the stability and reliability of the system in the face of abnormal and complex failures, and ensure that the operating system can continue to run and provide services to meet the system's requirements for product quality.

[0052] 2. The present invention utilizes relevant standard requirements in the testing field to design robustness test analysis criteria, guides testers to analyze potential failure modes in operating system-related documents according to the criteria, conducts failure mode and impact analysis, and further constructs a robustness test case set; it gives full play to the advantages of the FMEA method in the field of robustness testing, greatly increases the number of faults discovered, effectively improves the quality of robustness test cases, further improves the robustness of domestic operating systems, and has strong engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 A flow chart of a method for constructing operating system robustness test cases based on FMEA provided by the present invention.

[0055] Figure 2 A workflow diagram of a method for constructing operating system robustness test cases based on FMEA provided by the present invention.

[0056] Figure 3 A schematic diagram of the system structure for building a FMEA-based operating system robustness test case is provided in the present invention.

[0057] Figure 4 This is a structural block diagram of the computer device provided by the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] FMEA (Failure Mode and Effects Analysis) is a systematic method used to identify potential failure modes in systems, designs, manufacturing or processes, evaluate the impact of these failures, and determine appropriate improvement measures to reduce or eliminate the occurrence of failures. FMEA technology has been widely used in various fields and at all stages. In the software development stage, it can identify potential problems in requirements analysis, design decisions, and codes, help developers optimize designs and take preventive measures. In the software testing stage, it can guide the test team to design specific test cases for identified failure modes, help determine the focus and priority of testing, evaluate the severity and impact of discovered failures, and improve the effectiveness and pertinence of testing.

[0060] In order to rationally allocate resources for robustness testing of domestic operating systems, priority is given to testing typical computing, storage, and network-related content to improve the robustness of the operating system. The concept of the FMEA method is used to build a fault library, analyze fault modes, fault causes, impacts, and severity levels; extract fault data from relevant standards in the test field and related documents of domestic operating systems, and build a fault mode library and robustness test case set. The robustness of the operating system is improved through case construction and the sorting and selection of case sets.

[0061] Example 1

[0062] like Figure 1 As shown, an embodiment of the present invention discloses a method for constructing a robustness test case of an operating system based on FMEA, comprising:

[0063] Construct robustness test analysis criteria based on software-related standard requirements;

[0064] Determine the usage scenario based on the operating system to be analyzed;

[0065] Determine the failure modes of the usage scenarios based on robustness test analysis criteria;

[0066] Processing is performed based on the fault mode to obtain fault data;

[0067] Establishing a mapping relationship between the fault data and the software function items of the operating system to be analyzed;

[0068] Based on the mapping relationship, an operating system robustness test case library is obtained.

[0069] Example 2

[0070] like Figure 2 As shown, an embodiment of the present invention discloses a method for constructing a robustness test case of an operating system based on FMEA, comprising:

[0071] Robustness test analysis criteria establishment phase:

[0072] Construct robustness test analysis criteria based on software-related standard requirements:

[0073] Preferably, constructing robustness test analysis criteria specifically includes:

[0074] Software-related standard requirements include: software security design guidelines and software testing guidelines;

[0075] Based on the software safety design guidelines and software testing guidelines, robustness testing specifications and key points are obtained;

[0076] Based on the robustness test specifications and key points, it is classified into three aspects: erroneous data processing, abnormal situation processing and illegal operation processing. Combined with the functional characteristics and operation characteristics of the operating system, the robustness test analysis criteria are constructed.

[0077] Preferably, the robustness test analysis criteria include: erroneous data processing, abnormal situation processing and illegal operation processing.

[0078] Preferably, the error data processing subcategories include: computing boundary scenarios, storage boundary scenarios and network boundary scenarios; the criterion names of computing boundary scenarios include: memory out-of-bounds access, memory address error, illegal pointer and array boundary excess; the criterion names of storage boundary scenarios include: writing exceeding disk quota, writing file format error and reading file does not exist; the criterion names of network boundary scenarios include: high-load communication, UDP packet header error and TCP packet length error.

[0079] Preferably, analyze and design the memory out-of-bounds access scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0080] 1. Array index error; 2. Pointer operation error; 3. Memory allocation error; 4. Kernel module error; 5. Driver error; 6. Improper use of kernel interface; 7. Buffer overflow; 8. Buffer underflow; 9. Multi-threaded shared buffer; 10. Use of uninitialized memory; 11. Memory access after release; 12. Repeated release of memory; 13. Improper synchronization; 14. Race condition; 15. Resource lock contention; 16. Buffer overflow attack; 17. Code injection attack; 18. Privilege escalation attack; 19. Incomplete protocol implementation; 20. Version incompatibility; 21. Protocol extension error; 22. Virtual memory management error; 23. Insufficient resource isolation; 24. Virtual device driver error.

[0081] Preferably, analyze and design for memory address error scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0082] 1. Invalid pointer passing; 2. Buffer overflow; 3. Address space conflict; 4. Uninitialized memory access; 5. Duplicate free; 6. Use after free; 7. Invalid page access; 8. Page table error; 9. Page replacement error; 10. Permission error; 11. Protection mechanism failure; 12. Race condition; 13. Synchronization failure; 14. Failed to properly validate input; 15. Hardware address error; 16. Uninitialized pointer; 17. Invalid memory mapping; 18. I / O synchronization error; 19. Buffer error; 20. Interrupt vector error; 21. Exception handling error; 22. Unhandled exception; 23. Vulnerability exploit; 24. Privilege escalation; 25. Buffer overflow.

[0083] Preferably, analyze and design for illegal pointer scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0084] 1. Use of uninitialized pointer; 2. Improper assignment; 3. Null pointer dereference; 4. Improper check; 5. Use after free; 6. Dangling pointer passing; 7. Out-of-bounds access; 8. Wrong pointer offset; 9. Memory leak; 10. Repeated release; 11. Incorrect release; 12. Race condition; 13. Unsynchronized access; 14. Incorrect cast; 15. Unaligned access; 16. Passing invalid pointer; 17. API usage error; 18. Invalid memory mapping; 19. I / O buffer error; 20. Buffer overflow attack; 21. Privilege escalation attack.

[0085] Preferably, analyze and design the array boundary overrun scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0086] 1. Failure to check array bounds; 2. Loop error; 3. Hard-coded index; 4. Misuse of dynamic allocation function; 5. Failure to resize; 6. Access to uninitialized memory; 7. Incorrect index calculation; 8. Loop nesting error; 9. Failure to check bounds of multiple dimensions; 10. Incorrect pointer arithmetic; 11. Incorrect type conversion; 12. Out-of-bounds pointer dereference; 13. Incorrect recursion termination; 14. Stack overflow; 15. Recursive condition error; 16. Unsynchronized access; 17. Dynamic change of array size; 18. Shared data competition; 19. Failure to check input size; 20. Buffer overflow; 21. Incorrect file reading and writing; 22. Hardware address access error; 23. Device buffer management error; 24. Incorrect handling of interrupt data; 25. Buffer overflow attack; 26. Privilege escalation attack; 27. Code injection.

[0087] Preferably, analyze and design the scenario of exceeding disk quota writing to confirm whether the software can effectively detect and handle the following failure modes:

[0088] 1. Quota calculation error; 2. Inconsistency check; 3. Log record error; 4. Metadata update error; 5. Space allocation error; 6. File extension error; 7. Race condition; 8. Synchronization mechanism error; 9. Permission management error; 10. Unchecked quota; 11. Cache management error; 12. Improper exception handling; 13. Patch conflict; 14. Version incompatibility; 15. Configuration error; 16. Backup and restore error; 17. Duplicate backup data; 18. Backup operation does not check quota; 19. Disk bad sector; 20. Disk controller error; 21. Hardware damage; 22. Virtual machine resource contention; 23. Host and virtual machine synchronization error; 24. Virtual disk mapping error; 25. Client cache failure; 26. Network delay; 27. Protocol implementation error; 28. Permission configuration error; 29. ​​Quota bypass; 30. Log audit failure.

[0089] Preferably, analyze and design scenarios for writing file format errors to confirm whether the software can effectively detect and handle the following failure modes:

[0090] 1. Metadata update error; 2. Space allocation error; 3. File extension error; 4. Improper data processing; 5. Incorrect file header writing; 6. Cache processing error; 7. Race condition; 8. Synchronization mechanism error; 9. Permission management error; 10. Patch conflict; 11. Version incompatibility; 12. Configuration error; 13. Cache consistency error; 14. I / O synchronization error; 15. Buffer error; 16. Permission configuration error; 17. Malicious attack; 18. Log audit failure; 19. Format specification error; 20. Standard version conflict; 21. Incomplete format information; 22. Transmission interruption; 23. Transmission error; 24. Protocol implementation error; 25. Mapping error; 26. Unsynchronized mapping; 27. Mapping conflict; 28. Corrupted backup data; 29. ​​Recovery operation error; 30. Repeated recovery.

[0091] Preferably, analyze and design the scenario of reading files that do not exist to confirm whether the software can effectively detect and handle the following failure modes:

[0092] 1. Metadata corruption; 2. File system inconsistency; 3. Hard disk error; 4. Path error; 5. Failed to check file existence correctly; 6. Concurrent access problem; 7. Permission error; 8. File deleted by other users; 9. Shared folder synchronization problem; 10. Patch conflict; 11. Version incompatibility; 12. Configuration error; 13. Cache not refreshed; 14. I / O error; 15. Buffer error; 16. Permission configuration error; 17. Security software interception; 18. Log audit failure; 19. Path splicing error; 20. Relative path error; 21. File link error; 22. Network delay or interruption; 23. Server synchronization problem; 24. Network configuration error; 25. Mapping error; 26. Unsynchronized mapping; 27. Mapping conflict; 28. Backup data loss; 29. ​​Recovery operation error; 30. Backup strategy error.

[0093] Preferably, analyze and design for high-load communication scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0094] 1. Insufficient bandwidth; 2. Network congestion; 3. Resource exhaustion; 4. Inefficient protocol; 5. Configuration error; 6. Protocol conflict; 7. Inadequate server performance; 8. Load balancing failure; 9. Resource contention; 10. I / O bottleneck; 11. Cache overflow; 12. I / O blocking; 13. Poor concurrency control; 14. Inefficient data processing; 15. Memory leak; 16. Excessive security checks; 17. Firewall misconfiguration; 18. Security vulnerabilities; 19. Poor scheduling strategy; 20. Resource allocation error; 21. System lock contention; 22. Topology bottleneck; 23. Single point of failure; 24. Insufficient redundant design; 25. Equipment performance bottleneck; 26. Driver incompatibility; 27. Equipment failure; 28. DDoS attack; 29. ​​Traffic hijacking; 30. Malware.

[0095] Preferably, analyze and design the UDP header error scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0096] 1. Data format error; 2. Field value error; 3. Data truncation or overflow; 4. Protocol stack implementation error; 5. Buffer management error; 6. Resource race; 7. Hardware failure; 8. Driver error; 9. Device configuration error; 10. Data packet corruption; 11. Network delay or jitter; 12. Data packet loss; 13. Firewall misinterception; 14. Security device modification; 15. Malicious attack; 16. Checksum algorithm error; 17. Checksum field update error; 18. Data packet modification; 19. Synchronization mechanism error; 20. Race condition; 21. Resource lock contention; 22. Buffer overflow; 23. Memory leak; 24. Memory fragmentation; 25. Incomplete protocol implementation; 26. Version incompatibility; 27. Protocol extension error; 28. Virtual network interface error; 29. ​​Network virtualization layer error; 30. Resource isolation error.

[0097] Preferably, analyze and design the TCP message length error scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0098] 1. Data format error; 2. Field value error; 3. Data truncation or overflow; 4. Protocol stack implementation error; 5. Segmentation processing error; 6. Resource race; 7. Hardware failure; 8. Driver error; 9. Device configuration error; 10. Data packet corruption; 11. Network delay or jitter; 12. Data packet loss; 13. Firewall misinterception; 14. Security device modification; 15. Malicious attack; 16. Checksum algorithm error; 17. Checksum field update error; 18. Data packet modification; 19. Synchronization mechanism error; 20. Race condition; 21. Resource lock contention; 22. Buffer overflow; 23. Memory leak; 24. Memory fragmentation; 25. Incomplete protocol implementation; 26. Version incompatibility; 27. Protocol extension error; 28. Virtual network interface error; 29. ​​Network virtualization layer error; 30. Resource isolation error.

[0099] Preferably, the abnormal situation handling subcategories include: computing boundary scenarios, storage boundary scenarios and network boundary scenarios; the criterion names of computing boundary scenarios include: process creation failure, computing timeout and simultaneous disk reading and writing; the criterion names of storage boundary scenarios include: file read timeout, file write timeout, abnormal interruption of file write process and abnormal interruption of file read process; the criterion names of network boundary scenarios include: abnormal interruption of network transmission, abnormal power off of network equipment and blocked network communication link.

[0100] Preferably, analyze and design the process creation failure scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0101] 1. Insufficient memory; 2. Limitation on the number of processes; 3. Exhaustion of file descriptors; 4. Insufficient user permissions; 5. Security policy restrictions; 6. Insufficient permissions for executing files; 7. Configuration file errors; 8. Missing environment variables; 9. Incompatible system parameters; 10. Initialization code errors; 11. Missing library dependencies; 12. Code logic errors; 13. CPU overload; 14. I / O bottlenecks; 15. Slow system response; 16. Memory failure; 17. Hard disk failure; 18. CPU failure; 19. Resource isolation issues; 20. Incorrect virtual environment configuration; 21. Container restrictions; 22. No patches; 23. Incompatible patches; 24. Exploitation of security vulnerabilities.

[0102] Preferably, analyze and design the calculation timeout scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0103] 1. CPU contention; 2. Memory contention; 3. I / O contention; 4. Deadlock; 5. Livelock; 6. Low-priority tasks occupy key resources; 7. Priority inheritance mechanism failure; 8. Insufficient bandwidth; 9. Network congestion; 10. Unstable network connection; 11. Excessive algorithm complexity; 12. Infinite loop; 13. Incorrect exception handling; 14. CPU failure; 15. Memory failure; 16. Hard disk failure; 17. Excessive resource sharing; 18. Configuration error; 19. Host contention; 20. Improper scheduling strategy; 21. Scheduler overload; 22. Frequent context switching.

[0104] Preferably, analyze and design for simultaneous disk read and write scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0105] 1. Insufficient bandwidth; 2. Long I / O queues; 3. Low processing power; 4. Lock conflicts; 5. File system fragmentation; 6. Metadata contention; 7. Cache consistency issues; 8. Buffer overflows; 9. Low cache hit rate; 10. Lock contention; 11. Race conditions; 12. Excessive synchronization overhead; 13. Bad disk sectors; 14. Mechanical failure; 15. Excessive temperature; 16. Improper I / O scheduling strategy; 17. Scheduler overload; 18. Unfair scheduling; 19. Network delay; 20. Bandwidth limitation; 21. Unstable connection; 22. Resource isolation issues; 23. Virtual disk delay; 24. Host resource contention; 25. Hot plug failure; 26. Driver incompatibility; 27. Hardware failure.

[0106] Preferably, analyze and design the file read timeout scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0107] 1. Insufficient disk bandwidth; 2. Long disk I / O queue; 3. Disk aging; 4. File system fragmentation; 5. Metadata competition; 6. File system corruption; 7. Cache consistency issues; 8. Cache overflow; 9. Low cache hit rate; 10. Lock contention; 11. Race condition; 12. Excessive synchronization overhead; 13. Bad disk sectors; 14. Mechanical failure; 15. Excessive temperature; 16. Improper I / O scheduling strategy; 17. Scheduler overload; 18. Unfair scheduling; 19. Network delay; 20. Bandwidth limitation; 21. Unstable connection; 22. Resource isolation issues; 23. Virtual disk delay; 24. Host resource contention; 25. Insufficient permissions; 26. Access control list conflict; 27. Security policy restrictions; 28. Code defects; 29. ​​Improper exception handling; 30. High algorithm complexity.

[0108] Preferably, analyze and design the file write timeout scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0109] 1. Insufficient write bandwidth; 2. Too long I / O queue; 3. Disk aging; 4. File system fragmentation; 5. Metadata lock contention; 6. File system corruption; 7. Cache consistency issues; 8. Buffer overflow; 9. Low cache hit rate; 10. Lock contention; 11. Race condition; 12. Excessive synchronization overhead; 13. Bad disk sectors; 14. Mechanical failure; 15. Excessive temperature; 16. Improper I / O scheduling strategy; 17. Scheduler overload; 18. Unfair scheduling; 19. Network delay; 20. Bandwidth limitation; 21. Unstable connection; 22. Resource isolation issues; 23. Virtual disk delay; 24. Host resource contention; 25. Insufficient permissions; 26. Access control list conflict; 27. Security policy restrictions; 28. Code defects; 29. ​​Improper exception handling; 30. High algorithm complexity.

[0110] Preferably, analyze and design the abnormal interruption scenario of the file writing process to confirm whether the software can effectively detect and handle the following failure modes:

[0111] 1. Disk failure; 2. Disk full; 3. Disk overload; 4. File system corruption; 5. File system mount problem; 6. File system metadata contention; 7. Cache consistency problem; 8. Buffer overflow; 9. Cache writeback failure; 10. Lock contention; 11. Race condition; 12. Excessive synchronization overhead; 13. Disk bad sectors; 14. Mechanical failure; 15. Excessive temperature; 16. Improper I / O scheduling strategy; 17. Scheduler overload; 18. Unfair scheduling; 19. Network delay; 20. Bandwidth limitation; 21. Unstable connection; 22. Resource isolation problem; 23. Virtual disk failure; 24. Host resource contention; 25. Insufficient permissions; 26. Access control list conflict; 27. Security policy restriction; 28. Code defect; 29. ​​Improper exception handling; 30. High algorithm complexity; 31. Power failure; 32. Unstable power supply.

[0112] Preferably, analyze and design the abnormal interruption scenario of the file reading process to confirm whether the software can effectively detect and handle the following failure modes:

[0113] 1. Disk failure; 2. Disk full; 3. Disk overload; 4. File system corruption; 5. File system mounting problem; 6. Metadata contention; 7. Cache consistency problem; 8. Buffer overflow; 9. Low cache hit rate; 10. Lock contention; 11. Race condition; 12. Excessive synchronization overhead; 13. Bad disk; 14. Mechanical failure; 15. Excessive temperature; 16. Improper I / O scheduling strategy; 17. Scheduler overload; 18. Unfair scheduling; 19. Network delay; 20. Bandwidth limitation; 21. Unstable connection; 22. Resource isolation problem; 23. Virtual disk failure; 24. Host resource contention; 25. Insufficient permissions; 26. Access control list conflict; 27. Security policy restrictions; 28. Code defects; 29. ​​Improper exception handling; 30. High algorithm complexity; 31. Power failure; 32. Unstable power supply.

[0114] Preferably, analyze and design the network transmission abnormal interruption scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0115] 1. Network card failure; 2. Router / switch failure; 3. Cable problem; 4. IP address conflict; 5. Incorrect subnet settings; 6. DNS configuration error; 7. Network congestion; 8. Insufficient bandwidth; 9. Improper quality of service settings; 10. Protocol mismatch; 11. Software error; 12. Firewall configuration error; 13. Malicious attack; 14. Intrusion detection false alarm; 15. Security policy restriction; 16. Driver incompatibility; 17. System update problem; 18. Resource contention; 19. Improper network isolation; 20. Virtual switch failure; 21. Resource contention; 22. Device power failure; 23. Unstable power supply.

[0116] Preferably, analyze and design the abnormal power-off scenarios of network devices to confirm whether the software can effectively detect and handle the following failure modes:

[0117] 1. Power failure; 2. Unstable power supply; 3. Power overload; 4. Poor heat dissipation; 5. Excessive ambient temperature; 6. Dust accumulation; 7. Component failure; 8. Aging problem; 9. Manufacturing defect; 10. Strong electromagnetic interference; 11. Grounding problem; 12. Incorrect power management settings; 13. Firmware vulnerability; 14. False triggering of power protection mechanism; 15. Power supply environment problem; 16. Excessive humidity; 17. Dust and dirt; 18. Failure of connected equipment; 19. Interface problem; 20. External device conflict; 21. Firmware update error; 22. Software vulnerability; 23. Configuration error; 24. Incorrect operation; 25. Configuration error; 26. Failure to follow operating specifications.

[0118] Preferably, analyze and design the network communication link blocking scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0119] 1. High traffic; 2. Burst traffic; 3. Broadcast storm; 4. Insufficient bandwidth; 5. Improper bandwidth management strategy; 6. Equipment overload; 7. Equipment failure; 8. Cache overflow; 9. Single point of failure; 10. Loop problem; 11. Incorrect routing configuration; 12. Protocol incompatibility; 13. Protocol attack; 14. Protocol implementation problem; 15. Excessive multicast traffic; 16. Excessive broadcast traffic; 17. DDoS attack; 18. Worm and virus propagation; 19. Firewall configuration problem; 20. Virtual network configuration problem; 21. Resource contention; 22. Virtual switch problem; 23. Software vulnerability; 24. Firmware error; 25. Update problem; 26. Configuration error; 27. Misoperation; 28. Unplanned maintenance.

[0120] Preferably, the illegal operation processing subcategories include: computing boundary scenarios, storage boundary scenarios, and network boundary scenarios; the criterion names of computing boundary scenarios include: continuous process creation and frequent process creation; the criterion names of storage boundary scenarios include: high concurrency and high frequency reading and writing; the criterion names of network boundary scenarios include: ultra-multi-link communication.

[0121] Preferably, analyze and design the continuous process creation scenario to confirm whether the software can effectively detect and handle the following failure modes:

[0122] 1. Memory leak; 2. Infinite loop creation; 3. Improper exception handling; 4. Virus infection; 5. Worm propagation; 6. Incorrect scheduled tasks; 7. Startup script problems; 8. Resource locking; 9. Race conditions; 10. Script errors; 11. Incorrect task scheduling; 12. Update errors; 13. Patch incompatibility; 14. Service restart strategy errors; 15. Daemon failure; 16. Misoperation; 17. Incorrect configuration.

[0123] Preferably, analyze and design for frequent process creation scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0124] 1. Memory leak; 2. Creation of infinite loops; 3. Improper exception handling; 4. Viruses or worms; 5. Malicious scripts; 6. Scheduled task configuration errors; 7. Startup script problems; 8. Resource locking and competition; 9. Deadlock; 10. Script errors; 11. Incorrect task scheduling; 12. Update errors; 13. Patch incompatibility; 14. Service restart strategy errors; 15. Daemon failure; 16. Misoperation; 17. Misconfiguration; 18. Monitoring system configuration errors; 19. Log system errors; 20. Insufficient resource quota; 21. Configuration restrictions.

[0125] Preferably, analyze and design for high-concurrency and high-frequency read and write scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0126] 1. Insufficient disk bandwidth; 2. Long seek time; 3. Too long I / O queue; 4. Metadata lock; 5. File fragmentation; 6. File system crash; 7. Insufficient memory; 8. Cache conflict; 9. Frequent page replacement; 10. Insufficient bandwidth; 11. High latency; 12. Network congestion; 13. Lock contention; 14. Deadlock; 15. Starvation; 16. Data race; 17. Transaction conflict; 18. Data loss; 19. Improper I / O scheduler; 20. File system configuration error; 21. Insufficient resource quota; 22. Disk failure; 23. Memory failure; 24. Network device failure; 25. Driver error; 26. Software bug; 27. Patch or update problem; 28. Configuration error; 29. ​​Operation error.

[0127] Preferably, analyze and design for ultra-multi-link communication scenarios to confirm whether the software can effectively detect and handle the following failure modes:

[0128] 1. Insufficient bandwidth; 2. Traffic congestion; 3. Connection limit; 4. Resource exhaustion; 5. Protocol conflict; 6. Incompatibility; 7. Improper routing; 8. Transmission error; 9. Encryption overhead; 10. Security vulnerability; 11. Network device limitation; 12. Hardware failure; 13. Improper configuration; 14. Insufficient parameter optimization; 15. Software bug; 16. Improper driver update; 17. Uneven load distribution; 18. Load balancer failure; 19. Configuration error; 20. Misoperation.

[0129] Robustness test requirements analysis phase:

[0130] Determine the usage scenario based on the operating system to be analyzed:

[0131] Preferably, the usage scenario is determined, specifically including:

[0132] Select the requirement documents, design documents and user documents of the operating system to be analyzed as relevant documents;

[0133] Decompose the steps based on relevant documents to obtain multiple complete usage scenarios containing one or more functional points.

[0134] Preferably, the relevant documents of this embodiment are: requirement specification, design description and user manual.

[0135] Preferably, the usage scenario is expressed in text form, covering aspects such as computing, storage, and network.

[0136] Determine the failure modes of the usage scenarios based on robustness test analysis criteria;

[0137] Preferably, determining the failure mode of the usage scenario specifically includes:

[0138] Based on the usage scenario and the robustness test analysis criteria, relevant analysis criteria are obtained;

[0139] Determine the failure mode of the usage scenario based on relevant analysis criteria.

[0140] Preferably, the matching process is specifically as follows:

[0141] Based on the usage scenarios and robustness test analysis criteria, the pre-trained semantic similarity model is input to obtain the usage scenario vector and the analysis criterion vector;

[0142] The cosine similarity is obtained based on the usage scenario vector and the analysis criterion vector;

[0143] The correlation analysis criterion is determined based on cosine similarity.

[0144] Preferably, this embodiment uses a deep learning-based semantic similarity model BERT (Bidirectional Encoder Representations from Transformers) to match test requirement features and quickly establish a correspondence between usage scenarios in operating system-related documents and robustness test analysis criteria; the BERT model is fully trained and encodes the contextual information of the text through a bidirectional Transformer architecture, thereby capturing the semantics of words in different contexts.

[0145] Preferably, based on the usage scenario and robustness test analysis criteria as the original text T i , perform cleaning and standardization operations to obtain the preprocessed text T' i ;

[0146] Based on the preprocessed text T' i Input to the BERT model, the BERT model generates a high-dimensional vector representation based on each input word, and then obtains the scene vector matrix Z i and the criterion vector matrix Z j ;

[0147] Based on the scene vector matrix Z i and the criterion vector matrix Z j Extract features respectively and get the usage scenario vector F i and the analysis criterion vector F j ;

[0148] Based on the usage scenario vector F i and the analysis criterion vector F j Calculate the cosine similarity cos_sim(F i ,F j ):

[0149]

[0150] The most relevant robustness test analysis criterion is determined based on cosine similarity as the relevant analysis criterion.

[0151] Preferably, analysis is performed based on relevant analysis criteria to determine potential failure modes in the usage scenario.

[0152] Process based on the fault mode to obtain fault data:

[0153] Preferably, obtaining fault data specifically includes:

[0154] Analyze the failure mode according to relevant standards to obtain all possible causes of failure;

[0155] Analyze the impact of the failure mode on the operational safety of the local and entire system, as well as the task completion and user experience of the operating system users, to obtain the failure impact;

[0156] Determine the severity of the impact based on the fault impact and obtain a severity level;

[0157] Failure data is constructed based on failure mode, failure cause, failure impact, severity level and usage scenario.

[0158] Preferably, this embodiment performs a fault cause analysis based on the relevant contents of 5.4.1.3 Fault Cause Analysis in the Failure Mode, Effects and Criticality Analysis Guide Standard Requirements to obtain all possible fault causes.

[0159] Preferably, the severity levels are divided into: Class I (serious impact), Class II (moderate impact), Class III (mild impact) and Class IV (no impact).

[0160] Preferably, each piece of fault data consists of a fault mode, a fault cause, a fault impact, a severity level, and a usage scenario, and all fault data constitute a fault database.

[0161] Preferably, taking the search function point of the operating system running a text editor scenario as an example, the robustness test requirements analysis of the operating system is performed, and the results are shown in Table 1:

[0162] Table 1 Search function FMEA analysis table

[0163]

[0164]

[0165] Robustness test case library construction phase:

[0166] Establish a mapping relationship between the fault data and the software function items of the operating system to be analyzed:

[0167] Preferably, establishing a mapping relationship specifically includes:

[0168] Software function items include: external input interface data, functional processing and external output interface data;

[0169] Establish a mapping between the fault cause and the external input interface data or the functional processing process, determine the value space and value equivalence class of the external input interface data, or determine the execution logic and constraint conditions of the functional processing process;

[0170] Establish a mapping based on the fault mode and the external output interface data, and determine the value space and value equivalence class of the external input interface data;

[0171] A mapping is established based on the severity level and the execution priority number of the robustness test cases.

[0172] Based on the mapping relationship, a test case library for robustness of the operating system is obtained;

[0173] Preferably, an operating system robustness test case library is obtained, specifically including:

[0174] Determine the content of robustness test cases based on the mapping relationship;

[0175] The initial test cases are composed based on the robustness test case content and execution priority number;

[0176] The initial test cases are sorted from high to low based on the execution priority number, and a logical operating system robustness test case library is obtained.

[0177] Preferably, the robustness test case content includes: conditions and inputs, purposes and actions, expected outputs and passing criteria.

[0178] Preferably, determining the input of the test case specifically includes:

[0179] The cause of the fault can determine the external input interface data that triggers the fault mode;

[0180] Determine the input interface data and value rules of related functions based on the cause of the fault;

[0181] Based on the value selection rules, a certain value or an equivalent value is selected as the input content of the robustness test case.

[0182] Preferably, if a "fault cause" corresponds to multiple functions, relevant execution logic should be selected for each function to jointly determine the input of the robustness test case.

[0183] Preferably, the specific process of determining the expected output of the test case is:

[0184] Combine failure modes to determine the expected output of the test case;

[0185] Determine the value selection rules of the corresponding external output interface data based on the fault database;

[0186] Based on the value selection rules, a certain value or equivalent value is selected as the expected output of the robustness test case.

[0187] Preferably, the specific process of determining the passing criteria of the test case is:

[0188] If the robustness of the system has been effectively implemented, the expected output of the robustness test case is: after the cause of the failure is imposed in the test environment, the operating system does not exhibit the corresponding failure mode or the failure mode is effectively controlled;

[0189] Determine the passing criteria for the test case based on the failure mode impact and the expected output of the test case.

[0190] Preferably, the passing criteria of the robustness test case should be completely consistent with the expected output. If the failure mode has a low impact level (slight or no impact), the passing criteria should lower the threshold.

[0191] Preferably, the higher the severity level, the higher the execution priority.

[0192] Preferably, the test cases are executed, and the execution results are recorded and analyzed. Based on the list sorted by severity, the test cases with high severity are tested first and focused.

[0193] Preferably, a robustness test example is constructed for the "incorrect search results" mentioned in Table 1 as shown in Table 2:

[0194] Table 2 Schematic diagram of search function robustness test cases

[0195]

[0196] Example 3

[0197] like Figure 3 As shown, a FMEA-based operating system robustness test case construction system includes: a criterion construction module, a usage scenario determination module, a failure mode determination module, a failure data acquisition module, a mapping relationship determination module and a test case construction module;

[0198] Criteria building module, used to build robustness test analysis criteria based on software-related standard requirements;

[0199] A usage scenario determination module, used to determine a usage scenario based on the operating system to be analyzed;

[0200] The failure mode determination module is used to determine the failure mode of the usage scenario based on the robustness test analysis criteria.

[0201] A fault data acquisition module is used to process based on the fault mode to obtain fault data;

[0202] A mapping relationship determination module, used to establish a mapping relationship between the fault data and the software function items of the operating system to be analyzed;

[0203] The test case construction module is used to obtain an operating system robustness test case library based on the mapping relationship.

[0204] Preferably, each module of this embodiment can implement the above method content correspondingly, which will not be described one by one here.

[0205] Example 4

[0206] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0207] Memory, used to store computer programs;

[0208] The processor, when used to execute the program stored in the memory, can implement a method for constructing an operating system robustness test case based on FMEA as in Example 1 or 2.

[0209] like Figure 4 As shown, the electronic device may include: a processor 41, a communication interface 42, a memory 43 and a communication bus 44, wherein the processor 41, the communication interface 42 and the memory 43 communicate with each other via the communication bus 44. The processor 41 may call the logic instructions in the memory 43 to execute a method for constructing a robustness test case of an operating system based on FMEA in Embodiment 1 or 2.

[0210] In addition, the logic instructions in the above-mentioned memory 43 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0211] Through the above technical solutions, it can be known that the present invention discloses a method and system for constructing robustness test cases of an operating system based on FMEA, which uses the relevant standard requirements in the testing field to design robustness test analysis criteria, guides testers to analyze potential failure modes in operating system related documents according to the criteria, conducts failure mode and impact analysis, and further constructs a robustness test case set, which has the following features:

[0212] Beneficial effects:

[0213] 1. The present invention can help software developers and testers fully discover the robustness problems in the operating system requirements and design stages, formulate requirements and design strategies to enhance the robustness of the operating system; enhance the stability and reliability of the system in the face of abnormal and complex failures, and ensure that the operating system can continue to run and provide services to meet the system's requirements for product quality.

[0214] 2. The present invention utilizes relevant standard requirements in the testing field to design robustness test analysis criteria, guides testers to analyze potential failure modes in operating system-related documents according to the criteria, conducts failure mode and impact analysis, and further constructs a robustness test case set; it gives full play to the advantages of the FMEA method in the field of robustness testing, greatly increases the number of faults discovered, effectively improves the quality of robustness test cases, further improves the robustness of domestic operating systems, and has strong engineering application value.

[0215] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other; for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part;

[0216] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention; various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention; therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing operating system robustness test cases based on FMEA, characterized in that: include: Construct robustness test analysis criteria based on software-related standard requirements; Determine the usage scenario based on the operating system to be analyzed; Determining a failure mode of the usage scenario based on the robustness test analysis criteria; Performing processing based on the fault mode to obtain fault data; Establishing a mapping relationship between the fault data and the software function items of the operating system to be analyzed; An operating system robustness test case library is obtained based on the mapping relationship.

2. The method for constructing operating system robustness test cases based on FMEA according to claim 1, characterized in that: Construct robustness test analysis criteria, including: The software-related standard requirements include: software security design guidelines and software testing guidelines; Obtain robustness test specifications and key points based on the software safety design guide and the software testing guide; Based on the robustness test specifications and key points, the robustness test analysis criteria are constructed by classifying them from three aspects: erroneous data processing, abnormal situation processing and illegal operation processing, combined with the functional characteristics and operation characteristics of the operating system.

3. The method for constructing operating system robustness test cases based on FMEA according to claim 1, characterized in that: Determine the usage scenarios, including: Selecting requirement documents, design documents and user documents of the operating system to be analyzed as relevant documents; Based on the relevant documents, the steps are decomposed to obtain a plurality of complete usage scenarios including one or more functional points.

4. The method for constructing operating system robustness test cases based on FMEA according to claim 1, characterized in that: Determine the failure mode of the usage scenario, specifically including: Based on the usage scenario, the robustness test analysis criteria are matched to obtain relevant analysis criteria; The failure mode of the usage scenario is determined based on the relevant analysis criteria.

5. The method for constructing operating system robustness test cases based on FMEA according to claim 4, characterized in that: The matching process is specifically as follows: Based on the usage scenario and the robustness test analysis criteria, the usage scenario vector and the analysis criterion vector are input into a pre-trained semantic similarity model to obtain a usage scenario vector and an analysis criterion vector respectively; Obtaining cosine similarity based on the usage scenario vector and the analysis criterion vector; The correlation analysis criterion is determined based on the cosine similarity.

6. The method for constructing operating system robustness test cases based on FMEA according to claim 4, characterized in that: Get fault data, including: Based on the failure mode, analyze according to relevant standards to obtain all possible causes of the failure; Analyze the impact of the failure mode on the operational safety of the local and entire system and the task completion and user experience of the operating system user to obtain the failure impact; Determine the severity of the impact based on the fault impact to obtain a severity level; The fault data is constructed based on the fault mode, the fault cause, the fault impact, the severity level and the usage scenario.

7. The method for constructing operating system robustness test cases based on FMEA according to claim 6, characterized in that: Establish a mapping relationship, including: The software function items include: external input interface data, function processing process and external output interface data; Establishing a mapping based on the fault cause and the external input interface data or the functional processing process, determining the value space and value equivalence class of the external input interface data, or determining the execution logic and constraint conditions of the functional processing process; Establishing a mapping based on the fault mode and the external output interface data, determining the value space and value equivalence class of the external input interface data; A mapping is established based on the severity level and the execution priority number of the robustness test case.

8. The method for constructing operating system robustness test cases based on FMEA according to claim 7, characterized in that: Get the operating system robustness test case library, including: Determining robustness test case content based on the mapping relationship; An initial test case is formed based on the robustness test case content and the execution priority number; The initial test cases are sorted from high to low based on the execution priority numbers to obtain a logical operating system robustness test case library.

9. The method for constructing operating system robustness test cases based on FMEA according to claim 7, characterized in that: The higher the severity level, the higher the execution priority.

10. A system for constructing operating system robustness test cases based on FMEA, applied to any one of the methods for constructing operating system robustness test cases based on FMEA as claimed in claims 1 to 9, characterized in that: include: Criteria construction module, usage scenario determination module, failure mode determination module, failure data acquisition module, mapping relationship determination module and test case construction module; The criterion building module is used to build robustness test analysis criteria based on software-related standard requirements; The usage scenario determination module is used to determine the usage scenario based on the operating system to be analyzed; The failure mode determination module is used to determine the failure mode of the usage scenario based on the robustness test analysis criterion; The fault data acquisition module is used to process based on the fault mode to obtain fault data; The mapping relationship determination module is used to establish a mapping relationship between the fault data and the software function items of the operating system to be analyzed; The test case construction module is used to obtain an operating system robustness test case library based on the mapping relationship.

Images