A method for software testability analysis, design and evaluation

Abstract

The embodiment of the application provides a software testability analysis, design and evaluation method, comprising: determining software testability characteristics of target software according to a preset software testability definition; determining application scenarios of software testability analysis, design and evaluation of the target software; determining software testability influencing factors and defect modes of the target software according to software testability characteristics and application scenarios of the target software, and software testability defect mode and influence analysis of the target software; developing software testability design according to the software testability influencing factors and defect modes of the target software, and forming software testability of the target software; and evaluating software testability of the target software according to a preset software testability measurement index system, obtaining software testability measurement results of the target software, and the software testability measurement results of the target software can be used to improve software testability of the target software.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a method for analyzing, designing, and evaluating software testability. Background Technology

[0002] As software is continuously developed, internal information is increasingly obscured, leading to biases in testers' understanding of the information and making later verification increasingly difficult. Software testability is an attribute of software that facilitates the application of testing methods, the discovery of existing software errors, and the faster correction of errors. Low levels of software testability result in poor control over the cost, time, and personnel allocation of software testing work, leading to low testing efficiency and impacting the overall quality of the software engineering process. Research on software testability can, to some extent, solve some of the problems faced in software testing. For example, it can help developers design easily testable software, thereby reducing testing difficulty and time. It can also help testers understand the varying degrees of difficulty in detecting errors in different parts of the software, and assist in allocating testing resources, thus improving the adequacy and reliability of software testing.

[0003] There is currently no unified understanding of the definition of software testability. There are still differences in the form of expression, content, and scope of application. There are also no mature methods for related analysis, design, and evaluation techniques. Summary of the Invention

[0004] This application addresses the defects and shortcomings of existing technologies by proposing a method for software testability analysis, design, and evaluation.

[0005] This application provides a method for software testability analysis, design, and evaluation, characterized in that the method includes:

[0006] Based on the pre-defined definition of software testability, determine the software testability characteristics of the target software;

[0007] Determine the application scenarios for software testability analysis, design, and evaluation of the target software;

[0008] Based on the software testability characteristics of the target software and the application scenario, the software testability defect patterns and impacts of the target software are analyzed to determine the influencing factors and defect patterns of the target software's software testability.

[0009] Based on the factors affecting the testability of the target software and the defect patterns, software testability design is carried out to form the testability of the target software;

[0010] Based on a pre-defined software testability metric system, the software testability of the target software is evaluated to obtain the software testability metric results of the target software. These results can be used to improve the design of the software testability of the target software.

[0011] In some embodiments, the preset software testability definition includes: observability, controllability, decomposability, simplicity, change stability, operability, understandability, test support, environmental adaptability, program traceability, and sensitivity.

[0012] In some embodiments, the application scenarios for determining the software testability analysis, design, and evaluation of the target software include:

[0013] For software development institutions, it predicts the verifiability of requirements, ensures that good requirements are put forward in the demonstration phase, assesses the testability risks of software, guides the testability design of software, and ensures the quality of software development.

[0014] For software testing and evaluation organizations, by measuring and providing software testability, the difficulty of software testing and verification can be reduced, and test case design and test resource allocation can be guided, while also weighting software quality.

[0015] For user use and maintenance, improving software testability can promptly identify internal states and defects, handle fault states in a timely manner during use, and facilitate later maintenance.

[0016] In some embodiments, the step of analyzing the software testability defect patterns and impacts of the target software based on the software testability characteristics of the target software and the application scenario, and determining the software testability influencing factors and defect patterns of the target software, includes:

[0017] Research and collection of software testability defect patterns;

[0018] Software testability defect patterns and their impact analysis.

[0019] In some embodiments, the software testability defect pattern and impact analysis includes:

[0020] Risk analysis of software testability itself;

[0021] Risk analysis for software requirements verification;

[0022] Risk analysis of software usage quality.

[0023] In some embodiments, the preset software testability metric system is constructed using the following method:

[0024] Determine the relationship between testability characteristics and testability manifestations;

[0025] Determine the relationship between factors affecting testability and testability characteristics.

[0026] In some embodiments, the evaluation of the software testability of the target software according to a preset software testability metric system includes:

[0027] Determine the measurement method for software testability metrics;

[0028] Software testability evaluation based on a fuzzy evaluation hierarchical model;

[0029] Determine the software testability level;

[0030] Determine the results of software testability metrics.

[0031] This application comprehensively defines software testability and its application scenarios. It proposes sub-characteristics of software testability, including understandability, controllability, observability, test support capability, simplicity, decomposability, adaptability, traceability, and sensitivity. It also identifies performance characteristics such as test case set size, test case generation difficulty, test environment setup difficulty, test control difficulty, test result collection difficulty, and test result interpretation difficulty. Furthermore, it constructs software testability defect patterns and conducts impact analysis, proposes a cause-effect graph-based analysis of software testability influencing factors, and establishes software testability design principles and a general evaluation index system for software testability measurement. It evaluates testability based on a fuzzy hierarchical model, defines software testability levels, and proposes methods for applying testability results and control improvement methods. This application represents a complete methodology for software testability analysis, design, and evaluation. Attached Figure Description

[0032] The accompanying drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0033] Figure 1 An overview of the definition of software testability;

[0034] Figure 2 An overview of scenarios for measuring software testability;

[0035] Figure 3 A schematic diagram illustrating the process of obtaining software testability patterns;

[0036] Figure 4 A schematic diagram illustrating the relationship between risk analysis methods for software testability issues or defects;

[0037] Figure 5 A process for developers to use testability risk assessment methodologies;

[0038] Figure 6 A flowchart illustrating the operational process for testers to conduct software testability risk assessments.

[0039] Figure 7 This is a diagram illustrating the factors affecting the size of the use case set.

[0040] Figure 8 Generate a diagram illustrating the difficulty factors for the use cases;

[0041] Figure 9 A diagram illustrating the difficulty of setting up a test environment;

[0042] Figure 10 A diagram illustrating the difficulty level applied to the test input;

[0043] Figure 11 This is a diagram illustrating the difficulty of controlling the test.

[0044] Figure 12 A diagram illustrating the difficulty of collecting test results;

[0045] Figure 13 A diagram illustrating the difficulty of the test results;

[0046] Figure 14 A schematic diagram of the process for improving software testability risk assessment and control.

[0047] Figure 15 A diagram illustrating the software testability evaluation index system;

[0048] Figure 16 A schematic diagram illustrating the technical approaches to researching software testability prediction and evaluation methods;

[0049] Figure 17 This is a schematic diagram of the factor weight calculation process based on AHP;

[0050] Figure 18 Types of program assertions;

[0051] Figure 19 Example diagram for designing an interface-oriented test structure;

[0052] Figure 20 Example diagram of the EFSM model for the task. Detailed Implementation

[0053] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0054] The embodiments of this application include the following specific steps.

[0055] The first step is to define software testability and analyze its characteristics.

[0056] Clarify the definition and connotation of software testability. Software testability is defined as the ability of software to be easily tested and its defects exposed.

[0057] In the embodiments of this application, such as Figure 1 As shown, software testability is defined as the ability of software to be easily tested and its defects exposed. The scope of software in this application embodiment may include, for example: software requirements, design, or implementation; software programs or documentation; statements, functions, modules, and components—products at all design levels of software are included within this scope; system software, application software, embedded software, etc.

[0058] Software testability is a design characteristic of software. Once software is developed, it possesses a certain degree of testability, theoretically independent of external conditions. To reflect this characteristic, this application describes software testability as "the ability of software to be easily tested and to expose defects," indicating that software testability is a capability of software and an internal attribute of software.

[0059] This application proposes that software testability includes the software's ability to be easily tested. This ability can be determined by the software's requirements and design, and can influence the software testing process. The higher the software's testability, the easier the testing is, and the lower the time or cost required for testing; conversely, the lower the software's testability, the more difficult the testing is, and the higher the time or cost required for testing.

[0060] In this application, the ability of software to readily expose defects includes the ability of software to manifest inherent defects as external failures, and is redescribed as the ability of software to readily expose defects. This ability can be determined by the software's implementation, influenced by the software's programming language and internal structure, and can affect the software's test results.

[0061] Analyze the testability characteristics of the software.

[0062] In this application, software testability may include the following sub-characteristics: observability, controllability, decomposability, simplicity, change stability, operability, understandability, test support, environmental adaptability, program traceability, and sensitivity.

[0063] Observability: The ability to easily observe external output and monitor internal state. Implementation: Each input has a unique output; state and variables are visible or queryable during runtime; past system states and variables are visible or queryable during runtime (e.g., transaction logs); all factors affecting output are visible; erroneous outputs are easily identified; internal errors are automatically detected through self-testing mechanisms; internal errors are automatically reported; source code is available.

[0064] Controllability: The ability of testers to directly control the various outputs, states, and variables that are controlled from the inputs of a software product. All possible outputs arise from a certain combination of inputs; all code can be executed through a certain combination of inputs; the state and variables of software and hardware can be directly controlled; input and output formats are consistent and structured; tests can be easily described, automated, and regenerated; interfaces and modules are easy to control; business processes and scenarios are easy to control.

[0065] Decomposability: Software modules and test scope are easy to decompose; the software system consists of independent modules; each software module can be tested independently; business processes and scenarios are easy to decompose.

[0066] Simplicity: Software should be as simple as possible while meeting the requirements, including simplicity of function, simplicity of result, and simplicity of code.

[0067] Change stability: This refers to software that does not require frequent changes, and whose changes are controlled; the fewer the changes, the less disruptive they are to testing. Software changes are infrequent; software changes are controllable; software changes do not affect existing tests; and software failures can be well recovered and isolated.

[0068] Operability: refers to the ability of software to be operated and used in a given configuration and environment. The system has few errors; there are no errors that prevent test execution; the product allows for simultaneous development and testing.

[0069] Easy to understand: The degree to which software information is easy to understand. The design can be well understood and follows industry standards; dependencies between internal, external, and shared components can be well understood; design changes are communicated; technical documentation is readily available; technical documentation is well-organized, clear, detailed, accurate, and stable; relevant environment configuration instructions and operating guides are provided.

[0070] Test support: This refers to the software's ability to support testing tools. The ease with which the software supports testing tools will greatly impact the testing process.

[0071] Environmental adaptability: This refers to the software's ability to adapt to various usage environments. On one hand, it considers the software's requirements for the hardware and software environment during runtime. Lower requirements indicate less dependence on the external environment, making it easier to set up a suitable test environment. On the other hand, it considers the software's flexibility, observing whether it can automatically change its configuration based on the environment or be manually modified through simple operations, such as automatically adjusting the window size according to the monitor resolution.

[0072] Program traceability refers to the ability of software to track its own functional operations, attributes, and behaviors. During program debugging, it may be necessary to track code execution, observe variable values, and monitor changes in memory and the stack to determine if the code is correct. The same applies to testing; sometimes it's also necessary to track the software's operations, attributes, and behaviors through continuous observation to determine if defects exist.

[0073] Sensitivity: This refers to the ability of software to easily expose its own defects. Human errors during development may leave defects in the program. These defects are not actively exposed; they can only be discovered when the code containing the hidden defects is executed, causing certain state errors, and the software propagates these error states to the outside world. Sensitivity describes the ability of software to transform defects into error states and then propagate these error states to the outside world. Software with high sensitivity is more likely to have its defects discovered during testing.

[0074] Software testability can be divided into two parts: the software's ease of testing and the software's ability to expose defects. The software's ability to expose defects can be represented by sensitivity, while other sub-characteristics can be used to represent its ease of testing.

[0075] The second step is to clarify the application scenarios for software testability analysis, design, and evaluation.

[0076] like Figure 2 As shown, the application scenarios in this application embodiment include:

[0077] First, it targets software R&D institutions, predicts the verifiability of requirements, ensures that good requirements are put forward during the demonstration phase, assesses the testability risks of software, guides the testability design of software, and ensures the development of good software.

[0078] Secondly, it is geared towards software testing and evaluation organizations. By measuring and providing software testability, it can reduce the difficulty of software testing and verification, guide test case design and allocate test resources, and weigh software quality. Software testability is an important factor in quality evaluation.

[0079] Third, it is user-oriented and maintenance-focused, improving software testability to promptly identify internal states and defects, handle fault states in a timely manner during use, and facilitate later maintenance. Here, the user could be, for example, a military unit.

[0080] The third step is to analyze the software testability defect patterns and their impacts.

[0081] Specifically, the third step includes the following steps:

[0082] Step 301: Research and collection of software testability defect patterns.

[0083] Step 301 is a preliminary step in software generation, which involves summarizing and categorizing historical defect information to obtain historical testability defect information corresponding to the target software.

[0084] The process involves reviewing literature and searching the internet, distributing questionnaires to testers, analyzing the testing obstacles encountered, and examining the external manifestations of the problems to determine which testability characteristics they affect. Based on the code (documentation) environment, the causes of the problems are analyzed, such as special code statements, complex structures, incorrect comments, unique unit structures, unclear relationships between code and documents, etc. Solutions are then proposed to address these causes, forming a software testability defect pattern. The specific process is as follows: Figure 3 As shown.

[0085] Based on the four phases of requirements analysis, preliminary design, detailed design, software coding, and software testing, a software testability defect pattern library is established, as shown in Table 1 below.

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] Table 1 Software Testability Defect Patterns

[0093] Step 302, Software testability defect mode and impact analysis.

[0094] SFMEA (System Failure Mode and Effects Analysis) is based on failure modes and focuses on the impact or consequences of software failures on the system or software. It analyzes the system or software architecture, performs causal reasoning and induction to identify weaknesses in software design, and proposes improvement measures and suggestions to ensure software quality. This paper draws on the SFMEA analysis process to conduct software testability defect mode and effect analysis.

[0095] like Figure 4 As shown, during the software requirements analysis, software design, and software coding phases, software developers select available testability problem patterns based on risk analysis methods for software testability issues or defects to write documents and code. Before software unit testing and component testing, software testers use risk analysis methods for software testability issues or defects while writing the software test plan.

[0096] Software developers should use Style I to fill in the form, as shown in Table 2 below. Software testers should use Style II to fill in the form, as shown in Table 3 below. The tables record the specific process and other elements used in this method.

[0097] Software Name: _________ Worksheet Responsible Person: _________

[0098] Software version _________ Date of completion _________

[0099]

[0100] Table 2 Risk Assessment Worksheet for Developers

[0101] Software version prior to risk assessment: ________ Worksheet responsible person: _________

[0102] Software version after risk assessment: _______ Date of completion: _______________

[0103] serial number The risk assessment worksheets are numbered II in sequence. Stage at which Describe the current stage of the software process (usually before testing, occurring simultaneously with test plan writing). Test level Explain the level of the test Testing phase Explain the current testing phase. Testing tools Explain the testing tools used. Problem Description Explanation of the test obstacles found based on the pattern Code location Indicate the location of the code where the problem occurred. According to the pattern Explanation of the underlying pattern Recommended solution Explanation of the suggested solutions described in the pattern Processing results Please describe the actual processing results and whether the code was modified. Impact Analysis Explain the impact of the treatment on the software and testing.

[0104] Table 3 Software Testability Risk Assessment Worksheet II

[0105] The process of using the developer testability risk assessment method is as follows: Figure 5 As shown, it includes:

[0106] (1) Prepare the software task specification and software testability pattern library.

[0107] (2) Select software testability patterns from the pattern library that can be used in the software development phase.

[0108] (3) Based on the selected pattern and the software task book, the pattern suggestions are designed into the software according to the software process.

[0109] (4) Annotate the documents and comment the code that involve software testability patterns during the design process, and fill these records into the worksheet.

[0110] (5) After the software is completed, submit all documents (including Risk Assessment Worksheet I) and code to the testers.

[0111] The operational procedures for testers to conduct software testability risk assessments are as follows: Figure 6 As shown, it includes:

[0112] (1) Prepare the software code and documentation.

[0113] (2) Determine the test environment: test level (unit test, integration test and system test), test phase and test tools. Fill in these three items in Risk Assessment Worksheet II.

[0114] (3) Select a pattern from the pattern library that can be used in this software phase.

[0115] (4) Read the relevant description of the selected pattern, use the principle of pattern matching, use tools or manual methods to search for matching problems in the code, record the location of the problem, analyze the testability characteristics of the software according to the pattern description and the actual situation, and record the problem, the location of the problem and the testability characteristics in the risk assessment worksheet II.

[0116] (5) Read the problematic code and the code before and after it, read the comments and related documents, check the solutions described in the pattern, and determine whether to use the solutions to modify the code based on the actual situation. If you modify the code, note the specific code before and after the modification, and analyze and explain the impact. If you do not modify the code, note the reason for not modifying it. Record these in Risk Assessment Worksheet I.

[0117] (6) After the inspection is completed, the software code is searched again until all problems in this software stage are found and resolved.

[0118] (7) Worksheet II is shown in the table below. After completion, submit it to the actual tester.

[0119] The impact of software testability is analyzed from the following three aspects:

[0120] (1) Risks to software testability itself: This risk mainly concerns the impact of software testability itself, and assesses the overall impact on metrics. Analyze the impact of software testability problems or defects on corresponding metrics, assess the overall impact on software testability based on the software testability measurement model, and classify the degree of impact.

[0121] (2) Risks related to software requirements verification: Based on the degree of change in software testability and the evolution of software testability issues, the impact on software verification is measured. The risks are mainly the opposite of the external manifestations of software testability, such as a large test case set, difficulty in generating test cases, difficulty in setting up the environment, difficulty in applying input, difficulty in collecting output data, difficulty in interpreting results, and failure to discover defects in a timely manner. Users design and optimize software testability based on these risks. Since optimization is involved, it is necessary to analyze the impact of this design and optimization method on the software: 1) What impact does it have on software functionality? 2) What impact does it have on which testability feature of the software? 3) Does it affect other testability features? 4) What impact does it have on the testing work to be carried out? 5) Are the test results valid? 6) If, due to practical limitations, it is impossible (unable) to optimize the software (modify the code)?

[0122] (3) Risks to software quality: In software systems with poor testability, internal information such as internal status, defects, internal exception handling, and internal operation results are concealed, making it impossible to detect fault status in a timely manner, resulting in reliability and security risks during use. Considering the impact of software testability and software testing, analyze the problems that may be caused during use, such as hidden defects leading to related usage problems, problems that are difficult to reproduce causing low stability, and certain information not being exposed causing misoperation, etc.

[0123] In this embodiment of the application, a cause-effect graph-based analysis of factors affecting software testability is presented.

[0124] 1. Define the characteristics of software testability.

[0125] Software testability characteristics refer to the testing obstacles faced by software during testing, which are mainly divided into seven categories: test case set size, test case generation difficulty, test environment setup difficulty, test input application difficulty, test control difficulty, test result collection difficulty, and test result interpretation difficulty.

[0126] 2. Based on the testability characteristics of the software, analyze the factors affecting software testability. Targeting the testability characteristics, identify the factors affecting software testability from aspects such as software code, software architecture, software functional features, and software documentation through a progressively refined approach, and use a cause-effect graph to describe the relationships between the influencing factors.

[0127] (1) Size of the use case set

[0128] like Figure 7 As shown, the factors influencing the size of the use case set include software sensitivity, software requirements, software design, software size, and software complexity. Software sensitivity is related to the programming language, the specific structure of the program, and the implementation of each statement (operators, operands, and their combinations), and its influence is highly complex. Software requirements include two sub-factors: Function Point Index (FPI) and Number of Non-Functional Requirements (NNFR). Software design includes four sub-factors: Number of Software Units (NSU), Unit Coupling (CBU), Fan-in Number (NFs), and Fan-out Number (NFO). Software size includes three sub-factors: Thousand Lines of Code (KLOC), Decision-to-Decision Paths (CDD-patches), and Definition-Use Path (du-path). Software complexity includes three sub-factors: Concurrency (CN), Cyclic Complexity (CC), and Synchronization Factors. Synchronization Factors can be further subdivided into Synchronization Points (SN) and Communication Relationships, with Communication Relationships further subdivided into Communication Number (CoN) and Communication Complexity (CoC).

[0129] (2) Difficulty in generating test cases

[0130] like Figure 8As shown, the factors influencing the generation of embedded software test cases include four aspects: testing tools, software description, testers, and software implementation. Testing tools and testers are not considered software attributes and therefore will not be discussed in detail. Software description is further subdivided based on certainty, completeness, detail, consistency, verifiability, and complexity, including Average Requirements Certainty (ARD), Average Unit Certainty (AUD), Average Requirements Completeness (ARI), Average Unit Completeness (AUI), Average External Interface Detail (ADIDL), Requirements Consistency (COR), Design Consistency (COD), Requirements / Design Consistency (CORAD), Design / Implementation Consistency (CODAI), Feasible Requirements Ratio (FRR), Predictable Requirements Ratio (ERR), and Quantitative Requirements Ratio (QRR). The study examines and discusses software implementation from the perspectives of modules, decision-making, and concurrency, including decision probabilistic complexity (DPC), decision input references (DIR), decision nesting (DN), input / output ratio (DRR), fan-in (NFs), fan-out (NFO), software sensitivity, concurrency (CN), synchronization points (SN), number of communications (CoN), shared variable dependency set size (LOSVDS), and shared variable accessibility (AN).

[0131] (3) Difficulty in setting up the test environment

[0132] like Figure 9 As shown, the factors affecting test environment setup are categorized into four types: supporting environment, testers, interface size, and interface complexity. Supporting environment and testers are not considered software attributes and therefore were not further subdivided. The more interfaces a software has and the more external units or systems it interacts with, the more complex the test environment becomes. Therefore, the number of external inputs (NEI), external outputs (NEO), fan-in (NFs), fan-out (NFO), unit coupling (CBU), and number of interconnected systems (NCS) are all identified as software factors affecting test environment setup. Similarly, parameters that characterize interface complexity, such as the size of the external input dependency set (LOEIDS), unit real-time performance (URT), standard input ratio (SIR), standard output ratio (SOR), and number of interface types (NIT), also affect the difficulty of test environment setup.

[0133] (4) Testing the difficulty of input application

[0134] like Figure 10As shown, factors such as input size, input complexity, self-testing capability, test execution tools, and testers can all affect the difficulty of applying test inputs during software testing. Removing the factors of test execution tools and testers, input size can be further refined into more specific factors such as the number of external inputs (NEI), fan-in numbers (NFs), decision-to-decision paths (DD-path), and definition-use paths (du-path). Input complexity includes factors such as concurrency (CN), shared variable accessibility (AN), size of shared variable dependency sets (LOSVDS), synchronization points (SN), number of communications (CoN), communication complexity (CoC), unit real-time performance (URT), size of external input dependency sets (LOEIDS), and minimum input cycle time (IPmin). Self-testing capability can be represented by parameters such as reset capability (RC), working mode control capability (MCC), intermediate state control capability (ISCC), and standard input ratio (SIR).

[0135] (5) Test control difficulty

[0136] like Figure 11 As shown, several factors influencing test control include software design, software implementation, operational mode control capability, reset capability, and intermediate state control capability. Software design includes two sub-elements: fan-in and fan-out. Software implementation includes sub-elements such as input / output ratio, shared variable dependency set, and interface type. Operational mode control capability should allow switching between operational and test modes, and be able to control any individual business process. Reset capability should allow resetting to the initial state at any time. Intermediate state observation capability: intermediate variables should be as observable as possible; the more errors hindering execution, the lower the operability, and the more difficult the test control becomes. The number of software changes, change control degree, and change impact degree affect change stability; the worse the change stability, the greater the difficulty of test control.

[0137] (6) Difficulty in collecting test results

[0138] like Figure 12 As shown, corresponding to the difficulty of applying test input, the factors affecting the difficulty of collecting test results fall into five categories: output size, output complexity, self-testing capability, test collection tools, and testers. Similarly, disregarding the factors of test tools and testers, output size can be represented by smaller factors such as fan-out number (NFO), external output number (NEO), decision-to-decision path (DD-path), definition-use path (du-path), and software sensitivity. Output complexity includes unit real-time performance (URT) and minimum output cycle time (OPmin). Self-testing capability, in addition to working mode control capability (MCC), also includes intermediate state observation capability (ISOC) and standard output ratio (SOR).

[0139] (7) Difficulty in interpreting test results

[0140] like Figure 13 As shown, in addition to non-software factors such as data analysis tools and testers, the software itself contains many factors that affect process tracking, test results, and expected results, which will affect the difficulty of interpreting the software's test results. Intermediate State Observation Capability (ISOC), Intermediate Result Assertion Ratio (IRAR), Precondition Assertion Ratio (CARpre), and Postcondition Assertion Ratio (CARpos) can be used to represent the process traceability capability of software; External Output Count (NEO), External Output Intuition (EOI), Fan-Out Intuition (FOI), Output Document Standardization (OFS), Decision-to-Decision Path (DD-path), and Definition-Use Path (du-path) can represent the scale and complexity of software output, thus reflecting the complexity of test results; Software Sensitivity indicates the ease or difficulty of test result output; Average Requirement Completeness (ARI), Predictable Requirement Ratio (ERR), Average Unit Completeness (AUI), Predictable Unit Ratio (EUR), and Average External Interface Detail (AEIDL), which characterize the completeness and detail of requirements and design, reflect the ease or difficulty of obtaining expected results from software, and therefore will also affect the interpretation of test results.

[0141] The fourth step is to conduct software testability design based on the factors affecting testability and defect patterns, combined with the characteristics of the software.

[0142] Software testability risk assessment and control improvement methods and processes are as follows: Figure 14 As shown in Table 4, software testability design criteria are formulated based on testability influencing factors and defect patterns. These criteria cover aspects such as requirement verifiability design, software operability design, software controllability design, software decomposability design, change stability design, understandability design, observability design, simplicity design, test support design, testability design patterns, sensitivity design, and code testability design.

[0143] The design can employ source code-oriented assertion design, interface-oriented test structure design, and state-oriented software probe design. For details, please refer to step ten.

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152] Table 4 Software Testability Design Criteria

[0153] The fifth step is to construct a general evaluation index system for measuring software testability.

[0154] 1. Determine the relationship between testability characteristics and testability manifestations.

[0155] Testability characteristics represent different aspects of software testability. Therefore, changes in testability characteristics will inevitably affect the overall testability of the software, leading to changes in certain testability manifestations. On the other hand, the quality of testability characteristics can also be judged through testability manifestations. For example, the difficulty in setting up a test environment for software may be due to poor software understandability, or it may be due to poor software controllability or observability, resulting in excessively high requirements for the test environment. Therefore, the relationship between testability manifestations and testability characteristics can also be analyzed. Table 5 below shows all possible relationships between testability characteristics and testability manifestations obtained through analysis from different perspectives, where "√" indicates a relationship between testability characteristics and testability manifestations.

[0156]

[0157] Table 5 Relationship between testability characteristics and testability manifestations

[0158] 2. Determine the relationship between factors affecting testability and testability characteristics.

[0159] By examining the relationships between influencing factors and testability performance, and between testability characteristics and testability performance, we can initially establish the relationship between influencing factors and testability characteristics. Taking FPI in Table 6 as an example, it is related to the size of the test case set. By consulting Table 6, we can see that the testability characteristics related to the size of the test case set are simplicity, decomposability, and sensitivity. Therefore, we can conclude that FPI may be related to simplicity, decomposability, and sensitivity.

[0160] This application uses the hierarchical structure principle of the Analytic Hierarchy Process (AHP) to classify the sub-characteristics and influencing factors of software testability. Combined with specific measurements of these influencing factors, a general evaluation index system for software testability is formed, specifically comprising testability objectives, sub-characteristic layers, and measurement layers. The software testability index system is as follows: Figure 15 As shown.

[0161] Each row in Table 6 represents a set of software testability characteristics that a testability influencing factor may affect. Whether the influencing factor affects the elements in this set and the manner of influence need further determination. Taking FPI as an example, further analysis of its relationship with simplicity, decomposability, and sensitivity reveals that: the higher the FPI, the more complex the software and the lower its simplicity; FPI has a negative impact on simplicity. When other factors remain unchanged, FPI has virtually no impact on the software's decomposability and sensitivity. Table 11 lists the impact of software testability influencing factors on each software testability characteristic, where "+" indicates a positive impact and "-" indicates a negative impact.

[0162]

[0163]

[0164] Table 6 shows the relationship between influencing factors and testability characteristics. Step 6: Clarify the measurement method for software testability indicators.

[0165] The specifics are shown in Table 7.

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Table 1. Measurement Methods for Software Testability Indicators

[0172] Step 7: Software testability evaluation based on a fuzzy evaluation hierarchical model.

[0173] Software testability prediction and evaluation methods primarily rely on collecting and analyzing software testability data through software testing and experimentation. Then, combined with a software testability index system, AHP (Abstract Hierarchy Process) is used to determine weights, followed by hierarchical fuzzy evaluation. Specific schemes are as follows: Figure 16 As shown.

[0174] 1. Data collection required for software testability measurement.

[0175] This section clarifies the relationship between software testing and experimentation data and software testability metrics. It outlines the collection and analysis of this data, specifying the types and stages of data collection, as well as the processing methods for qualitative and quantitative data, and considering testability from the perspective of testers. While software testing and experimentation provide access to relevant documentation from earlier stages, software testability metrics are derived from data collected throughout the entire lifecycle. This includes information such as system structure and software functional structure during the validation phase, specific requirements from requirements analysis, architecture and interface design from the design phase, data flow and control flow information from coding, test case execution information from testing, and fault information from experimentation. In essence, software testability data collection and analysis uses the software testing and experimentation phases as a link to guide the collection and analysis of data throughout the entire lifecycle.

[0176] 2. Calculation of single influencing factor index for software testability.

[0177] A software testability evaluation set categorizes the degree of software testability achieved. This paper divides these categorized metrics and elements into four levels, establishing the following software testability evaluation set:

[0178] CL={CL1(poor),CL2(pass),CL3(Good),CL4(Excellent)}

[0179] The quantitative standards for software testability evaluation index levels are shown in Table 8 below:

[0180] grade Poor Pass Good Excellent Indicator Score 0-59 60-79 80-89 90-100

[0181] Table 8. Quantitative Standards for Software Testability Evaluation Levels

[0182]

[0183]

[0184]

[0185]

[0186] In the formula, d1, d2, d3, and d4 are the center values ​​of the intervals (c1, c2), (c2, c3), (c3, c4), and (c4, c5), respectively.

[0187] P i Corresponding to U i The evaluation results, P ij Corresponding to U ij The evaluation results can be obtained as follows:

[0188]

[0189] W i =(w i1 w i2 ... w imt )

[0190]

[0191] Where m is the number of secondary indicators contained in the indicator set Ui, and CF is the quantified evaluation set CL.

[0192] 3. Preprocessing for single-factor measurement of software testability.

[0193] The original data with heterogeneous identification frameworks or evaluation functions needs to be transformed into evidence with a unified identification framework, and the original measurement data of quantitative or qualitative metrics need to be transformed into unified values ​​to solve the problem of information fusion of multiple types and dimensions in the software evaluation process. Discrete quantitative data is used to represent software testability, and a ranking method is used to measure the degree of software testability.

[0194]

[0195] n≥1, m≥1, s ij ∈{-1, 0, 1}

[0196] (1≤i≤m,1≤j≤n)

[0197] This indicates the corresponding indicator u ij It belongs to the category of indicators where the larger the better.

[0198] This indicates the corresponding indicator u ij It belongs to the category of indicators where smaller is better.

[0199] This indicates the corresponding indicator u ij It belongs to the category of indicators that are better the closer they are to the middle level.

[0200] The preprocessing algorithm for the measurement results is as follows:

[0201] Algorithm IPA

[0202] Input: matrices P and S

[0203] Output: Matrix P'

[0204] Step 1: Initialize matrix P.

[0205] Step 2 for If s ij =1, calculate

[0206]

[0207] Step 3 for If s ij =-1, calculate

[0208]

[0209] Step 4 for If s ij =0, calculate

[0210]

[0211] Step 5: Output P'.

[0212] 4. Factor weight calculation based on AHP.

[0213] The calculation steps are as follows: Figure 17 As shown.

[0214] Construct a comparison matrix A. Based on the comparison scale, as shown in Table 9 below, compare each attribute of the analysis object pairwise to represent the importance of each criterion C1, C2, ... Cn to target 0.

[0215] Factor comparison Quantized value Equally important 1 Slightly important 3 Obviously important 5 Strongly important 7 Extremely important 9 The median value between two adjacent judgments 2，4，6，8

[0216] Table 9 Comparison Table of Judgment Matrix

[0217] a ij Table 1 lists the nine importance levels and their assigned values ​​given by Saaty, representing the importance comparison results between elements Ci and Cj. The matrix formed by the pairwise comparison results is called the judgment matrix. The judgment matrix has the following properties:

[0218]

[0219] (3) Calculate the weighted phasor

[0220] The weights of each influencing factor are determined using the weighted arithmetic mean method, and the steps are as follows.

[0221] The weights of each valid judgment matrix are calculated using the square root method, and the product M of the elements in each row of the judgment matrix is ​​calculated. i The calculation formula is as follows:

[0222]

[0223] Calculate M i The nth power of the sum:

[0224] Normalize the vector:

[0225] Wi The desired eigenvectors, which are the corresponding weight coefficients.

[0226] (4) Consistency check

[0227] λ max Let be the largest eigenvalue of matrix A. Let CI be the consistency index. CI = 0 indicates complete consistency; CI close to 0 indicates satisfactory consistency; the larger CI is, the more serious the inconsistency.

[0228]

[0229] To measure the magnitude of CI, the random consistency metric RI is introduced:

[0230]

[0231] Among them, the random consistency index RI is related to the order of the judgment matrix. Generally, the larger the order of the matrix, the greater the possibility of random deviation from consistency. The corresponding relationship is shown in Table 10 below.

[0232] Matrix order 1 2 3 4 5 6 7 8 9 10 RI 0 0 0.58 0.90 1.12 1.24 1.32 1.41 1.45 1.49

[0233] Table 2 Standard values ​​of the average random consistency index RI

[0234] Different standards result in slight differences in the RI value.

[0235] Considering that deviations in consistency may be caused by random factors, when verifying whether the judgment matrix has satisfactory consistency, it is also necessary to compare CI with the random consistency index RI to obtain the test coefficient CR, as shown in the following formula:

[0236]

[0237] Generally, if CR < 0.1, the judgment matrix is ​​considered to have passed the consistency test; otherwise, it does not have satisfactory consistency.

[0238] 5. Evaluation of software testability features.

[0239] Evaluation Matrix

[0240]

[0241] n≥1, m≥1, r ij ∈[0,1]

[0242] (1≤i≤m,1≤j≤nn)

[0243] R = (r i1 r i2 …rin This describes the evaluation set of single influencing factors for the i-th secondary indicator under a certain primary indicator in the software testability evaluation index.

[0244] j∈[1,n], rij∈[0,1] describes the evaluation result of a single influencing factor obtained by calculating the i-th secondary indicator under a certain primary indicator in the software testability evaluation index using the single-factor influence evaluation formula in Definition 3.

[0245] SFCEST (Single Fuzzy Comprehensive Evaluation of Software Testability) u i It is the evaluation result of the i-th input indicator and ui∈[0,1], U` describes a first-level indicator (U`∈U) in the defined software testability indicator set, R i ` represents the evaluation set of a single influencing factor obtained using the membership function, W is the corresponding weight matrix, and CF is the quantified evaluation set CL.

[0246]

[0247] 6. Fuzzy comprehensive evaluation of software testability.

[0248] For each software testability metric, it is represented by a quintuple, i.e., CT = (N, V, W, M, FN), where N represents the unique identifier of this metric, V is the evaluation value of this metric, W represents the weight value of this metric, M is the level of this metric, and FN is the identifier of the parent metric containing this metric. k represents the level number of the metric set minus 1, CT = {ct1, ct2, ..., ctN} n}, where n is the total number of indicators in the indicator set. The algorithm calculation process is as follows:

[0249]

[0250] Step 8: Define the software testability level.

[0251] Based on the software's testability evaluation value, the level of internal information masking is determined by comprehensively assessing testability attributes and the software's overall testability. In the grading model, the grading value range is not equally divided. The value ranges increasing from the lowest level to the next trust level approximate a decreasing proportion according to the golden ratio.

[0252]

[0253]

[0254]

[0255]

[0256] Based on a decreasing interval division method approximating the golden ratio, a software internal information concealment level model is proposed, as shown in Table 11. The evaluation of software testability level requires the software's testability value T and attribute value yi to meet the software value requirements in the table. Five different trust levels are represented by I, II, III, IV, and V, with level I being the lowest, increasing sequentially, and level V being the highest.

[0257]

[0258] Table 11 Definition of Software Internal Information Concealment Levels

[0259] In Table 11, for each interval value, T can be rated at one and only one level, ensuring the uniqueness of its measurability level. Furthermore, level assessment requires the software to meet both the requirements of the software measurability metric and the corresponding individual attribute requirements for each attribute. For two software programs with the same metric, if one software program fails to meet the individual attribute requirement, its measurability level will decrease. The software measurability level model satisfies the "barrel principle," meaning that for a software program to reach a certain measurability value, the measurability of each measurable attribute must reach a certain limit; otherwise, it cannot ultimately achieve overall measurability.

[0260] Step 9: Application of software testability measurement results.

[0261] When analyzing the measurement objectives, we mentioned that measuring different software testability parameters is to meet the ultimate needs of different users, such as allocating test resources, determining the number of test cases, balancing software quality, and guiding testability design. Here, we analyze how to achieve the above objectives based on the obtained software testability measurement results.

[0262] 1. Allocate test resources

[0263] Allocate test resources using testability. Assume the software can be divided into n distinct parts S. i Let the testability of each part be Ti, i = 1, 2, ..., n, and the total resources used for testing be C. Then, based on the testability, the test resources for each part can be determined.

[0264]

[0265] 2. Determine the number of test cases.

[0266] Using the sensitivity analysis formula

[0267]

[0268] Where γ represents the minimum failure probability, C represents the confidence interval, N represents the number of test cases in the sensitivity analysis, and ε is a correction for the uncertainty of the sensitivity analysis. If we ignore the impact of the uncertainty caused by the sensitivity analysis and replace the minimum failure probability with software testability T, we can obtain a formula representing the relationship between software reliability confidence, software testability, and the number of test cases.

[0269] C≈1-(1-T) N

[0270] The formula above can be rewritten to obtain the formula for calculating the number of test cases.

[0271]

[0272] From the two formulas above, we can draw the following conclusions:

[0273] 1) The higher the testability and the more test cases there are, the higher the confidence level of software reliability; conversely, the lower the confidence level of software reliability.

[0274] 2) Software with low testability requires a greater number of tests to achieve a certain level of reliability confidence;

[0275] 3) Software with high testability can achieve a certain level of reliability confidence with only a small number of test cases.

[0276] When the number of test cases N exceeds the limitations of test resources, making it impossible to execute so many tests, other testing techniques such as code review and formal verification must be considered to ensure software quality. If other methods fail to reduce test resources, the software may be abandoned, and new software may be developed instead.

[0277] 3. Balancing software quality

[0278] Software quality formula

[0279] Q=f(R,P,F,Sa,Se,A,T,M)+ε

[0280] Where R represents reliability, P represents performance, F represents fault tolerance, Sa represents security, Se represents confidentiality, and A represents security.

[0281] ε represents availability, T represents testability, M represents maintainability, and ε represents other undefined software properties.

[0282] In the text

[0283] When the value of a certain attribute fails to reach a threshold, the software quality does not meet the requirements. Therefore, after measuring software testability, the software testability must be greater than the required threshold value before comparing it with other software attributes to determine whether the software quality meets the standard. When the software testability value is less than the threshold value, the software quality will inevitably fail to meet the requirements, and the design needs to be changed to improve the software testability.

[0284] Step 10: Conduct a special design improvement project on software testability based on "code-interface-state".

[0285] Software testability control or improvement refers to achieving specified software testability by changing the software design or program structure. Based on the calculated values ​​of each sub-characteristic, the testability characteristics to be improved are selected; influencing factors are identified, and based on the calculation results of these factors under the testability characteristics, it is determined which factors need improvement. After improvement, the calculations are re-performed to evaluate the effectiveness of the testability improvements.

[0286] This application improves software testability by designing specialized test structures. It addresses testing obstacles faced by unit testing, integration testing, and system testing by researching source code-oriented assertion design methods, interface-oriented test structure design methods, and state-oriented software probe design methods.

[0287] (1) Source code-oriented assertion design method

[0288] Developers most commonly insert assertions into source code to improve program observability and controllability. Many programming languages, such as C and C++, have macros called `assert` for writing assertions. These assertions are preprocessed and expanded into corresponding statements embedded in the program, enabling simple runtime monitoring. Testers reference assertions, bind assertion parameters, and configure assertion positions within preprocessed macros to reduce testing overhead. Types of program assertions include preconditions, postconditions, class invariants, and loop invariants. The main component of a program assertion is a program invariant expression. Six types of program assertion expressions are shown below. Figure 18 As shown.

[0289] (2) Interface-oriented test structure design method

[0290] To improve testability, a corresponding test structure is designed for the software interface. Control points / observation points can be inserted at the module boundaries, and messages can be sent and received interactively. The interface-oriented test structure includes three operating modes: observation mode, transparent mode, and test mode. Transparent mode is used during normal operation, with no control inputs and no observation outputs; observation mode is used to monitor the system during normal operation and can also control inputs; test mode is used to test modules isolated from the outside world, where inputs are passed to the observation output to observe the sending module, while control inputs are passed to the output to control the receiving module.

[0291] For example, a torpedo overload switch control software, running on a microcontroller, functions to enter a delay state after power-on, activating the overload detection function after a 5.6-second delay. This timing requirement is quite strict; the switch is activated only when the activation condition is met. This software runs within a device without a host computer display interface, making parameter monitoring and setting difficult. Therefore, the device includes a serial port and a structure to implement serial transmission. The presence or absence of the BK1 trigger signal determines whether the software enters a waiting-to-trigger mode or a communication mode. The BK1 trigger signal is critical and requires debouncing. The specific workflow is as follows... Figure 19 As shown.

[0292] This interface testing structure allows for the setting of overload trigger thresholds via serial communication using external devices, while simultaneously monitoring related parameters, thus improving software testability.

[0293] (3) State-oriented software probe design method.

[0294] In the software development phase, finite state machine (EFSM)-based specification transitions are integrated and applied. An effective EFSM generates a series of paths, each with a precondition in addition to a series of transitions. Systems with intermediate states are described using an extended finite state machine (ESM)-based language. During testing, a series of paths are generated from the finite state machine, each with a precondition in addition to a series of transitions. However, in some cases, the preconditions for certain paths cannot be met, rendering these paths infeasible. To avoid using infeasible paths, the specification description is rewritten in the form of a paradigmatic extended finite state machine. An example of a task EFSM model is shown below. Figure 20 As shown.

[0295] Setting software probes at path transitions or preconditions improves state observability. For system states, software probes are needed to achieve state observation and control. Common software probes can use assertion functions to set assertions in both preconditions and postconditions.

[0296] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0297] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for software testability analysis, design, and evaluation, characterized in that, The method includes: Based on the pre-defined definition of software testability, determine the software testability characteristics of the target software; Determine the application scenarios for software testability analysis, design, and evaluation of the target software; Based on the software testability characteristics of the target software and the application scenario, the software testability defect patterns and impacts of the target software are analyzed to determine the influencing factors and defect patterns of the target software's software testability. Based on the factors affecting the testability of the target software and the defect patterns, software testability design is carried out to form the testability of the target software; Based on a preset software testability measurement index system, the software testability of the target software is evaluated to obtain the software testability measurement result of the target software. The software testability measurement result of the target software can be used to improve the design of the software testability of the target software. The preset software testability definition includes: observability, controllability, decomposability, simplicity, change stability, operability, understandability, test support, environmental adaptability, program traceability, and sensitivity. The application scenarios for determining the software testability analysis, design, and evaluation of the target software include: For software development institutions, it predicts the verifiability of requirements, ensures that good requirements are put forward in the demonstration phase, assesses the testability risks of software, guides the testability design of software, and ensures the quality of software development. For software testing and evaluation organizations, by measuring and providing software testability, the difficulty of software testing and verification can be reduced, and test case design and test resource allocation can be guided, while also weighting software quality. For user use and maintenance, improving software testability can promptly detect internal states and defects, handle fault states in a timely manner during use, and facilitate later maintenance. The step of analyzing the software testability defect patterns and their impacts based on the software testability characteristics of the target software and the application scenario, and determining the influencing factors and defect patterns of the target software's software testability, includes: Research and collection of software testability defect patterns; Software testability defect patterns and impact analysis; The software testability defect patterns and impact analysis includes: Risk analysis of software testability itself; Risk analysis for software requirements verification; Risk analysis of software usage quality; The preset software testability measurement index system is constructed using the following method: Determine the relationship between testability characteristics and testability manifestations; Determine the relationship between factors affecting testability and testability characteristics; The evaluation of the testability of the target software based on a preset software testability metric system includes: Determine the measurement method for software testability metrics; Software testability evaluation based on a fuzzy evaluation hierarchical model; Determine the software testability level; Determine the results of software testability metrics.

Images