Generator controller software verification method oriented to airworthiness standard

Abstract

The invention belongs to the technical field of embedded software testing, and relates to a generator controller software verification method for airworthiness standards. The method comprises the following steps: performing software review and analysis; constructing a layered test environment; designing a test case and a test procedure, and performing review; performing test ready review; executing software testing in the layered testing environment; performing software demand coverage analysis for ensuring that a bidirectional traceability relationship exists between the test case and the software demand; software structure coverage analysis in the aspects of statement coverage, judgment coverage, MC / DC coverage, data coupling and control coupling is carried out; and establishing bidirectional tracing data between the test cases and the software requirements, between the test regulations and the test cases and between the test results and the test regulations, and performing test result review to form a software verification result. According to the method, the airworthiness standard process is quantified, the depth, efficiency and compliance of airborne software verification are remarkably improved through a layered test environment and an intelligent algorithm, and high safety is effectively guaranteed.

Description

Technical Field

[0001] This application belongs to the field of embedded software testing technology, and specifically relates to a method for verifying generator controller software oriented towards airworthiness standards. Background Technology

[0002] Embedded software is playing an increasingly important role in aerospace airborne systems, and its quality significantly impacts system operation, drawing considerable attention from relevant R&D departments. However, domestic research on embedded software testing theories, technologies, and tools is still in its early stages. Airborne embedded software, due to its real-time requirements, high security and reliability, and high hardware-software coupling, presents a significantly increased workload and difficulty in verification. Because of the unique and complex nature of embedded software, research progress in embedded software testing technology lags behind that of general software testing techniques. Since the 1970s, foreign countries have begun researching embedded software testing methods. Many research institutions have subsequently conducted in-depth testing research on the characteristics of embedded software, but progress in embedded software testing methodologies has been limited, leading to the development of tools to support embedded software testing. my country only began researching embedded software testing in the 1990s, lagging significantly behind international research levels, and has developed some targeted software testing tools. While some domestic and international research has explored software testing and its tools from different perspectives, their primary focus is not on airborne embedded software, and the range of tools covered is relatively limited. Summary of the Invention

[0003] To address the aforementioned issues, this application provides a generator controller software verification method oriented towards airworthiness standards. Software verification based on airworthiness certification standards can detect errors in the software to the greatest extent, ensuring software safety and quality and meeting the requirements of airworthiness certification.

[0004] The generator controller software verification method for airworthiness standards provided in this application mainly includes:

[0005] Step S1: Conduct software review and analysis;

[0006] Step S2: Build a layered testing environment;

[0007] Step S3: Design test cases and test procedures, and review them. The TBrun tool automatically generates test steps and you manually add basic elements to form the test cases, test procedures and test results.

[0008] Step S4: Conduct a test readiness review;

[0009] Step S5: Perform software testing in the layered testing environment;

[0010] Step S6: Perform software requirement coverage analysis to ensure a two-way traceability relationship between test cases and software requirements;

[0011] Step S7: Perform software structure coverage analysis in terms of statement coverage, decision coverage, MC / DC coverage, data coupling and control coupling;

[0012] Step S8: Establish bidirectional traceability data between test cases and software requirements, test procedures and test cases, and test results and test procedures; review test results; and form software verification results.

[0013] Preferably, step S1 further includes:

[0014] Step S11: Initiate the software review and analysis process, determine the review team members and their roles, organize peer reviews, and provide a review package;

[0015] Step S12: Review according to the checklist, and identify the review materials and versions;

[0016] Step S13: Record the issues, including the reviewer's name, document identifier, and issue number.

[0017] Step S14: Discuss and reach a consensus on important issues, with the review panel leader arbitrating any disputes;

[0018] Step S15: The author processes the review comments and provides feedback; the review panel confirms the processing status and then closes the window.

[0019] Step S16: The review team leader determines the review conclusion and closes the process.

[0020] Preferably, in step S2, the stratified testing environment includes:

[0021] Hardware and software integration testing environment: Based on a hardware-in-the-loop platform, it integrates some real hardware to verify the functions of the system;

[0022] Software integration testing and low-level testing environment: Use testing software tools to simulate module input / output and interactions between modules.

[0023] Preferably, the hardware and software integration test environment is specifically configured as a test platform that includes the target machine environment, and the test is completed by loading the executable code onto the target machine.

[0024] Preferably, the software integration testing and low-level testing environment uses the LDRATESTBED testing tool to simulate module input / output and interaction.

[0025] Preferably, in step S4, the test readiness review includes the following review items:

[0026] Have baselines been established for requirements, design, source code, and target code?

[0027] Have the test cases and procedures been reviewed and baselines established?

[0028] Whether to use the approved testing environment;

[0029] Test whether the tracking data is complete and correct;

[0030] Has the test schedule been provided to the relevant personnel?

[0031] Have the known issues with the software under test been resolved or has an extension been agreed upon?

[0032] Preferably, step S6 further includes:

[0033] Step S61: Analyze the traceability relationship between test cases and high-level and low-level software requirements to ensure that each requirement has a corresponding test case, each test case can be traced back to the corresponding requirement, and the test cases meet the normal testing and robustness testing criteria defined in the standard document.

[0034] Step S62: Address coverage defects by supplementing test cases, submitting issue reports, and implementing change control procedures.

[0035] Preferably, step S6 further includes:

[0036] The sufficiency index is calculated using the following formula. :

[0037]

[0038] in, , , These are the weighting coefficients. This represents the number of software requirement items covered by existing test cases. This represents the total number of software requirement items. This represents the total number of current test cases. For the i-th test case The complexity weighting factor of the covered code paths, A traceability score calculated to assess the completeness and accuracy of two-way traceability data;

[0039] When the sufficiency index exceeds the set value, the verification sufficiency is deemed to be met; otherwise, additional test cases are designed or existing test cases are optimized.

[0040] Preferably, step S7 further includes taking measures when the software structure coverage does not reach 100%, including:

[0041] When test cases or procedures are missing, add and execute additional test cases and test procedures.

[0042] When software requirements are insufficient, modify the requirements and supplement test cases and test procedures.

[0043] If dead code exists in the software, remove it and perform revalidation;

[0044] When inactive code exists in the software, analysis and testing show that the call has been blocked, or a specified configuration is established to meet the coverage target.

[0045] This application quantifies the airworthiness standard process and significantly improves the depth, efficiency, and compliance of airborne software verification through a layered testing environment and intelligent algorithms, effectively ensuring high safety. Attached Figure Description

[0046] Figure 1 This is a flowchart of a preferred embodiment of the generator controller software verification method for airworthiness standards in this application.

[0047] Figure 2 This application Figure 1 The software review and analysis flowchart of the embodiment shown is illustrated.

[0048] Figure 3 This application Figure 1 A schematic diagram of the layered testing environment of the embodiment shown.

[0049] Figure 4 This application Figure 1 A schematic diagram of the hardware / software integration test environment of the embodiment shown.

[0050] Figure 5 This application Figure 1 The illustrated embodiment is a schematic diagram of the software integration testing and low-level testing environment.

[0051] Figure 6 This is a schematic diagram of hardware / software integration test cases TC1 to TC2. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only some, not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0053] This application provides a method for verifying generator controller software oriented towards airworthiness standards, such as... Figure 1 As shown, it mainly includes:

[0054] Step S1: Conduct software review and analysis;

[0055] Step S2: Build a layered testing environment;

[0056] Step S3: Design test cases and test procedures, and review them. The TBrun tool automatically generates test steps and you manually add basic elements to form the test cases, test procedures and test results.

[0057] Step S4: Conduct a test readiness review;

[0058] Step S5: Perform software testing in the layered testing environment;

[0059] Step S6: Perform software requirement coverage analysis to ensure a two-way traceability relationship between test cases and software requirements;

[0060] Step S7: Perform software structure coverage analysis in terms of statement coverage, decision coverage, MC / DC coverage, data coupling and control coupling;

[0061] Step S8: Establish bidirectional traceability data between test cases and software requirements, test procedures and test cases, and test results and test procedures; review test results; and form software verification results.

[0062] In step S1 of this application, software review and analysis are performed. In some optional embodiments, step S1 further includes:

[0063] Step S11: Initiate the software review and analysis process, determine the review team members and their roles, organize peer reviews, and provide a review package;

[0064] Step S12: Review according to the checklist, and identify the review materials and versions;

[0065] Step S13: Record the issues, including the reviewer's name, document identifier, and issue number.

[0066] Step S14: Discuss and reach a consensus on important issues, with the review panel leader arbitrating any disputes;

[0067] Step S15: The author processes the review comments and provides feedback; the review panel confirms the processing status and then closes the window.

[0068] Step S16: The review team leader determines the review conclusion and closes the process.

[0069] like Figure 2 As shown, the software review and analysis process begins by initiating the process, identifying the names and roles of the review team members, such as the author, review team leader, and reviewers. The author or software manager organizes the peer review and provides the review team with a review package containing review materials, checklists, review report templates, and other materials required for the review. Next, the review is conducted according to the checklist, identifying the materials to be reviewed and their relevant versions in the peer review records. The review team then reviews the materials based on the checklist. During the review process, issues are recorded, including reviewer names, document identifiers and versions, document chapter or line numbers, issue numbers, and issues. Following this, issues are discussed and confirmed. Important opinions or issues are discussed during the review process to ensure all reviewers agree with the assessment. The review process involves several steps: First, reaching a consensus on the review issues. Second, resolving disputes by the peer review panel leader to ensure team agreement. Third, completing the review checklist based on all review comments. Finally, processing the comments involves the authors addressing them and the review panel confirming the processing status. The authors process the comments, develop solutions, modify the reviewed work according to the comments, prepare a peer review report, and submit the modified work to the review panel for confirmation. Finally, after the review panel leader and reviewers confirm that all comments have been addressed as required, the review panel leader finalizes the review conclusion and closes the review and analysis process.

[0070] Step S2 is used to build the test environment.

[0071] In some alternative implementations, in step S2, the stratified testing environment includes:

[0072] Hardware and software integration testing environment: Based on a hardware-in-the-loop platform, it integrates some real hardware to verify the functions of the system;

[0073] Software integration testing and low-level testing environment: Use testing software tools to simulate module input / output and interactions between modules.

[0074] In this embodiment, the test environment includes two types: a hardware / software integration test environment and a software integration test environment, as well as a low-level test environment. Figure 3 As shown, all software tests were performed in the target machine testing environment, which is a testing environment that includes both software and hardware. Testing was completed in this environment by loading the executable code onto the target machine. The hardware / software integration testing environment is as follows: Figure 4 As shown, based on the Hardware-in-the-Loop (HIL) platform, some real hardware is integrated to verify system functionality. The software integration testing and low-level testing environment are as follows: Figure 5 As shown, the LDRA TESTBED testing software tool is used to simulate module inputs and outputs and interactions between modules.

[0075] This application reduces the cost of repeated setup through a layered environment design, while ensuring the continuity of test scenarios at each stage.

[0076] Step S3 is used for the design and review of test cases and test procedures.

[0077] The requirement-based test case design selects normal range test cases and abnormal range test cases (robustness test cases). Both types of test cases take into account equivalence class partitioning (valid equivalence classes and invalid equivalence classes) and boundary values.

[0078] In software integration testing and low-level testing, according to the basic requirements of the DO-178C standard for test cases, test cases must have a unique identifier, traceability to requirements, conformity to requirements, and input / output information. The "tcf file" serves as the test case. A test case consists of the actual test case setup steps and documentation. The relevant information for the actual test case setup steps is automatically generated by the TBrun tool, while the documentation is implemented by manually supplementing basic elements according to the requirements of DO-178C. By supplementing the documentation in the tcf file, the airworthiness standard is reflected in the test cases.

[0079] Within the software verification team, cross-validation is employed. Software test cases and software test procedure checklists are reviewed, and items that fail the review are analyzed, corrected according to feedback, and then reviewed again.

[0080] In software integration testing / low-level testing, test case files, test procedure files, and test result files are automatically generated by tools. According to DO-178C requirements, if a tool is used that eliminates, reduces, or automates any processes specified in DO-178C, and the tool's output is not reviewed, then the tool needs to be qualified. To avoid additional qualification work when using the TBrun tool, its output requires manual review. Therefore, test cases, test procedures, and test results must all be reviewed accordingly.

[0081] For example, the high-level requirements of a generator controller software are:

[0082] When PGM_Mode=GENERATE, if CbitOver VoltageFault or CbitOpenPhaseFault is set to TRUE, PGM_Mode should be set to TRIP.

[0083] Based on the test case development principles in DO-178C, the design should be as follows: Figure 6 The hardware / software integration test cases are shown above. Based on the above test cases, write the test procedures, i.e., the test scripts.

[0084] Step S4 is used to conduct a test readiness review.

[0085] In some alternative implementations, step S4 includes the following review items:

[0086] Have baselines been established for requirements, design, source code, and target code?

[0087] Have the test cases and procedures been reviewed and baselines established?

[0088] Whether to use the approved testing environment;

[0089] Test whether the tracking data is complete and correct;

[0090] Has the test schedule been provided to the relevant personnel?

[0091] Have the known issues with the software under test been resolved or has an extension been agreed upon?

[0092] Analyze the items that fail the test readiness review, make corrections according to the feedback, and then conduct the review again.

[0093] Step S5 is used to perform software testing. This includes performing tests for qualification confidence level and recording the test results in a hardware / software integration testing environment, and performing tests for qualification confidence level and recording the test results using the testing tool LDRATESTBED in software integration testing and low-level testing environments.

[0094] Step S6 is used to perform software requirements coverage analysis.

[0095] In some alternative implementations, step S6 further includes:

[0096] Step S61: Analyze the traceability relationship between test cases and high-level and low-level software requirements to ensure that each requirement has a corresponding test case, each test case can be traced back to the corresponding requirement, and the test cases meet the normal testing and robustness testing criteria defined in the standard document.

[0097] Step S62: Address coverage defects by supplementing test cases, submitting issue reports, and implementing change control procedures.

[0098] In this embodiment, the traceability relationship between test cases and software requirements can be satisfied by analyzing the traceability data between software requirements and test cases. The normal testing and robustness testing criteria defined in the standard document DO-178C can be satisfied by the test case review activities. If the test cases have been fully reviewed, there is generally no need to pay too much attention to this goal during the requirement coverage analysis.

[0099] In some alternative implementations, step S6 further includes:

[0100] The sufficiency index is calculated using the following formula. :

[0101]

[0102] in, , , These are weighting coefficients that satisfy α + β + γ = 1. Their specific values ​​are adjusted according to the software security level. For example, for Level B software, recommended values ​​are α = 0.5, β = 0.3, and γ = 0.2. This represents the number of software requirement items covered by existing test cases. This represents the total number of software requirement items. This represents the total number of current test cases. For the i-th test case The complexity weighting factor of the covered code paths, The traceability score is calculated to assess the completeness and accuracy of two-way traceability data. Its value ranges from 0 to 1, and f is an adjustment coefficient.

[0103] When the sufficiency index exceeds the set value, the verification sufficiency is deemed to be met; otherwise, additional test cases are designed or existing test cases are optimized.

[0104] This embodiment provides an alternative method for software requirements coverage analysis, introducing a quantitative, multi-indicator validation adequacy assessment model (VCI). This model considers not only the breadth of requirements coverage (α-item), but also the depth of testing (β-item, path complexity) and process quality (γ-item, traceability score), thereby elevating the assessment of test adequacy from qualitative evaluation to quantitative calculation. This provides a scientific basis for deciding whether to proceed to the more time-consuming structural coverage analysis, improving the efficiency and reliability of the validation process.

[0105] It should be noted that in software verification, path complexity is an important metric for measuring the difficulty of software testing and the required coverage of test cases. It is directly related to the control flow complexity of the code. This application can use the classic McCabe cyclomatic complexity algorithm as its core calculation method because it provides a proven and quantifiable standard for measuring code complexity. For example, an effective calculation model is:

[0106] ;

[0107] Among them, test cases It will sequentially pass through M nested or sequential decision nodes. Let's consider the local cyclomatic complexity of the branch where the path is located at the k-th decision node. Assume a test case's path needs to satisfy: if(A&&B) is true, then proceed to the branch where if(C||D) is true. At the first decision (A&&B), the true branch has lower local complexity, but higher local cyclomatic complexity. It can be set to 1, at the second decision (C||D), the local complexity of the true branch is... If we also set it to 1, then the complexity weight of this path is:

[0108] .

[0109] If another path needs to traverse a branch containing complex loops and internal decisions, its local complexity value will be greater, and the calculated path complexity weight will also be significantly higher, which is in line with the original intention of measuring its "test difficulty".

[0110] The goal of this embodiment is to evaluate "validation adequacy." A test set containing a large number of highly complex test cases has a much greater validation depth and ability to uncover deep-seated defects than a test set that only covers simple paths. To effectively distinguish the quality differences between these two test sets, the value of the second term in the formula may be much greater than 1, significantly amplifying the contribution of complex tests to the overall evaluation. Simultaneously, to prevent the adequacy index from being completely hijacked by the path complexity, it is adjusted by a coefficient f, which is typically a constant value for the path complexity of empirically based test cases.

[0111] Step S7 is used to perform software structure coverage analysis.

[0112] In some alternative implementations, step S7 further includes taking measures when the software structure coverage does not reach 100%, including:

[0113] When test cases or procedures are missing, add and execute additional test cases and test procedures.

[0114] When software requirements are insufficient, modify the requirements and supplement test cases and test procedures.

[0115] If dead code exists in the software, remove it and perform revalidation;

[0116] When inactive code exists in the software, analysis and testing show that the call has been blocked, or a specified configuration is established to meet the coverage target.

[0117] In this embodiment, based on the software level and in accordance with DO-178C requirements, software structure coverage analysis involves statement coverage, decision coverage, MC / DC coverage, data coupling, and control coupling analysis. It should be noted that inactive code is typically classified into the following two categories:

[0118] 1) Code that will never be executed under any circumstances: For this type of inactive code, it is necessary to combine analysis and testing methods to verify that unintentional calls to inactive code have been blocked, isolated or eliminated.

[0119] 2) Code that will be executed in a specific configuration of the target machine environment: For this type of inactive code, the required specific configuration should be established, and corresponding test cases and test procedures should be developed to meet its required coverage target.

[0120] Finally, in step S8, tracking data is established and the test results are reviewed and analyzed.

[0121] For hardware / software integration testing, it is necessary to establish bidirectional traceability data between test cases and high-level software requirements, as well as some low-level software requirements; to establish bidirectional traceability data between test procedures and test cases; and to establish bidirectional traceability data between test results and test procedures. For software integration testing and low-level testing, it is necessary to establish bidirectional traceability data between test cases and low-level software requirements, as well as some high-level software requirements; to establish bidirectional traceability data between test procedures and test cases; and to establish bidirectional traceability data between test results and test procedures.

[0122] After the test is executed, the results should be reviewed to ensure the correctness of the test results and to ensure that any failed tests are analyzed and properly resolved. The test cases, test procedures, test results, trace data, software coverage analysis results, software review and analysis data, etc. are summarized to form the software verification results.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for verifying generator controller software oriented towards airworthiness standards, characterized in that, include: Step S1: Conduct software review and analysis; Step S2: Build a layered testing environment; Step S3: Design test cases and test procedures, and review them. The TBrun tool automatically generates test steps and you manually add basic elements to form the test cases, test procedures and test results. Step S4: Conduct a test readiness review; Step S5: Perform software testing in the layered testing environment; Step S6: Perform software requirement coverage analysis to ensure a two-way traceability relationship between test cases and software requirements; Step S7: Perform software structure coverage analysis in terms of statement coverage, decision coverage, MC / DC coverage, data coupling and control coupling; Step S8: Establish bidirectional traceability data between test cases and software requirements, test procedures and test cases, and test results and test procedures; review test results; and form software verification results.

2. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, Step S1 further includes: Step S11: Initiate the software review and analysis process, determine the review team members and their roles, organize peer reviews, and provide a review package; Step S12: Review according to the checklist, and identify the review materials and versions; Step S13: Record the issues, including the reviewer's name, document identifier, and issue number. Step S14: Discuss and reach a consensus on important issues, with the review panel leader arbitrating any disputes; Step S15: The author processes the review comments and provides feedback; the review panel confirms the processing status and then closes the window. Step S16: The review team leader determines the review conclusion and closes the process.

3. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, In step S2, the stratified testing environment includes: Hardware and software integration testing environment: Based on a hardware-in-the-loop platform, it integrates some real hardware to verify the functions of the system; Software integration testing and low-level testing environment: Use testing software tools to simulate module input / output and interactions between modules.

4. The generator controller software verification method for airworthiness standards according to claim 3, characterized in that, The hardware and software integration test environment is specifically configured as a test platform that includes the target machine environment, and the test is completed by loading the executable code onto the target machine.

5. The generator controller software verification method for airworthiness standards according to claim 3, characterized in that, The software integration testing and low-level testing environment uses the LDRATESTBED testing tool to simulate module input, output, and interaction.

6. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, In step S4, the test readiness review includes the following review items: Have baselines been established for requirements, design, source code, and target code? Have the test cases and procedures been reviewed and baselines established? Whether to use the approved testing environment; Test whether the tracking data is complete and correct; Has the test schedule been provided to the relevant personnel? Have the known issues with the software under test been resolved or has an extension been agreed upon? 7. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, Step S6 further includes: Step S61: Analyze the traceability relationship between test cases and high-level and low-level software requirements to ensure that each requirement has a corresponding test case, each test case can be traced back to the corresponding requirement, and the test cases meet the normal testing and robustness testing criteria defined in the standard document. Step S62: Address coverage defects by supplementing test cases, submitting issue reports, and implementing change control procedures.

8. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, Step S6 further includes: The sufficiency index is calculated using the following formula. : in, , , These are the weighting coefficients. This represents the number of software requirement items covered by existing test cases. This represents the total number of software requirement items. This represents the total number of current test cases. For the i-th test case The complexity weighting factor of the covered code paths, A traceability score calculated to assess the completeness and accuracy of two-way traceability data; When the sufficiency index exceeds the set value, the verification sufficiency is deemed to be met; otherwise, additional test cases are designed or existing test cases are optimized.

9. The generator controller software verification method for airworthiness standards according to claim 1, characterized in that, Step S7 further includes measures taken when the software structure coverage does not reach 100%, including: When test cases or procedures are missing, add and execute additional test cases and test procedures. When software requirements are insufficient, modify the requirements and supplement test cases and test procedures. If dead code exists in the software, remove it and perform revalidation; When inactive code exists in the software, analysis and testing show that the call has been blocked, or a specified configuration is established to meet the coverage target.

Images