A method for evaluating test priority based on airborne embedded software

Abstract

This invention discloses a test priority evaluation method based on airborne embedded software, relating to the field of software engineering verification. The method includes: decomposing the functional modules corresponding to the airborne embedded software based on requirements to obtain several decomposed functional modules; obtaining the security impact level score and importance level score of the decomposed functional modules; calculating a comprehensive level score based on the security impact level score and importance level score; defining priorities and their permissions based on the comprehensive level score; and formulating test priority result evaluation criteria based on the priority and permission definitions, thereby obtaining a test priority result matrix based on the criteria. This invention defines a high-security evaluation criterion for security impact analysis under functional failure modes, as well as the importance and evaluation standards of functional modules. It also defines a comprehensive evaluation criterion by combining software module maturity, importance, and project development stage, achieving both practical application needs and safe, reliable, and rapid deployment and testing.

Description

Technical Field

[0001] This invention relates to the field of software engineering verification, and in particular to a test priority evaluation method based on airborne embedded software. Background Technology

[0002] Software testing, as a method of software verification, plays a crucial role in the software engineering process. In GJB5000B, software verification and validation are assigned to a separate process area for practical requirements; in DO-178C, software verification objectives account for 50%, which sufficiently demonstrates the importance of software verification.

[0003] With the accelerating pace of software engineering project development cycles and the increasing rigor of software product delivery quality, prioritizing the testing of functional modules within different software development stages and limited development cycles to ensure that the software delivered at each stage meets customer requirements and avoids significant quality issues and major security risks that could lead to customer complaints is a pressing issue that needs to be addressed.

[0004] Most existing methods focus on prioritizing requirements or prioritizing test case execution. These methods aim to quickly and accurately develop requirements or efficiently execute test cases in batches during regression testing, and are generally applicable to non-airborne and non-embedded software domains. There is a lack of methods specifically for airborne and embedded software, and a lack of targeted research on how to quickly and reliably evaluate and select requirement-based tests for different development stages based on the criticality and safety characteristics of requirements. Summary of the Invention

[0005] In view of the above-mentioned shortcomings in the prior art, the present invention provides a test priority evaluation method based on airborne embedded software, which solves the problems of complex, costly and inefficient priority evaluation methods in the prior art.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is: a test priority evaluation method based on airborne embedded software, comprising:

[0007] Based on the requirements, the functional modules corresponding to the airborne embedded software are decomposed, resulting in several decomposed functional modules.

[0008] Based on the failure mode, the safety impact level score and importance level score of the decomposed functional module are obtained according to the defined safety impact level judgment criteria and functional module importance judgment criteria.

[0009] Determine whether each decomposed functional module is a reusable functional module, and adjust the security impact level score and importance level score of each functional module according to the defined functional module reuse criteria;

[0010] The overall rating score is calculated based on the safety impact level score and importance level score of each functional module. If the decomposed functional module is a non-reusable functional module, the overall rating score of the decomposed functional module is calculated directly. If the decomposed functional module is a reusable functional module, the safety impact level score and importance level score of the reusable functional module are updated according to the functional module reuse criteria before the overall rating score of the decomposed functional module is calculated.

[0011] Priority and priority permissions are defined based on the comprehensive grade score; test priority result evaluation criteria are formulated based on the definition of priority permissions, and a test priority result matrix is ​​obtained according to the criteria.

[0012] The beneficial effects of this invention are as follows:

[0013] 1. By establishing various guidelines, testers can quickly develop reasonable test work plans, determine the test scope, prioritize tasks, and implement test work in a purposeful and directional manner, thus avoiding the loss of human resources, time, and quality costs caused by conducting comprehensive testing within a limited time.

[0014] 2. Since the required functional modules and software security analysis results have been identified and reusable attributes have been marked during the development phase of the software project, the test priority result matrix can be quickly obtained based on the evaluation criteria in the evaluation method when entering the testing and verification phase. This allows for the determination of the required functional requirements to be tested and the testing work to be carried out quickly.

[0015] 3. The evaluation process of this invention is clear and simple, the evaluation criteria are quantified, clear and standardized, the algorithm is simple and easy to understand, the method is easy to operate, it is convenient for technical personnel to understand and use, reduces the complexity of test priority ranking, saves manpower costs, and improves work efficiency.

[0016] 4. This invention is an evaluation method developed based on multiple dimensions, including failure safety analysis, critical software characteristics, and reusability maturity. It is safe, reliable, and low-risk, and can intuitively and quickly determine the testing work required at each stage of software development, even with short development cycles and heavy delivery tasks. This ensures that the delivered software does not have any quality problems or security risks of a critical level or above, and meets the customer's delivery requirements at that stage. As the stages progress and the corresponding testing work is carried out, the quality problems and security risks of the delivered software will also be eliminated. Attached Figure Description

[0017] Figure 1The flowchart illustrates a test priority evaluation method based on airborne embedded software, provided as an example. Detailed Implementation

[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0019] like Figure 1 As shown, in one embodiment of the present invention, taking the software requirements of an atmospheric data computer as an example, a test priority evaluation method based on airborne embedded software includes the following steps:

[0020] S1. Based on the requirements, the functional modules corresponding to the airborne embedded software are decomposed to obtain several decomposed functional modules.

[0021] By communicating with customers, users, or other stakeholders, we collect requirements for the airborne embedded software, and then categorize and organize these requirements. Based on the results of the requirements analysis, the core functional modules are further decomposed into smaller functional modules, resulting in several decomposed functional modules. The goal is to make each decomposed functional module functionally singular and independent, reducing coupling between modules and facilitating better testing and understanding of the software functionality by developers.

[0022] In this embodiment, the decomposed functional modules include a CPU initialization module, an external interface initialization module, an internal interface initialization module, a parameter initialization module, a sensor feature data reading module, a correction data reading module, a CPU instruction checking module, a memory checking module, and a power detection module.

[0023] S2. Obtain the security impact level score and importance level score of the decomposition function module.

[0024] The specific steps include:

[0025] S21. Identify the failure modes of each decomposed functional module, that is, identify the faults or abnormal situations that may occur in each functional module during operation.

[0026] S22. Define the criteria for judging the degree of security impact of functional modules;

[0027] As shown in Table 1, the safety impact level includes four levels: high, medium, low, and minor. This quantifies the safety impact, facilitates intuitive understanding, and is used in subsequent assessments. Failure modes with a high safety impact level include: causing the product to malfunction or crash; hindering the achievement of critical product functions, leading to the loss of critical functions; causing fatal errors in pilot operations; and causing fatal consequences for the application of the interactive system. Failure modes with a medium safety impact level include: adversely affecting critical functions with no alternative solutions; hindering the achievement of important functions, leading to the loss of important functions; adversely affecting critical functions, but with known alternative solutions; causing serious errors in pilot operations; causing serious consequences for the application of the interactive system; and adversely affecting important functions with no alternative solutions. Failure modes with a low safety impact level include: hindering or causing the loss of general functions; adversely affecting important functions, but with known workarounds; causing minor errors in pilot operations; having minor consequences for the application of the interactive system; affecting user satisfaction but not the operation or functionality of the product software; and adversely affecting general functions with no workarounds. Failure modes with a minor safety impact level include: adversely affecting general functions, but with known workarounds; causing inconvenience to technicians but not hindering work completion; not meeting user requirements, hindering self-maintenance or auxiliary testing, but not affecting user needs; and other impacts that do not affect user needs.

[0028] Table 1

[0029]

[0030] "Cataclysmic" refers to damage to the aircraft or injury to personnel; "serious" refers to an increase in the crew's workload that causes discomfort, pain, or injury to the crew members; and "minor" refers to a slight increase in the crew's workload that causes slight discomfort to the crew members.

[0031] S23. Based on the criteria for judging the degree of safety impact of functional modules, perform a safety impact analysis on the failure modes corresponding to each decomposed functional module and obtain the degree of safety impact score A.

[0032] In this embodiment, when the CPU initialization module fails to initialize, the CPU malfunctions, causing the software to stop running or malfunction, and the product cannot operate normally. The safety impact level is "high," with a score of 6. When the external interface (bus, analog, discrete) initialization fails, the software input / output communication is abnormal, and the product cannot operate normally. The safety impact level is "high," with a score of 6. When the internal interface (AD, FD, interrupt) initialization fails, the software input and data acquisition are abnormal, and the product cannot operate normally. The safety impact level is "high," with a score of 6. When parameter initialization is incomplete, random numbers may participate in the program logic, leading to functional errors. The safety impact level is "medium," with a score of 5. When sensor features... When data reading fails / errors occur, the calculated atmospheric parameters are abnormal / inaccurate, resulting in the loss of the aircraft's main functions. The safety impact level is "high," with a corresponding score of 6 points. When correcting data reading failures / errors occurs, the calculated atmospheric data without correction may be inaccurate and may not meet user requirements for data accuracy / flight bus requirements, adversely affecting critical functions. The safety impact level is "medium," with a corresponding score of 5 points. When CPU instruction checks fail / error, if the CPU instruction check fails, it indicates a potential problem with the CPU's internal logic devices, which may affect the correctness of data calculation. If the data is abnormal and the CPU instruction check implementation is incorrect, it cannot correctly determine whether the CPU instruction is correct, leading to false alarms or no alarms, making troubleshooting and location difficult and affecting user satisfaction. Because this is a fault reporting mechanism, there are no task handling measures for actual faults. Furthermore, if a fault is not reported, feedback can be obtained through other means. Therefore, errors in the implementation of this function do not affect the implementation of the main functions, and the safety impact level is "low," corresponding to a score of 3 points. When the memory check fails / errors, if the memory check fails, it indicates a problem with the memory access area, preventing normal data access and potentially affecting the correct application of program variables, leading to abnormal program operation. If the memory check implementation is incorrect, this fault may be falsely reported or not reported, increasing the difficulty of troubleshooting and affecting user satisfaction. However, it does not affect the implementation of the main functions, and the safety impact level is "low," corresponding to a score of 3 points. When the power supply detection fails / errors, if the power supply detection fails, a faulty power supply will cause corresponding hardware malfunctions, leading to abnormal software operation. If the power supply detection implementation is incorrect, this fault may be falsely reported or not reported, increasing the difficulty of troubleshooting and affecting user satisfaction. However, it does not affect the implementation of the main functions, and the safety impact level is "low," corresponding to a score of 3 points.

[0033] S24. Define the criteria for judging the importance of functional modules;

[0034] As shown in Table 2, the importance level includes three levels: critical functions, important functions, and general functions. Among them, the basic configuration functions that ensure the operation of the product software and the main functional requirements proposed by users are critical functions; other functional requirements proposed by users besides critical functions that need to be implemented in the air are important functions; and software functions that do not affect flight safety and need to be implemented on the ground are general functions.

[0035] Table 2

[0036]

[0037] Among them, basic configuration functions refer to the initialization functions that enable the software to run on the target machine; core functional requirements refer to the product input, output, logic calculation / control functions implemented by the software to meet user needs.

[0038] S25. Determine the importance level score B of each decomposed functional module according to the functional module importance judgment criteria;

[0039] In this embodiment, the CPU instruction checking module, memory checking module, and power detection module are all classified as "Important Functions" with an importance score of 2. The remaining functional modules are classified as "Critical Functions" with an importance score of 3. This quantifies the importance of the functional modules, making them easier to understand and participate in subsequent evaluations. This embodiment does not include modules corresponding to general functions, which mainly refer to software functions that do not affect flight safety and require ground execution, such as ground maintenance functions.

[0040] S26. Determine whether each decomposed functional module is a reusable functional module;

[0041] S27. Define the criteria for functional module reuse;

[0042] Table 3 shows the correspondence between the security impact level (score) and the importance level (score) of reusable and non-reusable functional modules. For reusable functional modules, a downgraded score is used for the security impact level and importance level.

[0043] Table 3

[0044]

[0045] If the decomposed functional module is a non-reusable functional module, then the comprehensive grade score of the decomposed functional module is calculated directly;

[0046] If the decomposed functional modules are reusable functional modules, then according to the functional module reuse criteria, the safety impact level score and importance level score of the reusable functional modules are determined as A1 and B1, respectively.

[0047] All modules involved in this embodiment are non-reusable modules. If a module is reusable, then only the security impact level score and functional importance level score need to be updated according to the reuse criteria, and then the comprehensive level score can be calculated for the final evaluation.

[0048] S3. Calculate the comprehensive score based on the safety impact level score and the importance level score;

[0049] The formula for calculating the overall grade score is:

[0050] Overall rating score = Safety impact rating score × Importance rating score.

[0051] In this embodiment, the overall score of the CPU initialization module, external interface initialization module, internal interface initialization module and sensor feature data reading module is 18, the overall score of the parameter initialization module and correction data reading module is 15, and the rest are 6.

[0052] S4. Define priorities and priority permissions based on the overall rating score;

[0053] As shown in Table 4, the priority is divided into 6 domains based on the comprehensive grade score. A priority is defined for each domain and a corresponding permission is assigned to each priority, i.e., priority permission, in order to support the final evaluation of the priority results of the test.

[0054] Table 4

[0055]

[0056] S5. Based on the definition of priority permissions, and in conjunction with the development stage and software level, formulate the criteria for evaluating test priority results.

[0057] As shown in Table 5, the development phase includes three stages: ground joint testing, first flight after installation, and delivery flight; the software level is divided into: general software, important software, and critical software. By considering three dimensions—comprehensive score, development phase, and software importance—a test priority result evaluation criterion is formulated to comprehensively evaluate and provide a test priority result matrix for the corresponding required functional modules.

[0058] Table 5

[0059]

[0060] S6. Based on the test priority result evaluation criteria, a comprehensive evaluation matrix of test priority results for each required functional module is provided to directly guide the test work arrangement.

[0061] As shown in Table 6, the test priority result matrix of all functional modules is obtained after comprehensive evaluation based on the test priority result evaluation criteria.

[0062] Table 6

[0063]

[0064] Finally, software testing is conducted based on the priority result matrix.

[0065] This invention is an evaluation method developed based on multiple dimensions, including failure safety analysis, critical software characteristics, and reusability maturity. It is safe, reliable, and low-risk, and can intuitively and quickly determine the testing work required at each stage of software development, even with short development cycles and heavy delivery tasks. This ensures that the delivered software does not have any quality problems or security risks of a critical level or above, and meets the customer's delivery requirements at that stage. As the stages progress and the corresponding testing work is carried out, the quality problems and security risks of the delivered software will also be eliminated.

Claims

1. A test priority evaluation method based on airborne embedded software, characterized in that, include: Based on the requirements, the functional modules corresponding to the airborne embedded software are decomposed, resulting in several decomposed functional modules. Based on the failure mode, the safety impact level score and importance level score of the decomposed functional module are obtained according to the defined safety impact level judgment criteria and functional module importance judgment criteria. Determine whether each decomposed functional module is a reusable functional module, and adjust the security impact level score and importance level score of each functional module according to the defined functional module reuse criteria; The overall rating score is calculated based on the safety impact level score and importance level score of each functional module. If the decomposed functional module is a non-reusable functional module, the overall rating score of the decomposed functional module is calculated directly. If the decomposed functional module is a reusable functional module, the safety impact level score and importance level score of the reusable functional module are updated according to the functional module reuse criteria before the overall rating score of the decomposed functional module is calculated. Priority and priority permissions are defined based on the overall rating score; Based on the definition of priority authority, a test priority result evaluation criterion is formulated, and a test priority result matrix is ​​obtained according to the criterion. In the test priority result evaluation criterion, the test priority result is divided into 6 level domains according to the comprehensive level score. A priority is defined for each level domain, and a corresponding priority authority is assigned to each priority, including the development stage and software level standard applicable to the functional module test.

2. The method according to claim 1, characterized in that, The specific steps for obtaining the security impact level score and importance level score of the decomposition function module are as follows: Identify the failure modes of each decomposed functional module; Define the criteria for judging the degree of security impact of functional modules; Based on the criteria for judging the degree of safety impact of functional modules, a safety impact analysis is performed on the failure modes corresponding to each decomposed functional module, and a safety impact degree score A is obtained. Define the criteria for judging the importance of functional modules; The importance level score (B) of each decomposed functional module is determined according to the functional module importance judgment criteria.

3. The method according to claim 2, characterized in that, The importance level includes three levels: critical functions, important functions, and general functions. Among them, the basic configuration functions that ensure the operation of the product software and the functions of the main functional requirements proposed by users are critical functions; the functions of the product software that need to operate in the air and other functional requirements proposed by users other than critical functions are important functions; and the software functions that do not affect flight safety and need to be implemented on the ground are general functions.

4. The method according to claim 3, characterized in that, The safety impact level is divided into four levels: high, medium, low, and minor. Among them, the failure modes with a high safety impact level include: causing the product to malfunction or crash; hindering the realization of the product's critical functions; causing the loss of the product's critical functions; causing fatal errors in pilot operation; and causing fatal consequences to the application of interactive systems.

5. The method according to claim 4, characterized in that, Failure modes with a safety impact level of Medium include: adversely affecting critical functions with no alternative solutions; hindering the implementation of important functions, leading to the loss of important functions; adversely affecting critical functions but with known alternative solutions; causing serious errors in pilot operations; causing serious consequences for the application of interactive systems; and adversely affecting important functions with no alternative solutions.

6. The method according to claim 4, characterized in that, Failure modes with a low safety impact level include: hindering or causing the loss of general functions; adversely affecting important functions, but with known workarounds; causing minor errors in pilot operations; having minor consequences for the application of interactive systems; affecting user satisfaction but not the operation of the product software or the realization of its functions; and adversely affecting general functions with no workarounds.

7. The method according to claim 4, characterized in that, Failure modes with a minor safety impact level include: adversely affecting general functionality but with known alternative solutions; causing inconvenience to technicians but not hindering the completion of work; not meeting user requirements, hindering self-maintenance or auxiliary testing, and not affecting user requirements; and other impacts that do not affect user requirements.

8. The method according to claim 2, characterized in that, The formula for calculating the overall grade score is: Overall rating score = Safety impact rating score × Importance rating score.

9. The method according to claim 2, characterized in that, In the functional module reuse criteria, a downgraded score is used for the security impact level and importance level of reused functional modules.

Images