OCaml unit test system
The OCaml unit testing system, designed with a layered architecture, addresses the deficiencies in test case design and management and insufficient report generation in the OCaml testing framework for airworthiness certification. It achieves an efficient and standardized testing process and report generation, meeting the testing requirements of high-safety-requirements such as avionics.
Patent Information
- Application Number
- CN202511847297.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-02-10
AI Technical Summary
The existing OCaml testing framework has shortcomings in test case design and management, insufficient compatibility with complex functional features, and lack of standardized test report generation functions in the field of airworthiness certification, which cannot meet the testing requirements of high safety requirements such as avionics.
The OCaml unit testing system, designed with a layered architecture, includes a test driver layer, a test case specification layer, and a test suite function layer. It uses Makefile scripts to automate the test process, provides standardized test case templates and assertion modules, and generates test reports that conform to the DO-330 standard.
It improved the standardization and efficiency of testing, enhanced type compatibility, established a complete traceability system, reduced manual intervention and documentation workload, and met the requirements of airworthiness certification.
Smart Images
Figure CN121501682A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of software testing, in particular to an OCaml unit test system. BACKGROUND
[0002] With the increasing requirements of software safety for avionics systems, unit testing as an important part of software quality assurance plays a key role in airworthiness certification process. OCaml, as a functional programming language, has been widely used in avionics software development due to its strong type system and memory safety features. However, the existing OCaml test framework still has many deficiencies in meeting the airworthiness certification standards.
[0003] Firstly, in terms of test case design and management, the test case template of the existing framework lacks standardized structure and does not require the inclusion of key elements such as requirement number, preconditions, etc., resulting in the breakdown of the traceability relationship between testing and requirements, and the lack of information description such as test purpose, input parameters, expected output, etc., which does not cover the required element points of DO-330 standard. Secondly, the compatibility of complex functional characteristics is insufficient, when the tested object contains a large number of type definitions, pattern matching and high-order functions, part of the assertion mechanism relies on manually written type-specific comparison functions, and its weakly typed assertion cannot be detected at compile time, affecting the reliability and efficiency of testing. Finally, the existing test tool lacks standardized test report generation function, the output test result format is not unified, and lacks detailed information including test case number, expected result, actual execution result, judgment criterion, test result, etc., which cannot meet the requirements of airworthiness certification document specification, resulting in the need for additional manpower to organize documents and supplement information for the verification team, making it difficult to meet the testing needs of high safety requirement fields such as avionics. SUMMARY
[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide an OCaml unit test system oriented to airworthiness standards, to solve the technical problems of test case design and management defects, insufficient compatibility of complex functional characteristics, and lack of standardized test report generation function of the existing OCaml language test framework in the field of airworthiness certification, and to achieve the technical effects of improving test standardization, enhancing type compatibility, perfecting traceability system, and improving test efficiency.
[0005] To achieve the above-mentioned purposes and other related purposes, the present application provides an OCaml unit test system which adopts a layered architecture, including: The test driving layer is realized by a Makefile script, which is used for automatic management of OCaml test code compilation, execution, and generation and archiving of test result files; The test case specification layer, coupled to the test driver layer, is used to define and organize test cases based on standardized templates. Each test case declares its unique identifier, the airworthiness requirement number it traces, the test purpose, preconditions, input parameters, execution steps, and expected output through structured metadata annotations. The test suite functional layer, coupled to both the test case specification layer and the test driver layer, includes: The assertion module provides a set of assertion functions for verifying the behavior of a function under test; The report generation module is used to generate a formatted test report conforming to the DO-330 standard based on the test execution results; The assertion module includes a general assertion function, which is configured to use the built-in structural equality operator of the OCaml language to perform an automatic recursive comparison of the actual return value and the expected value of the function under test with all types.
[0006] In an optional embodiment of this application, the assertion module further includes: The floating-point assertion function is configured to determine the equality of two floating-point numbers by calculating whether the absolute value of the difference between them is less than a preset precision threshold (epsilon). An exception assertion function is configured to execute a function that may throw an exception and verify whether the type of the caught exception matches the predefined expected exception type. Set and map assertion functions are configured to use a two-way containment check algorithm to verify whether two hash tables, sets, or maps contain exactly the same elements or key-value pairs.
[0007] In one optional embodiment of this application, the test case procedure layer provides a test procedure to support the operation of each test case. The test procedure defines execution steps and assertion strategy flow, and the code of the test procedure is bidirectionally bound to the test case implementation.
[0008] In one optional embodiment of this application, the test procedure includes a core procedure, a configuration procedure, an operation management procedure, and an execution procedure; The core procedures provide standardized support for the execution of all tests; The operation management class procedure controls the test drive process of each function under test; The configuration procedures are used to form a complete environment management system; The execution class procedure serves as the implementation carrier of the test case and constitutes the main part of the test case code.
[0009] In an optional embodiment of this application, the configuration class procedure includes: The pre-configuration sub-procedure is used to specify information related to the function under test, reference data files involved in the test, create report storage paths, and specify print output paths; The post-configuration sub-procedure is used to dynamically select test cases to be executed, trigger the standardized report generation process, and display window prompts.
[0010] In an optional embodiment of this application, the execution class procedure includes: The return value verification subroutine calls a general assertion function or a floating-point tolerance assertion function to verify the correctness of the function's return value under normal input. The print output verification subroutine is configured to capture and output the print information or prompt information of the function under test to the generated report, and verify the correctness of the print information in the report; The abnormal behavior verification subroutine calls the abnormal assertion function to verify whether the error handling logic of the function under test meets expectations under boundary conditions of abnormal input or resource shortage. The global variable validation subroutine is configured to check whether the state changes of specified global variables are as expected before and after the execution of the function under test.
[0011] In an optional embodiment of this application, the test suite functional layer further includes: The data conversion module provides a conversion function library, which uses functions in the library to convert different types of return values, exception information, and global variable states of OCaml functions into standardized string formats for presentation in the test report by the report generation module.
[0012] In an optional embodiment of this application, the test suite functional layer further includes a test assistance module, which provides a series of auxiliary test framework operation functions, including a time acquisition function, a file read / write function, and a function for counting the number of test cases passed.
[0013] In an optional embodiment of this application, the test-driven layer is specifically used for: Define and manage the paths and compilation order of OCaml source code files, test suite files, and related dependent libraries; In response to the first command input by the user, a complete test process for a single function under test is triggered, including setting up the test environment, compiling the code, running the test program, and generating a unit test report for the function under test. In response to the second command input by the user, the automated execution sub-script is invoked to execute the complete test process of all tested functions in the system in batches and in sequence, and finally a summary-level test execution log and a detailed log containing detailed compilation and error tracing information are generated.
[0014] In an optional embodiment of this application, the summary-level test execution log records the following information: The name of each function under test and its final test execution status; The total number of test cases, the number of passed cases, and the pass rate for each function under test; The test execution time for each function under test; Global statistics on the execution of all tested functions.
[0015] In an optional embodiment of this application, a coverage collection tool integrated through the interface provided by the Makefile script is also included. The coverage collection tool includes one or more combinations of statement coverage, decision coverage, and modified condition / decision coverage.
[0016] This application's OCaml unit testing system, through a systematic layered architecture design, separates core testing functions (assertions, transformations, report generation), test case specification definitions, and test environment construction and execution processes. Utilizing a Makefile as a unified control center, it achieves one-click automation of the entire process from code compilation and test execution to report generation and result summarization, effectively addressing the technical shortcomings of existing OCaml testing frameworks in airworthiness certification environments. Compared to existing technologies, this application ensures testing activities comply with airworthiness standards through standardized test case templates and structured testing procedures, improving test standardization; it leverages OCaml language features to achieve full-type automatic assertions, effectively handling complex functional programming structures and enhancing type compatibility; it establishes a complete two-way traceability relationship between tests and requirements, providing a sufficient chain of evidence for airworthiness certification and improving the traceability system; through automated testing processes and standardized report generation, it significantly reduces manual intervention and documentation workload, improving testing efficiency; it is not only applicable to avionics software development tools but can also be used in other high-safety-requirement OCaml language development scenarios, expanding its application scope and providing an effective solution for software quality assurance in airworthiness certification and related fields. Attached Figure Description
[0017] Figure 1 This is an architecture diagram of the OCaml unit testing system provided in this application; Figure 2 A flowchart of the OCaml unit test system provided in this application. Detailed Implementation
[0018] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0019] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the illustrations only show the components related to this application and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0020] like Figure 1 and 2 As shown, this application provides an OCaml unit testing system (also known as an OCaml unit testing framework), which adopts a layered architecture design, including a test-driven layer (…). Figure 2 The test driver Makefile and test case specification layer (in the test driver Makefile) Figure 2 The test cases and test suite functional layers (in the test case) Figure 2 The test suite (Otestcase) has three core layers.
[0021] The test-driven layer, implemented using Makefile scripts, serves as the system's control center, automating the compilation, execution, and generation and archiving of OCaml test code and test result files. This layer defines and manages the paths and compilation order of OCaml source code files, test suite files, and related dependent libraries. It specifies the compilation paths for these files, ensuring the compilation process correctly identifies and processes various files, clearly defines the compilation objects, controls the compilation order, and avoids compilation errors caused by file dependencies. It defines the test program's execution logic and execution method, implementing flow control from compilation to test execution. Through Makefile directives, testers can complete test environment setup, code compilation, and test execution with a single click, improving testing efficiency and reducing the risk of human error.
[0022] The testing framework supports a fully automated test case execution and result aggregation process. When the user enters the first command, the test driver layer calls the automated run sub-script auto_run.mk to execute the complete test process of all tested functions in the system in batch order, and finally generates a summary-level test execution log run.log and a detailed log backup_output.log containing detailed compilation and error tracing information.
[0023] The summary-level test execution log records the name of each function under test and its final test execution status, the total number of test cases for each function under test, the pass rate and pass count, the test execution time for each function under test, and global statistics for the test execution of all functions under test. By reviewing the summary-level test execution log (run.log), testers can make a preliminary judgment on the overall test situation based on the test case pass rate information and quickly locate test cases that are abnormal or missing.
[0024] The detailed log fully records the execution trajectory of each use case, including the complete compile-time log of the use case execution and the error stack and context information when an exception occurs.
[0025] When modifying or supplementing test cases for a single function under test, in order to avoid the time-consuming full execution, simply enter the directory of the corresponding function under test and enter the corresponding command "make compiler" in the console. That is, when the user enters the second command, the test driver layer triggers the complete test process for the single function under test, including setting up the test environment, compiling the code, running the test program, and generating the unit test report for the function under test.
[0026] The test case specification layer is coupled to the test driver layer, used to define and organize test cases based on standardized templates. To meet airworthiness requirements, the design process comprehensively utilizes methods such as boundary value analysis, anomaly analysis, and equivalence class partitioning to conduct full coverage analysis of the input / output scenarios and logical branches of the function under test, ensuring compliance with airworthiness coverage metrics such as statements, decisions, and correction condition decisions. Each test case declares its unique identifier, the airworthiness requirement it traces, the test purpose, preconditions, input parameters, execution steps, and expected output through structured metadata annotations. Existing standardized templates ensure the standardization of testing activities, generating a complete requirement-test-verification traceability evidence chain for airworthiness certification.
[0027] The test case procedure layer provides a test procedure to support the execution of each test case. The test procedure defines execution steps and assertion strategy flows. The procedure design meets the requirements for controlling the test process, ensuring the verification process of the source code by the test cases through standardized steps. The code of the test procedure and the test cases are bidirectionally bound, ensuring that the test procedure code executes according to the test case logic, thereby guaranteeing the credibility of the test results and providing a process evidence chain that meets airworthiness requirements for tool certification.
[0028] Test procedures include four types: core procedures, configuration procedures, runtime management procedures, and execution procedures. Core procedures provide standardized support for all test execution, runtime management procedures control the test-driven process of each function under test, configuration procedures are used to form complete environment management, and execution procedures serve as the implementation carrier of test cases, containing the main part of the test case code.
[0029] Configuration procedures are placed before and after test cases to form a complete environment management system, divided into pre-configuration sub-procedures and post-configuration sub-procedures. The pre-configuration sub-procedures specify information about the function under test, reference data files involved in the test, create report storage paths, and specify print output paths. The post-configuration sub-procedures dynamically select test cases to execute, trigger the standardized report generation process, and display window prompts.
[0030] The execution procedures include return value verification subroutines, print output verification subroutines, abnormal behavior verification subroutines, and global variable verification subroutines. The return value verification subroutines call general assertion functions or floating-point tolerance assertion functions to verify the correctness of the function's return value under normal input. The print output verification subroutines are configured to capture and output the printed information or prompts from the function under test to the generated report, verifying the correctness of the printed information in the report. The abnormal behavior verification subroutines call abnormal assertion functions to verify whether the error handling logic of the function under test meets expectations under boundary conditions of abnormal input or missing resources. The global variable verification subroutines are configured to check whether the state changes of specified global variables meet expectations before and after the execution of the function under test.
[0031] The test suite functional layer is coupled to the test case specification layer and the test driver layer, respectively. As the core functional provider, it includes an assertion module, a report generation module, a data conversion module, and a test assistance module.
[0032] The assertion module provides a set of assertion functions for verifying the behavior of the function under test. The assertion module includes a general assertion function configured to automatically and recursively compare the actual return value of the function under test with the expected value across all types using the OCaml built-in structural equality operators. It fully leverages OCaml's structural equality capabilities to achieve automatic recursive comparison across all types, supporting assertions for basic types, custom ADTs (such as `type const_listY = | YTConst| DBConstConL of Yint*Dfloat_list*var list`), and complex data structures (such as `type programY = { type_blockY : type_declD list; const_blockY : const_declD list; node_blockY : nodeD list; node_mainY : AST.id}`), performing deep structural comparisons directly without additional conversions.
[0033] The assertion module also includes floating-point assertion functions, exception assertion functions, and set and mapping assertion functions to meet other testing needs.
[0034] Due to the precision limitations of floating-point numbers, conventional assertions are prone to misjudgment. The floating-point assertion function is configured to determine equality by calculating whether the absolute value of the difference between two floating-point numbers is less than a preset precision threshold, epsilon. This uses an error range comparison method, setting an acceptable precision threshold to determine whether two floating-point numbers are equal within a reasonable error range, ensuring the reliability of the floating-point test results. It is understandable that in other embodiments, the error range comparison method in the floating-point assertion function also employs different numerical approximation algorithms or error evaluation models, such as error analysis methods based on interval arithmetic, which can also achieve reliable determination of floating-point equality.
[0035] The exception assertion function is configured to execute a function that may throw an exception and verify whether the captured exception type matches the preset expected exception type. By capturing exceptions during function execution and matching them with the preset exception type, the test is determined to pass, effectively covering the testing of exception handling logic.
[0036] To meet the comparison needs of complex data structures such as hash tables (Hashtbl.t), sets, and maps, such as determining whether two OCaml hash tables (Hashtbl.t) contain completely identical sets of key-value pairs, the set and map assertion functions are configured to use a two-way containment check algorithm. This algorithm verifies whether two hash tables, sets, or maps contain completely identical elements or key-value pairs. The two-way checking mechanism verifies that each element or key-value pair in the two hash tables, sets, or maps exists in the other, avoiding omissions that may occur with one-way checks. This is particularly suitable for verifying the correctness of data caching systems and configuration parsing processes.
[0037] The data conversion module provides a conversion function library. This library's functions convert different types of OCaml function return values, exception information, and global variable states into standardized string formats for presentation in test reports by the report generation module. Based on the diversity of OCaml function return value types, complex ADT structures, and exception handling mechanisms, the data conversion module constructs a string conversion function library covering return values, exception information, and global variable states. During testing, through type recognition, it can dynamically match the conversion function corresponding to the type of the object under test, uniformly converting execution results, exception information, or global variable states into standard string formats.
[0038] The report generation module generates formatted test reports conforming to the DO-330 standard based on the test execution results. Following the DO-330 airworthiness standard, the module custom-develops various report generation functions tailored to different types of tested functions and their return value characteristics, combined with corresponding assertion functions. The generated test reports use a fixed format template, including key elements such as expected results, actual results, judgment criteria, test results, execution time, and print information, ensuring the traceability of the testing process and results and providing a complete chain of evidence for airworthiness certification.
[0039] The test assistance module provides a series of auxiliary test framework execution functions, including functions for obtaining time, file reading and writing, and functions for counting the number of test cases passed. Specifically, it outputs detailed results information after test case execution to the console window, allowing testers to intuitively view the pass rate of the test cases of the function under test during execution, promptly identify test anomalies, and improve testing efficiency.
[0040] In a preferred embodiment, the OCaml unit test system also includes coverage collection tools integrated through an interface provided by a Makefile script. The coverage collection tools include one or more of statement coverage, decision coverage, and modified condition / decision coverage, as well as their collection tools, to ensure that airworthiness coverage indicators such as SC statement, DC decision, and MC / DC modified condition decision are met.
[0041] In summary, the OCaml unit testing system of this application, through a systematic layered architecture design, separates the core testing functions (assertions, transformations, report generation), test case specification definitions, and test environment construction and execution processes. It also utilizes Makefile as a unified control center to achieve one-click automation of the entire process from code compilation, test execution, to report generation and result summarization, effectively solving the technical shortcomings of existing OCaml testing frameworks in airworthiness certification environments. Compared with existing technologies, this application ensures that testing activities comply with airworthiness standards by using standardized test case templates and structured test procedures, thus improving test standardization. It leverages the features of the OCaml language to achieve automatic assertions across all types, effectively handling complex functional programming structures and enhancing type compatibility. It establishes a complete two-way traceability relationship between tests and requirements, providing a sufficient chain of evidence for airworthiness certification and improving the traceability system. Through automated testing processes and standardized report generation, it significantly reduces manual intervention and documentation workload, improving testing efficiency. Furthermore, it is not only applicable to avionics software development tools but can also be used in other high-safety-requirement OCaml language development scenarios, expanding its application scope and providing an effective solution for software quality assurance in airworthiness certification and related fields.
[0042] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.
[0043] The above description of the embodiments shown in this application (including the content set forth in the abstract of the specification) is not intended to be an exhaustive enumeration or to limit this application to the precise forms disclosed herein. Although specific embodiments and examples of this application have been described herein for illustrative purposes only, various equivalent modifications are possible within the spirit and scope of this application, as will be recognized and understood by those skilled in the art. As indicated, these modifications can be made to this application in accordance with the above description of the embodiments described herein, and such modifications will be within the spirit and scope of this application.
[0044] Therefore, although this application has been described herein with reference to specific embodiments thereof, freedom of modification, various changes and substitutions are also within the scope of the above disclosure, and it should be understood that in some cases, certain features of this application may be adopted without departing from the scope and spirit of the application and without corresponding use of other features. Thus, many modifications can be made to adapt a particular environment or material to the substantive scope and spirit of this application. This application is not intended to be limited to the specific terminology used in the following claims and / or the specific embodiments disclosed as the best mode contemplated for carrying out this application, but this application will include any and all embodiments and equivalents falling within the scope of the appended claims. Therefore, the scope of this application will be determined only by the appended claims.
Claims
1. An OCam1 unit testing system, characterized in that, A layered architecture is adopted, including: The test-driven layer, implemented by Makefile scripts, is used to automate the compilation, execution, and generation and archiving of OCaml test code and test result files. The test case specification layer, coupled to the test driver layer, is used to define and organize test cases based on standardized templates. Each test case declares its unique identifier, the airworthiness requirement number it traces, the test purpose, preconditions, input parameters, execution steps, and expected output through structured metadata annotations. The test suite functional layer, coupled to both the test case specification layer and the test driver layer, includes: The assertion module provides a set of assertion functions for verifying the behavior of a function under test; The report generation module is used to generate a formatted test report conforming to the DO-330 standard based on the test execution results; The assertion module includes a general assertion function, which is configured to use the built-in structural equality operator of the OCaml language to perform an automatic recursive comparison of the actual return value and the expected value of the function under test with all types.
2. The OCam1 unit test system according to claim 1, characterized in that, The assertion module also includes: The floating-point assertion function is configured to determine the equality of two floating-point numbers by calculating whether the absolute value of the difference between them is less than a preset precision threshold (epsilon). An exception assertion function is configured to execute a function that may throw an exception and verify whether the type of the caught exception matches the predefined expected exception type. Set and map assertion functions are configured to use a two-way containment check algorithm to verify whether two hash tables, sets, or maps contain exactly the same elements or key-value pairs.
3. The OCam1 unit test system according to claim 1, characterized in that, The test case procedure layer is also used to support the operation of each test case through test procedures. The test procedures define execution steps and assertion strategy processes, and the code of the test procedures is bidirectionally bound to the implementation of the test cases.
4. The OCam1 unit test system according to claim 3, characterized in that, The test procedures include core procedures, configuration procedures, operation management procedures, and execution procedures. The core procedures provide standardized support for the execution of all tests; The operation management class procedure controls the test drive process of each function under test; The configuration procedures are used to form a complete environment management system; The execution class procedure serves as the implementation carrier of the test case and constitutes the main part of the test case code.
5. The OCam1 unit test system according to claim 4, characterized in that, The configuration procedures include: The pre-configuration sub-procedure is used to specify information related to the function under test, reference data files involved in the test, create report storage paths, and specify print output paths; The post-configuration sub-procedure is used to dynamically select test cases to be executed, trigger the standardized report generation process, and display window prompts.
6. The OCam1 unit test system according to claim 4, characterized in that, The execution procedures include: The return value verification subroutine calls a general assertion function or a floating-point tolerance assertion function to verify the correctness of the function's return value under normal input. The print output verification subroutine is configured to capture and output the print information or prompt information of the function under test to the generated report, and verify the correctness of the print information in the report; The abnormal behavior verification subroutine calls the abnormal assertion function to verify whether the error handling logic of the function under test meets expectations under boundary conditions of abnormal input or resource shortage. The global variable validation subroutine is configured to check whether the state changes of specified global variables are as expected before and after the execution of the function under test.
7. The OCam1 unit test system according to claim 1, characterized in that, The test suite functional layer also includes: The data conversion module provides a conversion function library, which uses functions in the library to convert different types of return values, exception information, and global variable states of OCaml functions into standardized string formats for presentation in the test report by the report generation module.
8. The OCam1 unit test system according to claim 1, characterized in that, The test suite functional layer also includes a test assistance module, which provides a series of auxiliary test framework operation functions, including functions for obtaining time, file reading and writing, and functions for counting the number of test cases passed.
9. The OCam1 unit test system according to claim 1, characterized in that, The test-driven layer is specifically used for: Define and manage the paths and compilation order of OCaml source code files, test suite files, and related dependent libraries; In response to the first command input by the user, the automated execution sub-script is invoked to execute the complete test process of all tested functions in the system in batches and in sequence, and finally a summary-level test execution log and a detailed log containing detailed compilation and error tracking information are generated. In response to a second command input by the user, a complete testing process is triggered for a single function under test, including setting up the test environment, compiling the code, running the test program, and generating a unit test report for the function under test.
10. The OCam1 unit test system according to claim 9, characterized in that, The summary-level test execution log records the following information: The name of each function under test and its final test execution status; The total number of test cases, the number of passed cases, and the pass rate for each function under test; The test execution time for each function under test; Global statistics on the execution of all tested functions.