Automatic testing method and system of embedded operating system

Abstract

The invention provides an automatic test method and system for an embedded operating system. The system comprises a remote server, a building machine, a test upper computer and hardware equipment, the hardware equipment comprises a tested board card and a test accompanying board card; the tested board card and the test accompanying board card are provided with embedded operating systems; the test upper computer is used for testing pre-configuration and executing an automatic test task; the remote server is connected with the test upper computer, and the test upper computer is connected with the hardware equipment; and the remote server is connected with the building machine. According to the method, the whole set of embedded operating system automatic testing process including remote server triggering, test case compiling, mirror image downloading, hardware power-on and power-off, case execution, result evaluation and test report reporting is realized to replace manual testing, the testing efficiency and accuracy are improved, and the product quality is improved.

Description

Technical Field

[0001] The present invention belongs to the field of embedded software testing technology, and in particular, relates to an automated testing method and system for an embedded operating system, and more particularly, to an automated testing method and system for a highly reliable embedded operating system. Background Art

[0002] Due to the diversity of hardware and the complexity of embedded operating systems themselves, the testing of embedded operating systems is very complicated, especially the testing of domestic embedded operating systems based on domestic processors. On the one hand, it is necessary to provide more flexible hardware peripherals and bus protocol test configurations to adapt to the situation that domestic processors have a lot of independent research and development content and strong customization. On the other hand, there are not many mature automated execution use cases for download and use in the domestic embedded operating system ecosystem. Most of the use cases need to be customized and completed manually. Testers need to prepare a large number of use cases to verify the functions and performance points of each module of the operating system.

[0003] For embedded hardware platforms with different hardware interfaces and bus protocols, there are many types of comprehensive combinations and the test environment is complex to set up. Manual testing may also involve interactive operations with the test host computer. The test steps are numerous and relatively professional, requiring high hardware knowledge and comprehensive hardware and software capabilities of testers. After the test is completed, the test results need to be manually checked and analyzed, and the whole process is very time-consuming and labor-intensive.

[0004] High-reliability embedded operating systems are mainly used in aviation, aerospace, military, rail transportation and other fields with high security levels, which require the operating system to have high reliability, security and stability. Accordingly, the test for high-reliability embedded operating systems will add more stress tests, abnormal tests, boundary tests, and copy tests for hardware interfaces and bus protocols on the basis of general functional tests and performance tests. These tests are more complex and time-consuming, and a considerable part of the tests require mechanical repetitive labor, which requires a set of automated testing tools for embedded operating systems to solve the current dilemma.

[0005] However, the existing automated testing frameworks on the market cannot meet the needs of automated testing of high-reliability embedded operating systems. The specific shortcomings are as follows:

[0006] 1. The actual situation of embedded systems, especially BSP driver testing and bus protocol testing, is not taken into consideration. An automated testing method that can replace manual testing and support multiple hardware interfaces and bus protocols, processors with different instruction set architectures, and hardware boards with different peripheral models is provided.

[0007] 2. Automatic configuration and compilation of test cases are not considered. Embedded operating system test cases running on different hardware platforms / instruction sets and different peripherals need to be configured with different parameters and recompiled.

[0008] 3. Interaction tests between embedded devices, between embedded devices and test host computers, and some reliability and stability tests, such as USB repeated plug-in and unplug tests, file system power-off and restart stability tests, network cable plug-in and unplug tests, and various network packet flow tests, cannot be connected to the automation framework for execution and require manual testing, or even cannot be completed manually.

[0009] 4. Failure to consider test anomalies, such as the situation where the test causes the task to get stuck or even the system to crash, making it impossible to continue testing;

[0010] 5. For some high-reliability embedded operating systems that do not support console command interaction and cannot issue test commands, there is no solution if interactive testing with the outside is required.

[0011] The following patent documents were found after searching:

[0012] Application number: CN202211646760.0, patent document CN115629996A discloses an embedded software automated testing method. The invention discloses an embedded software automated testing method, specifically related to the field of embedded software testing, used to solve the problem that most of the existing embedded software tests still use manual repeated functional testing, with extremely low test efficiency, incomplete coverage, and high error rate; including the test case management tool is successfully connected and sent to the embedded product in sequence according to the configured test cases; the embedded product activates the automated testing process after successfully receiving the automated testing plug-in; the key node replacement code segment takes effect, and the embedded software executes according to the established business process; the embedded business function code is executed, and the results of the business execution are sent to the test case management tool; the present invention is to inject the automated test case plug-in into the embedded product on demand through the test case management tool, realize the automated testing and management of the embedded software, completely get rid of the manual repetitive and tedious testing; greatly improve the testing efficiency and software quality.

[0013] Application number: CN202410500286.3, patent document CN118484380A discloses an embedded automated testing system and method. The application discloses an embedded automated testing system and method, which relates to the field of automated framework testing technology, including: configuring the test command of the test case through the configuration module, selecting the corresponding test case according to the test command through the execution scheduling module, executing the characteristic test of the test case and recording the test results through the execution test module, retrieving the test results through the test scheduling module, and displaying the test results in real time. Compared with the existing technology, it realizes the high automation, refined management and instant feedback of the embedded system testing process, and improves the testing efficiency.

[0014] Application number: CN202411021260.7, patent document CN118550839A discloses an embedded automatic test system and its test method. The invention provides an embedded automatic test system and its test method, including interconnected test modules and function modules, the function module provides corresponding function units for the test selected by the user in the test module; wherein the test module integrates multiple tests and test information and test equipment information corresponding to the test; the function module is provided with function units corresponding to the test, the function unit is a test environment matching the corresponding test configuration, the test results are parsed, the test results are generated into corresponding logs and reports and sent by email; multiple tests can be performed synchronously or sequentially, synchronously can save test time and improve test efficiency, while sequentially can maximize the effective use of test equipment, and has a wide range of applications.

[0015] The application number is CN202311683277.4, and the patent document CN117539791A discloses an automated testing system for embedded software. The invention discloses an automated testing system for embedded software, which relates to the field of software testing technology. The system discloses a board capability definition and firmware compilation layer, a test plan parsing and use case generation layer, a program deployment and test scheduling execution layer. By decoupling the board firmware generation from the hardware board, the problem that the traditional CICD system cannot well support the diversity and differences of hardware boards, resulting in cumbersome processes and low efficiency, is solved; the board hardware and program versions, the version management is unified, which is conducive to traceability management; the test cases are classified and managed uniformly, and are reused reasonably; the board test plan is automatically parsed, which is conducive to management and automated testing; the board program development and testing work are decoupled to improve the efficiency of code development, testing and function iteration.

[0016] In summary, most of the existing automated testing methods for embedded software are for automated testing of embedded application software rather than embedded operating systems. For some embedded systems with complex hardware peripherals, interactive testing with other devices or test host computers is required, and automation cannot be achieved.

[0017] In addition, the automatic configuration and compilation of test cases, the automated testing of some repeated stress tests, and the handling of test exceptions were not considered. In particular, for some high-reliability embedded operating systems that do not support console command interaction and cannot issue test commands, no solution has been found for how to conduct efficient automated testing.

[0018] This problem needs to be solved urgently. Summary of the invention

[0019] In view of the defects in the prior art, an object of the present invention is to provide an automated testing method and system for an embedded operating system.

[0020] According to the present invention, an automated testing system for an embedded operating system comprises: a remote server, a builder, a test host computer and hardware equipment; the hardware equipment comprises a board to be tested and a companion test board; the board to be tested and the companion test board are installed with an embedded operating system; the test host computer is used to test pre-configuration and execute automated test tasks;

[0021] The remote server is connected to the test host computer, and the test host computer is connected to the hardware device;

[0022] The remote server is connected to the build machine.

[0023] Preferably, the test host computer is installed with an automated test control module; the automated test control module is used for hardware parameter and test case configuration, execution of automated tests, and generation and uploading of test reports;

[0024] The remote server is installed with a remote control module and a code version control tool; the remote control module is used to trigger the builder to start building, and then trigger the automated test control module to work and collect test reports;

[0025] The code version control tool is used to provide configuration files and test codes;

[0026] The construction machine is installed with a test case construction module and an image or executable program download module; the test case construction module is used to obtain the configuration file and the test code, and generate a system image or executable program; the image or executable program download module is used to download the system image or executable program to the tested board and the accompanying test board of the hardware device.

[0027] Preferably, the automated test control module comprises: a hardware interface configuration unit, a test case configuration unit, a test case execution unit, a test result evaluation unit, a log unit and a test report reporting unit;

[0028] The hardware interface configuration unit is used to configure the parameters of the board and generate a configuration file;

[0029] The test case configuration unit is connected to the hardware interface configuration unit; the test case configuration unit is used to configure the test action of the automated test;

[0030] The test case configuration unit is connected to the test case execution unit; the test case execution unit is used to schedule the test actions on the hardware equipment, test host computer and other auxiliary hardware equipment according to the configuration of the test case configuration unit; the other auxiliary hardware equipment includes: a switch, a bus protocol analyzer, an industrial field bus slave station device, and an oscilloscope.

[0031] The test case execution unit is connected to the log unit; the log unit is used to record key logs in the process of executing the automated test; the key logs are the operation logs of the automated test control module, the result output after the embedded operating system executes the test command, and the result output obtained after the test case execution unit calls the external program;

[0032] The test case execution unit is connected to the test result evaluation unit; the test result evaluation unit is used to evaluate the test result of the test case execution unit;

[0033] The test result evaluation unit is connected to the test report reporting unit; the test report reporting unit is used to generate a test report by performing statistics and analysis on the test results, and report the test report to the remote control module of the remote server.

[0034] Preferably, the test case execution unit includes: a command issuing subunit, a program-controlled switch control subunit, and an external program calling subunit;

[0035] The command issuing subunit is used to issue test commands to the tested board and the accompanying tested board of the hardware device;

[0036] The program-controlled switch control subunit is used to control the power switch according to the test command;

[0037] The external program calling subunit is used to call the external program of the test host computer and obtain the result output of the interaction between the external program and the embedded operating system.

[0038] Preferably, the hardware device further includes a program-controlled switch;

[0039] The program-controlled switch control subunit generates a program-controlled switch control file, and controls the program-controlled switch according to the program-controlled switch control file to switch the power supply.

[0040] Preferably, the test case configuration unit configures the test actions including: test commands, interval actions between two consecutive commands, interaction actions with the test host computer and other auxiliary hardware devices, the sending method of the test command and the evaluation criteria of the test results;

[0041] The test command includes: a test command sent to the embedded operating system of the board under test, a test command sent to the embedded operating system of the accompanying test board, and an accompanying test command executed by the test host computer;

[0042] The interval action between two consecutive commands includes restarting to execute the next command, not restarting but executing the next command after a fixed interval, and immediately executing the next command.

[0043] The interactive actions include: starting timing, bus protocol communication, network transmission and reception, and power switch;

[0044] Preferably, the sending mode of the command sending subunit test command includes: sending via serial port, sending via Telnet and sending via TCP client / server;

[0045] The TCP client / server mode is sent and is suitable for a high-reliability embedded system without a console. It is implemented based on the automated test system of the embedded operating system, which instructs the embedded operating system to create a TCP server and wait for receiving a test command, instructs the command issuing subunit to create a TCP client to issue a test command, instructs the TCP server to query a command list to execute the test command and obtain a test result; instructs the TCP server to send the test result and the TCP client to obtain the test result.

[0046] According to an automated testing method for an embedded operating system provided by the present invention, an automated testing system based on the embedded operating system is implemented, comprising:

[0047] A new test project is created through the hardware interface configuration unit of the automated test control module, and the hardware parameters of the test project are configured to generate a configuration file; the configuration file is uploaded to the code version control tool of the remote server; the test case configuration unit of the automated test control module generates a test case configuration file by checking existing test cases and adding new test cases in the interface;

[0048] The remote control module triggers the test case construction module to start case compilation and construction, and the test case construction module generates an image or executable program according to the test code and the configuration file, and stores it in the image or executable program download module to wait for the hardware device to download;

[0049] The remote control module is used to trigger the test case execution unit of the automated test control module to start the automated test, and based on the test case configuration file, the board under test, the accompanying test board and the test host computer are controlled to load and start the image or executable program, thereby enabling the test result evaluation unit to generate the test result; the test case execution unit is used to enable the log unit to record the operation log during the test execution;

[0050] The code version control tools include Git and SVN;

[0051] According to the test result, the test report reporting unit generates a test report and then uploads the test report to the remote control module of the remote server, pushes it to the corresponding personnel, and ends.

[0052] Preferably, the triggering method of the automated test includes triggering the remote control module or directly triggering the automated test control module itself.

[0053] Preferably, the hardware parameters of the test items include: board peripheral conditions, board network configuration and operating system conditions of the board;

[0054] The board peripherals information includes: device path, device quantity and device type;

[0055] The board network configuration includes: board IP address, peer IP address and interface name;

[0056] The operating system running on the board includes: system version, 32-bit or 64-bit, and multi-core operation mode.

[0057] Preferably, the test case configuration unit selects a serial port mode, a network mode or a TCP client / server mode to send a test command; and sets a timeout threshold of the test case to 2 minutes;

[0058] Determine whether the test case execution time exceeds the timeout threshold. If yes, restart the hardware device to continue the next test; if no, do not process.

[0059] Preferably, the test report includes: the number of test cases, the number of passed cases, the number of failed cases, the defect rate and the regression completion rate.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The present invention realizes a complete set of embedded operating system automated testing processes from remote server triggering, test case compilation, image downloading, hardware power on and off, case execution, result evaluation, and test report reporting to replace manual testing, thereby improving test efficiency and accuracy and improving product quality.

[0062] 2. The remote trigger test and automatic compilation use cases provided by the present invention are convenient for accessing the continuous integration process. The test case construction module supports hardware configuration and test case compilation for processors with different instruction set architectures, hardware boards with different peripheral models, multiple hardware interfaces and bus protocols; the processors with different instruction set architectures include: ARM, LoongArch, MIPS, X86, PPC, C-sky, SPARC and RiscV; the hardware boards with different peripheral models include: NAND / NorFlash, SD / TF cards, eMMC and network cards of different models; the bus protocols include: PCI / PCI-E, Serial, RapidIO, CAN / CANFD and USB.

[0063] 3. The automated test control module provided by the present invention supports hardware parameter configuration, supports controlling multiple devices, and interactive testing between devices and test host computers, and solves the problem that domestic embedded operating systems based on a variety of domestic processors are difficult to truly automate due to their variable hardware, complex peripherals, complex hardware interfaces and bus protocols, and a large proportion of customized testing.

[0064] 4. The present invention solves the problem that repeated tests and abnormal tests such as reliability testing, abnormal testing, copy testing and stress testing are difficult to complete manually or take a long time, and provides processing measures for abnormalities occurring during test execution, which is more suitable for high-reliability embedded operating system testing.

[0065] 5. The present invention solves the problem that some high-reliability embedded operating systems do not support console command interaction and cannot issue commands.

[0066] 6. The present invention can realize the full automation process of test code configuration and compilation, image downloading and startup, and test case execution, including test case execution that replaces the manual interaction part, test result analysis and collation, and test result reporting, which greatly saves manpower and material resources for embedded operating system testing. On the basis of improving efficiency and test accuracy, it can also realize some stress and abnormal tests that cannot be done manually to support the reliability and stability verification of high-reliability embedded systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0068] Figure 1 A schematic diagram of the overall framework provided by the present invention;

[0069] Figure 2 A test case execution unit framework diagram provided by the present invention;

[0070] Figure 3 A schematic diagram of the command interaction flow in the case of no console provided by the present invention;

[0071] Figure 4 It is a schematic diagram of the overall execution flow of the automated test of the present invention. DETAILED DESCRIPTION

[0072] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0073] The technical problems to be solved by the present invention are embodied in the following aspects:

[0074] 1. The present invention provides a stable and reliable software and hardware testing environment for embedded operating system testing, especially BSP driver and various bus protocol testing, supports multiple hardware communication interfaces and bus protocols, and facilitates the function, performance and reliability testing of processors with different instruction set architectures, hardware boards with different peripheral models, and different communication interfaces and protocols.

[0075] 2. To address the problems of low efficiency and misjudgment in manual testing, the present invention implements a fully automated process of test code configuration and compilation, image downloading and startup, test case execution, test result analysis and organization, test result reporting, and test log storage; the test case execution includes replacing the manual operation part that interacts with the test host computer or other hardware devices.

[0076] 3. In order to solve the problem that a failure in a test during the automated test leads to the failure of the entire automated test, the present invention sets a timeout when an abnormal situation such as task freezing, system freeze, inability to exit the loop, etc. is encountered during the execution of a test. After the timeout, the test environment is forced to restart to continue executing other tests without affecting the overall test process.

[0077] 4. In order to solve the problem that some hardware operations and interactions between different devices are still manually tested and even some stress tests cannot be completed manually, the manual part of the test process is replaced by an automated test control unit, such as controlling the power on and off of the hardware board, continuous multiple plugging and unplugging of the USB interface, continuous plugging and unplugging of the network port, CAN communication and network communication with the test host computer, etc.

[0078] 5. In order to solve the problem that some high-reliability embedded operating systems do not support console command interaction and cannot issue test commands, resulting in complicated testing methods, the present invention uses the TCP protocol to create a TCP server on the embedded system side to replace the console to wait for commands to be issued.

[0079] According to the present invention, an automated testing system for an embedded operating system comprises: a remote server, a builder, a test host computer and hardware equipment; the hardware equipment comprises a board to be tested and a companion test board; the board to be tested and the companion test board are installed with an embedded operating system; the test host computer is used to test pre-configuration and execute automated test tasks;

[0080] The remote server is connected to the test host computer, and the test host computer is connected to the hardware device;

[0081] The remote server is connected to the build machine.

[0082] In other words, the overall framework diagram provided by the present invention is as follows Figure 1 As shown, the details are as follows:

[0083] First, the remote control module is responsible for using continuous integration tools, such as Jenkins, to automatically trigger an automated test activity when accessing the continuous integration environment. It first triggers the test case construction module to build the test case into an image or executable program, then starts the automated test control module to start executing automated tests. After the test is completed, it receives the test report reported by the automated test control module and pushes it as needed.

[0084] Second, the configuration files and test codes are managed by the code version control tool. After the test case construction module is triggered, the test code and configuration files stored in the code version control tool are pulled out. You can choose to compile with the operating system to generate a system image, or directly generate an executable program, put it into the image or executable program download module and wait for it to be downloaded to the board. The configuration file is automatically generated in the automated test control module through interface configuration; the code version control tool includes: Git, SVN.

[0085] Third, the automated test control module is the central control module of the entire automated test. It is generally run on a test host computer connected to the hardware board to control the hardware configuration, case selection and configuration, case execution, result evaluation, test report submission, and test log recording of the automated test. This will be discussed in detail later.

[0086] Fourth, the hardware board obtains the test code by loading the image or executable program of the image or executable program download module;

[0087] Specifically, if the download module stores an image, the image is remotely loaded to the board through the network port to run the embedded operating system. If the download module stores an executable program, the local embedded operating system is run first, and then the executable program is remotely loaded to the board to run. The hardware board is connected to the test host computer through peripherals such as the network port and serial port, and interacts with the automated test control module.

[0088] The automated test control module is the core module of the automated test of the present invention. It can be started by the remote control module, that is, access the continuous integration process through the remote control mode, and can also be used as an independent module, that is, stand-alone mode, to perform specific automated tests. Figure 1 As shown, the automated test control module includes several sub-units: a hardware interface configuration unit, a test case configuration unit, a test case execution unit, a test result evaluation unit, a log unit, and a test report reporting unit.

[0089] Hardware interface configuration unit: This unit is responsible for providing an interface to configure the parameters of the hardware board under test, including the board peripherals, board network configuration, and the operating system running on the board; after the configuration is completed, an ini format hardware configuration file is automatically generated and stored in the code version control tool. This file should be generated before starting the automated test.

[0090] Specifically, the board peripherals include: device type, device path, and device quantity. The board network configuration includes: board IP address, peer IP address, and interface name. The board operating system includes: system version, 32-bit or 64-bit, and multi-core operation mode.

[0091] Test case configuration unit: This unit pre-configures all test actions to be performed by automated testing on the interface. This should be completed before the automated testing begins. Existing test cases can be directly checked on the interface. Test cases that need to be added or modified can be added or modified on the interface. The test actions that need to be configured include:

[0092] Test commands to be sent to the embedded operating system and the method of sending commands, such as sending through the serial port, sending through Telnet, and sending through the TCP client / server mode; Test commands executed by the host computer, such as ping, ftp upload and download file commands, etc.; Interaction with the test host computer or other test boards, including: network transmission and reception with the test host computer, bus protocol communication with the bus protocol slave station device, system startup time timing on the test host computer, random power failure of the power switch on the test host computer to create an abnormal power failure scenario to test reliability, control the on and off of the USB interface power to create a USB repeated plug-in scenario to test the reliability of the USB driver, network communication with the test board to test network performance, etc.;

[0093] Whether to call other external software of the test host computer as accompanying test software, such as starting a CAN transceiver software to perform CAN protocol communication test with the board, starting an FTP client to upload and download files with the board FTP server; test case timeout, if no result is received after the timeout, force the system to restart and start the next test; whether to restart and start the next test after the test is completed to prevent the test from polluting the system environment and affecting other tests; the delay time between issuing consecutive commands; the number of times a single test case needs to be executed in a loop; check the regular expression statement output by the embedded operating system test; test result evaluation criteria, etc.

[0094] Test case execution unit: The test case execution unit framework is as follows Figure 2 As shown, it can control the interaction test between one hardware board, multiple hardware boards, and between a hardware board and a test host computer. The test case execution unit includes three sub-units: command issuing sub-unit, program-controlled switch control sub-unit, and external program calling sub-unit.

[0095] First, the command issuing subunit mainly issues test commands to the board under test and the accompanying board under test, and the host computer can connect to the network port and serial port of the board under test and the accompanying board under test, and issue commands to the embedded operating system and obtain the output results of the embedded operating system through the serial port, Telnet protocol, TCP client / server, etc. The command issuing subunit controls the order of command issuance, the interval time between two commands, the number of times the same command is repeatedly executed, and the order of sending commands to multiple hardware boards.

[0096] Second, the programmable switch control subunit mainly controls the switching of the power supply of the hardware board and the automatic random or timed plug-in test of the multi-channel hot-swappable device interface. For the first time to power on the board after the startup test, when an abnormality occurs during the test and causes a timeout, and when some tests pollute the environment and need to be restarted for a new test, the programmable switch control subunit can be relied on to control the power on and off restart of the board. For the reliability and stability tests usually required by high-reliability embedded operating systems, such as abnormal power-off copy machine tests, repeated plug-in and unplug device tests, repeated restart tests, etc., which are difficult to complete manually, they can be achieved by relying on the programmable switch control subunit. The multi-channel hot-swappable device interface includes: USB, network port.

[0097] Third, the external program calling subunit is mainly used for some tests that require calling the external program of the test host computer to interact with the embedded operating system, such as calling a third-party CAN communication program to communicate with the CAN port of the embedded operating system using the standard CAN protocol to test the function of the CAN driver, calling the network test software of the test host computer to communicate with the embedded operating system using the network protocol to test the network function, etc. The external program calling subunit does not interfere with the execution of the external program, but is only responsible for starting the external program and receiving the execution results of the external program, and then evaluating the received results. This requires that the external program called is simple to execute and has clear results. Complex external programs are not suitable for this process.

[0098] The above three sub-units are not performed independently, but are completed together under the overall scheduling of the test case execution unit according to the configuration made by the test case configuration unit. For example, a network communication abnormal power-off stress test will first call the command issuing sub-unit to issue a network communication command, and then start the external network test software of the test host computer to communicate with the embedded operating system. During the communication, the test host computer randomly calls the programmable switch control sub-unit within a certain time range to power off the board under test, and then powers on the board under test to resume network communication. Repeat the above steps, and comprehensively judge whether an abnormal situation occurs based on the results returned by the embedded operating system and the test host computer software.

[0099] According to an automated testing method for an embedded operating system provided by the present invention, an automated testing system based on the embedded operating system is implemented, comprising:

[0100] A new test project is created through the hardware interface configuration unit of the automated test control module, and the hardware parameters of the test project are configured to generate a configuration file; the configuration file is uploaded to the code version control tool of the remote server; the test case configuration unit of the automated test control module generates a test case configuration file by checking existing test cases and adding new test cases in the interface;

[0101] The remote control module triggers the test case construction module to start case compilation and construction, and the test case construction module generates an image or executable program according to the test code and the configuration file, and stores it in the image or executable program download module to wait for the hardware device to download;

[0102] The remote control module is used to trigger the test case execution unit of the automated test control module to start the automated test, and based on the test case configuration file, the tested board, the accompanying test board and the test host computer are controlled to load and start the image or executable program, thereby enabling the test result evaluation unit to generate the test result; the test case execution unit is used to enable the log unit to record the operation log during the test execution;

[0103] The code version control tools include Git and SVN;

[0104] According to the test result, the test report reporting unit generates a test report and then uploads the test report to the remote control module of the remote server, pushes it to the corresponding personnel, and ends.

[0105] Specifically, for some high-reliability embedded operating systems, console command interaction is not supported and commands cannot be issued. The TCP client / server command issuing process provided by the present invention is as follows: Figure 3 shown.

[0106] Step 1: First, configure a list of test command names that may be executed in the high-reliability embedded operating system, and create a TCP server to wait for receiving commands;

[0107] Step 2: The command issuing subunit creates a TCP client, sends the test command name to the TCP server, and then waits for the test result;

[0108] Step 3: The TCP server queries the test command list. If the test command is not found, an error is returned. If the test command is found, the test command is executed.

[0109] Step 4: The execution result of the test command is returned to the test result evaluation unit through the serial port, network, etc., and the test is completed.

[0110] Log unit: The log unit records three types of logs: the log of the automated test control module's own operation, the log formed by obtaining all outputs of the embedded operating system, and the log formed by obtaining the output of external program results.

[0111] 1. The log of the automated test control module itself includes the startup and initialization of the module itself, the control record of the programmable switch, the commands issued, the record of the test result evaluation, etc., which can be used as auxiliary information for locating problems after the test is completed.

[0112] 2. Obtain the logs formed by all outputs of the embedded operating system and store them by date. You can choose whether to filter out the printouts during the hardware boot period. This is the main source of information to assist in locating problems after the test is completed.

[0113] 3. The log formed by obtaining the standard output of the external program is the standard output result of the external program collected after the automated test control module starts the external program of the test host computer, such as the result log output after starting the standard CAN transceiver software to communicate with the embedded operating system.

[0114] Test result evaluation unit: This unit evaluates the results output by the embedded operating system based on the test case configuration file generated by the test case configuration unit. For most test cases, the test will directly output "PASS" if it passes. The test result evaluation unit only needs to determine whether the output contains the keyword "PASS" or "FAIL" to collect the results. For a small number of test cases, further judgment is required in the test result evaluation unit, including:

[0115] Performance test: Some performance values ​​are related to hardware device parameters. Therefore, the test result evaluation unit determines whether the performance value returned by the embedded operating system reaches the qualified threshold according to the performance qualified threshold set in the test case configuration file. If the result is yes, it passes; if the result is no, it fails.

[0116] Interaction testing between multiple devices: For interaction testing between embedded operating systems of different devices, and between embedded operating systems and test host computers, it is difficult to directly judge the test results only in the system under test. The test result evaluation unit can make a comprehensive judgment on the outputs of multiple parties based on the test case configuration file and give a result on whether the test has passed.

[0117] For example, for the abnormal disconnection test of the TCP connection between the tested board and the accompanying tested board, after determining that the TCP port of the accompanying tested board has been closed, determine whether the TCP port of the tested board is closed accordingly and the port can be released. If the result is yes, the test passes; if the result is no, the test fails.

[0118] Tests that require observation of printed results: For a very small number of test cases, direct results cannot be given, and it is necessary to observe the keywords or output patterns in the output to make judgments. The test result evaluation unit adds regular expressions to these test cases to judge the output and identify the test results.

[0119] For example, to determine whether the dynamic loading function successfully loads the "my_module" module into the operating system, first execute the command: module_info, and then use the regular expression ".*my_module.*" to check whether the module is loaded successfully.

[0120] After the test is completed, the test result evaluation unit summarizes all test cases and their corresponding results into a list and displays it on the interface. Clicking on a test case will locate the output interface corresponding to the case, making it convenient to view the test site. If it is a stand-alone mode, the list can be used as the test result output, and the test results can be exported.

[0121] Test report reporting unit: This unit counts and analyzes the test results summarized by the test result evaluation unit, and counts the total number of test cases, the number of passed cases, the number of failed cases, the defect rate and other data. If it is a regression test, count the number of regression test cases, the number of passed cases, the regression completion rate, etc., and then record information such as the tester, test time, test environment, and number of test rounds, and form a test report to report to the remote server. After receiving the test report, the remote server pushes it to relevant personnel as needed, and the test is completed.

[0122] The present invention is applicable to the automated testing of embedded operating systems, especially embedded operating systems, and has obvious advantages in automated testing of domestic embedded systems with diverse hardware, diverse BSP drivers and bus protocol support, and not many ready-made automated test cases. The remote control mode is applicable to automatically triggering version testing and collecting test reports in the continuous integration process, and the stand-alone mode is applicable to module testing within a defined range, reproducing problems in specific use cases, and certain reliability tests, abnormal tests, copy tests, or stress tests. The test code and test case configurations can be reused on different hardware platforms. When testing a new hardware platform, it is only necessary to configure the hardware parameters and check the required use cases.

[0123] Implementation - Remote Control Mode Example:

[0124] Hardware environment: Two remote servers, namely Jenkins server and Git server, one build machine, and one test host computer. All the above hardware are in the same local area network; one hardware board under test, namely the tested board, which is connected to the test host computer through the serial port and network port, and exchanges information such as command issuance and result output through the serial port protocol.

[0125] Software deployment: The code version control tool runs on the Git server, the remote control module runs on the Jenkins server, the test case building module and the image or executable program download module run on the building machine, the automated test control module runs on the test host computer, and the high-reliability embedded operating system under test runs on the board under test.

[0126] An automated test step is as follows Figure 4 As shown, the description is as follows:

[0127] First, create a new test project in the automated test control module interface, and configure the hardware parameters of the project through the interface, including serial port information, network port information, bus device information, and block device information, and generate a hardware configuration file Board_Config.ini and upload it to the Git server;

[0128] Specifically, the serial port information includes: serial port number, baud rate, data bit and stop bit; the network port information includes: network port ID, test host computer IP, development board IP and network card name; bus device information includes: CAN port, GPIO, EtherCAT master station, etc.; the block device information includes: SATA, eMMC, SD card, etc.;

[0129] Secondly, configure the program-controlled switch unit, set the ports of the relays of hot-swappable devices such as the control power supply, USB port, and network port, and generate the program-controlled switch control file Realy_Config.ini.

[0130] Third, select the tests to be performed in the test project from the existing test set to form a test case configuration file, namely Project_Config.ini. This example configures the serial port to send commands and sets the serial port ID. The timeout of the test case is set to 2 minutes and the operating system is restarted after the test case is executed. Use regular expressions to check the results returned by the embedded operating system.

[0131] For example, the regular expression for checking whether ping is successful is: ^.*4 packets transmitted, 4 packets received, 0% packet loss.*round-trip\\(ms\\)min / avg / max=0 / 0 / 0.*$; the regular expression for checking whether the watchdog is restarted after reset is: .*P:power on...*.

[0132] Fourth, log in to the Jenkins server and select the project number that triggers the test project.

[0133] Fifth, after being triggered, the build machine is automatically enabled to execute the build script, pull the code from the Git server, download and install the cross-compilation environment from the storage location of the compilation environment, and compile the test case. In this example, the test case is selected to be linked with the system to generate a system image with the test case.

[0134] Sixth, the generated system image is stored in the image or executable program download module, waiting to be loaded;

[0135] Seventh, the test host computer continuously runs a program and waits for the signal to start the test. After the Jenkins server obtains the signal that the compilation is completed, it automatically sends a signal to the test host computer remotely to start the test and informs it of the test items to be executed. The test host computer starts the automated test control module to start the test, and then Jenkins starts waiting for the test results to be returned. If the result is not received after the set time, Jenkins sends a test termination signal to the test host computer and exits the waiting state.

[0136] Eighth, the automated test control module starts to execute the test according to the test case configuration file of the test project. After the board is powered on, the board will remotely load the image file stored in the TFTP server of the image or executable program download module, start the embedded operating system, and then the automated test control module will issue commands, start other software auxiliary tests of the test host computer, control the network / bus communication between the board under test and other auxiliary hardware devices, control the restart of the board under test, control the power on and off of the accompanying test board, and issue commands, etc. If the Jinkens test termination signal is received during this period, the automated test will be forced to exit;

[0137] Ninth, each test case will evaluate the test results after obtaining the result output of the hardware board, and finally output the test case result list on the interface. The results are of three types: passed, failed, and unknown.

[0138] Tenth, after all the tests are completed, the test results are analyzed and statistically generated to generate a test report, which includes the total number of test cases, the number of passed cases, the number of failed cases, the defect rate, the regression completion rate and other test case execution statistics, as well as test activity information such as testers, test time, test environment, and number of test rounds. The automated test control module sends it to the Jenkins server, and the Jenkins server pushes the results, and the test is completed.

[0139] Implementation - Single machine mode example:

[0140] Hardware environment: a test host computer; a hardware board under test, i.e. the tested board; the tested hardware board is connected to the test host computer through the serial port and network port, and the serial port protocol is used to send commands, output results and other information exchanges.

[0141] Software deployment: The test host computer runs the test code construction environment and automated test control module; the hardware equipment under test runs a highly reliable embedded operating system.

[0142] First, create a new test project, select the tests to be performed in the project from the existing test set, and form a test case configuration file.

[0143] Second, compile the test case in the test host computer compilation environment. In this example, the test case is selected and compiled with the system to generate a system image of the test case to be tested;

[0144] Third, store the generated system image on the TFTP server and wait for remote loading;

[0145] Fourthly, the automated test control module starts to execute the test according to the test case configuration file of the test project. After the board is powered on, the board will remotely load the image file of the TFTP server, start the embedded operating system, and then the automated test control module will issue commands, start other software auxiliary tests of the test host computer, control the network / bus communication between the board under test and other auxiliary hardware devices, control the restart of the board under test, control the power on and off of the accompanying test board, and issue commands, etc. If you want to interrupt the test during this period, you can click the forced exit button on the interface to exit the automated test;

[0146] Fifth, each test case will evaluate the test results after obtaining the result output of the hardware board. After all tests are completed, the test results will be analyzed and statistically generated to generate a test report, and the tester can view the test report on the interface.

[0147] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0148] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0149] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An automated testing system for an embedded operating system, characterized in that: include: Remote server, builder, test host computer and hardware equipment; the hardware equipment includes a board under test and a companion test board; the board under test and the companion test board are installed with an embedded operating system; the test host computer is used to test pre-configuration and execute automated test tasks; The remote server is connected to the test host computer, and the test host computer is connected to the hardware device; The remote server is connected to the build machine.

2. The automated testing system for an embedded operating system according to claim 1, characterized in that: The test host computer is installed with an automated test control module; the automated test control module is used to configure hardware parameters and test cases, perform automated tests, and generate and upload test reports; The remote server is installed with a remote control module and a code version control tool; the remote control module is used to trigger the builder to start building, and then trigger the automated test control module to work and collect test reports; The code version control tool is used to provide configuration files and test codes; The construction machine is installed with a test case construction module and an image or executable program download module; the test case construction module is used to obtain the configuration file and the test code, and generate a system image or executable program; the image or executable program download module is used to download the system image or executable program to the tested board and the accompanying test board of the hardware device.

3. The automated testing system for an embedded operating system according to claim 2, characterized in that: The automated test control module includes: a hardware interface configuration unit, a test case configuration unit, a test case execution unit, a test result evaluation unit, a log unit and a test report reporting unit; The hardware interface configuration unit is used to configure the parameters of the board and generate a configuration file; The test case configuration unit is connected to the hardware interface configuration unit; the test case configuration unit is used to configure the test action of the automated test; The test case configuration unit is connected to the test case execution unit; the test case execution unit is used to schedule the test actions of the hardware device, the test host computer and other auxiliary hardware devices according to the configuration of the test case configuration unit; the other auxiliary hardware devices include: a switch, a bus protocol analyzer, an industrial field bus slave device and an oscilloscope; The test case execution unit is connected to the log unit; the log unit is used to record key logs in the process of executing the automated test; the key logs are the operation logs of the automated test control module, the result output after the embedded operating system executes the test command, and the result output obtained after the test case execution unit calls the external program; The test case execution unit is connected to the test result evaluation unit; the test result evaluation unit is used to evaluate the test result of the test case execution unit; The test result evaluation unit is connected to the test report reporting unit; the test report reporting unit is used to generate a test report by performing statistics and analysis on the test results, and report the test report to the remote control module of the remote server.

4. The automated testing system for an embedded operating system according to claim 3, characterized in that: The test case execution unit includes: a command issuing subunit, a program-controlled switch control subunit, and an external program calling subunit; The command issuing subunit is used to issue test commands to the tested board and the accompanying tested board of the hardware device; The program-controlled switch control subunit is used to control the power switch according to the test command; The external program calling subunit is used to call the external program of the test host computer and obtain the result output of the interaction between the external program and the embedded operating system.

5. The automated testing system for an embedded operating system according to claim 4, characterized in that: The hardware device also includes a program-controlled switch; The program-controlled switch control subunit generates a program-controlled switch control file, and controls the program-controlled switch according to the program-controlled switch control file to switch the power supply.

6. The automated testing system for an embedded operating system according to claim 4, characterized in that: The test case configuration unit configures the test actions, including: test commands, interval actions between two consecutive commands, interaction actions with the test host computer and other auxiliary hardware devices, the sending method of the test command and the evaluation criteria of the test results; The test command includes: a test command sent to the embedded operating system of the board under test, a test command sent to the embedded operating system of the accompanying test board, and an accompanying test command executed by the test host computer; The interval action between two consecutive commands includes restarting to execute the next command, not restarting but executing the next command after a fixed interval, and immediately executing the next command. The interactive actions include: starting timing, bus protocol communication, network transmission and reception, and power switch; Preferably, the sending mode of the command sending subunit test command includes: sending via serial port, sending via Telnet and sending via TCP client / server; The TCP client / server mode of sending is suitable for a high-reliability embedded system without a console, and is implemented based on the automated testing system of the embedded operating system described in claims 1 to 5, wherein the embedded operating system is instructed to create a TCP server and wait for receiving a test command, and the command issuing subunit is instructed to create a TCP client to issue a test command, and the TCP server is instructed to query a command list to execute the test command and obtain a test result; the TCP server is instructed to send the test result, and the TCP client obtains the test result.

7. An automated testing method for an embedded operating system, characterized in that: An automated testing system for an embedded operating system according to any one of claims 1 to 6 is implemented, comprising: A new test project is created through the hardware interface configuration unit of the automated test control module, and the hardware parameters of the test project are configured to generate a configuration file; the configuration file is uploaded to the code version control tool of the remote server; the test case configuration unit of the automated test control module generates a test case configuration file by checking existing test cases and adding new test cases in the interface; The remote control module triggers the test case construction module to start case compilation and construction, and the test case construction module generates an image or executable program according to the test code and the configuration file, and stores it in the image or executable program download module to wait for the hardware device to download; The remote control module is used to trigger the test case execution unit of the automated test control module to start the automated test, and based on the test case configuration file, the tested board, the accompanying test board and the test host computer are controlled to load and start the image or executable program, thereby enabling the test result evaluation unit to generate the test result; the test case execution unit is used to enable the log unit to record the operation log during the test execution; According to the test result, the test report reporting unit generates a test report and then uploads the test report to the remote control module of the remote server, pushes it to the corresponding personnel, and ends.

8. The automated testing method for an embedded operating system according to claim 7, characterized in that: The triggering method of the automated test includes triggering the remote control module or directly triggering the automated test control module itself; The code version control tools include Git and SVN; The hardware parameters of the test items include: the board peripherals, the board network configuration and the operating system running on the board; The board peripherals include: block device path, device quantity and device type; The board network configuration includes: board IP address, peer IP address and interface name; The operating system running on the board includes: system version, 32-bit or 64-bit, and multi-core operation mode.

9. The automated testing method for an embedded operating system according to claim 8, characterized in that: The test case configuration unit selects a serial port mode, a network mode or a TCP client / server mode to send a test command; and sets a timeout threshold of the test case to 2 minutes; Determine whether the test case execution time exceeds the timeout threshold. If yes, restart the hardware device to continue the next test; if no, do not process.

10. The automated testing method for an embedded operating system according to claim 9, characterized in that: The test report includes: the number of test cases, the number of passed cases, the number of failed cases, the defect rate and the regression completion rate.

Images