Embedded device rtos multi-test set automatic testing method

Abstract

The application relates to an automatic test method for embedded device RTOS multiple test sets, which comprises the following steps: preparing a compiling machine; building a jenkins, adding a compiling task on the jenkins, and triggering the compiling machine at a fixed time; adding an upper computer, triggering a test task when the test set is successfully compiled, and making the lower computer complete automatic test under the control instruction of the upper computer; a tftp server directory for storing a test sequence file and a test set file collection is arranged on the upper computer; a storage medium for storing a test serial number is connected to the lower computer, the test serial number is obtained from the storage medium, and corresponding test cases are found from the tftp server directory based on the test serial number to perform automatic test. The application realizes automatic test of multiple test sets compiled every day on the lower computer hardware through the jenkins to cooperatively control the compiling server and the lower test computer, and test results are archived after test, so that problems can be traced back in the future.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology for embedded devices, and more specifically to an automated testing method for multiple test sets of embedded device RTOS. Background Technology

[0002] Current RTOS automated testing solutions mostly use QEMU for automated testing of multiple test sets, and finally output test reports to determine the compatibility, functionality, and stability of the RTOS kernel code. QEMU allows for rapid simulation testing via a host PC, but the QEMU emulator cannot fully simulate real lower-level hardware. Currently, there is a lack of a solution that automatically controls the real lower-level hardware to perform centralized automated testing and code regression verification of multiple test sets generated daily.

[0003] In RTOS embedded operating systems, there are not many mature automated execution solutions available for download and use for lower-level hardware. Most test solutions need to be customized and rely on manual execution. Testers need to manually operate and execute test cases to verify the functionality, performance, and stability of various modules of the operating system.

[0004] The QEMU emulator cannot fully simulate real lower-level hardware. Code regression verification requires triggering test suite compilation and regression verification on the test suite daily or after each code commit. Manually controlling the embedded lower-level machine to perform regression testing on multiple test sets is time-consuming and labor-intensive. There is an urgent need for an automated method to control the lower-level machine to perform regression testing on multiple test sets. Simultaneously, the results of each regression test should be recorded to facilitate rapid problem localization and backtracking.

[0005] Most existing automated testing methods for embedded software are designed for embedded application software rather than automated testing of RTOS embedded operating systems. For test sets consisting of multiple test cases in embedded RTOS systems, a restart is required after the test cases are executed. It is not possible to load all test programs and execute the test cases in sequence on the lower-level development board. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an automated testing method for multiple test sets of RTOS in embedded devices, comprising the following steps:

[0007] Step S1: Prepare a compiler to generate a test suite, which includes multiple test cases;

[0008] Step S2: Set up Jenkins, add compilation tasks to Jenkins, and make the compilation machine trigger periodically;

[0009] Step S3: Add a host computer to Jenkins and connect the host computer to the slave development board. When the test set is successfully compiled on the compiler, Jenkins triggers the test task, so that the slave computer completes the automated test under the control instructions of the host computer.

[0010] The host computer has a tftpserver directory for storing test sequence files and test set files. The test sequence files are used to store the test number and test set file name corresponding to the test cases, and the test set file set is used to store all test cases.

[0011] The lower-level machine is connected to a storage medium that stores the serial number to be tested. The lower-level machine obtains the serial number to be tested from the storage medium and finds the test case corresponding to the serial number to be tested from the TFTP server directory based on the serial number to be tested for automated testing.

[0012] Step S3 includes:

[0013] Step S31: Set up the initial environment of the lower-level development board;

[0014] Step S32: Start the storage medium connected to the lower-level machine;

[0015] Step S33: Load the test sequence number stored in the storage medium into the environment variables of the lower-level machine, so that the lower-level machine knows the test sequence number corresponding to the test case to be executed in this round of testing, that is, the test sequence number of this round;

[0016] Step S34: The lower-level machine loads the test cases of the current round and the test numbers of the next round of test cases corresponding to the current round of test cases from the tftp server directory of the upper-level machine into the lower-level machine's memory according to the test sequence number of the current round.

[0017] Step S35: Write the next round test sequence number as the test sequence number into the storage medium.

[0018] Step S36: Execute the test cases for this round on the lower-level machine;

[0019] Step S37: After the current round of test cases is completed, the lower-level development board will be automatically restarted. The lower-level development board will then start executing from step S33 again to execute the next round of test cases until all test cases are completed.

[0020] The storage medium is an external storage medium independent of the lower-level machine.

[0021] The storage medium is a USB flash drive.

[0022] The test sequence number in the test sequence file is represented by case_num= It exists in the form of, in which, " " indicates the test sequence number;

[0023] Test case filenames begin with case= It exists in the form of, in which, " "" indicates the filename of the test case corresponding to the test number.

[0024] In step S35, when the lower-level machine loads the current round of test cases and the corresponding next round test sequence number from the tftp server directory of the upper-level machine into the lower-level machine's memory, it converts the next round test sequence number into little-endian binary format and writes it.

[0025] This also includes: when the host computer detects the test termination condition, ending the test, exiting the serial port connection, generating a timestamp, parsing the test log to generate a test report, saving the test report to a file named after the timestamp, and sending it back to Jenkins.

[0026] The test termination conditions correspond to two scenarios:

[0027] One method is to achieve the timeout period;

[0028] Another method is: In step S34, the lower-level machine loads the test case file name of the current test case from the tftp server directory of the upper-level machine, which is named "autotestend".

[0029] The host computer and the slave computer development board are connected via the serial port protocol.

[0030] When the test set is successfully compiled on the compiler, Jenkins categorizes the test set based on the different lower-level machines before triggering the test task.

[0031] This invention enables automated testing of multiple test suites, including unit tests and functional tests, for RTOS on an embedded lower-level machine. By using Jenkins to control the collaborative operation between the build server and the test lower-level machine, automated testing of multiple test suites compiled daily is achieved on the lower-level hardware. After testing, the results are archived for easy backtracking of issues. Attached Figure Description

[0032] Figure 1: Flowchart of the implementation of the embedded device RTOS multi-test set automated testing method of the present invention.

[0033] Figure 2 This invention provides an operational diagram of adding a host computer to Jenkins.

[0034] Figure 3 Another schematic diagram illustrating the addition of a host computer to Jenkins in this invention. Detailed Implementation

[0035] To gain a better understanding of the technical solution and beneficial effects of the present invention, the technical solution of the present invention and its beneficial effects are described in detail below with reference to the accompanying drawings.

[0036] To avoid interference between RTOS test suites, a reboot is often used after each test is completed. Testing RTOS on lower-level hardware (development boards of various architectures and manufacturers) typically involves loading the test suite using u-boot. To avoid intrusive modifications to u-boot, only u-boot's general methods for loading test suites can be used, and the hardware reboots after each test case is executed. After rebooting, u-boot is re-entered. Therefore, u-boot needs to know the next test case to be executed before rebooting, requiring the test sequence number to be stored on a medium. This invention uses a USB flash drive to store the test sequence number. By customizing u-boot's environment variables, each time the system boots into u-boot, the u-boot command prompt automatically loads the test sequence number from the USB flash drive, then downloads the corresponding test case from the host computer, loads it into memory, and executes the test case from memory.

[0037] To automate the testing of multiple test sets compiled daily, this invention introduces a collaborative solution between the Jenkins-controlled build server and the host test server. After the build server completes its daily compilation, the host test server is triggered via the Jenkins pipeline to begin a copying task. Once the test sets are copied, the host server connects to the serial port of the slave hardware via the expect interaction method, sends initial test commands, and controls the slave hardware to execute the test tasks. After the test is completed, the test logs in the serial port are analyzed to generate test results, which are then uploaded back to Jenkins for archiving.

[0038] lower combination Figure 1 A specific implementation of the present invention will be described in detail.

[0039] I. Preparatory Work

[0040] Taking the multi-test set test of rtems (an RTOS) as an example, the following steps 1-3 are included.

[0041] 1. Prepare a compiler to compile rtems so that it can generate testcases test sets.

[0042] 2. Set up Jenkins, add build tasks, and schedule the build process to run daily. After compilation, package the test suite.

[0043] 3. Adding a remote test machine (host machine Node) to Jenkins: In one specific implementation, a PC can be added as a node for deployment and testing. Please refer to [the relevant documentation / reference]. Figure 2 and Figure 3 As shown, the specific method is as follows: After clicking "create", open the configuration page. In the "Launch method" section, select "Launch agents via SSH", add Credentials credentials, and enter the username and password for the remote test machine. Define a label for this remote test machine as "dailybuild-test", which will be used later. Simultaneously, the remote test machine needs to be configured with an environment, most importantly, the JDK environment needs to be configured to match the JDK version used to start Jenkins. This is so that Jenkins can log in to the remote test machine (corresponding to...). Figure 1 The term "test host computer" (hereinafter referred to as "host computer") is used in this context.

[0044] II. Automated Test Execution Methods

[0045] 1. Create an automated test task node of type pipeline in Jenkins. Its configuration conditions are as follows: when the test suite is compiled successfully, this task node is triggered to execute. The host machine is found by the tag "dailybuild-test" defined for the host machine in Jenkins during the preparation process. The automated test task node triggers the execution of the test task on it to complete the test on the corresponding slave machine.

[0046] When multiple lower-level machines need to be tested simultaneously, multiple test task nodes can be defined in parallel mode in the pipeline. This method allows for the specification of test commands for different lower-level machines, avoiding issues such as chaotic serial port outputs and lost output logs.

[0047] 2. Connect the host computer to the slave development board. Depending on the type of slave, the slave development board may be an ARM development board or a MIPS development board. Both connect to the host computer via the serial port protocol and receive control commands from the host computer.

[0048] 3. Delete old files from the last run on the host computer to avoid occupying too much disk space.

[0049] 4. The host computer copies the latest stable branch compilation output, i.e., the test set, from the compiler. The test set is a series of test case-exe files. Each exe file runs a single test. After the test is completed, the lower computer development board will be restarted.

[0050] 5. Write and execute a test classification script on the host computer to classify the test set: Since the test set contains test sets from multiple slave computers (products of slave computers with different architectures), it is necessary to write a classification script to place the test sets for different slave computers in different folders.

[0051] 6. Because the test suite contains multiple test cases, and the lower-level development board restarts after each test case is executed, and the development board is unaware of which .exe file in the test suite it ran before restarting, it is necessary to generate test sequence files on the upper-level computer and store the test sequence files and the test suite file set in the tftpserver directory (corresponding to...). Figure 1 The "TFTP server sequence source" allows the lower-level development board to access these files via the TFTP protocol. The test sequence file stores the test number and filename corresponding to each test case, while the test suite file collection stores all test cases. The test sequence file, in conjunction with environment variables customized based on the development board's general u-boot environment, controls the lower-level machine to automatically execute tests, loading the next numbered .exe file upon restarting, until all .exe files have been tested.

[0052] The test sequence number in the test sequence file is represented by case_num= It exists in the form of " " indicates the test sequence number, and the test case file name begins with case= It exists in the form of " "" indicates the filename of the test case corresponding to the test number.

[0053] 7. Control the communication between the host computer and the slave computer, send initialization commands, and cause the slave computer to start executing test cases in a loop according to the test sequence.

[0054] Specifically, the host computer uses the expect script to execute the picocom program to connect to the serial port of the slave computer, automatically and interactively inputs and executes the customized run init environment variable command in the u-boot environment variable of the slave computer, sets the timeout, causes the slave computer development board to execute the initialization command, and cyclically executes the test cases starting from t1 of the test sequence, including the following steps 7-1 to 7-8.

[0055] 7-1. Set up the initial environment.

[0056] 7-2. Insert the storage medium into the lower-level machine. In this invention, a USB flash drive is preferred. The USB flash drive stores the test sequence number to be tested. That is, after each round of test cases is executed, the USB flash drive stores the test sequence number corresponding to the next round of test cases, which will be referred to as the next round test sequence number below.

[0057] 7-3. Boot the USB drive using the command usb start.

[0058] 7-4. The lower-level machine loads the contents of the file caseno.bin from the USB drive into its memory by executing the customized loadno environment variable command, and loads case_no=<test number> from the USB drive into the lower-level machine's u-boot environment variables by using the env import command. This allows the lower-level machine to know the test number corresponding to the test cases to be executed in this round of testing, hereinafter referred to as the test number of this round.

[0059] 7-5. The lower-level machine executes the load_seq environment variable command to load the test case file name, test case, and next round test number from the tftp server directory of the upper-level machine into the environment variables according to the current test sequence number.

[0060] 7-6. Execute saveno to write the next round of test sequence number to the caseno.bin file on the USB drive.

[0061] `saveno` contains two sub-steps: `updateno` and `writeno`. `updateno` writes the next round of test sequence number along with "case_no=", a total of 12 bytes, into memory. `writeno` writes these 12 bytes from memory into a file on the USB drive.

[0062] Specifically, `updateno` writes 12 bytes of content to memory. The first two bytes are "case_no=" (occupying exactly 8 bytes), and the last word (4 bytes) is the test sequence number, which has been converted to little-endian binary format. For example, assuming the next test sequence number is t2, it is specified as 0x00000032. This is because the u-boot commonly used on lower-level development boards cannot directly read the sequence number from the file content and convert it into an environment variable. Furthermore, the filename length is not fixed, and simply saving the next test filename to the USB drive does not provide a clear indication of the test's progress. The number of bytes occupied by the test sequence number is fixed; the number of bytes loaded is specified to ensure the successful execution of the `env import` command.

[0063] For example, assuming the test sequence number of the current test phase is t180, and the test sequence number of the next test phase is t181, then case_num needs to convert the test sequence number into a little-endian binary format byte:

[0064] case=pthread_exit_test.exe / / Specifies the test case file name

[0065] case_num=0x00313831 / / The sequence number of the next test case, which is 181 here. Converted to little-endian binary single-byte content, it is 0x00313831.

[0066] It should be noted that case_num= This is to test the sequence number file in the tftp_server directory. `case_no` is an intermediate variable.

[0067] `case_num={[binary number]}` is used in the `update_no` procedure. It is written into memory, and the entire process takes up exactly 12 bytes.

[0068] The result after writing:

[0069] The first two bytes, "case_no=", occupy 8 bytes.

[0070] The next byte, "0x00313831", occupies one byte.

[0071] These 12 bytes can be imported into the environment variable case_no=181 by importing the address 12 bytes in u-boot;

[0072] Then, load file number 181 using tftpboot t${case_no}.

[0073] 7-7. Execute the test cases for this round of testing on the lower-level machine using the run test command.

[0074] 7-8. The test program automatically restarts the lower-level development board. After restarting, the u-boot environment of the development board will automatically execute the command specified by bootcmd, and start executing from step 7-3 again to carry out the next round of test case execution until all test cases have been executed.

[0075] 8. When the host computer detects the test termination condition, it ends the test, exits by executing "send \x01\18" to send "ctrl A ctrl x", exits the serial port connection, generates a timestamp, and parses the test log to generate a test report (corresponding to...). Figure 1 The program "Generate a test report from the test results" and "Upload the report" save the test report to a file named with a timestamp and send it back to Jenkins.

[0076] During this process, the host computer records the output of picocom through the logfile of the expect script and records it in parallel to the test log (corresponding to...). Figure 1 The "Upload Development Board Test Logs" method effectively synchronizes the picocom serial port output in real time and prints the output synchronously to the Jenkins Console Output page. Using tee or other redirection methods may result in lost test logs and non-real-time output synchronization.

[0077] The test termination condition corresponds to two cases:

[0078] (1) The following test serial numbers and test case file names were detected:

[0079] case_num=0x00000031;

[0080] case=autotestend;

[0081] (2) The timeout period has been reached.

[0082] The environment variables for automated testing on the lower-level machine are cumbersome, while flash media has a short lifespan. If the environment variables are stored using the internal flash media within the u-boot, each time the test sequence number is saved, the entire flash data block needs to be refreshed using an erase-then-write method. If automated testing is performed once a day, this process may need to be executed hundreds of times, leading to flash damage. Replacing the flash requires specialized equipment and a certain level of operator skill, which is time-consuming and labor-intensive. Therefore, this invention uses an external USB flash drive to store and refresh test sequence numbers, and executes the tests using the u-boot environment of the lower-level machine development board. After each round of test cases is completed, the development board is restarted, which has the following advantages:

[0083] 1. Versatility. Embedded development board manufacturers typically provide u-boot. The development board will boot into u-boot upon power-up. The test suite (exe) is a bare-metal program that can be executed within the u-boot environment. To perform unit and functional testing on the RTOS, separate module test programs are often written. Therefore, restarting the development board after running one test case before running another avoids interference between different test cases.

[0084] 2. By storing the test sequence number on a USB flash drive, the lower-level development board can understand the test cases that need to be executed in the current test cycle after restarting. This can also extend the lifespan of the flash media inside the development board and avoid intrusive modifications to u-boot to adapt to automated testing.

[0085] In this invention, U-Boot is an open-source, cross-platform bootloader widely used in embedded systems. It is responsible for initializing the hardware, loading the operating system (such as the Linux kernel), and passing the necessary startup parameters.

[0086] This invention utilizes Jenkins to control the compiler, host test server, and slave test server in a collaborative automated testing process. It can fully automate testing in multi-test suites where the test uses u-boot as the initial platform and baremental class executables, represented by RTOS test cases, require restarts. The operation is simple and requires no manual intervention. Automated tests can be scheduled daily or triggered after each code merge, providing timely regression verification of code quality on real hardware. Test results are archived using Jenkins, facilitating problem backtracking.

[0087] Although the present invention has been described using the above preferred embodiments, it is not intended to limit the scope of protection of the present invention. Any changes and modifications made by those skilled in the art to the above embodiments without departing from the spirit and scope of the present invention shall still fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. An automated testing method for multiple test sets of RTOS for embedded devices, characterized in that, Includes the following steps: Step S1: Prepare a compiler to generate a test suite, which includes multiple test cases; Step S2: Set up Jenkins, add compilation tasks to Jenkins, and make the compilation machine trigger periodically; Step S3: Add a host computer to Jenkins and connect the host computer to the slave development board. When the test set is successfully compiled on the compiler, Jenkins triggers the test task, so that the slave computer completes the automated test under the control instructions of the host computer. The host computer has a TFTP server directory for storing test sequence files and test set files. The test sequence files are used to store the test sequence number and test set file name corresponding to the test cases, and the test set file set is used to store all test cases. The lower-level machine is connected to a storage medium that stores the serial number to be tested. The lower-level machine obtains the serial number to be tested from the storage medium and finds the test case corresponding to the serial number to be tested from the TFTP server directory based on the serial number to be tested for automated testing.

2. The automated testing method for multiple test sets of embedded device RTOS as described in claim 1, characterized in that, Step S3 includes: Step S31: Set up the initial environment of the lower-level development board; Step S32: Start the storage medium connected to the lower-level machine; Step S33: Load the test sequence number stored in the storage medium into the environment variables of the lower-level machine, so that the lower-level machine knows the test sequence number corresponding to the test case to be executed in this round of testing, that is, the test sequence number of this round; Step S34: The lower-level machine loads the test cases of the current round and the test numbers of the next round of test cases corresponding to the current round of test cases from the tftp server directory of the upper-level machine into the lower-level machine's memory according to the test sequence number of the current round. Step S35: Write the next round test sequence number as the test sequence number into the storage medium. Step S36: Execute the test cases for this round on the lower-level machine; Step S37: After the current round of test cases is completed, the lower-level development board will be automatically restarted. The lower-level development board will then start executing from step S33 again to execute the next round of test cases until all test cases are completed.

3. The automated testing method for multiple test sets of embedded device RTOS as described in claim 2, characterized in that: The storage medium is an external storage medium independent of the lower-level machine.

4. The automated testing method for multiple test sets of embedded device RTOS as described in claim 3, characterized in that: The storage medium is a USB flash drive.

5. The automated testing method for multiple test sets of embedded device RTOS as described in claim 2, characterized in that: The test sequence number in the test sequence file is in the format of case_num= It exists in the form of, in which, " indicates the test sequence number; Test case filenames begin with case= It exists in the form of, in which, "" indicates the filename of the test case corresponding to the test number.

6. The automated testing method for multiple test sets of embedded device RTOS as described in claim 5, characterized in that: In step S35, when the lower-level machine loads the current round of test cases and the corresponding next round test sequence number from the tftp server directory of the upper-level machine into the lower-level machine's memory, it converts the next round test sequence number into little-endian binary format and writes it.

7. The automated testing method for multiple test sets of embedded device RTOS as described in claim 5, characterized in that, Also includes: When the host computer detects the test termination condition, it ends the test, disconnects from the serial port connection, generates a timestamp, parses the test log to generate a test report, saves the test report to a file named after the timestamp, and sends it back to Jenkins.

8. The automated testing method for multiple test sets of embedded device RTOS as described in claim 7, characterized in that, The test termination condition corresponds to two cases: One method is to achieve the timeout period; Another method is: In step S34, the lower-level machine loads the test case file name of the current test case from the tftp server directory of the upper-level machine, which is named "autotestend".

9. The automated testing method for multiple test sets of embedded device RTOS as described in claim 1, characterized in that, The host computer and the slave computer development board are connected via the serial port protocol.

10. The automated testing method for multiple test sets of embedded device RTOS as described in claim 1, characterized in that: When the test set is successfully compiled on the compiler, Jenkins categorizes the test set based on the different lower-level machines before triggering the test task.

Images