A real-time automated testing method for Preempt_RT kernel

Abstract

A method for automated real-time testing of a Preempt_RT kernel comprises: pre-configuring a central control node with a Preempt_RT kernel image package and a test tool set to complete environmental preparation; deploying the image package of the Preempt_RT kernel to be tested and remotely configuring the environment on a device under test through the central control node; generating a test matrix for the device under test according to a test strategy, calling a test tool set to perform a test on the device under test according to the test matrix, and obtaining a test result; analyzing the test result through the central control node to obtain key indicators of the Preempt_RT kernel to be tested, and further determining whether the Preempt_RT kernel to be tested has a performance improvement. The present invention can automatically complete the entire process from environmental preparation, kernel deployment, test execution, parameter tuning, and result analysis, thereby achieving rapid and accurate evaluation of changes in the real-time performance of the Preempt_RT kernel.

Description

Technical Field

[0001] The present invention relates to the technical field of kernel automated testing, and in particular to a method for automated real-time testing of a Preempt_RT kernel. Background Art

[0002] Real-time operating systems (RTOSs) play a vital role in fields with extremely high requirements for time determinism, such as aerospace, industrial automation, automotive electronics, financial trading, and communication base stations. By integrating the Preempt_RT patch, the Linux kernel can significantly improve its preemption capabilities and interrupt response determinism, enabling it to achieve hard real-time or quasi-hard real-time characteristics, leading to its increasingly widespread application in these fields. However, the real-time performance of the Preempt_RT kernel is a global, systemic metric that is highly sensitive to subtle changes in kernel subsystems such as the scheduler, interrupt handling, memory management, and locking mechanisms. During the development, iteration, and maintenance of the Preempt_RT kernel or related drivers, any code modifications, configuration adjustments, or patch updates made by developers may inadvertently introduce negative impacts on global real-time performance.

[0003] Currently, common methods for verifying the real-time performance of the Preempt_RT kernel include: 1. Manual testing: Test engineers manually configure the test environment (e.g., selecting a CPU, setting kernel startup parameters, and installing test tools), run benchmark tools such as cyclictest (periodic task latency testing), oslat (operating system latency testing), and stress-ng (system stress testing), manually record metrics such as maximum latency and jitter under different loads and configurations, and compare them with baseline data. 2. Semi-automated scripting: Use a few scattered scripts to assist in executing some test commands or collecting results, but environment deployment, parameter combination, and result analysis and comparison often still require significant manual intervention. 3. General testing frameworks: For example, the Real-Time Test Suite (LTP-RT) in the Linux TestProject (LTP) can be used. However, its configuration, execution, and result analysis often require adaptation and customization, and its integration with other stress tests and parameter combinations is not highly automated.

[0004] Therefore, verifying the real-time performance of the Preempt_RT kernel suffers from the following drawbacks: 1. Inefficiency: Manual or semi-automated testing is time-consuming and labor-intensive, especially when covering multiple hardware platforms, kernel versions, configuration parameters, and stress scenarios. This leads to long testing cycles and fails to meet the needs of rapid development iterations. 2. Poor consistency: Manual operations are prone to errors. Test environment configurations (such as GRUB parameters, core binding policies, and background workloads) may vary slightly between testers and at different times, resulting in a lack of comparability in test results and difficulty in accurately assessing performance changes. 3. Insufficient coverage: Manual testing fails to systematically cover the impact of various parameter combinations (such as scheduling policies, CPU isolation, interrupt affinity, and different stress types and intensities) on real-time performance. 4. Complicated result analysis: Large amounts of raw test data (such as latency logs and statistical data) must be manually collated, analyzed, and compared, which is error-prone, inefficient, and difficult to quickly identify performance regressions. 5. Lag in feedback: Low testing efficiency prevents performance issues from being discovered early in development, increasing the cost and risk of later remediation. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to solve the technical problems of low efficiency, poor consistency and insufficient coverage of Preempt_RT kernel real-time testing.

[0006] The present invention provides a method for automatically testing the real-time performance of a Preempt_RT kernel, comprising the following steps:

[0007] S1, pre-configuring the central control node according to preset configuration items and receiving test task requests through the central control node; the preset configuration items include the image package of the Preempt_RT kernel to be tested, the test tool set and the baseline result database;

[0008] S2, deploying the image package of the Preempt_RT kernel to be tested to the device under test through the central control node; ensuring that the device under test is in a testable state;

[0009] S3, remotely configure the environment of the device under test through the central control node; generate a test matrix for the device under test according to the test strategy through the central control node, call the test tool set to perform tests on the device under test according to the test matrix, and obtain the test results of the Preempt_RT kernel under test on the device under test;

[0010] S4, the test results are summarized and analyzed through the central control node to obtain the key indicators of the Preempt_RT core to be tested, and the key indicators are compared with the comparison baselines in the baseline result database to determine the change values ​​of the key indicators. According to the change values, it is determined whether there is performance improvement of the Preempt_RT core to be tested.

[0011] A further improvement of the present invention is that the preset configuration items also include: a test script and a configuration file template.

[0012] A further improvement of the present invention is that the image package of the Preempt_RT kernel to be tested includes: the image of the Preempt_RT kernel to be tested and related module files.

[0013] A further improvement of the present invention is that the specific process of deploying the image package of the Preempt_RT kernel to be tested to the device to be tested through the central control node is as follows:

[0014] Enable the central control node to remotely connect to the device under test through the network;

[0015] Transfer the image package of the Preempt_RT kernel to be tested from the central control node to the device to be tested;

[0016] Install the image package of the Preempt_RT kernel to be tested on the device to be tested;

[0017] Modify the boot loader configuration of the device under test according to the configuration file template through the test script to start the Preempt_RT kernel under test;

[0018] Restart the device under test, verify and record the device status and kernel version information of the Preempt_RT kernel under test.

[0019] A further improvement of the present invention is that the remote environment configuration of the device to be tested by the central control node includes:

[0020] At least one of modifying kernel startup parameters, setting CPU affinity, adjusting system parameters, shutting down or limiting non-essential background services, setting scheduling policies and priorities of test matrices, and loading kernel modules.

[0021] A further improvement of the present invention is that the specific process of calling the test tool set to perform the test on the device under test according to the test matrix and obtaining the test result of the Preempt_RT kernel under test on the device under test is as follows:

[0022] Traverse the test matrix to obtain multiple groups of multi-dimensional parameter combinations in the test matrix;

[0023] The test tool set is used to perform tests on the device under test concurrently or sequentially according to each set of multi-dimensional parameter combinations, and obtain sub-test results corresponding to each set of multi-dimensional parameter combinations;

[0024] All subtest results are combined to obtain the test result of the Preempt_RT kernel under test on the device under test.

[0025] A further improvement of the present invention is that S4 further includes:

[0026] S401: Generate a test report based on the test results and push the test report to the developer mailing list.

[0027] Beneficial effects of the present invention:

[0028] In terms of testing efficiency, the central control node remotely coordinates the devices under test to achieve end-to-end automation. The entire process from kernel deployment to result analysis requires no or very little human intervention. The testing time is shortened from days or weeks to hours or even shorter, which can adapt to agile development and CI / CD processes.

[0029] In terms of test consistency and repeatability, the test matrix generated by the central control node according to the test strategy includes multi-dimensional parameter combinations. The test process is automatically executed strictly according to the environmental configuration file template and parameter combination, eliminating variables and errors introduced by manual operations, ensuring that each test is carried out under precisely controlled conditions, and the test results are highly comparable and reliable.

[0030] In terms of test coverage and test depth, the combination of the test toolset and the test matrix can systematically and automatically cover a large number of parameter combinations and stress scenarios, more comprehensively evaluate the impact of different factors on real-time performance, and more easily discover hidden performance issues and regressions under boundary conditions.

[0031] In terms of test result analysis, the automated collection, aggregation, and comparative analysis of test results can quickly compare the current Preempt_RT kernel performance with the baseline, promptly discover and quantify changes in real-time indicators (such as maximum latency), and help developers quickly locate and fix problems.

[0032] In short, automated testing reduces the need for professional test engineers to manually perform repetitive tasks, lowering labor costs and enabling more frequent and efficient real-time testing. By providing efficient and reliable real-time verification, it helps continuously monitor and ensure the real-time performance of the Preempt_RT kernel during the development process, ultimately improving the stability and reliability of products based on this kernel. DETAILED DESCRIPTION

[0033] In order to have a further understanding of the technical solution and beneficial effects of the present invention, the technical solution of the present invention and the beneficial effects produced are described in detail below.

[0034] This paper proposes a method for automated real-time testing of Preempt_RT kernels, using a central control node (or test management server) to remotely coordinate and manage one or more devices under test (DUTs). This method provides an end-to-end, highly integrated automated solution specifically for real-time testing of Preempt_RT kernels, achieving significant improvements in efficiency, consistency, coverage, and results analysis. A central control node receives test tasks, stores test resources (kernels, scripts, configurations, baselines), schedules automated test processes, and aggregates and analyzes test results. At least one DUT runs the Preempt_RT kernel under test and is connected to the central control node via a network.

[0035] In one embodiment of the present invention, a Linux server serves as a central control node, and several target machines serve as devices under test (DUTs). It should be noted that despite the presence of multiple DUTs in this embodiment, the central control node coordinates and manages each DUT in a consistent manner, regardless of order. Therefore, the execution of the method steps is identical between a single DUT and multiple DUTs.

[0036] The present invention provides a method for automated testing of the real-time performance of the Preempt_RT kernel, which divides the process of automated testing of the real-time performance of the Preempt_RT kernel into four parts: automated environment preparation and configuration, automated kernel deployment and booting, automated test environment setup and parameterized execution, and automated result aggregation and comparative analysis.

[0037] Automated environment preparation and configuration is used to pre-configure and receive test task requests at the central control node.

[0038] Automated kernel deployment and booting is used to remotely deploy the Preempt_RT kernel to be tested to the DUT and remotely configure the kernel startup parameters and runtime environment of the DUT.

[0039] Automated test environment setup and parameterized execution are used to automatically execute real-time tests and optional stress tests on the DUT according to parameterized strategies.

[0040] Automated result aggregation and comparative analysis is used to collect test data and system status from the DUT, compare the collected data with the baseline, determine performance changes, and generate reports.

[0041] Specifically, the testing process is as follows:

[0042] S1, automated environment preparation and configuration, the specific process is as follows:

[0043] S101, configure the kernel image to be tested: prepare the image package of the Preempt_RT kernel to be tested (e.g., the .deb or .rpm package of the Preempt_RT kernel to be tested, or the kernel source code and compilation script) on the central control node;

[0044] S102, configure the test tool set: prepare the installation package or deployment script of the test tool set (rt-tests) on the central control node. The test tool set may include: standard real-time testing tools (such as cyclictest, oslat, and hwlatdetect) and stress generation tools (such as stress-ng, iperf, dd, or LTP-RT suite).

[0045] S103, configuring test scripts: installing and configuring automation tools, such as Python scripts, on the central control node for logic control and data processing.

[0046] S104, configuring a configuration file template: prepare a GRUB configuration file template on the central control node, including placeholders for setting isolcpus, nohz_full, and rcu_nocbs.

[0047] S105: Establish a baseline result database to store test configurations, raw results, and analyzed metrics, and set a comparison baseline. Setting the comparison baseline involves selecting a reference kernel version (which can be the last stable version), running a complete test process on the target hardware (which can be a specific stable hardware platform), and storing the results in the baseline result database as the baseline.

[0048] S106 , accepting a test task request, which includes kernel version information of the Preempt_RT kernel to be tested, information about the device to be tested, and a test strategy, where the test strategy includes a test set, a parameter range, and a comparison baseline.

[0049] S2, automated kernel deployment and booting, the specific process is:

[0050] S201, when a test task is triggered (eg, when code is submitted to a specific branch, the test task is triggered), automatic kernel deployment and booting are performed.

[0051] S202 , enabling the central control node to remotely connect to a designated DUT (i.e., the device under test in this test process; if there are multiple DUTs, connect to each DUT and execute subsequent tasks separately) through the network.

[0052] S203, when the central control node is remotely connected to the DUT, the following operations are automatically performed.

[0053] S204: Check the status of the DUT. Securely transfer the image package of the Preempt_RT kernel to be tested from the central control node to the DUT via SCP or similar methods.

[0054] S205 , executing a command (such as the dpkg -i or rpm -ivh command) on the DUT to install the image package of the Preempt_RT kernel to be tested.

[0055] S206, dynamically modify GRUB configuration: According to the test strategy (such as isolating CPUs 1-3), the test script automatically reads the GRUB configuration file template, locates the kernel boot line, adds or modifies parameters such as isolcpus=1-3, nohz_full=1-3, and rcu_nocbs=1-3, and executes update-grub or an equivalent command to update the boot menu.

[0056] S207, execute the reboot command to restart the DUT.

[0057] S208, after the DUT is restarted, reconnect the central control node and the DUT via SSH, execute uname -a to confirm that the Preempt_RT kernel to be tested has been successfully loaded, and record the version information of the Preempt_RT kernel to be tested. At this point, the DUT is in a testable state.

[0058] S3, automated test environment setup and parameterized execution, the specific process is as follows:

[0059] S301 , enabling the central control node to continue to remotely connect to the DUT via the network and execute subsequent steps.

[0060] S302, the process of configuring the runtime environment of the DUT through the central control node is as follows:

[0061] (1) Set CPU affinity according to the test strategy. For example, use the taskset command to bind the test process (cyclictest) to be run to a specified isolated CPU core (such as CPU 1).

[0062] (2) Move interrupts that may interfere away from the isolated CPU core, for example, by modifying / proc / irq / / smp_affinity implementation;

[0063] (3) Set the scheduling policy and priority of the real-time task, for example: chrt -f -p 99 <pid>;

[0064] (4) Load specific kernel modules or adjust sysctl parameters as needed.

[0065] It should be noted that in this step S302, the central control node automatically configures the runtime environment of the device under test according to the preset or task-specified test strategy, including but not limited to: modifying kernel boot parameters (through GRUB or other mechanisms, such as setting isolcpus, nohz_full, rcu_nocbs, etc. for CPU isolation and tickless configuration), setting CPU affinity (binding test tasks and interrupt service routines to specific CPU cores), adjusting system sysctl parameters, shutting down or restricting non-essential background services, etc. In this embodiment, one or several of the situations such as S302 (1)-(4) are expanded in detail to illustrate the environment configuration of the DUT during runtime. The kernel refers to the kernel of the Preempt_RT to be tested, which has been deployed to the DUT in S2.

[0066] S303, the process of parameterizing and concurrently testing the DUT through the central control node is as follows:

[0067] (1) Define the parameter space and test matrix. For example, the central control node generates a test matrix containing multi-dimensional parameter combinations according to the test strategy. An example of a test matrix (taking cyclictest for real-time testing and stress-ng for stress testing) is as follows:

[0068] Example 1 of parameter combinations in the test matrix: cyclictest parameters: duration, interval, priority, and CPU binding;

[0069] Example 1 of parameter combinations in the test matrix: stress-ng parameters: stress type (--cpu, --vm, --io, --net), number of instances, duration, and bound core CPU (usually bound to a non-isolated core, such as CPU 0).

[0070] It should be noted that this embodiment only provides one set of test matrices for real-time testing and one set of test matrices for stress testing. Test matrices with other test focuses can be generated as needed. Other test focuses may include: different scheduling strategies (FIFO, RR), different task priorities, different test loads (CPU-intensive, memory-intensive, I / O-intensive, network-intensive), different stress intensity levels, and different CPU core utilization strategies (isolated core vs. non-isolated core testing).

[0071] (2) System configuration: GRUB parameter combinations (different isolation strategies), scheduler settings. Corresponding to the dynamic modification of GRUB configuration in S206, GRUB is implemented here. Specifically, after replacing the kernel of the Preempt_RT to be tested in the DUT remote environment configuration, changing different configuration items will affect the test results of each parameter combination. For example, for a parameter combination in the test matrix, different GRUB parameter configurations with bound core isolation strategies will produce different test results. Therefore, through GRUB parameter configuration, more diverse tests can be achieved.

[0072] (3) Automated iterative execution, for example: enabling the central control node to automatically coordinate the test matrix through a cyclic task queue, and execute each parameter combination in the test matrix sequentially or in parallel. Examples of executing parameter combinations in the test matrix are as follows:

[0073] Example 1: cyclictest (prio=99, interval=1ms, on CPU 1) + stress-ng (--cpu 1, on CPU 0)

[0074] Example 2: cyclictest (prio=99, interval=1ms, on CPU 1) + stress-ng (--io 4, on CPU 0)

[0075] (4) Automatically record the standard output (including latency statistics) of the test tool (such as the cyclictest tool in the example of (3) above) during each test execution, or redirect the detailed latency data to a test result file for storage. Simultaneously monitor the system status and provide feedback.

[0076] During the execution process, S304 automatically collects log files from the DUT. After the test is complete, the DUT's test result files are obtained. The log files and test results contain information such as raw test data (such as latency statistics and histogram data from cyclictest), system logs, performance counters, and resource usage (CPU, memory, and I / O). All generated test result files and log files are transferred from the DUT to the central control node using the scp command.

[0077] S4, automated result aggregation and comparative analysis, the specific process is as follows:

[0078] S401: A Python script on the central control node automatically parses the test result file and log file output by the test tool (e.g., cyclictest) in step S304, extracts key metrics (e.g., maximum latency, average latency, duration data, standard deviation, and latency distribution), associates the key metrics with the test parameter combination (the parameter combination typically includes data such as kernel version, GRUB parameters, and stress type), and stores them in the baseline result database.

[0079] S402: Automated comparison and threshold determination are performed for this test using the information in the baseline database. The specific process is as follows:

[0080] (1) Use Python scripts to query the test results corresponding to the key indicators of this test and the comparison baseline corresponding to this test from the baseline result database;

[0081] (2) Determine the percentage change of the key indicator based on the test results of the baseline and the current test;

[0082] (3) Compare the percentage change of key indicators with the preset threshold, for example, if the maximum delay increases by more than 10%, it will be marked as a failure / warning;

[0083] (4) Distribute the test results corresponding to key indicators (such as latency distribution, etc.) and use statistical methods to compare whether there are significant changes in the distribution, and further determine whether there is a significant performance regression or improvement in the Preempt_RT kernel to be tested.

[0084] S403: Automatically generate a test report in HTML or PDF format based on the test result file and log file, clearly showing the comparison results, differences, and potential problems. The test report may include:

[0085] (1) Summary of the test environment (i.e., information about the Preempt_RT kernel to be tested, information about the DUT hardware, and configuration parameters used during the entire automated test process);

[0086] (2) Key indicator comparison table (i.e., the comparison results of the key indicators of this test with the comparison baseline, which can be displayed as the percentage difference between the two, and the key indicators can be further marked as passed or failed based on the comparison results of the difference percentage with the preset threshold);

[0087] (3) Delay histogram comparison chart;

[0088] (4) Details of the test cases that failed or changed significantly (i.e., the parameter combinations in the test matrix in S303 (3)).

[0089] S404: Push the test report generated in S403 (or the important results in the key indicators) to the developer's mailing list.

[0090] It should be noted that S1-S4 in this embodiment are all automated processes, which remotely coordinate and manage at least one device under test through a central control node, and can realize the entire process of automated deployment, automated configuration, automated test execution, automated result collection, and automated analysis and reporting. It provides an automated, standardized, efficient, and highly reliable Preempt_RT kernel real-time testing method, which significantly improves the test efficiency and the credibility of the results, and realizes rapid and accurate evaluation of the real-time performance changes of the Preempt_RT kernel.

[0091] Beneficial effects of the present invention:

[0092] In terms of testing efficiency, the central control node remotely coordinates the devices under test to achieve end-to-end automation. The entire process from kernel deployment to result analysis requires no or very little human intervention. The testing time is shortened from days or weeks to hours or even shorter, which can adapt to agile development and CI / CD processes.

[0093] In terms of test consistency and repeatability, the test matrix generated by the central control node according to the test strategy includes multi-dimensional parameter combinations. The test process is automatically executed strictly according to the environmental configuration file template and parameter combination, eliminating variables and errors introduced by manual operations, ensuring that each test is carried out under precisely controlled conditions, and the test results are highly comparable and reliable.

[0094] In terms of test coverage and test depth, the combination of the test toolset and the test matrix can systematically and automatically cover a large number of parameter combinations and stress scenarios, more comprehensively evaluate the impact of different factors on real-time performance, and more easily discover hidden performance issues and regressions under boundary conditions.

[0095] In terms of test result analysis, the automated collection, aggregation, and comparative analysis of test results can quickly compare the current Preempt_RT kernel performance with the baseline, promptly discover and quantify changes in real-time indicators (such as maximum latency), and help developers quickly locate and fix problems.

[0096] In terms of reducing testing costs and manpower dependence, automated testing reduces the need for professional test engineers to manually perform repetitive tasks, reduces labor costs, and enables real-time testing to be performed more frequently and efficiently.

[0097] In terms of promoting the quality improvement of the Preempt_RT kernel, by providing efficient and reliable real-time verification methods, it helps to continuously monitor and ensure the real-time performance of the Preempt_RT kernel during the development process, and ultimately improve the stability and reliability of products based on this kernel.

[0098] Although the present invention has been described using the above preferred embodiments, they are not intended to limit the scope of protection of the present invention. Any person skilled in the art may make various changes and modifications to the above embodiments without departing from the spirit and scope of the present invention. These changes and modifications are still within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.< / pid>

Claims

1. A Preempt_RT kernel real-time automated testing method, characterized in that: include: S1, pre-configuring the central control node according to preset configuration items and receiving test task requests through the central control node; the preset configuration items include the image package of the Preempt_RT kernel to be tested, the test tool set and the baseline result database; S2, deploying the image package of the Preempt_RT kernel to be tested to the device under test through the central control node; ensuring that the device under test is in a testable state; S3, remotely configure the environment of the device under test through the central control node; generate a test matrix for the device under test according to the test strategy through the central control node, call the test tool set to perform tests on the device under test according to the test matrix, and obtain the test results of the Preempt_RT kernel under test on the device under test; S4, the central control node summarizes and analyzes the test results to obtain key indicators of the Preempt_RT core to be tested, compares the key indicators with the comparison baseline in the baseline result database to determine the change value of the key indicators, and determines whether there is performance improvement of the Preempt_RT core to be tested based on the change value; Remote environment configuration of the device under test through the central control node includes: At least one of modifying kernel startup parameters, setting CPU affinity, adjusting system parameters, shutting down or limiting non-essential background services, setting scheduling policies and priorities for the test matrix, and loading kernel modules; The specific process of calling the test tool set to perform tests on the device under test according to the test matrix and obtaining the test results of the Preempt_RT kernel under test on the device under test is as follows: Traverse the test matrix to obtain multiple groups of multi-dimensional parameter combinations in the test matrix; The test tool set is used to perform tests on the device under test concurrently or sequentially according to each set of multi-dimensional parameter combinations, and obtain sub-test results corresponding to each set of multi-dimensional parameter combinations; All subtest results are combined to obtain the test result of the Preempt_RT kernel under test on the device under test.

2. A Preempt_RT kernel real-time automated testing method according to claim 1, characterized in that: The preset configuration items also include: Test script and configuration file templates.

3. A Preempt_RT kernel real-time automated testing method as claimed in claim 2, characterized in that: The image package of the Preempt_RT kernel to be tested includes: The image of the Preempt_RT kernel to be tested and related module files.

4. A Preempt_RT kernel real-time automated testing method as claimed in claim 3, characterized in that: The specific process of deploying the image package of the Preempt_RT kernel to the device under test through the central control node is as follows: Enable the central control node to remotely connect to the device under test through the network; Transfer the image package of the Preempt_RT kernel to be tested from the central control node to the device to be tested; Install the image package of the Preempt_RT kernel to be tested on the device to be tested; Modify the boot loader configuration of the device under test according to the configuration file template through the test script to start the Preempt_RT kernel under test; Restart the device under test, verify and record the device status and kernel version information of the Preempt_RT kernel under test.

5. A Preempt_RT kernel real-time automated testing method according to claim 1, characterized in that: The S4 also includes: S401: Generate a test report based on the test results and push the test report to the developer mailing list.