PreemptRT kernel real-time performance automatic testing method

Abstract

The invention discloses a method for automatically testing the real-time performance of a PreemptRT kernel. The method comprises the following steps of: pre-configuring a mirror image package and a test tool set of the PreemptRT kernel for a central control node to complete environment preparation; mirror image package deployment and remote environment configuration of a to-be-tested PreemptRT kernel are carried out on the to-be-tested equipment through the central control node; generating a test matrix of the to-be-tested equipment according to the test strategy, and calling the test tool set to test the to-be-tested equipment according to the test matrix to obtain a test result; and analyzing the test result through the central control node to obtain a key index of the PreemptRT kernel to be tested, and further determining whether the performance of the PreemptRT kernel to be tested is improved or not. According to the invention, the whole process of environment preparation, kernel deployment, test execution, parameter tuning and result analysis can be automatically completed, and rapid and accurate evaluation of the real-time performance change of the PreemptRT kernel is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of kernel automated testing, and particularly relates to a method for automated testing of the real-time performance of the Preempt_RT kernel. Background Art

[0002] Real-time operating systems (RTOS) play a crucial role in fields with extremely high requirements for time determinacy, such as aerospace, industrial automation, automotive electronics, financial transactions, and communication base stations. By integrating the Preempt_RT patch, the Linux kernel can significantly improve its preemption ability and interrupt response determinacy, enabling it to have hard real-time or quasi-hard real-time characteristics, and thus has been increasingly widely used in the above fields. However, the real-time performance of the Preempt_RT kernel is a global and systematic indicator, and it is very sensitive to subtle changes in kernel subsystems (such as schedulers, interrupt handling, memory management, lock mechanisms, etc.). During the research, iteration, and maintenance of the Preempt_RT kernel or related drivers, any code modification, configuration adjustment, or patch update made by developers may inadvertently introduce negative impacts on the global real-time performance.

[0003] Currently, the common methods for verifying the real-time performance of the Preempt_RT kernel include: 1. Manual testing: Test engineers manually configure the test environment (such as selecting the CPU, setting kernel startup parameters, installing test tools), run benchmark test tools such as cyclictest (periodic task latency test), oslat (operating system latency test), stress-ng (system stress test), etc., manually record indicators such as the maximum latency and jitter under different loads and configurations, and perform manual comparison with the baseline data. 2. Semi-automated scripts: Use some scattered scripts to assist in executing some test commands or collecting results, but environment deployment, parameter combination, result analysis, and comparison usually still require a large amount of manual intervention. 3. General test frameworks: Such as using the real-time test suite (LTP-RT) in Linux TestProject (LTP), but its configuration, execution, and result analysis often require adaptation and customization, and the degree of automation in linkage with other stress tests and parameter combinations is not high.

[0004] Therefore, there are the following drawbacks in verifying the real-time performance of the Preempt_RT kernel: 1. Low efficiency: The manual or semi-automated test process is time-consuming and laborious. Especially when it is necessary to cover multiple hardware platforms, kernel versions, configuration parameters, and stress scenarios, the test cycle is long, which cannot meet the development needs of rapid iteration. 2. Poor consistency: Manual operations are prone to introducing errors. There may be subtle differences in the test environment configurations (such as GRUB parameters, core binding policies, background loads, etc.) of different testers and at different time points, resulting in the lack of comparability of test results and making it difficult to accurately judge performance changes. 3. Insufficient coverage: It is difficult for manual tests to systematically cover the impacts of various parameter combinations (such as scheduling policies, CPU isolation, interrupt affinity, different stress types and intensities) on real-time performance. 4. Complicated result analysis: A large amount of original test data (such as latency logs, statistical data) needs to be manually sorted, analyzed, and compared, which is prone to errors and has low efficiency, making it difficult to quickly locate performance regressions. 5. Delayed feedback: The low test efficiency results in the inability to detect performance problems in the early stage of development in a timely manner, increasing the cost and risk of later repairs. Summary of the Invention

[0005] In order to overcome the above defects, the present invention is proposed to solve the technical problems such as low efficiency, poor consistency, and insufficient coverage in the real-time testing of the Preempt_RT kernel.

[0006] The present invention provides an automated testing method for the real-time performance of the Preempt_RT kernel, including the following steps: S1, pre-configure the central control node according to preset configuration items, and receive test task requests through the central control node; the preset configuration items include the image package of the Preempt_RT kernel to be tested, the test tool set, and the baseline result database; S2, deploy the image package of the Preempt_RT kernel to be tested to the device to be tested through the central control node; determine that the device to be tested is in a testable state; S3, perform remote environment configuration on the device to be tested through the central control node; generate a test matrix for the device to be tested by the central control node according to the test strategy, and call the test tool set to execute tests on the device to be tested according to the test matrix to obtain the test results of the Preempt_RT kernel to be tested on the device to be tested; S4, summarize and analyze the test results through the central control node to obtain the key indicators of the Preempt_RT kernel to be tested, compare the key indicators with the comparison baselines in the baseline result database to determine the change values of the key indicators, and determine whether there is a performance improvement in the Preempt_RT kernel to be tested according to the change values.

[0007] A further improvement of the present invention is that the preset configuration items further include: test scripts and configuration file templates.

[0008] A further improvement of the present invention is that the image package of the to-be-tested Preempt_RT kernel includes: the image of the to-be-tested Preempt_RT kernel and related module files.

[0009] A further improvement of the present invention is that the specific process of deploying the image package of the to-be-tested Preempt_RT kernel to the to-be-tested device by the central control node is as follows: Enable the central control node to remotely connect to the to-be-tested device through the network; Transmit the image package of the to-be-tested Preempt_RT kernel from the central control node to the to-be-tested device; Install the image package of the to-be-tested Preempt_RT kernel on the to-be-tested device; Modify the boot loader configuration of the to-be-tested device according to the configuration file template through the test script for starting the to-be-tested Preempt_RT kernel; Restart the to-be-tested device, verify and record the status of the to-be-tested device and the kernel version information of the to-be-tested Preempt_RT kernel.

[0010] A further improvement of the present invention is that the remote environment configuration of the to-be-tested device by the central control node includes: Modifying at least one of kernel startup parameters, setting CPU affinity, adjusting system parameters, shutting down or restricting unnecessary background services, setting the scheduling policy and priority of the test matrix, and loading kernel modules.

[0011] A further improvement of the present invention is that the specific process of calling the test tool set to execute tests on the to-be-tested device according to the test matrix and obtaining the test results of the to-be-tested Preempt_RT kernel on the to-be-tested device is as follows: Traverse the test matrix to obtain multiple groups of multi-dimensional parameter combinations in the test matrix; Execute tests on the to-be-tested device concurrently or sequentially according to each group of multi-dimensional parameter combinations through the test tool set to obtain sub-test results corresponding to each group of multi-dimensional parameter combinations; Combine all sub-test results to obtain the test results of the to-be-tested Preempt_RT kernel on the to-be-tested device.

[0012] A further improvement of the present invention is that S4 further includes: S401, generating a test report according to the test results and pushing the test report to the developer email list.

[0013] Advantages of the present invention: In terms of test efficiency, the central control node remotely coordinates the devices under test to achieve end-to-end automation. From kernel deployment to result analysis, the whole process requires little or no manual intervention. The test time is shortened from several days or weeks to several hours or even shorter, which can adapt to agile development and CI / CD processes.

[0014] In terms of test consistency and repeatability, the test matrix generated by the central control node according to the test strategy contains multi-dimensional parameter combinations. The test process is automatically executed strictly in accordance with the environment configuration file template and parameter combinations, eliminating variables and errors introduced by manual operations, ensuring that each test is carried out under precisely controlled conditions, and the test results have high comparability and credibility.

[0015] In terms of test coverage and test depth, through the combination of the test toolset and the test matrix, a large number of parameter combinations and stress scenarios can be systematically and automatically covered, more comprehensively evaluating the impact of different factors on real-time performance, and making it easier to discover hidden performance problems and regressions under boundary conditions.

[0016] In terms of test result analysis, the automatic collection, aggregation, and comparative analysis of test results can quickly compare the current performance of the Preempt_RT kernel with the baseline, timely discover and quantify changes in real-time metrics (such as maximum latency), and help developers quickly locate and fix problems.

[0017] All in all, automated testing reduces the need for professional test engineers to manually perform repetitive tasks, reduces labor costs, and enables real-time testing to be carried out more frequently and efficiently. By providing an efficient and reliable means of real-time verification, it helps to continuously monitor and ensure the real-time performance of the Preempt_RT kernel during the development process, ultimately improving the stability and reliability of products based on this kernel. Detailed implementation manners

[0018] To have a further understanding of the technical solutions and beneficial effects of the present invention, the technical solutions of the present invention and the beneficial effects they produce are described in detail below.

[0019] A real-time automation testing method for the Preempt_RT kernel proposed by the present invention remotely coordinates and manages one or more devices under test through a central control node (or called a test management server). It provides an end-to-end and highly integrated automation solution specifically for the real-time testing of the Preempt_RT kernel, and has made remarkable progress in terms of efficiency, consistency, coverage, result analysis, etc. Among them, a central control node is used to receive test tasks, store test resources (kernel, script, configuration, baseline), schedule the automated test process, and aggregate and analyze test results; at least one device under test is used to run the Preempt_RT kernel under test and is connected to the central control node through a network.

[0020] In an embodiment of the present invention, a Linux server is used as the central control node, and several target machines are used as devices under test (DUT). It should be noted that although there are several devices under test in this embodiment, since the central control node coordinates and manages each device under test uniformly and without a sequential order, there is no difference in the execution of method steps between one device under test and multiple devices under test.

[0021] The real-time automation testing method for the Preempt_RT kernel of the present invention divides the process of automating the real-time testing of the Preempt_RT kernel into four major parts, namely: automated environment preparation and configuration, automated kernel deployment and boot, automated test environment setup and parameterized execution, and automated result aggregation and comparative analysis.

[0022] Automated environment preparation and configuration is used to perform pre-configuration on the central control node and receive test task requests.

[0023] Automated kernel deployment and boot is used to remotely deploy the Preempt_RT kernel under test to the DUT and remotely configure the kernel startup parameters and runtime environment of the DUT.

[0024] Automated test environment setup and parameterized execution is used to automatically perform real-time testing and optional stress testing on the DUT according to the parameterization strategy.

[0025] Automated result aggregation and comparative analysis is used to collect test data and system status from the DUT, compare the collected data with the baseline, judge performance changes, and generate reports.

[0026] Specifically, the test process is as follows: S1, Automated environment preparation and configuration, the specific process is: S101, Configure the kernel image to be tested: Prepare the image package of the Preempt_RT kernel to be tested on the central control node (e.g., the.deb or.rpm package of the Preempt_RT kernel to be tested, or the kernel source code and compilation script). S102, Configure the test toolset: Prepare the installation package or deployment script of the test toolset (rt-tests) on the central control node. The test toolset can include: standard real-time test tools (such as cyclictest, oslat, and hwlatdetect, etc.), and stress generation tools (such as stress-ng, iperf, dd, or the LTP-RT suite, etc.).

[0027] S103, Configure the test script: Install and configure an automation tool, such as a Python script, on the central control node for logical control and data processing.

[0028] S104, Configure the "configuration file template": Prepare the GRUB configuration file template on the central control node, which contains placeholders for setting isolcpus, nohz_full, and rcu_nocbs, etc.

[0029] S105, Establish a baseline result database for storing test configurations, raw results, and analyzed metrics, and set a comparison baseline. The setting of the comparison baseline is: Select a reference kernel version (which can be the previous stable version), run a complete test process on the target hardware (which can be a specific stable hardware platform), and store the results in the baseline result database as the baseline.

[0030] S106, Accept the test task request, which includes the kernel version information of the Preempt_RT kernel to be tested, the information of the device to be tested, and the test strategy. The test strategy includes the test set, parameter range, and comparison baseline, etc.

[0031] S2, Automated kernel deployment and boot, the specific process is as follows: S201, When the test task is triggered (e.g., when the code is committed to a specific branch, triggering the test task), perform automated kernel deployment and boot.

[0032] S202, Make the central control node remotely connect to the specified DUT (i.e., the device to be tested in this test process. When there are multiple DUTs, connect to the DUTs separately and execute the subsequent tasks separately).

[0033] S203, After the central control node remotely connects to the DUT, automatically perform the following operations.

[0034] S204, Check the status of the DUT. Securely transfer the image package of the Preempt_RT kernel to be tested from the central control node to the DUT via scp or a similar method.

[0035] S205, Execute commands on the DUT (such as: dpkg -i or rpm -ivh commands) to install the image package of the Preempt_RT kernel to be tested.

[0036] S206, Dynamically modify the GRUB configuration: According to the test strategy (such as: isolate CPUs 1 - 3), automatically read the GRUB configuration file template through a test script, locate the kernel startup line, add or modify parameters such as isolcpus=1-3, nohz_full=1-3, rcu_nocbs=1-3, etc., and execute the update-grub or equivalent command to update the boot menu.

[0037] S207, Execute the reboot command to restart the DUT.

[0038] S208, After waiting for the DUT to restart, reconnect the central control node to the DUT via SSH, execute uname -a to confirm that the Preempt_RT kernel to be tested has been successfully loaded, and record the version information of the Preempt_RT kernel to be tested. At this time, the DUT is in a testable state.

[0039] S3, Set up the automated test environment and perform parameterized execution. The specific process is as follows: S301, Keep the central control node remotely connected to the DUT via the network and execute the subsequent steps.

[0040] S302, The process of configuring the runtime environment of the DUT through the central control node is as follows: (1) Set the CPU affinity according to the test strategy. For example: Use the taskset command to bind upcoming test processes such as cyclictest to the specified isolated CPU core (such as CPU 1); (2) Move interrupts that may cause interference away from the isolated CPU core. For example: Achieved by modifying / proc / irq / / smp_affinity; (3) Set the scheduling policy and priority of real-time tasks. For example: chrt -f -p 99 <pid>; (4) Load specific kernel modules or adjust sysctl parameters as needed.

[0041] It should be noted that in this step S302, the central control node automatically configures the runtime environment of the device under test according to the preset or task-specified test strategy, including but not limited to: modifying kernel boot parameters (through GRUB or other mechanisms, such as setting isolcpus, nohz_full, rcu_nocbs, etc. for CPU isolation and tickless configuration), setting CPU affinity (binding test tasks and interrupt service routines to specific CPU cores), adjusting system sysctl parameters, shutting down or restricting non-essential background services, etc. In this embodiment, one or several of the situations such as S302 (1)-(4) are expanded in detail to illustrate the environment configuration of the DUT during runtime. The kernel refers to the kernel of the Preempt_RT to be tested, which has been deployed to the DUT in S2.

[0042] S303, the process of parameterizing and concurrently testing the DUT through the central control node is as follows: (1) Define the parameter space and test matrix. For example, the central control node generates a test matrix containing multi-dimensional parameter combinations according to the test strategy. An example of a test matrix (taking cyclictest for real-time testing and stress-ng for stress testing) is as follows: Example 1 of parameter combinations in the test matrix: cyclictest parameters: duration, interval, priority, and CPU binding; Example 1 of parameter combinations in the test matrix: stress-ng parameters: stress type (--cpu, --vm, --io, --net), number of instances, duration, and bound core CPU (usually bound to a non-isolated core, such as CPU 0).

[0043] It should be noted that this embodiment only provides one set of test matrices for real-time testing and one set of test matrices for stress testing. Test matrices with other test focuses can be generated as needed. Other test focuses may include: different scheduling strategies (FIFO, RR), different task priorities, different test loads (CPU-intensive, memory-intensive, I / O-intensive, network-intensive), different stress intensity levels, and different CPU core utilization strategies (isolated core vs. non-isolated core testing).

[0044] (2) System configuration: GRUB parameter combinations (different isolation strategies), scheduler settings. Corresponding to the dynamic modification of GRUB configuration in S206, GRUB is implemented here. Specifically, after replacing the kernel of the Preempt_RT to be tested in the DUT remote environment configuration, changing different configuration items will affect the test results of each parameter combination. For example, for a parameter combination in the test matrix, different GRUB parameter configurations with bound core isolation strategies will produce different test results. Therefore, through GRUB parameter configuration, more diverse tests can be achieved.

[0045] (3) Automated iterative execution, for example: enabling the central control node to automatically coordinate the test matrix through a cyclic task queue, and execute each parameter combination in the test matrix sequentially or in parallel. Examples of executing parameter combinations in the test matrix are as follows: Example 1: cyclictest (prio=99, interval=1ms, on CPU 1) + stress-ng (--cpu 1, on CPU 0) Example 2: cyclictest (prio=99, interval=1ms, on CPU 1) + stress-ng (--io 4, on CPU 0) (4) Automatically record the standard output (including latency statistics) of the test tool (such as the cyclictest tool in the example of (3) above) during each test execution, or redirect the detailed latency data to a test result file for storage. Simultaneously monitor the system status and provide feedback.

[0046] During the execution process, S304 automatically collects log files from the DUT. After the test is complete, the DUT's test result files are obtained. The log files and test results contain information such as raw test data (such as latency statistics and histogram data from cyclictest), system logs, performance counters, and resource usage (CPU, memory, and I / O). All generated test result files and log files are transferred from the DUT to the central control node using the scp command.

[0047] S4, automated result aggregation and comparative analysis, the specific process is as follows: S401: A Python script on the central control node automatically parses the test result file and log file output by the test tool (e.g., cyclictest) in step S304, extracts key metrics (e.g., maximum latency, average latency, duration data, standard deviation, and latency distribution), associates the key metrics with the test parameter combination (the parameter combination typically includes data such as kernel version, GRUB parameters, and stress type), and stores them in the baseline result database. S402: Automated comparison and threshold determination are performed for this test using the information in the baseline database. The specific process is as follows: (1) Use Python scripts to query the test results corresponding to the key indicators of this test and the comparison baseline corresponding to this test from the baseline result database; (2) Determine the percentage change of the key indicator based on the test results of the baseline and the current test; (3) Compare the percentage change of key indicators with the preset threshold, for example, if the maximum delay increases by more than 10%, it will be marked as a failure / warning; (4) Distribute the test results corresponding to key indicators (such as latency distribution, etc.) and use statistical methods to compare whether there are significant changes in the distribution, and further determine whether there is a significant performance regression or improvement in the Preempt_RT kernel to be tested.

[0048] S403: Automatically generate a test report in HTML or PDF format based on the test result file and log file, clearly showing the comparison results, differences, and potential problems. The test report may include: (1) Summary of the test environment (i.e., information about the Preempt_RT kernel to be tested, information about the DUT hardware, and configuration parameters used during the entire automated test process); (2) Key indicator comparison table (i.e., the comparison results of the key indicators of this test with the comparison baseline, which can be displayed as the percentage difference between the two, and the key indicators can be further marked as passed or failed based on the comparison results of the difference percentage with the preset threshold); (3) Delay histogram comparison chart; (4) Details of the test cases that failed or changed significantly (i.e., the parameter combinations in the test matrix in S303 (3)).

[0049] S404: Push the test report generated in S403 (or the important results in the key indicators) to the developer's mailing list.

[0050] It should be noted that in this embodiment, S1 - S4 are all automated processes. By remotely coordinating and managing at least one device under test through a central control node, it is possible to achieve the entire process of automated deployment, automated configuration, automated test execution, automated result collection, and automated analysis and reporting. This provides an automated, standardized, highly efficient, and highly reliable real - time testing method for the Preempt_RT kernel, significantly improving the test efficiency and the credibility of the results, and enabling a rapid and accurate assessment of the real - time performance changes of the Preempt_RT kernel.

[0051] Advantages of the present invention: In terms of test efficiency, by remotely coordinating the devices under test through a central control node, end - to - end automation is achieved. The entire process from kernel deployment to result analysis requires little or no manual intervention. The test time is shortened from several days or weeks to a few hours or even less, which can adapt to agile development and CI / CD processes.

[0052] In terms of test consistency and repeatability, the test matrix generated by the central control node according to the test strategy contains multi - dimensional parameter combinations. The test process is automatically executed strictly in accordance with the environment configuration file template and parameter combinations, eliminating variables and errors introduced by manual operations, ensuring that each test is carried out under precisely controlled conditions, and the test results have high comparability and credibility.

[0053] In terms of test coverage and test depth, through the combination of a test toolset and a test matrix, it is possible to systematically and automatically cover a large number of parameter combinations and stress scenarios, more comprehensively evaluate the impact of different factors on real - time performance, and more easily discover hidden performance problems and regressions under boundary conditions.

[0054] In terms of test result analysis, the automated collection, aggregation, and comparative analysis of test results can quickly compare the current performance of the Preempt_RT kernel with the baseline, timely discover and quantify changes in real - time metrics (such as maximum latency), and help developers quickly locate and fix problems.

[0055] In terms of reducing test costs and human dependence, automated testing reduces the need for professional test engineers to manually perform repetitive tasks, reduces labor costs, and enables real - time testing to be carried out more frequently and efficiently.

[0056] In terms of promoting the quality improvement of the Preempt_RT kernel, by providing an efficient and reliable real - time verification method, it helps to continuously monitor and ensure the real - time performance of the Preempt_RT kernel during the development process, and ultimately improves the stability and reliability of products based on this kernel.

[0057] Although the present invention has been described by using the above preferred embodiments, it is not intended to limit the protection scope of the present invention. Any person skilled in the art can make various changes and modifications to the above embodiments without departing from the spirit and scope of the present invention, and still fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the claims.< / pid>

Claims

1. A real-time automation testing method for the Preempt_RT kernel, characterized in that, Including: S1. Pre - configure the central control node according to preset configuration items, and receive test task requests through the central control node; the preset configuration items include the image package of the Preempt_RT kernel to be tested, a test tool set, and a baseline result database. S2. Deploy the image package of the Preempt_RT kernel to be tested to the device to be tested through the central control node; determine that the device to be tested is in a testable state. S3. Configure the remote environment of the device to be tested through the central control node; generate a test matrix for the device to be tested according to the test strategy through the central control node, and call the test tool set to execute tests on the device to be tested according to the test matrix to obtain the test results of the Preempt_RT kernel to be tested on the device to be tested. S4. Aggregate and analyze the test results through the central control node to obtain the key indicators of the Preempt_RT kernel to be tested, compare the key indicators with the comparison baselines in the baseline result database to determine the change value of the key indicators, and determine whether there is a performance improvement in the Preempt_RT kernel to be tested according to the change value.

2. The real-time automation testing method for the Preempt_RT kernel according to claim 1, wherein The preset configuration items further include: Test scripts and configuration file templates.

3. The real-time automation testing method for the Preempt_RT kernel according to claim 2, characterized in that, The image package of the Preempt_RT kernel to be tested includes: The image of the Preempt_RT kernel to be tested and related module files.

4. The real-time automated testing method for the Preempt_RT kernel according to claim 3, wherein, The specific process of deploying the image package of the Preempt_RT kernel to be tested to the device to be tested through the central control node is as follows: Enable the central control node to remotely connect to the device to be tested through the network. Transfer the image package of the Preempt_RT kernel to be tested from the central control node to the device to be tested. Install the image package of the Preempt_RT kernel to be tested on the device to be tested. Modify the boot loader configuration of the device to be tested according to the configuration file template through the test script to start the Preempt_RT kernel to be tested. Restart the device to be tested, verify and record the status of the device to be tested and the kernel version information of the Preempt_RT kernel to be tested.

5. A real-time automation testing method for the Preempt_RT kernel as described in claim 1, characterized in that, Configuring the remote environment of the device to be tested through the central control node includes: Modifying kernel startup parameters, setting CPU affinity, adjusting system parameters, shutting down or restricting unnecessary background services, setting the scheduling policy and priority of the test matrix, and loading at least one of the kernel modules.

6. The real-time automation test method of a Preempt_RT kernel as described in claim 1, wherein The specific process of calling the test tool set to execute tests on the device to be tested according to the test matrix to obtain the test results of the Preempt_RT kernel to be tested on the device to be tested is as follows: Traverse the test matrix to obtain multiple groups of multi - dimensional parameter combinations in the test matrix. Call the test tool set to sequentially execute tests on the device to be tested according to each group of multi - dimensional parameter combinations, either concurrently or sequentially, to obtain sub - test results corresponding to each group of multi - dimensional parameter combinations. Combine all sub - test results to obtain the test results of the Preempt_RT kernel to be tested on the device to be tested.

7. The real-time automation test method of the Preempt_RT kernel according to claim 1, characterized in that S4 further includes: S401. Generate a test report according to the test results and push the test report to the developer email list.