Testing and evaluating method for virtual system in multiple scenes
By employing multi-scenario testing and evaluation methods for virtualization systems, fine-grained metrics for evaluating virtualization systems are quantitatively assessed. This addresses the shortcomings of existing virtualization systems in terms of functional integrity, real-time performance, and resource isolation, enabling efficient and stable system optimization and security assessment.
Patent Information
- Application Number
- CN202511273804.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-08
- Publication Date
- 2025-12-12
AI Technical Summary
Existing virtualization systems have deficiencies in functional integrity, real-time performance, resource isolation, and anomaly handling capabilities, making them prone to defects in harsh scenarios. Existing testing methods are unable to perform multi-dimensional, full-load, and cross-scenario system testing.
This paper presents a testing and evaluation method for virtualization systems across multiple scenarios. Through scripting, automation, and platform-based approaches, it integrates data recording, virtual machine management, and network communication testing. It quantitatively evaluates fine-grained metrics such as interrupt response capability, event handling capability, CPU context latency, and memory allocation/release time, and supports verification in multiple scenarios including high concurrency, full load, and exception injection.
It enables effective evaluation of virtualization systems in various scenarios, provides scientific basis for system optimization, ensures stable operation under high load, and improves the reliability and security of virtualization systems.
Smart Images

Figure CN121116780A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of software testing, in particular to a test and evaluation method for a virtualization system in multiple scenarios. BACKGROUND
[0002] In recent years, with the rapid development of heterogeneous computing, car-grade high-performance chips and multi-core system-level chips, the application scenarios of embedded systems show a significant trend of function fusion, resource intensification and real-time enhancement. In the safety field of aerospace, automotive electronics, industrial control and medical equipment, on the one hand, rich-function operating systems such as Linux and Android are needed to provide complete ecosystems such as graphical interfaces, network protocol stacks and artificial intelligence reasoning frameworks, and on the other hand, real-time operating systems are needed to complete microsecond-level or sub-microsecond-level deterministic control tasks. The traditional single-chip single operating system architecture cannot meet the above two types of needs at the same time; while the multi-chip multi-operating system solution brings multiple bottlenecks such as board-level area, power consumption, communication delay and reliability. Therefore, the industry has generally turned to a single-chip multi-operating system hybrid deployment architecture, and the key to realizing this architecture is an embedded virtualization system.
[0003] The current mainstream virtualization systems include: KVM (Kernel-based Virtual Machine) relies on the Linux kernel to provide virtualization capabilities in the form of a loadable module, and the ecosystem is mature, but it lacks control of the jitter introduced by interrupt delay and context switching, and the real-time performance is not good; Xen adopts a microkernel architecture, supports ARM and x86 platforms, has a Domain-0 privileged domain management model, and is widely used in server and embedded scenarios, but the configuration is complex and real-time extension requires additional patches; Xvisor is an open source Type-1 virtualization system for embedded systems, with concise code and strong scalability, supporting multi-core scheduling and device trees, but lacking commercial-level security certification; Jailhouse is mainly based on static hardware partitioning, achieving high real-time performance for bare metal and Cell strong isolation, but its flexibility is limited and only supports fixed resource configuration. The above systems collectively build the current virtualization technology ecosystem, providing underlying support for chip-level resource partitioning, fault isolation and cross-domain communication, and have gradually become the core infrastructure of intelligent automotive domain controllers, industrial Internet of Things edge gateways, medical imaging terminals and other devices. Therefore, the functional correctness, real-time performance, resource isolation and exception handling capability of the virtualization system itself directly determine the reliability, security and feasibility of the upper hybrid deployment scheme.
[0004] Although various virtualization systems have their own characteristics in architecture and implementation, their common defects are gradually exposed in large-scale deployment and rigorous scenario verification. Functional lack of integrity: existing verification focuses on resource visibility and basic allocation, ignoring task preemption links, event channels, interrupt nesting, and other fine-grained semantics. For example, when a high-priority real-time task triggers scheduling through a virtual interrupt, if the virtualization layer does not correctly forward or delay injection, it will directly lead to Deadline violation; event notification queues may appear out of order in multi-core concurrent scenarios, causing inter-task synchronization failure. Performance perturbation is difficult to quantify: EL2 / EL1 exception level switching, two-stage address translation, I / O semi-virtualization or direct path introduced by virtualization will all add additional delay. However, existing technical documents mostly only give coarse-grained conclusions on relative bare-metal loss, lacking fine-grained characterization of context switch delay, TLB flush frequency, cache partition invalidation rate, and other micro-architecture level indicators, resulting in a lack of data basis for system-level timing analysis and resource reservation strategies. Insufficient verification of resource isolation: when multiple clients run CPU-intensive or memory bandwidth-intensive loads in parallel, resources such as shared LLC (Last Level Cache), DRAM channels, and interrupt controllers will have implicit competition. Traditional testing only verifies isolation under light load or zero concurrency conditions, which cannot reflect real-world conditions, resulting in excessive jitter of real-time tasks in high-load scenarios, and even triggering safety monitor alarms. Exception handling link is fragile: the virtualization layer needs to capture and report events such as page fault exceptions, illegal I / O access, ECC errors, and device failures generated by clients. If the virtualization system capture delay, log loss or recovery strategy is improper, it will amplify the cross-operating system cascade failure. Currently, there is a lack of quantitative evaluation methods for the whole process of exception triggering, capturing, reporting, and recovery.
[0005] In summary, if virtualization systems do not undergo multi-dimensional, full-load, and cross-scenario systematic testing during the development stage, they are likely to expose defects in the terminal field, leading to project rework, certification failure, and even safety accidents.
[0006] At the current stage, the testing methods for virtualization systems generally stay at the level of functional verification: through manual scripts to check whether the client can recognize the number of virtual CPUs, memory capacity, or use single benchmark tools such as cyclictest, UnixBench, lmbench to measure delay and throughput rough values; data collection relies on serial console to capture items one by one, which is low in efficiency and easy to miss. At the same time, the switching of test scenarios requires re-burning of images and manual switching of hardware jumpers, making it difficult to form a continuous integration and regression testing closed loop. SUMMARY
[0007] To solve the problems in the prior art, the application provides a test and evaluation method for a virtualization system in multiple scenarios, and the executable file of the compiled test script is deployed in a virtualization client to test the performance of a high-level client operating system to determine the performance loss of a bottom-layer virtualization system on computer resources.
[0008] To support efficient execution of the test process, the application builds an application platform, and the functions include data recording, virtual machine management and network communication testing. Based on the test script, the virtualization system is subjected to functional testing, performance testing, isolation special testing and abnormal processing testing. The application can perform detection on indexes including interrupt response capability, event processing capability, CPU context delay, switching times and usage, memory allocation / release time, memory allocation fragmentation rate, thread communication single message delay, batch throughput rate, CPU / memory isolation performance, abnormal capture rate and abnormal processing response time, so as to confirm the performance of the virtualization system in multiple scenarios and provide a scientific basis for system optimization.
[0009] The test and evaluation method for the virtualization system in multiple scenarios includes the following steps:
[0010] Step 1: Create a project and configure system parameters.
[0011] Step 2: Integrate the test program into the project to generate an image file.
[0012] Step 3: Package the complete virtualization system image.
[0013] Step 4: Transmit the image to the development board through a network protocol.
[0014] Step 5: Execute a command to start the system and log in to the system console through a serial port.
[0015] Step 6: Call and execute the functional testing, performance testing and security testing scripts.
[0016] In the application, the virtualization system test case deployed in the virtualization system is each functional script written in a programming language and its dependent tool.
[0017] The virtualization system environment constructed in the virtualization system is a real-time operating system and a supporting integrated development environment.
[0018] The transmission process of the virtualization system constructed in the virtualization system is to package the generated image file to generate a complete virtualization system image, which can be transmitted to the development board through a network protocol and used for deployment of the test method in different scenarios.
[0019] The test configuration is executed on a virtualization system performance test method, and the configuration information includes engineering system parameters, a number of consoles and CPUs, performance data, and a storage path of a log, and is used for test environment construction and request initiation and data reception of performance testing.
[0020] In the application, the function test includes basic virtualization function test, task preemption mechanism, event management function test, and interrupt processing mechanism test, and specifically includes the following sub-steps:
[0021] Step A1: Obtain the number of CPUs perceivable by the operating system through the lscpu command;
[0022] Step A2: Obtain the number of available memories of the operating system;
[0023] Step A3: Analyze whether the record conforms to the allocated virtualization resource amount;
[0024] Step A4: Create threads with different priorities;
[0025] Step A5: Run the low-priority thread and hold the semaphore resource, and suspend the high-priority thread;
[0026] Step A6: Wake up the suspended state thread in disorder, and record the priority inverted task;
[0027] Step A7: Construct a multi-priority multi-thread benchmark program, and implant an event flag;
[0028] Step A8: Verify and record the event function and concurrency correctness, and measure the event delay;
[0029] Step A9: Measure the statistical interrupt corresponding delay, interrupt nesting depth, and interrupt enable / shield correctness.
[0030] The basic virtualization function test is performed by checking whether the number of CPUs perceivable by the operating system conforms to the allocated virtualization resource amount through the lscpu command, and analyzing whether the number of available memories of the operating system conforms to the allocated virtualization resource amount;
[0031] The task preemption mechanism test includes task preemption time consumption test, task execution order test, and task inversion analysis; wherein the task preemption time consumption test is how long the high-priority thread needs to wait after issuing a request to obtain the resource occupied by the low-priority thread in the case of implementing the priority scheduling strategy; and specifically includes the following experiments:
[0032] Experiment 1: Create a low-priority thread, and preempt a semaphore; the high-priority task is created first to wait for the semaphore, and the time consumption of the low-priority task releasing the semaphore to the high-priority task acquiring the semaphore is tested;
[0033] Experiment 2: Keep a low-priority thread looping on a compute-intensive task, create a high-priority task to preempt the CPU immediately; record the difference between the preemption trigger time and the high-priority task execution time stamp;
[0034] Experiment 3: Randomly create threads with different priorities, all in suspended state, wake them up in random order to simulate real scheduling; for each time a priority inversion happens, record it in the log; the metrics include average response time, preemption flag; the priority inversion is determined by the task waiting time, the industry usually limits the maximum response jitter of real-time tasks to 1-5 ms, and 3 ms is the conservative value of the upper limit of the above standard, which can cover 99% of real-time scenarios. In addition, the current public benchmark such as cyclictest official document considers "> 3 ms as abnormal jitter" as the default alarm line, so for high-priority tasks waiting more than 3 ms, it is considered that the scheduling is unreasonable, a certain degree of inversion occurs, and the actual order of the corresponding scheduling is printed out to determine whether inversion has indeed occurred;
[0035] The event management function test includes success rate of event delivery, ensuring that events communicated between tasks will not be lost; synchronous / asynchronous event delay, quantifying the speed at which the system processes events; and concurrent event processing accuracy, verifying that the system can process multiple simultaneous events without error under high load.
[0036] The interrupt handling mechanism test includes testing the interrupt response delay in the system, calculating the interrupt nesting depth, and counting the correct rate of interrupt enable / disable; wherein,
[0037] The interrupt response delay test is the time difference between the interrupt trigger and the start of the interrupt service program execution, which evaluates the response speed of the system to the interrupt event.
[0038] The interrupt nesting depth test verifies whether the system supports the interrupt nesting mechanism, tests whether a high-priority interrupt can successfully interrupt the execution of a low-priority interrupt handler, and counts the number of interrupt nesting levels; the
[0039] The interrupt enable / disable correctness test is to execute the interrupt enable and disable operations multiple times and count the number of successful times to evaluate the correctness and stability of the interrupt control function.
[0040] Preferably, first, a multi-priority multi-thread benchmark program is constructed in each client, and an event flag, a synchronous / asynchronous event sending / receiving interface and a high-precision time stamp acquisition module are implanted to form a controllable concurrent load; second, event functions and concurrent correctness are verified, including: by injecting an eight-level priority, synchronous / asynchronous mixed event stream into an event queue, recording event creation, delivery, reception success rate and sequence consistency, to verify that there is no loss of messages, no wrong sequence and the priority strategy takes effect; in the same virtual machine and cross-virtual machine scenarios, N events are triggered in parallel, where N is not less than the number of virtual CPU (vCPU) cores allocated to the client to ensure that each vCPU core has at least one event to process, the event processing result and system state are monitored, and it is confirmed that there is no error in event parallel processing under high load; finally, the whole path delay from the publication of a synchronous event and an asynchronous event to the complete processing of the target thread is measured; the minimum, average and maximum delay values are repeatedly sampled and output to quantitatively evaluate the event management performance.
[0041] In the present application, the performance test includes: CPU performance test, memory allocation performance test, process communication performance test and deadlock detection and recovery mechanism test; specifically including the following sub-steps:
[0042] Step B1: creating threads of different priorities;
[0043] Step B2: running a low-priority thread and holding a semaphore resource, suspending a high-priority thread;
[0044] Step B3: counting the number of context switches and context delays;
[0045] Step B4: running a unified stream benchmark test set to generate continuous memory read / write load;
[0046] Step B5: counting data for each memory allocation-access-release cycle;
[0047] Step B6: creating more than two threads, and fixedly applying / holding two sets of mutual exclusion resources;
[0048] Step B7: starting the deadlock detection interface, and counting detection and recovery delays;
[0049] Step B8: introducing a thread that holds resources for a long time but has no circular dependency, and counting the number of false positives and recovery success rate.
[0050] The CPU performance test is for evaluating the efficiency of system task scheduling and processor resource management. The number of context switches in a unit of time is counted by simulating a multi-thread or multi-process concurrent execution scene, by adding a context switch awareness hook in a thread, counting before the operation, and finally obtaining the number of context switches of programs with different characteristics. The context delay is measured, i.e. the specific time required to complete a task switch. The implementation is to create threads with different priorities, keep the low-priority thread running and hold the semaphore resource when creating, and suspend the high-priority process to wait for the semaphore. When the low-priority thread releases the semaphore, the switching time is obtained by comparing the semaphore release timestamp with the high-priority thread start running timestamp. Through the analysis of the two core reference indicators, the performance of the CPU scheduler is quantitatively evaluated, which provides a basis for system optimization and resource allocation strategy, and ensures that the system can still run efficiently and stably under high load.
[0051] The memory allocation performance test is for evaluating the memory allocation / release time. A uniform stream benchmark test set is run, the iteration number and data block size are fixed, and a controllable continuous memory read-write load is generated to exclude external disturbances. A high-precision timer embedded in the benchmark program is used to record the time consumption of each memory allocation-access-release cycle, and the minimum, average, maximum time consumption and optimal bandwidth are extracted. After repeated sampling, the average time consumption difference is calculated to evaluate the allocation overhead introduced by the virtualization system.
[0052] The inter-process communication performance test is for evaluating the efficiency and stability of the system in transmitting data between multiple processes or threads, including single message transmission delay: the time required from sending a message from the sending end to receiving the message at the receiving end; batch message throughput: the number of messages that the system can successfully send and receive in a unit of time; communication error rate: including packet loss, duplicate messages, out-of-order, etc.
[0053] The deadlock detection performance test is for evaluating the efficiency and accuracy of the system in handling deadlock problems, including deadlock detection execution time, the time spent from triggering deadlock detection to completing detection; recovery success rate, the proportion of successfully recovering the blocked threads after the deadlock is detected; false deadlock false positive rate, the frequency of false deadlock reporting by the system in the absence of actual deadlock.
[0054] Preferably, firstly, a controllable deadlock scenario is constructed, 2-N threads are created, two groups of mutual exclusion resources are applied / held by a fixed order, a cyclic waiting graph is artificially formed, and a system deadlock detection mechanism is triggered; secondly, the detection and recovery delay is quantified, the deadlock detection interface is started, the time consumed from the formation of the cyclic waiting to the system determining the deadlock is recorded, and the time consumed from the determination being completed to the lowest priority thread being terminated and the remaining threads continuing to run is recorded; the multi-thread scale test is repeated and statistical values are extracted; finally, the correctness and false positives are verified, under the same load, long-time resource holding threads without cyclic dependency are introduced, it is verified that the system only produces a deadlock determination for a real cyclic waiting, and the number of false positives is zero; and the recovery success rate is recorded to confirm the correctness and stability of the deadlock resolution strategy.
[0055] In the application, the safety performance test comprises: an isolation performance test and an abnormality processing robustness test; and specifically comprises the following sub-steps:
[0056] Step C1: using the stress command to make the CPU and the memory enter a load;
[0057] Step C2: running a benchmark program with a fixed calculation amount, and recording the execution time consumption in multiple scenarios;
[0058] Step C3: calculating the average time consumption deviation rate of the high-load scenario and the baseline scenario and determining the isolation;
[0059] Step C4: injecting three types of abnormalities into the system under test by calling;
[0060] Step C5: recording the measurement abnormality capture and the corresponding time delay, and verifying the robustness index.
[0061] The isolation performance test is running a CPU-intensive test program and a memory access test program under different load states of the Linux virtual machine; verifying whether the virtualization system can maintain resource isolation under high-load conditions; ensuring that the resources of different virtual machines in the virtualization environment do not interfere with each other; detecting the isolation capability of the virtualization system through different load scenarios to ensure that the task execution time is stable under high-load conditions of the Linux virtual machine;
[0062] The abnormality processing robustness test is triggering a preset abnormality and counting the abnormality processing success rate; recording the complete time chain from the abnormality triggering to the processing and verifying the response time; checking the accuracy of the error code return and verifying the system state recovery condition; observing whether the system can correctly return an error and set an appropriate error code when an operation is performed on a closed message queue descriptor; whether the system can detect that the message exceeds the limit and return an EMSG-SIZE error when a super-large message is sent to a message queue; whether the system can correctly capture and return an ESRCH error when an event is sent to a terminated thread;
[0063] First, three types of exceptions are injected into the system under test through API calls: operations on message queue descriptors that have been closed; sending oversized messages with lengths exceeding the upper limit; sending events to terminated threads; each type of exception triggers 100 times, forming a repeatable exception sequence; second, exception capture and response time measurement, using the system-provided exception hook and high-precision timer, records the time consumption between each exception trigger and the system capture and return of the error code; since the time consumption of a single exception trigger to the system capture and return of the error code is extremely short and has random fluctuations, the stable mean value needs to be obtained through multiple statistics. On the other hand, in view of the requirement of IEC 61508 and other industry standards for software unit robustness verification "≥ 100 times of repetition", the average time consumption of 100 times of triggering is taken as the exception response time; finally, the robustness is verified, the capture success rate of 100 times of exception triggering is calculated, and whether 100% is reached is confirmed; at the same time, the returned error code and the system state recovery are checked to see if they meet the expectations; if the capture rate is 100% and the average response time is less than 100 microseconds, it is determined that the exception handling mechanism has reliability and real-time performance.
[0064] The virtualization system function, performance and security test method provided by the application, the test range covers indexes including interrupt response capability, event processing capability; CPU context delay, switching times, usage rate, memory allocation / release time, memory allocation fragmentation rate, thread communication single message delay, batch throughput rate; CPU / memory isolation performance, exception capture rate, exception handling response time, so as to confirm that the application can effectively evaluate the performance of the virtualization system in multiple scenarios, and provide a scientific basis for system optimization.
[0065] The test and evaluation method for virtualization system in multiple scenarios provided by the application solves the pain points of the prior art through scripting, automation and platformization: more than 20 indexes in three dimensions of function, performance and security are encapsulated as programmable test cases, an image is generated by one key and is distributed to the target board in batches through the network protocol; during the test process, fine-grained data such as CPU context delay, memory fragmentation rate, interrupt response time and deadlock recovery success rate are collected in real time; high concurrency, full load, exception injection and other multiple scenarios are supported for repeatable verification, thereby realizing the leap from single-point verification to systematic evaluation. BRIEF DESCRIPTION OF DRAWINGS
[0066] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, brief descriptions will be given to the drawings needed to be used in the embodiments or prior art descriptions. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.
[0067] Figure 1is a flow framework diagram of the virtualization system of the present application.
[0068] Figure 2 is a performance test method flow diagram of the virtualization system of the present application.
[0069] Figure 3 is a performance test method flow diagram of the virtualization system of the present application.
[0070] Figure 4 is a security test method flow diagram of the virtualization system of the present application.
[0071] Figure 5 is a task preemption mechanism experimental result display diagram of the function test of the embodiment of the present application.
[0072] Figure 6 is a context switching number experimental result display diagram of the performance test of the embodiment of the present application.
[0073] Figure 7 is an isolation test experimental result display diagram of the security test of the embodiment of the present application. DETAILED DESCRIPTION
[0074] The application will be further described in detail in combination with the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present application are the general knowledge and common sense in the art, and the present application does not have special limitations.
[0075] The multi-scene-oriented virtualization system test method provided by the present application deploys the executable file after compiling the test case in the virtualization client, so as to perform function, performance and security tests on the virtualization system.
[0076] The test range that can be performed by the compiled test case includes functional test, performance test and security test. For the functional test, basic virtualization function, task preemption mechanism, event management function and interrupt mechanism test are performed; for the performance test, CPU performance, memory allocation performance, process communication performance and deadlock detection and recovery mechanism test are performed; for the security test, isolation and abnormality processing robustness test are performed. In this way, the performance of the virtualization system is effectively evaluated in multiple scenarios.
[0077] In order to have a further understanding of the technical solutions and beneficial effects of the present application, the technical solutions of the present application and the beneficial effects generated thereby will be described in detail below in combination with the drawings.
[0078] Figure 1 is a function test method flow diagram of the virtualization system of the present application, and in the embodiment, the method comprises:
[0079] 1. Create a project and configure system parameters;
[0080] 2、Test procedure integration into the project, generating an image file;
[0081] 3、Packaging the complete virtualization system image;
[0082] 4、Transmitting the image to the development board through a network protocol;
[0083] 5、Executing a command to start the system and logging into the system console through a serial port;
[0084] 6、Calling the function test, performance test and security test procedures.
[0085] Figure 2 is the functional test method flowchart of the virtualization system of the application, in the embodiment, the method comprises:
[0086] 1、Basic virtualization function test;
[0087] Specifically, the lscpu command is used to check whether the number of CPUs that can be perceived by the operating system conforms to the number of allocated virtualization resources, and whether the number of available memories of the operating system conforms to the number of allocated virtualization resources is analyzed.
[0088] 2、Task preemption mechanism test;
[0089] The task preemption mechanism is divided into task preemption time consumption test, task execution order test and task inversion analysis. The task preemption time consumption test is essentially that, in the case of implementing a priority scheduling strategy, how long the high-priority thread needs to wait after making a request before it can obtain the resources occupied by the low-priority thread.
[0090] The application provides three experiments: Experiment 1: Create a low-priority thread and preempt a semaphore (a resource not preempted by priority). The high-priority task waits for the semaphore first when it is created. The experiment tests the time consumption of the low-priority task releasing the semaphore to the high-priority task acquiring the semaphore. Experiment 2: Keep the low-priority thread executing a compute-intensive task in a loop, and create a high-priority task to preempt the CPU immediately. Record the difference between the preemption trigger time and the time stamp of the high-priority task execution. Experiment 3: Randomly create threads with different priorities, which are all in a suspended state when created. Through disorderly wake-up, the real scheduling is simulated. For each time the priority inversion occurs, it is recorded in the log. The measurement indexes include average response time and preemption mark. The inversion is determined by the task waiting time. If the high-priority task waits for more than 3 ms, it is considered that the scheduling is unreasonable, and a certain degree of inversion occurs. At this time, the actual order of the corresponding scheduling is printed out for judgment, and whether the inversion really occurs is checked.
[0091] 3、Recording the context switching time consumption and task execution order;
[0092] 4. Event management function test;
[0093] The event management function test mainly focuses on its reliability, performance and concurrent processing capability, including event delivery success rate, ensuring that event communication between tasks will not be lost; synchronous / asynchronous event delay, quantifying the corresponding speed of the system processing events; and concurrent event processing accuracy, verifying whether the system can process multiple simultaneous events without error under high load.
[0094] The specific process is first to build a multi-priority multi-thread benchmark program in each client, and implant event flag, synchronous / asynchronous event transceiver interface and high-precision timestamp acquisition module to form controllable concurrent load. Secondly, verify the event function and concurrent correctness, including: by injecting eight-level priority, synchronous / asynchronous mixed event stream into the event queue, recording event creation, delivery, reception success rate and sequence consistency to verify that the message is lossless, error-free and priority policy effective; in the same virtual machine and cross-virtual machine scenario, trigger N events concurrently, where N is not less than the number of virtual CPU (vCPU) cores allocated to the client to ensure that each vCPU core has at least one event to process, monitor event processing results and system state, and confirm that there is no error in event parallel processing under high load. Finally, measure the full path delay from the publication of synchronous events and asynchronous events to the complete processing of the target thread; repeat sampling and output the minimum, average and maximum delay values to quantitatively evaluate the event management performance.
[0095] 5. Record event delivery success rate and synchronous / asynchronous event delay;
[0096] 6. Interrupt handling mechanism test;
[0097] The interrupt handling mechanism test includes testing the interrupt response delay in the system, calculating the interrupt nesting depth, and counting the interrupt enable / shield correctness. The interrupt response delay test specifically measures the time difference between the interrupt trigger and the start of the interrupt service program execution, to evaluate the response speed of the system to the interrupt event. The interrupt nesting depth test specifically verifies whether the system supports the interrupt nesting mechanism, tests whether the high-priority interrupt can successfully interrupt the execution of the low-priority interrupt handling program, and counts the number of interrupt nesting layers. The interrupt enable / shield correctness test specifically executes the interrupt enable and shield operation multiple times, counts the number of successful times, and evaluates the correctness and stability of the interrupt control function.
[0098] 7. Record interrupt response delay, interrupt nesting depth and interrupt enable / shield correctness.
[0099] Figure 3 is the performance test method flowchart of the virtualization system of the present application, in the embodiment, the method comprises:
[0100] 1. CPU performance test;
[0101] CPU performance test mainly evaluates the efficiency of system task scheduling and processor resource management. The number of context switches per unit time is counted by simulating multi-thread or multi-process concurrent execution scenarios. The number of context switches is obtained by adding a context switch awareness hook in the thread and counting before the operation. The context delay is measured, which is the specific time required to complete a task switch. The implementation is to create threads with different priorities, and the low-priority thread holds the semaphore resource while running. The high-priority process is suspended and waits for the semaphore. When the low-priority thread releases the semaphore, the switching time is obtained by comparing the semaphore release timestamp with the high-priority thread start running timestamp.
[0102] By analyzing these two core reference indicators, the performance of the CPU scheduler can be quantitatively evaluated, providing a basis for system optimization and resource allocation strategies to ensure that the system can still run efficiently and stably under high load.
[0103] 2. Count the number of context switches and context delays;
[0104] 3. Memory allocation performance test;
[0105] Memory allocation performance test mainly evaluates memory allocation / release time. By running a unified stream benchmark test set (Copy / Scale / Add / Triad), the iteration number and data block size are fixed to generate controllable continuous memory read-write load to exclude external disturbances. The high-precision timer embedded in the benchmark program is used to record the time consumption of each memory allocation-access-release cycle, and the minimum, average, and maximum time consumption and optimal bandwidth are extracted. After repeated sampling, the average time consumption difference is calculated to evaluate the allocation overhead introduced by the virtualization system.
[0106] 4. Record memory allocation / release time;
[0107] 5. Process communication performance test;
[0108] Process communication performance test mainly evaluates the efficiency and stability of data transmission between multiple processes or threads. It mainly includes single message transmission delay: the time required from sending a message from the sender to receiving the message at the receiver; batch message throughput rate: the number of messages that can be successfully sent and received by the system per unit time; communication error rate: including packet loss, duplicate messages, out-of-order, etc.
[0109] 6. Record single message transmission delay, batch message throughput rate, and communication error rate;
[0110] 7. Deadlock detection and recovery mechanism test;
[0111] The deadlock detection performance test mainly evaluates the efficiency and accuracy of the system in dealing with deadlock problems. Mainly includes the execution time of deadlock detection, the time spent from triggering deadlock detection to completing detection; recovery success rate, after the deadlock is detected, the proportion of the system that can successfully recover the blocked thread; false deadlock false positive rate, the frequency of false deadlock reported by the system without actual deadlock.
[0112] The specific process is to first construct a controllable deadlock scenario, create 2-N threads, and artificially form a circular waiting graph by fixedly applying / holding two groups of mutual exclusion resources to trigger the system deadlock detection mechanism. Secondly, the detection and recovery delay is quantified, the deadlock detection interface is started, the time from the formation of the circular waiting to the system determining the deadlock (detection delay) is recorded, and the time from the completion of the determination to the lowest priority thread being terminated and the remaining threads continuing to run (recovery delay) is recorded; Repeat the multi-thread scale test and extract the statistical value. Finally, verify the correctness and false positives, introduce threads that hold resources for a long time but have no circular dependency under the same load, verify that the system only makes deadlock determination for real circular waiting, and the false positive number is zero; At the same time, record the recovery success rate to confirm the correctness and stability of the deadlock resolution strategy.
[0113] 8, record the execution time of deadlock detection, recovery success rate and false deadlock false positive rate.
[0114] Figure 4 The safety test method flowchart of the virtualization system of the application, in the embodiment, the method comprises:
[0115] 1, isolation performance test;
[0116] The isolation performance test is specifically running a CPU-intensive test program and running a memory access test program under different load states of the Linux virtual machine. Verify whether the virtualization system can maintain resource isolation under high load conditions; Ensure that the resources of different virtual machines in the virtualization environment do not interfere with each other, especially the allocation of CPU and memory resources; Detect the isolation capability of the virtualization system through different load scenarios, and ensure that the task execution time is stable under high load conditions of the Linux virtual machine.
[0117] 2, abnormal processing robustness test;
[0118] The abnormality processing robustness test specifically triggers preset abnormalities and counts the abnormality processing success rate; records the complete time chain from abnormality triggering to processing and verifies the response time; checks the error code return accuracy (such as EBADF / EMSGSIZE) and verifies the system state recovery condition. When observing the message queue descriptor that has been closed, whether the system can correctly return an error and set a suitable error code (such as EBADF); when sending a super-large message to the message queue, whether the system can detect that the message exceeds the limit and return an EMSG-SIZE error; when sending an event to a terminated thread, whether the system can correctly capture and return an ESRCH error.
[0119] The specific process is as follows: first, three types of abnormalities are injected into the system to be tested through API calls: operations are performed on the message queue descriptor that has been closed; a super-large message whose length exceeds the upper limit is sent; an event is sent to a terminated thread. Each type of abnormality is triggered 100 times to form a repeatable abnormality sequence. Secondly, abnormality capture and response time measurement are performed. By using the abnormality hook provided by the system and the high-precision timer, the time consumption between each abnormality triggering and the system capturing and returning an error code (EBADF, EMSGSIZE, ESRCH) is recorded; the average time consumption of 100 times of triggering is counted as the abnormality response time. Finally, the robustness is verified, the capture success rate of 100 times of abnormality triggering is calculated, and whether 100% is reached is confirmed; at the same time, whether the returned error code and the system state recovery meet the expectation is checked. If the capture rate is 100% and the average response time is less than 100 µs, it is determined that the abnormality processing mechanism has reliability and real-time performance.
[0120] Figure 5 is a task preemption mechanism experimental result display diagram of the function test of the embodiment of the application.
[0121] The task preemption mechanism is divided into task preemption time consumption test, task execution sequence test, and task inversion analysis. The task preemption time consumption test is essentially that, in the case of implementing a priority scheduling strategy, how long the high-priority thread needs to wait after making a request to obtain the resource occupied by the low-priority thread.
[0122] As Figure 5As shown, the application quantitatively verifies the task preemption mechanism in two scenarios on a typical embedded hardware platform: Experiment 1 simulates semaphore resource preemption, and the minimum, maximum and average context switch delays are 875 ns, 2041 ns and 1051 ns respectively; Experiment 2 simulates CPU-intensive load preemption, and the minimum, maximum and average context switch delays are 875 ns, 2916 ns and 1058 ns respectively; the average of the two sets of data is about 1 µs, and the minimum is consistent, indicating that high-priority tasks can preempt the resources occupied by low-priority tasks in microseconds; although the maximum delay is increased due to the fluctuations introduced by the computational load in Experiment 2, it is still much lower than the millisecond tolerance threshold of real-time systems, thereby proving that the preemption mechanism of the application has the technical advantages of determinism, low delay and high responsiveness in typical real-time application scenarios.
[0123] Figure 6 is the context switch number experiment result display graph of the performance test of the embodiment of the application.
[0124] The context switch number test mainly adds a context switch awareness hook in the thread, counts before the operation, and finally obtains the context switch number of programs with different characteristics.
[0125] The experimental data show that when the thread concurrency degree is expanded from 4 to 10, the context switch frequency of the I / O-intensive load increases from 78 times / s to 109 times / s in the virtualization environment, while the same load jumps from 214 times / s to 764 times / s in the bare machine environment, and the additional loss introduced by the virtualization layer is amplified from 63.6% to 85.7%; on the contrary, the switching frequency difference of the computational-intensive load in the two environments is always less than 0.5%. This difference directly verifies the effectiveness of the "scenario-index" mapping rule defined by the method: for I / O-sensitive scenarios, "interrupt trigger rate and event processing delay" should be included in the performance loss quantification model; for computational-sensitive scenarios, a simplified CPU time slice drift indicator can be used. Further comparison of the curve slopes under different concurrency degrees shows that the bare machine scheduler expands approximately linearly with the number of threads, while the virtualization environment expands sublinearly due to the fixed limitation of the virtualization system scheduling granularity, thereby providing a predictable performance boundary in the multi-tenant high-density deployment scenario. This result not only proves the repeatability and discrimination of the test method on real hardware platforms, but also provides a calibration benchmark for subsequent joint modeling based on multiple indicators such as interrupt response delay, which can be directly used to guide parameter optimization and resource reservation strategy formulation of virtualization systems in multiple scenarios.
[0126] Figure 7 is the isolation test experiment result display graph of the security test of the embodiment of the application.
[0127] The test program was run on the operating system under the condition that the CPU and memory were put into heavy load by running Stress command on the Linux virtual machine managed on the same virtualization system. By comparing the running time of the test program under the condition of Linux idle, the final conclusion is that the difference of the running time of the program is very small, and the virtualization system realizes the effective isolation of resources. The results are shown in Table 6.
[0128] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they have the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the application.
[0129] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.
[0130] The protection scope of the present application is not limited to the above embodiments. Changes and advantages that can be thought of by those skilled in the art without departing from the spirit and scope of the present application are included in the present application, and are protected by the appended claims.
Claims
1. A testing and evaluation method for virtualization systems in multiple scenarios, characterized in that, The executable file compiled from the test script is deployed on the virtualization client, and the performance of the higher-level guest operating system is tested to determine the performance consumption of the underlying virtualization system on computer resources; the method includes the following steps: Step 1: Create a project and configure system parameters; Step 2: Integrate the test program into the project and generate an image file; Step 3: Package the complete virtualization system image; Step 4: Transfer the image to the development board via network protocol; Step 5: Execute the command to start the system and log in to the system console via the serial port; Step 6: Call and execute the functional test, performance test, and security test scripts.
2. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 1, characterized in that, The test cases for the virtualized system deployed in the virtualization system are the functional scripts written in programming languages and their dependent tools; The virtualization system environment built in the virtualization system is a real-time operating system and a supporting integrated development environment; In the virtualization system, the transmission process of building the virtualization system involves packaging the generated image file to create a complete virtualization system image, which can be transmitted to the development board via network protocol for deployment of testing methods in different scenarios. The test configuration is executed on the virtualization system performance testing method. Its configuration information includes engineering system parameters, number of consoles and CPUs, storage path of performance data and logs, and is used for building the test environment and initiating performance test requests and receiving data.
3. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 1, characterized in that, The functional tests include: basic virtualization function testing, task preemption mechanism testing, event management function testing, and interrupt handling mechanism testing; specifically, they include the following sub-steps: Step A1: Obtain the number of CPUs that the operating system can detect using the lscpu command; Step A2: Obtain the amount of available memory in the operating system; Step A3: Analyze whether the records match the allocated virtualization resource amount; Step A4: Create threads with different priorities; Step A5: Run the low-priority thread and hold the semaphore resource, suspend the high-priority thread; Step A6: Wake up suspended threads out of order and record the priority of inverted tasks; Step A7: Build a multi-priority, multi-threaded benchmark program and insert event flags; Step A8: Verify and record the event functionality and concurrency correctness, and measure the event latency; Step A9: Measure and statistically analyze interrupt response latency, interrupt nesting depth, and interrupt enable / mask accuracy.
4. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 3, characterized in that, The basic virtualization function test uses the lscpu command to check whether the number of CPUs that the operating system can perceive matches the amount of virtualization resources allocated, and analyzes whether the amount of memory available in the operating system matches the amount of virtualization resources allocated. And / or, The task preemption mechanism test includes task preemption time test, task execution order test, and task inversion analysis. Specifically, the task preemption time test, under a priority scheduling strategy, determines how long a high-priority thread must wait after making a request before acquiring resources held by a low-priority thread. This includes the following experiments: Experiment 1: Create a low-priority thread to preempt a semaphore; when a high-priority task is created, it first waits for the semaphore. The experiment tests the time taken from when the low-priority task releases the semaphore to when the high-priority task acquires the semaphore. Experiment 2: Keep a low-priority thread running a computationally intensive task in a loop, and create a high-priority task to immediately preempt the CPU; record the difference between the preemption trigger time and the timestamp of the high-priority task execution. Experiment 3: Randomly create threads of different priorities, all in a suspended state upon creation, and simulate real scheduling by waking them up out of order; record each instance of priority inversion in the log; measure metrics including average response time and preemption flag; inversion is determined by task waiting time, and high-priority tasks waiting for more than 3ms are considered to have unreasonable scheduling and a certain degree of inversion, and the actual scheduling order is printed out to check whether inversion has indeed occurred; And / or, The event management function test focuses on its reliability, performance, and concurrent processing capabilities, including event delivery success rate to ensure that events are not lost during inter-task communication; synchronous / asynchronous event latency to quantify the system's response speed in processing events; and concurrent event processing accuracy to verify whether the system can handle multiple simultaneous events without errors under high load. And / or, The interrupt handling mechanism test includes testing the system's interrupt response latency, calculating the interrupt nesting depth, and statistically analyzing the interrupt enable / disable accuracy. The interrupt response latency test is the time difference between the interrupt trigger and the start of the interrupt service routine execution, which evaluates the system's response speed to interrupt events; The interrupt nesting depth test verifies whether the system supports interrupt nesting mechanisms, tests whether a high-priority interrupt can successfully interrupt a low-priority interrupt handler that is currently executing, and counts the interrupt nesting level; The interrupt enable / disable accuracy test involves repeatedly executing interrupt enable and disable operations, counting the number of successful operations, and evaluating the correctness and stability of the interrupt control function.
5. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 4, characterized in that, First, a multi-priority, multi-threaded baseline program is built in each client machine, and event flags, synchronous / asynchronous event sending and receiving interfaces, and high-precision timestamp acquisition modules are implanted to form a controllable concurrent load; Secondly, verifying the event functionality and concurrency correctness includes: injecting an eight-level priority, synchronous / asynchronous mixed event stream into the event queue, recording the success rate and order consistency of event creation, delivery, and reception to verify that messages are not lost or out of order and that the priority strategy is effective; concurrently triggering N events in the same virtual machine and cross-virtual machine scenarios, where N is not less than the number of virtual CPU (vCPU) cores allocated to the guest machine to ensure that each vCPU core has at least one event to process, monitoring the event processing results and system status to confirm that there are no errors in parallel event processing under high load; finally, measuring the full path latency of synchronous and asynchronous events from publication to complete processing by the target thread; Repeated sampling and output of minimum, average, and maximum latency values to quantitatively evaluate event management performance.
6. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 1, characterized in that, The performance tests include: CPU performance testing, memory allocation performance testing, inter-process communication performance testing, and deadlock detection and recovery mechanism testing; specifically, they include the following sub-steps: Step B1: Create threads with different priorities; Step B2: Run the low-priority thread and hold the semaphore resource, suspend the high-priority thread; Step B3: Count the number of context switches and context latency; Step B4: Run a unified stream benchmark set to generate continuous memory read / write loads; Step B5: Compile data for each memory allocation-access-release cycle; Step B6: Create two or more threads to request / hold two sets of mutually exclusive resources in a fixed order; Step B7: Start the deadlock detection interface and calculate the detection and recovery latency; Step B8: Introduce threads that hold resources for a long time but have no circular dependencies, and count the number of false alarms and the recovery success rate.
7. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 6, characterized in that, The CPU performance test evaluates the efficiency of system task scheduling and processor resource management. The number of context switches per unit time is calculated by simulating multi-threaded or multi-process concurrent execution scenarios. Context switch awareness hooks are added to threads, and the count is performed before each operation to obtain the context switch count for programs with different characteristics. Context latency, i.e., the specific time required to complete one task switch, is measured by creating threads of different priorities. Low-priority threads keep running and hold semaphore resources upon creation, while high-priority processes are suspended waiting for the semaphore. When a low-priority thread releases the semaphore, the switching time is obtained by comparing the semaphore release timestamp with the timestamp of the high-priority thread starting execution. By analyzing these two core reference indicators, the performance of the CPU scheduler is quantitatively evaluated, providing a basis for system optimization and resource allocation strategies, and ensuring that the system can still run efficiently and stably under high load. And / or, The memory allocation performance test evaluates memory allocation / release time by running a unified stream benchmark test set with a fixed number of iterations and data block size to generate a controllable continuous memory read / write load to eliminate external disturbances. A high-precision timer embedded in the benchmark program is used to record the time consumed in each memory allocation-access-release cycle, extracting the minimum, average, and maximum times and the optimal bandwidth. After repeated sampling, the average time difference is calculated to evaluate the allocation overhead introduced by the virtualization system. And / or, The inter-process communication performance test is used to evaluate the efficiency and stability of the system in transmitting data between multiple processes or threads, including single message transmission latency: the time required from the sending end to the receiving end receiving the message; batch message throughput: the number of messages that the system can successfully send and receive per unit time; communication error rate: including packet loss, duplicate messages, out-of-order messages, etc. And / or, The deadlock detection performance test is to evaluate the efficiency and accuracy of the system in handling deadlock problems, including the deadlock detection execution time, which is the time spent from triggering the deadlock detection to completing the detection. Recovery success rate: the percentage of blocked threads that the system can successfully recover after a deadlock is detected; false deadlock rate: the frequency at which the system falsely reports a deadlock when there is no actual deadlock. Preferably, First, a controllable deadlock scenario is constructed by creating 2-N threads that artificially form a circular wait graph by requesting / holding two sets of mutually exclusive resources in a fixed order, triggering the system's deadlock detection mechanism. Second, the detection and recovery latency are quantified by starting the deadlock detection interface and recording the time from the formation of the circular wait to the system's deadlock determination, as well as the time from the determination to the termination of the lowest priority thread and the continuation of the remaining threads. Multi-threaded tests are repeated, and statistical values are extracted. Finally, the correctness and false alarms are verified by introducing threads that hold resources for a long time but have no circular dependencies under the same load. This verifies that the system only determines deadlock for genuine circular waits, and the number of false alarms is zero. The recovery success rate is also recorded to confirm the correctness and stability of the deadlock resolution strategy.
8. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 1, characterized in that, The security performance test includes: isolation performance test and anomaly handling robustness test; specifically, it includes the following sub-steps: Step C1: Use the stress command to load the CPU and memory; Step C2: Run a benchmark program with a fixed amount of computation and record the execution time in multiple scenarios; Step C3: Calculate the average time deviation rate between the high-load scenario and the baseline scenario and determine the isolation. Step C4: Inject three types of exceptions into the system under test by calling the function; Step C5: Record the measurement anomaly capture and corresponding delay to verify robustness indicators.
9. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 8, characterized in that, The isolation performance test involves running CPU-intensive test programs and memory access test programs under different load conditions on a Linux virtual machine to verify whether the virtualization system can maintain resource isolation under high load conditions. Ensure that resources of different virtual machines do not interfere with each other in the virtualization environment; test the isolation capability of the virtualization system under different load scenarios to ensure stable task execution time under high load conditions of Linux virtual machines; The robustness test for exception handling involves triggering a preset exception and calculating the exception handling success rate; recording the complete time chain from exception triggering to handling and verifying the response time; checking the accuracy of error code returns and verifying the system state recovery; observing whether the system can correctly return errors and set appropriate error codes when operating on a closed message queue descriptor; whether the system can detect that the message exceeds the limit and return an EMSG-SIZE error when sending an excessively large message to the message queue; and whether the system can correctly capture and return an ESRCH error when sending an event to a terminated thread.
10. The testing and evaluation method for virtualization systems in multiple scenarios as described in claim 9, characterized in that, First, three types of exceptions are injected into the system under test via API calls: performing operations on a closed message queue descriptor; sending an excessively large message exceeding the length limit; and sending an event to a terminated thread. Each type of exception is triggered 100 times, forming a repeatable exception sequence. Second, exception capture and response latency are measured. Using the exception hooks and high-precision timers provided by the system, the time elapsed between each exception trigger and the system capturing and returning an error code is recorded. The average time elapsed for the 100 triggers is calculated as the exception response latency. Finally, robustness is verified by calculating the capture success rate of the 100 exception triggers to confirm whether it reaches 100%. At the same time, the returned error codes and system state recovery are checked to see if they meet expectations. If the capture rate is 100% and the average response latency is less than 100µs, the exception handling mechanism is deemed to have reliability and real-time performance.
Citation Information
Cited By
Distributed MQTT high-concurrency connection testing method based on ZeroMQ
CN121486245A
