System and method for operating system fault injection and experiment orchestration based on virtualization technology

Abstract

This invention discloses an operating system fault injection and experiment orchestration system and method based on virtualization technology. Without modifying the guest operating system source code, the system, on the host side, applies controlled perturbations to the processor register state variables, memory data variables, and input / output state variables perceptible to the guest operating system. This effectively simulates the impact of SEU and MBU effects caused by space radiation or extreme environments on the system, and reproduces the resulting data errors, anomalies, or system crashes. A target descriptor containing GPA and verification information is encapsulated by a guest agent at the guest layer and a deterministic workload is run. The host side performs security verification and fault injection based on this descriptor, and confirms the injection result through a closed-loop verification mechanism with at least two independent paths. This invention achieves a safe, controllable, and reproducible radiation-resistant reliability assessment.

Description

Technical Field

[0001] This invention belongs to the field of software fault tolerance and reliability testing technology, and in particular relates to a system and method for operating system fault injection and experiment orchestration based on virtualization technology. Background Technology

[0002] With the increasing computing power of spacecraft payloads and the development of space-based cloud and space-based intelligent computing platforms, more and more critical missions, such as on-orbit target detection and recognition, real-time processing of remote sensing data, and inter-satellite collaboration and autonomous decision-making, are beginning to run complex operating systems, virtualization layers, and intelligent inference frameworks in orbit. The space radiation environment contains high-energy particles, cosmic rays, and space ionization effects. Combined with factors such as reduced critical charge of devices under advanced processes and process fluctuations, single-event upsets (SEUs) and multiple-bit upsets (MBUs) can easily occur in processor registers, on-chip caches, main memory, and storage media, leading to problems such as abnormal process exits, system crashes, service suspensions, or deviations in mission results. For operating systems designed for spaceborne intelligent computing platforms, their fault tolerance under radiation interference directly affects the reliability of on-orbit missions and service continuity. Therefore, quantifiable and reproducible fault tolerance assessments and fault mechanism analyses are necessary.

[0003] Existing fault tolerance assessment methods mainly include: hardware irradiation experiments based on irradiation sources, fault triggering using hardware mechanisms (such as machine checks and fault injection interfaces), and fault injection simulation at the software layer. Among these, irradiation experiments can realistically reflect on-orbit radiation effects, but they are costly and lack controllability and repeatability. Hardware-supported fault injection usually depends on specific chip capabilities or platform interfaces, limiting its coverage and making it difficult to perform fine-grained control over complex interactions at the software level, such as virtualization stacks and cloud-native runtimes. Software-simulated fault injection has advantages such as low cost, programmability, and easy scalability, making it suitable for large-scale experiments and regression verification on the ground. Therefore, it has significant value in the reliability assessment of space-based intelligent computing platforms.

[0004] However, in the virtualized multi-user scenario of aerospace intelligent computing, existing software fault injection schemes still face several key problems. This is due to the reshaping of "address semantics" and "state visibility" by the virtualization architecture, as well as the uncertainties brought about by cross-layer concurrency. Specifically, the addresses used by applications and the kernel within the guest machine are typically virtual addresses. These are first converted to guest physical addresses (GPAs) via the guest page table, then mapped to host physical addresses (HPAs) by a two-level address translation mechanism, and finally placed into the physical organization of the actual storage medium. This link exhibits dynamic changes across multiple rounds under the influence of factors such as address space randomization, differences in loading order, memory allocator state, restart cycles, and resource scheduling. This makes it difficult to guarantee a stable target landing point when relying solely on a single address value for injection, easily leading to false injections caused by target page frame reuse or mapping drift. Meanwhile, even if the host machine completes a write to a certain physical location, it does not necessarily mean that the disturbance will "take effect" in the client machine's semantics. Registers and multi-level caches may mask the flip results in memory, TLB / EPT caches may cause inconsistencies in address translation results in a short period of time, and ECC error correction, parity checking, and upper-level verification recovery mechanisms may transparently mask the flip or prematurely terminate its propagation in an abnormal form, thus producing injection "pseudo-success" or "pseudo-failure," which manifests as injection reporting success but the client machine not observing the disturbance, or injection reporting failure but the disturbance has already occurred. In addition, fault injection, workload triggering, system state sampling, and result archiving naturally span the clock domain and execution domain of both the host and client machines, and are significantly affected by scheduling jitter, interrupts, and I / O latency. Without a unified round identifier and atomic recording mechanism, it is difficult to establish a replayable causal chain, making experiments difficult to program, compare, and reproduce. More critically, modifications to the client's state by the host are high-privilege operations. Without rigorous verification of source authorization, field integrity, address range, and mapping attribution, there is a risk of unauthorized writes to the host or other tenant's memory, potentially leading to uncontrollable security and stability consequences. Therefore, a fault injection and experimental orchestration scheme is needed that can establish a verifiable cross-layer target localization and injection loop without intruding on the client's source code, and that uses strong verification to constrain security boundaries and unified orchestration to achieve consistent records throughout the entire process.

[0005] Furthermore, existing fault injection implementations related to virtualization technologies often rely on simulator extensions, debugging interfaces, or intrusive modifications to the client and host machines. These solutions still suffer from problems such as manual target address location, weak cross-round reproducibility, a single injection verification chain, and a lack of unified management of experimental plans and results. Therefore, there is an urgent need for a fault injection and experimental orchestration scheme for aerospace intelligent computing virtualization technology. This scheme should achieve reproducible, verifiable, and scalable single-bit and multi-bit fault injection, while ensuring secure isolation and controllable boundaries. It should also integrate this with workload triggering, state sampling, and result recording into a closed loop, thereby supporting the fault tolerance assessment and reliability optimization of the intelligent computing platform's operating system and virtualization stack. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and proposes an operating system fault injection and experimental orchestration system and method based on virtualization technology. Under the premise of ensuring secure isolation and controllable boundaries, it achieves a reproducible, verifiable, and scalable closed loop for fault injection and data acquisition.

[0007] In view of this, the present invention provides an operating system fault injection and experiment orchestration system based on virtualization technology, used to equivalently simulate single-event flip and multi-event flip effects induced by space radiation or extreme environments. The system includes: A client agent, deployed at the client layer, is used to trigger or coordinate the execution of deterministic workloads. During the deterministic phase, it acquires the original target location data and verification data corresponding to the injection target, generates a target descriptor, and reports it to the host machine. The target descriptor contains injection target information and verification information. The client layer runs on the virtualization platform of the host machine. The host controller, deployed at the host layer, is used to generate experimental rounds and injection plans, control the virtualization platform to load a brand-new client-side operating system based on the original image, establish controlled communication with the client agent, establish fault injection configuration, and also receive target descriptors. After performing source authorization, mapping range verification, security boundary verification, field integrity verification, and validation of the target descriptors, it calls the fault injector to prepare for fault injection. The fault injector, deployed at the host layer, is used to perform fault injection operations on a specified target based on the injection target information of the target descriptor; and the closed-loop verification module, deployed at the host layer, is used to obtain the injection results at the target location through two independent paths after the fault injection is completed. One path is a direct read-back path from the host side, and the other is a path visible to the guest or a virtual machine monitoring interface read-back path. The consistency of the results returned by the two independent paths is used to determine whether the fault injection is effective.

[0008] As an improvement to the above system, the system also includes a state sampling and result classification module, which is used to collect the system state and workload output before and after injection, and automatically classify the experiment into masking, silent data error, crash, suspension or performance degradation based on the injection verification results, and output an expandable tag set.

[0009] As an improvement to the above system, the system also includes an experimental result atomic recording module, which is used to record experimental round identifiers, target descriptors, injection configuration parameters, verification results and result classification information to support experimental result analysis and experimental reproduction.

[0010] As an improvement to the above system, the virtualization platform is used to provide virtualization resources to the client layer and to provide the host machine with a virtual machine state access interface for the client layer.

[0011] As an improvement to the above system, the injection target information includes the client physical address or I / O logical block address, page offset, and bit index; the verification information includes the original value digest. The target descriptor shall include at least: round identifier, virtual machine identifier, target type, specific location of fault injection, and stage identifier.

[0012] As an improvement to the above system, the client agent locks at least one injection target according to deterministic rules in a predetermined phase and obtains the client physical address and page offset or bit index corresponding to the injection target.

[0013] As an improvement to the above system, the fault injection configuration established by the host controller includes at least: fault domain type, flip bit, injection timing or running phase / time window, and random seed.

[0014] As an improvement to the above system, the fault injector includes at least one of a memory fault injector, a CPU fault injector, and an I / O fault injector; wherein, The memory fault injector is used to convert the client's physical address into the host's virtual address, and after completing the security verification, performs bit flipping, setting, clearing, replacement or other preset fault injection operations on the target memory; the bit flipping is random or selected according to a preset strategy; The CPU fault injector is used to read and write registers through the debug interface and perform bit flip, set, clear, replace or other preset fault injection operations in the virtual machine paused state before switching to the running state; The I / O fault injector is used to intercept virtual block device I / O requests and perform bit flipping, setting, clearing, replacement or other preset fault injection operations based on the mapping from logical sector to mirror offset.

[0015] As an improvement to the above system, the security verification includes at least: process identity, storage mapping range, and whitelist verification.

[0016] On the other hand, this invention proposes a method for operating system fault injection and experiment orchestration based on virtualization technology. According to the above system implementation, the method includes: Step 1: Initialization. The host controller loads the experimental configuration, generates a random seed and round plan, controls the virtualization platform to load a brand-new guest layer operating system from the original image, and establishes controlled communication with the guest agent. Step 2: Target release. During the deterministic phase, the client agent locks the injection target and generates a target descriptor, which is then received by the host controller. Step 3: Legality verification and injection plan generation. After the host controller performs source authorization, mapping range verification, security boundary verification, field integrity verification and verification of the target descriptor, it calls the fault injector to prepare for fault injection. Step 4: Fault injection execution. The fault injector performs bit flipping, setting, clearing, replacement or other preset fault injection operations on the specified target according to the injection target information of the target descriptor. Step 5: Closed-loop verification. After each fault injection, the closed-loop verification module obtains the injection result of the target location through two independent paths. One path is the direct read-back path on the host side, and the other is the visible path on the client or the read-back path through the virtual machine monitoring interface. Based on the consistency of the results returned by the two independent paths, it determines whether the fault injection is effective and records the evidence. Step 6: Triggering and Sampling. The host controller continues or triggers the client workload to collect system status and output results. Step 7: Classification and Atom Storage. The state sampling and result classification module automatically classifies the experimental results, and the experimental results atom recording module writes the atom into storage. Step 8: Reproducibility experiment. The host controller replays the experiment rounds with the same configuration and random seed to verify reproducibility.

[0017] Compared with the prior art, the advantages of the present invention are: Automatic localization and stable reproduction across rounds: By combining the host machine's initial loading of the client-side operating system based on the original image, a consistent initial running environment can be further ensured across different experimental rounds. Through the client agent generating and reporting target descriptors in each round, deterministic release and stable localization of the injection target are achieved. The host machine can perform batch injection without manually searching for the target address. By binding the target localization to the round execution context, even if the virtual address, physical page mapping, or runtime layout changes between different rounds, deterministic release and stable reproduction of the injection target can still be achieved, significantly improving the automation level of batch experiments and regression verification.

[0018] Verifiable and auditable: Closed-loop verification is a mandatory step, which reads back the values ​​before and after the injection through at least two independent paths to verify consistency, significantly reducing the probability of false success and false failure, and forming an auditable chain of evidence to improve the credibility of experimental conclusions.

[0019] Unified orchestration and strategy randomization: A unified configuration file describes strategies such as random seed, random bit, random register, random offset, multiple flips per round, and trigger timing, supporting large-scale experiments and regression validation.

[0020] Multi-fault domain coverage: A single controller uniformly schedules three types of SEU and MBU models (CPU, memory, and I / O), covering key fault domains of the space-based intelligent computing operating system. This reduces the cost of toolchain fragmentation and supports fault models such as SEU and MBU. Compared to existing technologies that build tools and design processes separately for different fault domains, this invention achieves unified expression, execution, and recording of multi-fault domain injection capabilities on the same experimental platform. This provides more comprehensive coverage of critical vulnerable parts in the space-based intelligent computing operating system and reduces the adaptation, maintenance, and analysis costs caused by fragmented experimental toolchains.

[0021] Security boundary strengthening: Taking the authorization and verification of target descriptors as the core, and forcibly verifying the identity of virtualization processes, target address mapping range and write granularity on the host side, effectively reducing the risk of accidentally writing to the host's critical metadata area or other tenant memory.

[0022] Data integrity and ease of analysis: Each round of automatic collection of system status, survival detection results and workload output supports the classification of results into masking, silent data errors, crashes, suspensions or performance degradation, etc., providing a data closed loop for the fault tolerance hardening and optimization of the space-based intelligent computing operating system.

[0023] Through the cooperation of various modules, the problems of fault injection target address drift, unverifiable injection results and ambiguous security boundaries in virtualized environments have been solved, and a radiation-resistant reliability assessment that is safe, controllable and reproducible has been achieved. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall system structure of the present invention; Figure 2 This is a flowchart of a memory fault injection process; Figure 3 This is a flowchart of the CPU fault injector injection process; Figure 4 This is the I / O fault injection flowchart; Figure 5 This is the overall flowchart for fault injection. Detailed Implementation

[0025] To facilitate understanding of this invention, some of the terms used are defined as follows: Single-event flip (SEU): SEU refers to a soft error in which a single bit is flipped. Multiple bit flip (MBU): MBU refers to a soft error in which multiple bits are flipped, and these multiple bits may have spatial or temporal correlation. Host: A host is a physical computing node that hosts a virtualization platform and runs virtual machine instances; Guest: The guest refers to the virtual machine instance. Client agent: A component within the client machine for target location and status reporting; Guest OS: A guest OS refers to the operating system that runs within a guest machine. Client Physical Address (GPA): GPA refers to the physical address from the perspective of the client-side operating system; Deterministic workload: refers to a workload that produces stable and comparable output under fixed input and fixed execution rules, used to stably locate the injection target and output a verification digest for result determination; Target descriptor: Injection target description information generated by the client agent and sent to the host controller.

[0026] This invention proposes an operating system architecture-level fault injection and experiment orchestration scheme based on a virtualization platform. This scheme is compatible with implementations such as KVM / QEMU / Libvirt, but is not limited to them; it can also be adapted to other virtualization or isolated runtime environments.

[0027] The key to this invention lies in constructing a target descriptor mechanism to address address drift and security boundary issues. Specifically, the target descriptor transforms the positioning reference of the injection target from a virtual address / object address that changes with each round to the client's physical address (GPA) and page offset / bit index, which are re-resolved by the client during the deterministic phase of the current round. Since the client outputs the GPA and a positioning field for verification on the physical page frame where the target object actually resides in each round, even if address drift occurs due to address space randomization, restarts, or memory reclamation scheduling, the host machine can still stably and definitively locate the actual injection target and execute the injection based on the target descriptor of the current round. Furthermore, the target descriptor serves as the sole authorized input for host machine injection. Before injection, the host machine must perform source authorization verification, field integrity verification, and verification field validation on the descriptor. Injection is only performed on descriptors that pass the validation and meet the policy requirements. In memory injection scenarios, the GPA must be converted to a host-accessible address via a monitoring interface. Before writing, the identity of the virtualization process hosting the virtual machine, the writable attributes of the target address's mapping range, and the optional whitelist must be forcibly verified. This limits the write boundary to the target virtual machine's object memory backend mapping area, preventing accidental writing to the host machine or other tenant memory. "Closed-loop verification" is designed as a mandatory step, requiring at least two independent paths to read back and verify the consistency of values ​​before and after injection, forming a verifiable chain of evidence.

[0028] 1) Overall System Structure Host controller: Used to generate experimental rounds and injection plans, configure random seeds and strategies, trigger client workloads, execute delayed or conditional scheduling, call CPU, memory, and I / O fault injectors, collect host and client system status, output structured experimental results, and perform atomic recording and summarization.

[0029] Client Agent and Deterministic Workload: Running within the client operating system, the deterministic workload executes deterministic tasks with fixed inputs and generates verification results; the client agent acquires the target location-related raw data generated by the workload in a predetermined phase of each round of operation, parses its client physical address GPA and page offset, generates and reports the target descriptor; it can also report key OS statuses (heartbeat, log digest, critical process liveness, etc.) for result classification.

[0030] Target descriptor generation and authorization channel: The target descriptor must contain at least the round identifier, virtual machine identifier, target type (CPU, memory, I / O), GPA or I / O offset, byte index, bit index, target stage identifier, original value digest and verification field; the host machine only performs injection on target descriptors that meet the policy and verification, and performs authorization and consistency verification on the source of the descriptor to prevent forged targets or unauthorized injection, forming the core grip of the security boundary.

[0031] Memory fault injector: Based on the GPA, offset, and bit index in the target descriptor, complete the address translation link (GPA to host-accessible address), and force verification of the virtualization process identity and the writable attribute of the mapping area where the target address is located before writing; then perform single-byte or limited-word-length bit-flipping write; after completion, perform closed-loop verification, including at least two independent paths of consistency verification: host-side readback and guest-visible path readback (or GPA readback through the virtual machine monitoring interface), and record the evidence.

[0032] CPU Fault Injector: Based on the set of registers and bit indexes specified by the target descriptor, the virtual machine is placed in a consistent state (e.g., CPU is paused) at a controlled time, the target register value is read and single-bit or multi-bit flips are performed and then written back; the modified result is then verified through an independent readback path and the operation is resumed; fixed register or random register strategies are supported, and the injection stage and timestamp are recorded for reproduction.

[0033] I / O fault injector: Based on the mapping from LBA to mirror offset in the I / O path of the virtual block device, it performs controlled bit flipping on read and write data, so as to realize the controllable simulation of silent data corruption while maintaining the I / O success semantics and the integrity of the file system structure.

[0034] State sampling and result classification module: Collects the state of the host and client before and after injection (heartbeat, logs, critical service status, crash indicators, performance / latency indicators, etc.), automatically classifies the results into masking, silent data error, crash, suspension or performance degradation, etc., and outputs an expandable tag set.

[0035] Experimental Results and Atomic Recording Module: The plan, injection events, verification evidence, system status and workload results of each round of experiments are output in a structured manner, and an atomic writing strategy is used to ensure the consistency of "plan, execution, verification and results", which facilitates statistical analysis, regression verification and audit traceability.

[0036] 2) Methodology and Flow (Closed-loop verification is a mandatory step) 1. Initialization: The host controller loads the experimental configuration, generates a random seed and round plan, controls the virtualization platform to load a brand-new client-side operating system based on the original image, establishes a controlled communication channel with the client agent, and initializes the recording process.

[0037] 2. Target Release: The client agent allocates and locks the target memory region or target object at the specified stage of the workload, generates a target descriptor and writes it to the result file or reports it to the host machine through a controlled channel; the target descriptor contains fields and verification information for security verification and reproduction.

[0038] 3. Legality verification and injection plan generation: The host controller performs source authorization, field integrity and verification field verification on the target descriptor, and selects the fault type, injection timing (fixed delay, stage trigger, event trigger, etc.) and flip bit (single bit or multiple bits) according to the strategy.

[0039] 4. Fault Injection Execution: The corresponding injector is called according to the injection plan; the memory fault injector performs address translation and verifies the process identity and the mapped writable range before performing bit flipping; the CPU fault injector reads and modifies registers after pausing the virtual machine; the I / O fault injector performs controlled bit flipping on the specified offset based on LBA mapping.

[0040] 5. Closed-loop verification: For each injection, at least two independent paths must be used to perform readback consistency verification, and the values ​​before injection, after injection, expected flip bits, and verification results must be recorded. The two paths must include at least one of the host-side readback path and the client-side visible path (or the monitoring interface reads back the client address) to eliminate false successes or false failures caused by address drift and mapping changes.

[0041] 6. Triggering and Sampling: Continue or trigger the execution of deterministic workloads on the client side, collect the system status, log summaries and output results of the host and client systems, and form a comparable state snapshot.

[0042] 7. Classification and Atomic Write to Disk: Generate structured results from the current round plan, injection events, verification evidence, system status, and workload results and write them atomically; simultaneously generate summary CSV or JSONL for statistical analysis and visualization.

[0043] 8. Reproducibility Experiment: Replay the round plan based on the same configuration and random seed to ensure that the experiment can be reproduced across restarts and rounds.

[0044] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0045] Example 1 Embodiment 1 of the present invention proposes an operating system fault injection and experiment orchestration system based on virtualization technology. It is used to equivalently simulate single-event upsets and multi-event upsets induced by space radiation or extreme environments. The system includes: A client agent, deployed at the client layer, is used to trigger or coordinate the execution of deterministic workloads. During the deterministic phase, it acquires the original target location data and verification data corresponding to the injection target, generates a target descriptor, and reports it to the host machine. The target descriptor contains injection target information and verification information. The client layer runs on the virtualization platform of the host machine. The host controller is used to generate experimental rounds and injection plans, control the virtualization platform to load a brand-new guest layer operating system based on the original image, establish controlled communication with the guest agent, establish fault injection configuration, and also to receive target descriptors. After performing source authorization, mapping range verification and security boundary verification, field integrity and verification of the target descriptors, it calls the fault injector to prepare to execute fault injection. The fault injector is used to perform fault injection operations on a specified target based on the injection target information of the target descriptor. The closed-loop verification module is used to obtain the injection results at the target location through two independent paths after the fault injection is completed. One path is a direct read-back path from the host machine, and the other is a visible path from the client machine or a read-back path from the virtual machine monitoring interface. The module determines whether the fault injection is effective based on the consistency of the results returned by the two independent paths.

[0046] The state sampling and result classification module is used to collect system state and workload output before and after injection. Combined with the injection verification results, the experiment is automatically classified into masking, silent data error, crash, suspension or performance degradation, and an expandable tag set is output.

[0047] The experimental results atomic recording module is used to record experimental round identifiers, target descriptors, injection configuration parameters, verification results, and result classification information to support experimental result analysis and experimental reproduction.

[0048] like Figure 1 As shown, the interaction relationships between the host controller, client agent, target descriptor, and fault injector are illustrated.

[0049] It should be noted that this embodiment uses a KVM / QEMU virtualization environment and a Linux operating system as examples for illustration. However, in practical applications, this invention is also applicable to other virtualization platforms such as Xen, VMware, and Hyper-V, as well as other guest operating systems such as Windows and VxWorks. The specific instructions mentioned in this document (such as virsh and QMP) are merely one specific way to implement the technical concept of this invention and should not be construed as limiting the scope of protection of this invention.

[0050] 1) Host Controller The host machine is responsible for round scheduling, virtual machine workload triggering, injector invocation, status acquisition, and result persistence. The system runs a controller program on the host machine, which interacts with the target virtual machine through virtualization management tools and interfaces such as virsh, QMP, and QGA. Before each round of experiments, the controller controls the virtualization platform to start or restore a completely new guest operating system instance based on a preset original image, ensuring a consistent initial software state and operating environment for each round of experiments. The controller generates a unique round identifier (rid) for each round of experiments (e.g., UUID / timestamp + sequence number) and creates a directory structure to store the results of that round. This directory must contain at least the following files: injection.json, system_state.json, guest_output.json, and summary.json / summary.csv.

[0051] The host controller triggers the client workload script by executing guest-exec through the client agent interface. The workload writes structured output to the file system. The controller then triggers the injector script after a random or fixed delay within the run window, and after each round, it pulls the client output, collects the system status, and writes it to a summary record.

[0052] To improve stability across rounds, the controller performs a liveness probe and environment consistency check before the start of each round. The check includes at least: virtual machine status (running or paused), QEMU process existence, image path consistency, and interface availability for injection. If the probe fails, the fail_reason is recorded and the virtual machine can be reset / restarted to avoid cascading failures in subsequent rounds.

[0053] Interface example: virsh: Start, stop, and reset virtual machines, and obtain domain name and PID clues (by querying libvirt); QMP: Executes monitoring commands such as memory read (xp), address translation (gpa2hva), virtual machine stop or cont; QGA: Triggers guest-exec, pulls client output files, and performs liveness detection (guest-ping / guest-info).

[0054] 2) Virtual Machines and Deterministic Load Modules The system performs deterministic computation tasks within the virtual machine and outputs the GPA of the memory injection target. The system runs a controller program on the host side and outputs a fixed-length checksum to determine whether the injection causes a silent data error (SDC). The controller interacts with the target virtual machine through virtualization management tools and interfaces virsh, QMP, and QGA.

[0055] To achieve "automatic location and reproducibility", the workload allocates an anonymous memory page as the target page at the beginning of each round of operation, writes the initial value pattern to the preset offset known_off, and outputs the guest physical address GPA (or page base address + offset) of the page, so that the host machine can complete the memory injection without manual location.

[0056] The fault injector includes at least one of the following: memory fault injector, CPU fault injector, and I / O fault injector.

[0057] 3) Memory Fault Injector Module The memory fault injector reads the byte value at the target GPA through the virtual machine monitoring interface and translates the GPA into a host virtual address HVA; after completing security verification, it uses / proc / <pid>` / mem` performs a bit flip on the target byte and verifies the injection's effectiveness by reading back via a dual-path consistency check. For example... Figure 2 The diagram shown is a flowchart of a memory fault injection process, which includes address translation, security verification, and closed-loop verification steps. 4) CPU Fault Injector Module The CPU injector exposes register read / write interfaces via a QEMU debug stub (gdbstub). While the virtual machine is paused, it uses a gdb batch script to toggle individual bits of selected registers and performs read-back verification and logging before resuming operation. For example... Figure 3 The diagram shown is a flowchart of the CPU fault injector injection process, which includes steps for pausing the virtual machine, reading and modifying registers, and resuming operation.

[0058] 5) I / O fault injector module The I / O injector performs bit flipping on the virtual machine disk image file. To improve semantic consistency and controllability, an optional mapping mechanism of "virtual block device logical sector to LBA image offset" can be used before flipping the target bytes, thereby avoiding uncontrollable damage to metadata caused by simple offset flipping. For example... Figure 4 The diagram shown is an I / O fault injection flowchart, illustrating the memory bit flipping logic based on LBA mapping. Figure 5 The diagram shown is the overall flowchart of fault injection, illustrating the data flow during fault injection in the system.

[0059] The state sampling and result classification module stores information such as guest_test, injection, and system_state according to a unified schema, and writes the summary of each round to the summary file in an atomic append-only manner using JSONL / CSV. Each summary must contain at least the following fields: rid, fault_type, target (GPA / HVA / LBA), before / after / expected, verify_path, alive, checksum, classification, and fail_reason.

[0060] The specific processing steps include: Client-side deterministic workloads, target location information acquisition, and target descriptor reporting: The client workload performs deterministic matrix calculations and outputs a fixed-length checksum hash, which is used to determine whether an injection has caused silent data errors.

[0061] To address the issue of injection targets changing with each round or restart, the deterministic workload allocates one page of anonymous memory (target_page) in each round of operation. Within this page, an initial value (pattern) is written at a preset offset (known_off) to generate raw data and verification data related to the location of the injection target.

[0062] The client agent reads the address information of the target_page during a predetermined phase of the deterministic workload and obtains the client physical address (GPA) corresponding to that address by combining it with the page table mapping information. Subsequently, the client agent encapsulates the GPA, known_off, pattern, page_size, round identifier, virtual machine identifier, target type, phase identifier, and verification fields into a target descriptor and reports it to the host controller through a controlled channel or writes it to a structured output file for the host controller to read.

[0063] The page table mapping information can be read from / proc / self / pagemap. In systems with access restrictions, authorized client agents can also obtain the GPA information through auxiliary modules, reserved page frames, or other equivalent methods to ensure the feasibility of the solution.

[0064] When the host controller needs to perform memory injection and does not specify mem_gpa, it reads the gpa_hex field to automatically determine the injection target; if both the page base address and offset are provided, the controller calculates the final injection address gpa_target = gpa_base + offset.

[0065] Memory single-bit / multi-bit injection and security verification The memory fault injector first reads the byte value at the target GPA through the QEMU monitoring interface, and then obtains the corresponding host virtual address HVA through the gpa2hva command.

[0066] The injector performs a security check before writing, the security check including at least: (a) Locate the PID of the virtualization process hosting the virtual machine in the host process list and read / proc / <pid>Use ` / cmdline` to confirm that it is a virtualization process such as `qemu-system`; (b) Read / proc / <pid> / maps traverses the writable mapping region to confirm that the HVA falls within the writable range and that the range corresponds to the object memory backend mapping region; (c) Optionally, a whitelist verification may be performed. The whitelist shall contain at least one or more of the following: virtual machine identifier (domain name / UUID), virtualization process command line fingerprint, memory backend file path, and image path. Write operations shall only be permitted if the verification passes. After successful verification, the injector is located in / proc / <pid>In / mem, a read-modify-write operation is performed on the HVA location to obtain the flipped after, and the expected value is calculated.

[0067] To avoid false successes / failures, the injector performs consistency verification through at least two independent readback paths: one is the host machine reading back the HVA, and the other is the virtual machine monitoring command reading the GPA again. When both readback results are consistent with expected, the injection is considered successful; otherwise, it is recorded as verify_mismatch and an explainable reason is output (such as mapping change, unwritable, address drift, etc.).

[0068] Furthermore, to simulate MBU, the injector can select multiple bits or bytes to flip in a spatially related distribution within the same cache line / page, or trigger multiple flips in a time-related distribution within the same round, and write the flip set to injection.json to ensure reproducibility.

[0069] CPU register bit injection: Before injection, the CPU fault injector pauses the virtual machine and starts a gdbstub debug stub listening on a specified port via the virtual machine monitoring command. The injector calls gdb to connect to the debug stub in batch mode, reads the current value before of the target register reg, calculates the value after bit flipping, and writes it back to the register.

[0070] The injector immediately reads the register value after after writing back and verifies that after is consistent with expected; optionally, it records the vCPU number, register name, toggle bit index, and virtual machine state snapshot at the time of injection to form an auditable register injection evidence chain.

[0071] After injection, the debug stub is closed and the virtual machine is restored to continue running. This embodiment supports selecting a fixed register by parameters or randomly selecting a register from a candidate register set; it also optionally supports specifying a vCPU or limiting a register set to adapt to architectures such as ARM / RISC-V.

[0072] Virtual disk I / O bit injection (offset flipping and LBA semantic flipping): The I / O fault injector is deployed in the I / O path of the virtual block device on the host side, intercepting block device read and write requests initiated by the virtual machine. The injector uses the LBA as the error injection target. Before the block request is completed and the data is returned to the upper layer, it maps the target LBA to the physical byte offset position in the underlying disk image, ensuring the successful completion of the block I / O request. To reduce the risk of corrupting file system metadata, the injector prioritizes controlled data areas as injection targets, such as test files, reserved blocks, or test partitions, and can perform metadata exclusion checks on the target area before injection (e.g., not located in the superblock, log, or index area). Furthermore, error injection can be configured to be deterministic or probabilistically triggered to simulate silent data corruption caused by hardware transient failures, storage controller anomalies, or cosmic rays.

[0073] Experimental setup, error model randomization, and results recording: The host controller supports manually specifying injection parameters and also supports orchestration via a JSON configuration file. The configuration file can enable random seed, random injection time, random bit flip, random offset within memory pages, and a set of multiple flip counts per round.

[0074] To improve physical consistency, the controller also supports "error model-driven" injection plan generation: (a) SEU model: randomly select a single bit to flip within a given window; (b) MBU model: Select multiple bits to flip based on their distance distribution within the same cache line or page; (c) MBU time-dependent model: multiple flips are triggered in the same round according to a Poisson process or at fixed intervals; In memory mode, the controller calculates the page base address based on the gpa_hex, known_off and page size reported by the client, and selects a random offset within the same page to obtain the final injection address, thereby ensuring that the target always falls within the mapped and injectable target page.

[0075] To ensure data consistency in long-term batch experiments, the controller acquires an exclusive lock on the file and executes flush+fsync when writing to the summary CSV / JSONL, achieving atomic append writing; and archives the results of each round (injection.json, system_state.json, guest_output.json) in the rid directory to form a traceable chain of evidence.

[0076] The controller performs liveness detection using QGA and QMP, and categorizes the results by combining injection verification fields with workload checksums and execution results: MASKED (consistent output and successful injection), SDC (inconsistent output and successful injection), CRASH (client crash / restart), and HANG (timeout with no response). The categorization results and evidence fields are written to summary.json.

[0077] The classifier rules used are: if alive=false and a change in reboot_id is detected → CRASH alive=false and no reboot indication and timeout exceeded → HANG if alive=true and checksum is inconsistent and injection verification successful → SDC if alive=true and checksum matches and injection verification successful → MASKED `alive=true`, checksum is consistent, but performance / latency metrics exceed baseline thresholds → performance degradation. Optional deformations and extensions Expand memory injection from single-byte to multi-byte or cross-page injection, and set a stricter whitelist for the write range (e.g., limit it to a specific memory backend file segment). Extend CPU injection to include injection of a specified CPU, a specified instruction pointer / flag combination, or an adaptation of register sets for architectures such as ARM / RISC-V; The I / O injection is extended to logical sector (LBA) addressing for virtual block devices, which is converted to a mirror offset via a mapping table; Add a result classifier that combines workload_ok, alive with injection verification fields to automatically determine categories such as MASKED, SDC, CRASH, and HANG, and outputs a JSONL digest; Add security policies: only allow injection to whitelisted domains and whitelisted mirror paths, and take more comprehensive snapshots of the system status before and after injection (e.g., CPU status, memory backend mapping summary, I / O queue metrics).

[0078] Example 2 Embodiment 2 of the present invention provides a method for operating system fault injection and experiment orchestration based on virtualization technology. According to the system implementation of Embodiment 1, the method includes: Step 1: Initialization. The host controller loads the experimental configuration, generates a random seed and round plan, controls the virtualization platform to load a brand-new guest layer operating system from the original image, and establishes controlled communication with the guest agent. Step 2: Target release. During the deterministic phase, the client agent locks the injection target and generates a target descriptor, which is then received by the host controller. Step 3: Legality verification and injection plan generation. After the host controller performs source authorization, mapping range verification, security boundary verification, field integrity verification and verification of the target descriptor, it calls the fault injector to prepare for fault injection. Step 4: Fault injection execution. The fault injector performs bit flipping, setting, clearing, replacement or other preset fault injection operations on the specified target according to the injection target information of the target descriptor. Step 5: Closed-loop verification. After each fault injection, the closed-loop verification module obtains the injection result of the target location through two independent paths. One path is the direct read-back path on the host side, and the other is the visible path on the client or the read-back path through the virtual machine monitoring interface. Based on the consistency of the results returned by the two independent paths, it determines whether the fault injection is effective and records the evidence. Step 6: Triggering and Sampling. The host controller continues or triggers the client workload to collect system status and output results. Step 7: Classification and Atom Storage. The state sampling and result classification module automatically classifies the experimental results, and the experimental results atom recording module writes the atom into storage. Step 8: Reproducibility experiment. The host controller replays the experiment rounds with the same configuration and random seed to verify reproducibility.

[0079] In summary, this invention, without modifying the client-side operating system source code, implements controlled perturbations on the processor register state variables, memory data variables, and input / output state variables perceptible to the client-side operating system. This effectively simulates the impact of SEU and MBU effects caused by space radiation or extreme environments on the system, and reproduces the resulting data errors, anomalies, or system crashes. The system encapsulates a target descriptor containing GPA and verification information through a client-side agent at the client layer and runs a deterministic workload. The host side performs security verification and fault injection based on this descriptor, and confirms the injection result through a closed-loop verification mechanism with at least two independent paths.

[0080] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.< / pid> < / pid> < / pid> < / pid>

Claims

1. A virtualization-based operating system fault injection and experiment orchestration system for equivalently simulating single-event upsets and multi-event upsets induced by space radiation or extreme environments, characterized in that... The system includes: A client agent, deployed at the client layer, is used to trigger or coordinate the execution of deterministic workloads. During the deterministic phase, it acquires the original target location data and verification data corresponding to the injection target, generates a target descriptor, and reports it to the host machine. The target descriptor contains injection target information and verification information. The client layer runs on the virtualization platform of the host machine. The host controller, deployed at the host layer, is used to generate experimental rounds and injection plans, control the virtualization platform to load a brand-new client layer operating system based on the original image, establish controlled communication with the client agent, establish fault injection configuration, and also receive target descriptors. After performing source authorization, mapping range verification and security boundary verification, field integrity and verification of the target descriptors, it calls the fault injector to prepare to execute fault injection. A fault injector, deployed at the host layer, is used to perform fault injection operations on a specified target based on the injection target information of the target descriptor; and The closed-loop verification module, deployed at the host layer, is used to obtain the injection results at the target location through two independent paths after the fault injection is completed. One path is a direct read-back path from the host side, and the other is a path visible to the guest or a read-back path from the virtual machine monitoring interface. The consistency of the results returned by the two independent paths is used to determine whether the fault injection has taken effect. 2.The virtualization-based operating system fault injection and experiment orchestration system of claim 1, wherein, The system also includes a state sampling and result classification module, which is used to collect the system state and workload output before and after injection, and automatically classify the experiment into masking, silent data error, crash, suspension or performance degradation based on the injection verification results, and output an expandable tag set.

3. The virtualization-based operating system fault injection and experiment orchestration system of claim 2, wherein, The system also includes an experimental result atomic recording module, which records experimental round identifiers, target descriptors, injection configuration parameters, verification results, and result classification information to support experimental result analysis and experiment reproduction.

4. The operating system fault injection and experiment orchestration system based on virtualization technology according to claim 3, characterized in that, The virtualization platform is used to provide virtualization resources to the client layer and to provide the host machine with a virtual machine state access interface for the client layer.

5. The operating system fault injection and experiment orchestration system based on virtualization technology according to claim 3, characterized in that, The injection target information includes the client's physical address or I / O logical block address, page offset, and bit index; The verification information includes a digest of the original value; The target descriptor shall include at least: round identifier, virtual machine identifier, target type, specific location of fault injection, and stage identifier.

6. The operating system fault injection and experiment orchestration system based on virtualization technology according to claim 5, characterized in that, The client agent locks at least one injection target according to deterministic rules in a predetermined phase and obtains the client physical address and page offset or bit index corresponding to the injection target.

7. The virtualization-based operating system fault injection and experiment orchestration system of claim 3, wherein, The fault injection configuration established by the host controller includes at least: fault domain type, flip bit, injection timing or running phase / time window, and random seed.

8. The virtualization-based operating system fault injection and experiment orchestration system of claim 3, wherein, The fault injector includes at least one of a memory fault injector, a CPU fault injector, and an I / O fault injector; wherein... The memory fault injector is used to convert the client's physical address into the host's virtual address, and after completing the security verification, performs bit flipping, setting, clearing, replacement or other preset fault injection operations on the target memory; the bit flipping is random or selected according to a preset strategy; The CPU fault injector is used to read and write registers through the debug interface and perform bit flip, set, clear, replace or other preset fault injection operations in the virtual machine paused state before switching to the running state; The I / O fault injector is used to intercept virtual block device I / O requests and perform bit flipping, setting, clearing, replacement or other preset fault injection operations based on the mapping from logical sector to mirror offset.

9. The virtualization-based operating system fault injection and experiment orchestration system of claim 8, wherein, The security verification includes at least: process identity, storage mapping range, and whitelist verification.

10. A method for operating system fault injection and experiment orchestration based on virtualization technology, implemented according to any one of claims 3-9, wherein the method comprises: Step 1: Initialization. The host controller loads the experimental configuration, generates a random seed and round plan, controls the virtualization platform to load a brand-new guest layer operating system from the original image, and establishes controlled communication with the guest agent. Step 2: Target release. During the deterministic phase, the client agent locks the injection target and generates a target descriptor, which is then received by the host controller. Step 3: Legality verification and injection plan generation. After the host controller performs source authorization, mapping range verification, security boundary verification, field integrity verification and verification of the target descriptor, it calls the fault injector to prepare for fault injection. Step 4: Fault injection execution. The fault injector performs bit flipping, setting, clearing, replacement or other preset fault injection operations on the specified target according to the injection target information of the target descriptor. Step 5: Closed-loop verification. After each fault injection, the closed-loop verification module obtains the injection result of the target location through two independent paths. One path is the direct read-back path on the host side, and the other is the visible path on the client or the read-back path through the virtual machine monitoring interface. Based on the consistency of the results returned by the two independent paths, it determines whether the fault injection is effective and records the evidence. Step 6: Triggering and Sampling. The host controller continues or triggers the client workload to collect system status and output results. Step 7: Classification and Atom Storage. The state sampling and result classification module automatically classifies the experimental results, and the experimental results atom recording module writes the atom into storage. Step 8: Reproducibility experiment. The host controller replays the experiment rounds with the same configuration and random seed to verify reproducibility.

Images