A Distributed Task Scheduling Method and System Based on One-Time Virtual Machines

Abstract

This invention relates to the fields of distributed computing and virtualization technology, and provides a distributed task scheduling method and system based on disposable virtual machines. The method includes: receiving a task request submitted by a user; performing dependency resolution on the task request based on a dependency database to obtain a task dependency graph; creating a disposable virtual machine instance for the current task to be executed according to the execution order of each task in the task dependency graph; sending the execution data required by the current task to the disposable virtual machine instance, instructing the disposable virtual machine instance to execute the task in an isolated environment using the execution data; obtaining the task artifacts generated by the disposable virtual machine instance and storing them in a shared storage layer; and reclaiming resources from the disposable virtual machine instance after the current task to be executed is completed. This invention achieves complete isolated execution and automated flow scheduling of distributed tasks by automatically orchestrating tasks based on the dependency graph and combining it with a dynamic creation and destruction mechanism for disposable virtual machines.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and virtualization technology, and in particular to a distributed task scheduling method and system based on a one-time virtual machine. Background Technology

[0002] As software development and deployment scales up, distributed task scheduling has become a core infrastructure for scenarios such as software package building, automated testing, and continuous integration. In these scenarios, tasks typically have complex dependencies and require high levels of consistency, isolation, and execution efficiency in the execution environment.

[0003] Currently, existing task scheduling and execution solutions mainly rely on container technologies (such as Docker) or the node pool mechanism of continuous integration platforms (such as Jenkins and GitLab CI), but these solutions still have shortcomings in terms of isolation, dependency management, and intelligent scheduling.

[0004] Specifically, when dealing with task sets with complex dependencies, existing technologies often require users to manually configure dependencies and execution order. This is not only labor-intensive and error-prone, but also makes it difficult for scheduling systems to automatically optimize parallel execution paths, resulting in underutilization of computing resources. Secondly, existing container technologies typically share the host machine kernel, resulting in low isolation levels and security vulnerabilities when kernel modules or sensitive operations are involved. Furthermore, to improve efficiency, continuous integration platforms often use a reuse model for execution nodes. File caches and environment variables left over from previous task executions are often difficult to clean up completely, leading to contamination of the runtime environment for subsequent tasks. This not only affects the determinism of task results but may also cause errors that are difficult to troubleshoot. Summary of the Invention

[0005] This invention provides a distributed task scheduling method and system based on a one-time virtual machine, which solves the defects in the prior art such as task scheduling relying on manual configuration and insufficient isolation of the task execution environment leading to uncertain task execution results.

[0006] This invention provides a distributed task scheduling method based on a one-time virtual machine, comprising: Receive a task request submitted by a user, and based on a pre-built dependency database, perform dependency parsing on the task request to obtain the task dependency graph required to complete the task request; The execution order of each task to be executed is determined according to the task dependency graph, and an independent one-time virtual machine instance is created for the current task to be executed according to the execution order. Send the execution data required for the currently pending task to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data; Obtain the task artifacts generated by the one-time virtual machine instance, store the task artifacts in the shared storage layer, and use the task artifacts for subsequent tasks to call; After the current task to be executed is completed, the resources of the one-time virtual machine instance are reclaimed, and the next task to be executed is continued until all tasks in the task dependency graph are completed.

[0007] According to a distributed task scheduling method based on one-time virtual machines provided by the present invention, the step of creating an independent one-time virtual machine instance for the currently pending task according to the execution order includes: Based on a pre-made read-only base image containing the operating system and runtime environment, an incremental overlay image is generated for the currently pending task. The state information of the incremental overlay image is cleaned up, and a task execution proxy service is injected into the cleaned incremental overlay image to obtain the processed incremental overlay image. Based on the preset hardware configuration template, the read-only base image, and the processed incremental overlay image, a virtual machine configuration file is generated. The virtual machine configuration file is then used to create and start the one-time virtual machine instance, thereby skipping the redundant step of querying operating system information through the system installation tool. After the one-time virtual machine instance starts up, the task execution agent service is polled through its health check interface to wait for it to be ready in order to establish a communication connection with the one-time virtual machine instance.

[0008] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, the resource reclamation of the one-time virtual machine instance includes: Send a forced shutdown command to the one-time virtual machine instance; Delete the definition information of the one-time virtual machine instance and the files corresponding to the incremental overlay image.

[0009] According to the distributed task scheduling method based on a one-time virtual machine provided by the present invention, the construction steps of the dependency database include: Offline scan of the metadata files of all candidate task objects, and extract the object identifier, dependency list, and mapping relationship between package group and sub-package of the candidate task object from the metadata files; Based on the object identifiers and dependency lists of each candidate task object, an initial dependency graph is constructed, and a depth-first traversal is performed on the initial dependency graph. During the traversal, visited nodes are detected to identify circular dependencies. If a circular dependency is detected, the corresponding circular edge is silently disconnected to obtain an acyclic task dependency graph. Based on the extracted mapping relationship between the package group and the sub-package, a forward mapping from the package base to the sub-package and a reverse mapping from the sub-package to the package base are established. The acyclic task dependency graph, along with the established forward and reverse mappings, are serialized and stored to obtain the dependency database.

[0010] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, the step of sending the execution data required for the currently pending task to the one-time virtual machine instance includes: The execution data is packaged into a binary stream in archive format; The binary stream is transmitted to the task execution agent service in the one-time virtual machine instance via a network transmission protocol. The task execution agent service is used to decompress the binary stream to a temporary directory and move the decompressed content in the temporary directory to the target execution path through an atomic renaming operation.

[0011] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, the step of instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data includes: The one-time virtual machine instance is controlled to access the local dependency repository in the shared storage layer through the file system protocol. The local dependency repository stores the task artifacts generated by the previous task and the dependency resources of the current task to be executed. By polymorphically calling the standard interface corresponding to the currently pending task, a task execution instruction is issued to the one-time virtual machine instance. The task execution instruction is used to trigger the one-time virtual machine instance to perform closed computation in an isolated environment where external network connections are prohibited, using the task artifacts and dependency resources in the local dependency repository and the execution data. The standard interface is provided through a preset build backend abstraction layer, which is statically configured with a standard interface that is uniformly adapted to different build systems.

[0012] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, after the step of instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data, the method further includes: Receive the task execution logs transmitted back in real time by the one-time virtual machine instance via a streaming protocol; The task execution log is parsed to update the running status of the currently pending task in real time based on the parsing results.

[0013] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, the step of obtaining the task artifacts generated by the one-time virtual machine instance and storing the task artifacts in a shared storage layer includes: A request to download the task artifacts is sent to the one-time virtual machine instance. The one-time virtual machine instance is used to verify whether the source path specified in the request is within a preset security path whitelist. If it is, the generated task artifacts are packaged and sent back. The system receives the task artifacts returned by the one-time virtual machine instance, writes the task artifacts into the local dependency repository in the shared storage layer, and calls an update command through a global lock mechanism to update the package database maintained in the local dependency repository.

[0014] According to a distributed task scheduling method based on a one-time virtual machine provided by the present invention, after the step of determining the execution order of each task to be executed according to the task dependency graph, the method further includes: The task dependency graph is divided into multiple execution levels using a topological sorting algorithm, where tasks within the same execution level are independent of each other. Accordingly, the step of creating an independent, one-time virtual machine instance for the currently pending task according to the execution order includes: According to the order of the execution levels, multiple one-time virtual machine instances are created concurrently for multiple tasks to be executed in the current execution level.

[0015] According to the present invention, a distributed task scheduling method based on a one-time virtual machine is provided, wherein the task dependency graph is divided into multiple execution levels using a topological sorting algorithm, including: Calculate the in-degree of each task node in the task dependency graph, and place the task node with an in-degree of zero into the first execution level. The in-degree represents the number of direct predecessor tasks that the task node depends on. After processing the task nodes in the current execution level, update the in-degree of subsequent task nodes, and put the task nodes with an in-degree of zero after the update into the next execution level; Repeat the steps of updating the in-degree of subsequent task nodes and placing task nodes with an in-degree of zero after the update into the next execution level, until all task nodes in the task dependency graph are placed into the corresponding execution level.

[0016] A distributed task scheduling method based on a one-time virtual machine provided by the present invention further includes: Monitor the execution status of each pending task; If the execution status of any pending task is detected as failed, all downstream tasks that depend on any pending task are retrieved according to the task dependency graph. Mark each downstream task as blocked to cancel the scheduled execution of each downstream task.

[0017] A distributed task scheduling method based on a one-time virtual machine provided by the present invention further includes: When creating each one-time virtual machine instance, a corresponding child cancellation signal is derived from the root-level cancellation signal controller, which is created during system startup. Upon receiving an external interrupt command or a system abnormal exit signal, the root-level cancellation signal controller is activated to propagate the cancellation signal to all child-level cancellation signals. In response to the child-level cancellation signal, concurrent resource reclamation operations are performed on all one-time virtual machine instances.

[0018] The present invention also provides a distributed task scheduling system based on a one-time virtual machine, comprising: An orchestration controller is configured to run on a host machine to execute a distributed task scheduling method based on a one-off virtual machine as described above. A one-time virtual machine cluster consists of multiple temporarily created virtual machine instances, each of which is used to receive instructions from the orchestration controller and execute tasks. A shared storage layer, including a base image repository, an overlay image directory, and a local dependency repository, is used to connect the orchestration controller and the one-off virtual machine cluster.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the distributed task scheduling method based on a one-time virtual machine as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed task scheduling method based on a one-time virtual machine as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed task scheduling method based on a one-time virtual machine as described above.

[0022] The distributed task scheduling method and system based on disposable virtual machines provided by this invention parses task requests and generates task dependency graphs based on a pre-built dependency database. This enables automatic and accurate planning of the execution logic and sequence between complex tasks, effectively avoiding the tediousness and potential errors of manually configuring dependencies. More importantly, this invention dynamically creates an independent disposable virtual machine instance for each task to be executed and immediately destroys the instance and cleans up resources after the task is completed. This mechanism not only ensures that each task runs in an interference-free isolated environment, avoiding the state residue and environmental pollution problems common in traditional reuse environments, but also greatly improves the determinism and repeatability of task execution results. Furthermore, by collecting the artifacts to a shared storage layer, it ensures smooth invocation of subsequent tasks while achieving immediate release and efficient recycling of computing resources. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the distributed task scheduling method based on a one-time virtual machine provided by the present invention; Figure 2 This is a schematic diagram of the structure of the distributed task scheduling system based on a one-time virtual machine provided by the present invention; Figure 3 This is a schematic diagram of the one-time virtual machine lifecycle management process provided by the present invention; Figure 4 This is a flowchart of the DAG scheduling process based on a pre-computed dependency database provided by the present invention; Figure 5 This is a schematic diagram of a bidirectional file transfer protocol based on HTTP tar streams provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] To facilitate understanding of the technical solutions provided by this invention, some key terms involved in this invention will first be explained.

[0027] Kernel-based Virtual Machine (KVM) is a virtualization technology integrated into the Linux kernel that allows multiple isolated virtual machine instances to be created and run on a physical host, each with its own operating system and resources.

[0028] QEMU Copy-On-Write version 2 (QCOW2) is a disk image file format that supports advanced features such as copy-on-write, snapshots, and incremental backups, and is widely used in virtual machine storage management.

[0029] Copy-on-Write (COW) is a resource management optimization strategy that allows multiple processes or virtual machines to share the same read-only underlying data. Modified parts are copied only when a write operation occurs, significantly reducing storage space usage and copying time.

[0030] In the QCOW2 virtual machine image system, an overlay is a temporary writable layer created on top of a read-only base image. All runtime modifications are written to the overlay, while the base image remains unchanged, facilitating the rapid creation and destruction of virtual machine instances.

[0031] A Directed Acyclic Graph (DAG) is a data structure consisting of vertices and directed edges that do not contain cycles. It is widely used to represent task dependencies and is used in this invention to calculate the topological sorting order of software package construction.

[0032] Network File System (NFS) is a distributed file system protocol that allows multiple computers to share and access the same storage resource over a network, enabling transparent remote access to files.

[0033] Libvirt is a set of open-source application programming interfaces, daemons, and management tools used to uniformly manage different virtualization technologies (such as KVM, Xen, etc.) and provide functions such as virtual machine creation, startup, and monitoring.

[0034] In this invention, a local dependency repository (Depot) specifically refers to a local file system directory that stores task artifacts (such as built software packages). It is accessible to the virtual machine via a file protocol (file: / / ) to freeze dependency versions and avoid obtaining variable dependencies from network sources.

[0035] A hygienic build refers to performing build tasks in a completely isolated and repeatable environment, ensuring that the build process is not contaminated by the state of the host machine or other build tasks. Each build starts from the same initial state, with fixed dependency versions, guaranteeing the determinism and repeatability of the build results.

[0036] The Orchestrator is the central control component in this invention. It runs on the physical host machine and is responsible for core business logic such as task scheduling, virtual machine lifecycle management, dependency calculation, and task artifact collection. It is a stateful management node.

[0037] A task execution agent is a lightweight HTTP service program that runs inside a temporary virtual machine. It receives commands from the orchestration controller and executes them within the virtual machine. It is responsible for file upload and download, command execution, and log streaming. It is a stateless execution node.

[0038] Embedded relational database (SQLite) is a lightweight, serverless database engine that stores data in a single file. In this invention, it is used to persist task states and build history.

[0039] The Kahn algorithm is a classic algorithm for topological sorting of directed acyclic graphs. It generates a topological sequence by iteratively deleting nodes with an in-degree of zero. This invention uses a variant of the Kahn algorithm to achieve hierarchical parallel scheduling of dependent tasks.

[0040] The virtualization shell tool (virsh) is a command-line management tool provided by Libvirt, used to interact with the virtualization daemon and perform operations such as defining, starting, stopping, and destroying virtual machines.

[0041] The virtual machine image cleanup tool (virt-sysprep) is part of the Libvirt toolset. It is used to remove specific system configurations from virtual machine images, such as DHCP (Dynamic Host Configuration Protocol) leases, SSH (Secure Shell) keys, machine IDs, etc., making the images suitable for batch copying as templates.

[0042] Guestfish is an interactive shell tool provided by Libvirt that allows direct access to and modification of the file system within a virtual machine disk image while the virtual machine is powered off. It can be used to inject files or modify configurations.

[0043] Tape Archive (tar stream) is an archive format that packages multiple files into a single byte stream and supports streaming read and write. In this invention, the tar stream is transmitted via the HTTP protocol to achieve efficient file transfer between the host machine and the virtual machine.

[0044] Newline Delimited JSON (NDJSON) is a text format where each line is an independent JSON object, facilitating streaming processing. In this invention, it is used for real-time transmission of build log events.

[0045] Arch Linux package build tool (makepkg) is the standard package build command for Arch Linux distributions. It reads the PKGBUILD script and executes the entire process of compiling, packaging, and generating package files.

[0046] Microsoft's cross-platform C++ package manager (Visual C++ Package Manager, vcpkg) is an open-source C / C++ dependency management tool that supports Windows, Linux, and macOS platforms.

[0047] Topological sorting refers to the linear sorting of nodes in a directed acyclic graph (DAG) such that for any directed edge from node A to node B, A always appears before B in the sorted result. This is used to determine the legal order of task execution.

[0048] AbortSignal is an asynchronous operation cancellation mechanism that allows the main process to send a cancellation signal to a subtask. The subtask listens for the signal and immediately terminates execution and cleans up resources upon receiving it. This invention is used to implement cascading resource cleanup when the orchestration controller exits.

[0049] As software systems become increasingly complex, distributed task scheduling and environment isolation technologies have become core infrastructure for software development, testing, and deployment. This is especially true in areas such as package building, continuous integration (CI), and automated testing, where tasks often have complex dependencies and require strict consistency and isolation of the execution environment.

[0050] Currently, the development of this technology has evolved from single-machine sequential execution and manual environment configuration to containerized isolation and distributed parallel scheduling. For example, in software package building scenarios, early build systems mainly executed tasks sequentially on a single host, relying on manual configuration of the build environment, which was inefficient and prone to environmental contamination. With the popularization of cloud computing and virtualization technologies, solutions based on containers (such as Docker) and continuous integration platforms (such as Jenkins and GitLab CI) have gradually become mainstream. However, these solutions still have shortcomings in terms of isolation, dependency management, and intelligent scheduling, as detailed below: First, insufficient isolation leads to environmental pollution and security risks. While mainstream container technologies like Docker provide lightweight isolation, containers essentially share the host machine's kernel. In sensitive scenarios involving kernel module compilation, system library replacement, and compiler toolchain testing, the isolation level of containers is insufficient to guarantee complete independence between tasks, posing a risk of kernel vulnerabilities leading to escape. Furthermore, existing CI platforms (such as Jenkins and GitLab CI) typically employ a Runner node reuse model, where multiple tasks are executed sequentially within the same Runner environment. Although cleanup scripts are usually executed between tasks, they cannot completely remove state pollution such as file system caches, residual environment variables, or temporary files. This can cause subsequent tasks to be affected by previous tasks, resulting in non-repeatable build results or even build failures that are difficult to troubleshoot. For example, a globally installed Python package in one task might affect dependency resolution in subsequent tasks, causing build failures that are difficult to detect.

[0051] Second, dependency management relies on network repositories, leading to poor version stability. Docker and CI platforms typically pull dependency packages from remote repositories (such as PyPI, npm registry, and Maven Central) in real-time during the package build process. Version updates or service unavailability of upstream repositories can cause build failures or inconsistent results. For example, if a new version of a dependency package is released, and the build script does not lock the version number, builds at different times may retrieve different versions of the dependency, resulting in different build results. Network latency and bandwidth limitations also significantly increase build time, especially in batch build scenarios, where repeatedly downloading the same dependency wastes resources. Although this problem can be mitigated by using a private caching server, cache management itself increases system complexity and makes it difficult to guarantee cache consistency.

[0052] Third, scheduling efficiency is low, and complex dependencies cannot be automatically inferred. Traditional CI tools (such as Jenkins and GitLab CI) typically require users to manually declare dependencies and execution order between tasks in configuration files. For complex package build scenarios (such as building a Linux distribution containing hundreds of packages), manually maintaining dependency configurations is extremely labor-intensive and error-prone. Although some workflow engines (such as Airflow) support DAG definitions, users still need to explicitly write code to describe dependencies. They cannot automatically infer transitive dependencies from package metadata (such as the depends field in PKGBUILD), causing the scheduler to fail to fully recognize and utilize parallel capabilities. For example, packages that could be built in parallel without dependencies may be executed serially due to improper configuration, resulting in wasted computing resources.

[0053] Fourth, traditional virtual machines (VMs) are slow to start and unsuitable for short-lifecycle tasks. While traditional full virtualization technologies (such as KVM) offer the strongest isolation, VM startup times are long (typically tens of seconds or even minutes) and resource overhead is high (each VM requires an independent kernel and system processes), making them unsuitable for large-scale concurrent short-lifecycle tasks. Creating VMs using tools like virt-install requires time-consuming operations such as operating system information queries (osinfo-query) and hardware configuration generation, further increasing latency. This limits the application of VM isolation technology in real-world system builds, and most scenarios still rely on container solutions with weaker isolation.

[0054] Fifth, poor ecosystem compatibility and high migration costs. Systems like Nix and Guix manage dependencies using purely functional methods, storing all dependencies in isolated hash paths to achieve deterministic builds. However, this requires rewriting all packages into dedicated Nix expressions or Guix schemes, making them incompatible with existing package management ecosystems (such as Arch Linux's PKGBUILD, Debian's dpkg, and vcpkg). Migrating tens of thousands of existing packages requires a huge manpower investment and is difficult to keep up with upstream community updates. For organizations with a large number of PKGBUILD or other build scripts, adopting Nix means rewriting the entire build definition, which is prohibitively expensive.

[0055] Sixth, resource management is complex and lacks unified lifecycle control. In existing technologies, the lifecycle management of containers, virtual machines, and Runner nodes is typically scattered across different components, lacking a unified resource reclamation mechanism. For example, Docker containers may leave behind zombie containers and dangling volumes when they exit abnormally, requiring additional cleanup scripts for periodic processing. Jenkins build workspaces require manual configuration of cleanup strategies; otherwise, they consume significant disk space. When Airflow worker nodes terminate abnormally, task states may be inconsistent, requiring manual intervention for recovery. These issues increase operational complexity and reduce system reliability and automation.

[0056] To address the shortcomings of the existing technologies, this invention provides a distributed task scheduling method and system based on a one-time virtual machine, aiming to solve the following technical problems: (1) Provide a completely isolated task execution environment to eliminate environmental pollution. By creating an independent virtual machine instance for each task, complete isolation based on hardware virtualization is achieved. Each virtual machine has an independent operating system kernel and file system. The virtual machine is destroyed immediately after the task is completed, completely eliminating state pollution and security risks between tasks. This solves the problem of insufficient isolation caused by container sharing of the kernel and the problem of environmental residue caused by Runner reuse, ensuring that each task is executed in a completely clean environment and guaranteeing the repeatability and consistency of the build results.

[0057] (2) Implement dependency version freezing to eliminate the instability of network dependencies. All built packages are stored in a local dependency repository (Depot). Build tasks within the virtual machine retrieve dependencies from the local repository without accessing external network sources. Combined with a pre-computed dependency database, this ensures that all dependencies required for task execution are ready in the local repository and their versions remain fixed. This solves the version instability and network latency issues caused by pulling dependencies from network sources, achieving a completely offline deterministic build.

[0058] (3) Automatically deduce task dependencies to improve scheduling efficiency. By pre-parsed package metadata (such as the depends field of PKGBUILD), a complete dependency graph is automatically constructed, and a topological sorting algorithm is used to calculate the hierarchical execution order of tasks. Tasks without dependencies within the same level can be executed in parallel. This solves the problems of large workload and error-proneness in manually configuring dependencies, makes full use of computing resources, maximizes parallel scheduling capabilities, and significantly shortens the total execution time of batch tasks.

[0059] (4) Optimize virtual machine startup speed to adapt to short-lifecycle tasks. The QCOW2 image format with copy-on-write (COW) is adopted, and multiple virtual machines share the same read-only base image, recording modifications only in their respective overlay layers, which greatly reduces image creation time and storage space usage. By directly using the virsh tool and pre-generated XML configuration to start the virtual machine, the redundant operation of virt-install is skipped, and the virtual machine startup time is optimized from several minutes to the level of ten seconds, making the virtual machine isolation technology suitable for large-scale concurrent short-lifecycle task scenarios.

[0060] (5) Maintain compatibility with the existing ecosystem and reduce migration costs. No modification to existing build scripts and toolchains is required; direct compatibility with mainstream package management ecosystems (such as makepkg, vcpkg, dpkg, etc.) is achieved. By building a backend abstraction layer to support different build systems, only adapters need to be written for new build tools, allowing reuse of the entire scheduling and isolation framework. This addresses the migration cost issue of functional package managers requiring rewriting build definitions, fully utilizing existing package repositories and build scripts in the community.

[0061] (6) Achieve unified resource lifecycle management and improve system reliability. A cascading resource cleanup is achieved through the AbortSignal mechanism. When the orchestration controller exits, a cancellation signal is automatically propagated. All running virtual machines receive the signal and immediately terminate and clean up the overlay image, requiring no manual intervention. Combined with the persistent task state in the database, unfinished tasks can be automatically recovered after a system restart. This solves the resource leakage and state inconsistency problems in existing solutions, achieving fully automated resource management and reducing operational complexity.

[0062] The method provided by this invention can be applied to scenarios requiring environment isolation and task orchestration, such as software package building, continuous integration, machine learning experiment management, and security testing. For ease of understanding, the technical solution provided by this invention will be described in detail below using a software package building scenario as an example. It should be noted that all actions involving the acquisition of information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the invention is located, and with authorization from the owner of the corresponding device.

[0063] Figure 1 This is a flowchart illustrating the distributed task scheduling method based on a one-time virtual machine provided by the present invention, as shown below. Figure 1 As shown, this method is primarily executed by an orchestration controller running on the host machine. It aims to perform build or test tasks through a completely isolated virtualized environment, ensuring deterministic task execution and environmental purity. The method specifically includes the following steps: Step 110: Receive the task request submitted by the user, and perform dependency parsing on the task request based on the pre-built dependency database to obtain the task dependency graph required to complete the task request.

[0064] Specifically, a task request typically refers to a build or test command submitted by a user through a command-line tool or API interface, such as requesting to build a specific software package (e.g., openssl) or execute a test task. To accurately identify the context of this task, the system utilizes a pre-built dependency database. This dependency database is not built by fetching network data in real-time at runtime, but rather is a static data set generated by offline scanning of the metadata files (e.g., the depends field in PKGBUILD) of all candidate task objects (e.g., all software packages in a Linux distribution), extracting information such as package names and dependency lists.

[0065] Upon receiving a task request, the orchestration controller loads the database into memory and uses efficient data structures such as hash tables to query not only the task's direct dependencies but also recursively resolve all levels of transitive dependencies. For example, if a user requests to build task A, where A depends on B, and B depends on C, the system will resolve the complete dependency chain. Ultimately, these dependencies are constructed into a task dependency graph, typically a directed acyclic graph (DAG), which clearly describes all related task nodes and their sequential dependencies.

[0066] Step 120: Determine the execution order of each task to be executed according to the task dependency graph, and create an independent one-time virtual machine instance for the current task to be executed according to the execution order.

[0067] Specifically, after obtaining the task dependency graph, it is necessary to determine which tasks can be executed in parallel and which must be executed sequentially. The system typically uses a topological sorting algorithm (such as a variant of Kahn's algorithm) to analyze the nodes in the graph, calculate the in-degree of each node (i.e., the number of direct predecessor tasks that the node depends on), and put tasks with an in-degree of zero into the execution queue as the current tasks to be executed.

[0068] For each currently pending task in the queue, the system creates a separate, one-time virtual machine instance. "One-time" here means that the virtual machine's lifecycle is strictly bound to a single task; that is, each task corresponds to one virtual machine. Unlike traditional long-running virtual machines or containers, this virtual machine is dynamically created specifically for the current task, ensuring that the initial state of the execution environment is absolutely clean and unaffected by residual files or configurations from previous tasks.

[0069] Step 130: Send the execution data required for the current task to be executed to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the current task to be executed in an isolated environment using the execution data.

[0070] Specifically, after the virtual machine starts, the orchestration controller needs to transfer execution data into it. This execution data includes, but is not limited to, source code, build scripts, and configuration parameters. To ensure efficient transmission, the source code directory on the host machine can be packaged and sent to the agent program (also known as the agent service) inside the virtual machine via network communication protocols.

[0071] Subsequently, the orchestration controller instructs the virtual machines to execute tasks using this execution data in an isolated environment. This isolated environment is implemented using hardware-assisted virtualization technology (such as KVM), where each virtual machine has its own independent operating system kernel, file system, and network protocol stack. This means that even if high-risk operations such as kernel module loading or system library replacement are involved in task execution, or if malicious code escapes, the impact is confined to the virtual machine and will not affect the host machine or other concurrently executing tasks, thus ensuring extremely high security.

[0072] Step 140: Obtain the task artifacts generated by the one-time virtual machine instance, and store the task artifacts in the shared storage layer. The task artifacts are used for subsequent task invocation.

[0073] Specifically, after a task is completed (such as compiling or packaging), corresponding task artifacts (such as binary software packages, test reports, etc.) are generated. The orchestration controller or an agent inside the virtual machine extracts these artifacts from the virtual machine and persists them to a shared storage layer. This shared storage layer can be a directory mounted on a network file system (NFS), which connects the host machine and all virtual machines.

[0074] Understandably, the purpose of storing artifacts in the shared storage layer is to satisfy dependencies. For example, if task B depends on task A, and task A completes, its artifacts are stored in the shared location; the virtual machine of task B, which is subsequently started, can directly mount or read the artifacts of task A from this shared location without rebuilding task A, thus achieving reuse of the build results.

[0075] Step 150: After the current task to be executed is completed, the resources of the one-time virtual machine instance are reclaimed, and the next task to be executed is continued until all tasks in the task dependency graph are completed.

[0076] Specifically, when a task finishes execution (whether successfully or not), the orchestration controller immediately performs resource reclamation operations to prevent environmental contamination. This includes forcibly shutting down virtual machines, deleting virtual machine definitions, and clearing all temporary files generated during the virtual machine's operation (such as disk overlays). This mechanism ensures that no state remains.

[0077] Subsequently, the scheduling logic checks the task dependency graph and updates the dependency state of the remaining tasks (e.g., reducing the in-degree of subsequent nodes). If a new task with an in-degree of zero is found, it is used as the next task to be executed, and steps 120 to 150 above are repeated until all tasks in the entire dependency graph have been processed.

[0078] The method provided in this invention, by constructing a task dependency graph based on dependency resolution and dynamically creating a completely isolated, one-time virtual machine for each task, fundamentally eliminates the environmental pollution problem in distributed task scheduling. Each task runs in an independent kernel-level isolated environment, avoiding the security risks caused by shared kernels in container solutions and the mutual interference between tasks caused by environmental remnants, thereby ensuring a high degree of determinism and repeatability of task execution results. Simultaneously, by storing artifacts in a shared storage layer, effective reuse of dependency artifacts is achieved. Combined with the mechanism of immediately destroying virtual machines after task completion, both environmental purity and automated resource recycling are ensured.

[0079] Based on any of the above embodiments, step 120, which involves creating an independent one-time virtual machine instance for the currently pending task according to the execution order, includes: Step 121: Based on the pre-made read-only base image containing the operating system and runtime environment, generate an incremental overlay image for the currently executed task.

[0080] It should be noted that the embodiments of the present invention are a further detailed description of the creation of a one-time virtual machine instance, which makes full use of the copy-on-write and fast startup mechanisms in virtualization technology.

[0081] Specifically, when creating a separate, one-time virtual machine instance for each task to be executed, the system employs copy-on-write technology to avoid completely copying a large operating system image (typically several gigabytes in size) for each task. First, a read-only base image is pre-created, typically in QCOW2 format, which pre-installs an operating system (such as Arch Linux, Windows, etc.), a build toolchain, and runtime libraries. This base image is shared and read-only across all concurrent tasks.

[0082] When the orchestration controller decides to execute a task, it invokes virtualization tools (such as the `qemu-img create` command) to create a new QCOW2 file, i.e., an incremental overlay image. This overlay image points to the aforementioned read-only base image via a backing file mechanism. At this point, the overlay image only records incremental modification data generated during subsequent execution; its initial size is extremely small, and its creation time is typically less than one second. This mechanism allows hundreds of concurrent tasks to share the same base image, significantly saving storage space and improving creation speed.

[0083] Step 122: Clean up the state information of the incremental overlay image, inject task execution proxy service into the cleaned incremental overlay image, and obtain the processed incremental overlay image.

[0084] Specifically, although the overlay image is newly created, its base image may contain residual system configurations from the time of creation. To prevent IP conflicts or duplicate identifiers from multiple concurrently started virtual machines, the overlay must be cleaned up. The orchestration controller calls an image cleanup tool (such as virt-sysprep) to directly modify the overlay file while the virtual machines are not running, removing configurations that could cause concurrent conflicts, such as DHCP leases, SSH keys, machine IDs, and network card MAC address configurations.

[0085] Subsequently, the system uses a virtual machine disk access tool (such as guestfish) to inject a task execution agent service into the image. This agent is a lightweight HTTP service program (typically only a few hundred lines of code), placed in the system's executable path (e.g., / usr / local / bin / agent), and configured to start automatically at boot. This offline injection method avoids the latency and instability of downloading the agent over the network after the virtual machine starts. After cleaning and injection, the image becomes a ready-to-process incremental overlay image.

[0086] Step 123: Generate a virtual machine configuration file based on the preset hardware configuration template, the read-only base image, and the processed incremental overlay image. Use the virtual machine configuration file to create and start the one-time virtual machine instance to skip the redundant step of querying operating system information through the system installation tool.

[0087] Specifically, in this step, to adapt to the needs of different scenarios and pursue ultimate startup performance, the system's internal virtual machine driver module is designed with two virtual machine startup paths: one is the traditional path, which uses traditional tools such as virt-install for creation and startup. Although this path has good compatibility, it requires redundant and time-consuming operations such as querying the operating system information database (i.e., osinfo query) and dynamically generating hardware configurations during execution, making it unsuitable for large-scale concurrent tasks with short lifecycles.

[0088] The other is the fast path, which is the preferred core startup method adopted in this embodiment of the invention. It completely skips the redundant operations of virt-install and the runtime operating system information query. Specifically, the orchestration controller directly generates a complete virtual machine XML configuration file based on a preset hardware configuration template (including complete hardware configuration information such as CPU, memory, disk, and network), the read-only base image, and the above-processed incremental overlay image, combined with the specific requirements of the current task. In this configuration file, the disk device is specified as the above-processed incremental overlay image.

[0089] Subsequently, the orchestration controller invokes virtualization shell tools, such as executing `virsh define` to register virtual machines and `virsh start` to start them. By pre-generating XML configuration files and directly invoking them, time-consuming operations are successfully avoided, allowing virtual machines to start within seconds.

[0090] Step 124: After the one-time virtual machine instance starts up, the task execution agent service is polled for health check interface to wait for the task execution agent service to be ready in order to establish a communication connection with the one-time virtual machine instance.

[0091] Specifically, after the virtual machine starts, the operating system kernel loads and starts user-space services, including the previously injected task execution agent service. The orchestration controller accesses the health check interface of this agent service in a round-robin manner via the HTTP protocol (e.g., GET / health).

[0092] Understandably, during the initial startup of the virtual machine, the network stack is not yet ready, and the request may fail or time out (the timeout period can be set to 60 seconds). The orchestration controller will retry at a preset frequency (e.g., every 0.5 seconds) until it receives an HTTP 200 OK response, indicating that the Agent within the virtual machine has successfully started and is ready to receive instructions. At this point, the communication connection between the orchestration controller and the virtual machine instance is formally established, and subsequent file transfers and command executions will be conducted through this connection.

[0093] The method provided in this invention combines QCOW2 copy-on-write technology and offline image cleanup and injection technology, while introducing two virtual machine startup paths. In particular, by adopting a fast path that directly generates configuration files and calls the virsh command, it skips the extremely time-consuming redundant operations such as operating system information lookup in traditional startup paths, reducing virtual machine creation and startup time from minutes to seconds (e.g., less than 10 seconds). Simultaneously, incremental overlay technology reduces storage overhead by more than 90%. This makes a high-frequency creation and destruction mode of one task corresponding to one virtual machine feasible and efficient in practical engineering, preserving the strong isolation of virtual machines while achieving high startup speed and resource utilization.

[0094] Based on any of the above embodiments, step 150, the resource reclamation of the one-time virtual machine instance, includes: Send a forced shutdown command to the one-time virtual machine instance; Delete the definition information of the one-time virtual machine instance and the files corresponding to the incremental overlay image.

[0095] Specifically, once a task is completed (whether due to successful completion, compilation failure, or cancellation due to timeout), the orchestration controller will immediately trigger a resource reclamation process, which is designed to be a fast and thorough cleanup operation.

[0096] Specifically, the orchestration controller first sends a forced shutdown command (such as calling `virshdestroy`) to the hypervisor. This command is equivalent to unplugging the power supply to a physical server, directly terminating the virtual machine's CPU execution and releasing its memory. For one-time virtual machines, since their generated data is already streamed or stored in shared storage in real time, this forced shutdown is safe and efficient.

[0097] Secondly, after the virtual machine stops running, the orchestration controller will delete the definition information of the one-time virtual machine instance (such as by calling virsh undefine), remove it from the registration list of the virtualization hypervisor, and release the management resources it occupies.

[0098] Finally, the system performs file-level cleanup, directly deleting the incremental overlay image file corresponding to the task. Since the base image is read-only and shared, deleting the overlay file only erases all temporary states and modifications generated during the current task's runtime, without affecting other tasks or the underlying environment. The entire destruction and cleanup process typically takes about 2 seconds.

[0099] This invention, through a forced shutdown and cleanup mechanism, ensures that computing and storage resources are immediately released after a task ends, making them available for subsequent tasks. More importantly, this destruction method eliminates any residual state that may have been generated during task execution, such as temporary files, modified configurations, and residual processes, thereby ensuring that the next task can begin execution in a completely reset and clean environment.

[0100] Based on any of the above embodiments, the steps for constructing the dependency database include: Offline scan of the metadata files of all candidate task objects, and extract the object identifier, dependency list, and mapping relationship between package group and sub-package of the candidate task object from the metadata files; Based on the object identifiers and dependency lists of each candidate task object, an initial dependency graph is constructed, and a depth-first traversal is performed on the initial dependency graph. During the traversal, visited nodes are detected to identify circular dependencies. If a circular dependency is detected, the corresponding circular edge is silently disconnected to obtain an acyclic task dependency graph. Based on the extracted mapping relationship between the package group and the sub-package, a forward mapping from the package base to the sub-package and a reverse mapping from the sub-package to the package base are established. The acyclic task dependency graph, along with the established forward and reverse mappings, are serialized and stored to obtain the dependency database.

[0101] It should be noted that the embodiments of the present invention mainly describe the construction process of the dependency database. This process is usually completed in the offline stage before the task scheduling begins, and is the basis for realizing dependency version freezing and fast dependency resolution.

[0102] Specifically, the orchestration controller performs a traversal scan of all candidate task objects (such as all package source code repositories) in locally stored data during the offline phase. Candidate task objects here refer to all target objects in the system that could potentially be scheduled for execution (such as various components in a Linux distribution). The target of the scan is the metadata file corresponding to each package (such as Arch Linux's .SRCINFO).

[0103] The system parses these text-formatted metadata files, extracting key information such as object identifiers and dependency lists. Here, the object identifier can be the unique name of a software package, such as openssl or glibc. The dependency list refers to the necessary resources to complete the task, such as the names of dependent software packages. Furthermore, it extracts the mapping relationship between package groups and subpackages, because in the actual package management ecosystem, a large source code project (package group, pkgbase) often generates multiple split components (subpackages) after being built. This process is completely offline, without relying on real-time queries from external networks, thus avoiding uncertainties caused by network fluctuations and upstream source changes.

[0104] After extracting the dependency information of all objects, the system first constructs a global initial dependency graph in memory based on the extracted identifiers and dependency lists. This graph covers package group relationships, sub-package mappings, and transitive dependencies at all levels. For example, if package A depends on package B, and package B depends on package C, a complete link A→B→C will be established in the graph. During this process, the system also handles logic such as circular dependency detection. This is because in a large software ecosystem, unreasonable circular dependencies can easily occur (e.g., package A depends on package B, and package B depends on package A). To ensure that the subsequent topological sorting algorithm can work properly and avoid getting stuck in an infinite recursion loop, the system performs a depth-first traversal of the initial dependency graph. During the traversal, the system records the node states on the current access path. Once an accessed node is detected (i.e., the current node points to a node that is already being accessed in the current traversal path), a circular dependency can be accurately identified. At this point, the system adopts a silent disconnection strategy, that is, directly deletes the circular edge that caused the cycle in the graph without interrupting the entire scanning process, and finally cleans the graph to obtain a strictly directed acyclic task dependency graph.

[0105] Furthermore, to address the mismatch between dependency declarations and actual build units, the system establishes a bidirectional mapping data structure in memory. On one hand, a forward mapping is established from the package base to sub-packages, recording which specific components will be produced after a source code project is built. On the other hand, a reverse mapping is established from sub-packages to the package base. For example, when resolving to a task that depends on the component libfoo, but libfoo itself cannot be built independently, but is a sub-package of the foo package base (pkgbase), the system can automatically look up the actual source task node that needs to be scheduled for building through this reverse mapping, thus ensuring the accuracy of dependency resolution.

[0106] After completing the circular dependency detection and bidirectional mapping establishment of the graph, the system merges these three core memory data structures, serializes and stores them, and generates a static structured text file (such as the JSON format archdb.json file). This file is the dependency database. During subsequent task scheduling, the orchestration controller only needs to load this file into memory to achieve O(1) time complexity dependency lookup through hash table lookup, without having to repeatedly parse the metadata file.

[0107] The method provided in this invention achieves dependency freezing through offline pre-computation and serialization storage mechanisms. This means that all dependencies are deterministic and immutable during task execution, eliminating the dependency instability problem caused by real-time network dependency retrieval in traditional build systems (i.e., inconsistent results obtained from builds at different times). More importantly, by automatically detecting and silently breaking circular dependency edges through depth-first traversal, the robustness of the system in the face of large and complex real-world software package ecosystems is improved, preventing the scheduler from crashing or deadlocking due to circular dependencies. Simultaneously, the loading and querying speed of JSON files is extremely fast, significantly improving parsing efficiency during large-scale task scheduling.

[0108] Based on any of the above embodiments, step 130, which involves sending the execution data required for the currently pending task to the one-time virtual machine instance, includes: Step 131: Package the execution data into a binary stream in archive format; Step 132: The binary stream is transmitted to the task execution agent service in the one-time virtual machine instance via a network transmission protocol. The task execution agent service is used to decompress the binary stream to a temporary directory and move the decompressed content in the temporary directory to the target execution path through an atomic renaming operation.

[0109] It should be noted that the embodiments of the present invention mainly describe the process of securely transferring execution data (such as source code and build scripts) from the host machine to the virtual machine.

[0110] Specifically, once the virtual machine starts and is ready, the host machine needs to send the source code directory required by the current task. To improve transmission efficiency, the orchestration controller first packages the target directory (e.g., / path / to / source) on the host machine. This packaging typically uses an archive-formatted binary stream, such as a tar stream or a compressed tar.gz stream. The advantage of streaming is that it avoids generating a large intermediate temporary file on the host machine's disk; instead, it allows reading the source file while sending it over the network, reducing disk I / O overhead.

[0111] The orchestration controller sends the aforementioned binary stream to a specific interface (such as / upload) of the task execution agent service (Agent) running inside the virtual machine via a network transport protocol (specifically, a POST request using the HTTP protocol).

[0112] Upon receiving a request, the agent inside the virtual machine performs the following operations to ensure data integrity: First, the agent directly reads the network stream and decompresses it to a temporary directory (such as / tmp / upload-). <uuid>At this point, the target execution path (e.g., / build / package_name) has not yet been modified. Only after the entire stream has been successfully received and fully decompressed will the Agent invoke an atomic operation provided by the operating system (such as the `mv` command or the `rename` system call) to rename the temporary directory to the target execution path. It should be understood that this mechanism is called an atomic write strategy. If a network interruption or decompression error occurs during transmission, the temporary directory will be discarded, and the target path will not be left with corrupted files that were only partially transmitted.

[0113] This invention, through HTTP streaming and atomic renaming mechanisms, ensures both high efficiency in file transfer between the host machine and the virtual machine, and data consistency under extreme conditions. Even in the event of network instability or abnormal transmission interruption, the execution environment within the virtual machine will not be corrupted by files, thus guaranteeing the reliability of task execution.

[0114] Based on any of the above embodiments, step 130, instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data, includes: Step 133: Control the one-time virtual machine instance to access the local dependency repository in the shared storage layer through the file system protocol. The local dependency repository stores the task artifacts generated by the previous task and the dependency resources of the current task to be executed. Step 134: By polymorphically calling the standard interface corresponding to the currently pending task, a task execution instruction is issued to the one-time virtual machine instance. The task execution instruction is used to trigger the one-time virtual machine instance to perform closed-loop computation in an isolated environment where external network connections are prohibited, using the task artifacts and dependency resources in the local dependency repository and the execution data. The standard interface is provided through a preset build backend abstraction layer, which is statically configured with a standard interface that is uniformly adapted to different build systems.

[0115] It should be noted that the embodiments of the present invention mainly describe how a virtual machine performs tasks in an isolated environment using a local dependency repository and execution data. This process does not depend on an external network and the result is highly deterministic. At the same time, by introducing an abstraction layer architecture, seamless compatibility with different build systems is achieved.

[0116] Specifically, after the virtual machine starts and completes initialization, and successfully receives the execution data (such as source code) transmitted by the host machine, the orchestration controller controls the virtual machine to mount the shared storage layer directory of the host machine to the file system inside the virtual machine through a file system protocol (such as the NFS protocol). This shared directory contains a core component, namely the local dependency repository (Depot).

[0117] Here, Depot is a local file directory that stores data according to a specific structure (such as build backends, build configurations, and processor architecture hierarchical organization). It not only stores pre-prepared basic dependency packages but also the task artifacts generated by preceding tasks in the task dependency graph. For example, if the current task is to build B, and B depends on A, and A has just been built, then A's artifact packages already exist in this NFS-mounted Depot directory, which B can directly access.

[0118] Before issuing specific instructions, the orchestration controller first determines the specific commands required for the task through a preset build backend abstraction layer. Since there may be various different package management ecosystems in actual applications (such as Arch Linux's makepkg, Microsoft's vcpkg, Debian's dpkg, etc.), this build backend abstraction layer is designed to provide a unified standard interface that adapts to different build systems. For example, it defines standard interfaces including resolving dependencies, computing paths, and executing builds.

[0119] For the currently pending task, the orchestration controller will invoke the corresponding standard interface (such as the `build` interface) through polymorphic calls, based on the build ecosystem to which the task belongs. This polymorphic mechanism allows the orchestration controller to ignore the details of the underlying build commands; instead, the specific build backend adapter generates the corresponding shell commands. For example, for an Arch Linux task, the interface will generate the command `makepkg --noconfirm -s`.

[0120] Subsequently, the orchestration controller issues task execution instructions to the virtual machine. The computation process triggered by this instruction takes place in a closed, isolated environment. Specifically, the virtual machine is configured to disable external network connections, and its internal package manager is forced to retrieve dependencies only from the local dependency repository. Therefore, during task execution, the virtual machine strictly utilizes the dependency resources in the local dependency repository, task artifacts, and previously passed execution data (source code) for computation. This mechanism ensures that all the dependency resources and execution data (i.e., source code) required by the task come from the local controlled environment, eliminating the impact of network latency and upstream source changes.

[0121] The method provided in this invention, by combining a local dependency repository mounted on NFS and a closed network environment, severs the uncertain connection between the build process and the external Internet. This not only solves the problem of build failure caused by network jitter, but more importantly, it achieves dependency freezing, thereby improving the security and traceability of the software supply chain.

[0122] Based on any of the above embodiments, after step 130, the method further includes: Receive the task execution logs transmitted back in real time by the one-time virtual machine instance via a streaming protocol; The task execution log is parsed to update the running status of the currently pending task in real time based on the parsing results.

[0123] It should be noted that the embodiments of the present invention mainly describe the mechanism of real-time log feedback and status update during task execution. This mechanism enables the host machine to monitor the internal operating status of the virtual machine in real time.

[0124] Specifically, during the execution of the aforementioned task instructions (such as compiling code) by the virtual machine, a large amount of standard output (stdout) and standard error (stderr) information is generated. To avoid waiting for all tasks to complete before transmitting the logs all at once, this system adopts a streaming transmission protocol.

[0125] Specifically, the agent service inside the virtual machine captures the output of the child processes in real time and encapsulates it into structured data packets (such as NDJSON format). Each data packet contains the log content, timestamp, and log level. The agent streams these data packets to the orchestration controller using chunked encoding of the HTTP response.

[0126] The orchestration controller, acting as the receiver, reads this HTTP stream in real time. Each line of JSON data read is parsed. The parsed log content is written to a persistent log file on the host machine for users to view in real time. Simultaneously, the orchestration controller updates the running status of the currently pending tasks based on the parsing results, such as runtime and last active time. When the stream ends, the last message typically contains the command's exit code. The orchestration controller uses this exit code (e.g., 0 for success, non-zero for failure) to ultimately update the task status to success or failure.

[0127] This invention enables monitoring of remotely isolated tasks through the NDJSON streaming protocol. Users can see the build progress in real time without logging into the virtual machine, and can immediately locate the cause in the host machine logs if an error occurs.

[0128] Based on any of the above embodiments, step 140 specifically includes: Step 141: Send a request to the one-time virtual machine instance to download the task artifacts. The one-time virtual machine instance is used to verify whether the source path specified in the request is within the preset security path whitelist. If it is, the generated task artifacts are packaged and sent back. Step 142: Receive the task artifact returned by the one-time virtual machine instance, write the task artifact into the local dependency repository in the shared storage layer, and call the update command through the global lock mechanism to update the package database maintained in the local dependency repository.

[0129] Specifically, after the current task to be executed successfully completes its computation, the orchestration controller sends a request to the disposable virtual machine instance to download the task artifacts (e.g., by calling the copyOut download interface provided by the agent service). To prevent malicious build scripts or test code from attempting to read sensitive system-level files within the virtual machine, the task execution agent service inside the virtual machine implements a mandatory security protection mechanism. Upon receiving the download request, the disposable virtual machine instance first verifies whether the source path specified in the request is within a preset safe path whitelist. For example, this whitelist strictly restricts reading only predefined build and log output directories such as / tmp, / depot, and / var / log. If the verification passes, confirming that the source path is valid and within the whitelist, the agent service packages the generated task artifacts (e.g., packaged as a binary stream in tar.gz format) and sends them back to the host machine's orchestration controller.

[0130] Subsequently, the orchestration controller receives the task artifacts returned by the one-time virtual machine instance and writes them directly to the corresponding hierarchical directory of the local dependency repository (Depot) in the shared storage layer. Because the scheduling system of this invention supports highly parallel execution of multiple tasks, it is highly likely that multiple virtual machine build tasks will complete simultaneously and attempt to update the underlying database of the same local repository at the same time. To prevent data write conflicts, after completing the physical write of the artifact file, the orchestration controller acquires exclusive modification rights through a global locking mechanism (e.g., using system-level file locks or SQLite database transactions to achieve global mutual exclusion queuing), and then calls an update command (e.g., calling the `repo-add` command for the Arch Linux ecosystem) to update the package database maintained in the local dependency repository. The global lock is released immediately after the update is completed, ensuring that the local package database during continuous integration is always up-to-date and undamaged, thus enabling subsequent tasks that depend on the artifact to immediately and correctly parse and pull the dependency from the repository.

[0131] This invention improves data security in the build and test environments by introducing a strict security path whitelist verification mechanism on the agent side of a one-time virtual machine instance, effectively blocking malicious operations that read sensitive system files without authorization. At the same time, by using a global lock mechanism to control the update of the package database of the local dependency repository in the shared storage layer, it avoids the risk of corruption or inconsistency of the underlying database files when writing to large-scale distributed concurrent tasks, ensuring the high reliability and data consistency of the entire distributed scheduling system in high-concurrency scenarios.

[0132] Based on any of the above embodiments, after the step of determining the execution order of each task to be executed according to the task dependency graph in step 120, the method further includes: The task dependency graph is divided into multiple execution levels using a topological sorting algorithm, where tasks within the same execution level are independent of each other. Accordingly, in step 120, creating an independent, one-time virtual machine instance for the currently pending task according to the execution order includes: According to the order of the execution levels, multiple one-time virtual machine instances are created concurrently for multiple tasks to be executed in the current execution level.

[0133] It should be noted that the embodiments of the present invention mainly describe how to use topological sorting to transform complex task dependency graphs into executable execution levels, thereby maximizing parallel scheduling efficiency.

[0134] Specifically, after constructing the task dependency graph, the orchestration controller does not directly queue all tasks linearly, but first runs a topological sorting algorithm (such as a variant of the Kahn algorithm). The core purpose of this algorithm is to identify which tasks are independent at the current moment, that is, all their preconditions have been met.

[0135] The system uses algorithms to categorize these tasks into the same set, which is called an execution level. For example, in a scenario of building a Linux system, basic packages such as glibc and linux-api often do not depend on other packages, and they would be classified as Level 0. Packages such as zlib and gcc that depend on glibc would be classified as Level 1. Crucially, tasks within the same execution level are independent of each other. This means that all tasks in Level 0 (such as glibc and linux-api) have no sequential requirements and can theoretically begin execution simultaneously.

[0136] Based on the above division, the orchestration controller can adopt a hierarchical concurrent scheduling strategy. First, the system locks the current execution level (e.g., Level 0). For all multiple tasks to be executed within this level, the scheduler will attempt to start them simultaneously. Specifically, the system checks the host machine's resource capacity (such as the number of CPU cores and memory size). If resources allow (e.g., the host machine has 64 CPU cores, and Level 0 has 10 tasks), the system will immediately create multiple one-time virtual machine instances concurrently. This means that each of the 10 tasks will have its own independent virtual machine, and these virtual machines will start and begin building almost simultaneously (in parallel). Only after all tasks in Level 0 have completed execution will the system enter Level 1 and start all tasks in Level 1 concurrently again.

[0137] This invention significantly improves overall build efficiency by transforming the dependency graph into a hierarchical structure and implementing concurrent scheduling within each hierarchy. Compared to traditional serial builds or scheduling based on simple queues, this method fully utilizes the computing power of multi-core servers. In actual testing, for build tasks containing hundreds of software packages, this solution can reduce the total time from tens of hours to several hours, achieving an order-of-magnitude performance improvement.

[0138] Based on any of the above embodiments, the step of dividing the task dependency graph into multiple execution levels using a topological sorting algorithm includes: Calculate the in-degree of each task node in the task dependency graph, and place the task node with an in-degree of zero into the first execution level. The in-degree represents the number of direct predecessor tasks that the task node depends on. After processing the task nodes in the current execution level, update the in-degree of subsequent task nodes, and put the task nodes with an in-degree of zero after the update into the next execution level; Repeat the steps of updating the in-degree of subsequent task nodes and placing task nodes with an in-degree of zero after the update into the next execution level, until all task nodes in the task dependency graph are placed into the corresponding execution level.

[0139] It should be noted that the embodiments of the present invention mainly describe the specific calculation logic of the topology sorting algorithm, that is, how to dynamically generate the execution level through in-degree calculation.

[0140] Specifically, when the algorithm starts, it first traverses the entire task dependency graph and calculates the in-degree of each task node. In a DAG graph, the in-degree of a node represents how many edges point to it, that is, how many other tasks it depends on (i.e., the number of direct predecessor tasks).

[0141] Task nodes with an in-degree of zero mean they have no outstanding dependencies and can be executed immediately. The system collects these nodes and places them in the execution queue of the first execution level (usually marked as Level 0).

[0142] Once the orchestration controller has completed the scheduling and execution of all tasks within the current execution level (e.g., Level 0) (i.e., the virtual machines are destroyed and their artifacts are stored), the algorithm logically considers these nodes to be removed from the graph. At this point, the system iterates through all subsequent task nodes (i.e., downstream tasks that depend on them) of these completed nodes and decrements the in-degree of these subsequent nodes by 1 (or subtracts the corresponding number of dependencies). During the update process, if the in-degree of a subsequent node becomes zero, it means that all its dependencies are ready. The system collects these newly generated task nodes with an in-degree of zero and places them into the next execution level (e.g., Level 1).

[0143] The system repeats the above steps in a loop, continuously generating Level 2, Level 3, and so on, until no nodes remain in the graph. Ultimately, all task nodes are assigned to a defined level, forming an ordered list of execution sequences.

[0144] Based on any of the above embodiments, the method further includes: Monitor the execution status of each pending task; If the execution status of any pending task is detected as failed, all downstream tasks that depend on any pending task are retrieved according to the task dependency graph. Mark each downstream task as blocked to cancel the scheduled execution of each downstream task.

[0145] It should be noted that the embodiments of the present invention mainly describe how the system uses a dependency graph to perform fast failure propagation when a task fails, so as to avoid ineffective resource waste.

[0146] Specifically, throughout the distributed task scheduling process, the orchestration controller monitors the execution status of all running or waiting tasks in real time through various mechanisms (such as HTTP polling, streaming log parsing, and process exit code capture). These statuses typically include pending, running, completed, failed, and blocked.

[0147] Once the orchestration controller detects that the state of a task (e.g., task A) has changed to failure, the system immediately triggers the failure handling logic. First, the system queries the task dependency graph in memory. This graph not only records the dependencies of each task but also maintains inverse dependencies—that is, which tasks depend on the current task. Using this inverse index, the system can quickly retrieve all downstream tasks that directly or indirectly depend on task A. For example, if task B depends on task A and task C depends on task B, then B and C are both downstream tasks.

[0148] After retrieving all affected downstream tasks, the orchestration controller updates the status of these tasks to blocked. Tasks in the blocked state will be canceled from scheduling, meaning the system will no longer allocate virtual machine resources for these tasks or attempt to start them.

[0149] This invention utilizes a dependency graph for failure propagation, avoiding a large amount of unnecessary computation. In traditional build systems, if the underlying base package fails to build, hundreds of packages that depend on it may continue to attempt to build and eventually all fail, wasting significant computation time and resources. This invention, however, can identify the entire failure chain and immediately stop scheduling related tasks, saving substantial time on unnecessary computation and significantly improving the overall system performance.

[0150] Based on any of the above embodiments, the method further includes: When creating each one-time virtual machine instance, a corresponding child cancellation signal is derived from the root-level cancellation signal controller, which is created during system startup. Upon receiving an external interrupt command or a system abnormal exit signal, the root-level cancellation signal controller is activated to propagate the cancellation signal to all child-level cancellation signals. In response to the child-level cancellation signal, concurrent resource reclamation operations are performed on all one-time virtual machine instances.

[0151] It should be noted that the embodiments of the present invention mainly describe how the system uses the Abort Controller to achieve cascading resource cleanup, ensuring that no orphan virtual machines are left behind in the event of abnormal exit or user interruption.

[0152] Specifically, at the start of the orchestration controller, the system initializes a global root-level cancel signal controller. When the scheduler decides to start a new task and create a one-time virtual machine instance for it, it derives a corresponding child AbortSignal from this root-level controller. This child signal is bound to a specific virtual machine instance, forming a tree-like signal propagation structure.

[0153] During system operation, situations may arise requiring immediate termination. For example, a user might press Ctrl+C on the console (sending a SIGINT signal), the operating system might send a termination signal, or the orchestration controller itself might encounter an unrecoverable fatal error (such as database corruption). When the orchestration controller captures these external interrupt commands or abnormal system exit signals, it immediately calls the root controller's `abort()` method. Because the signal controller uses a tree structure, this cancellation action propagates to all child cancellation signals.

[0154] Each virtual machine instance's management thread registers to listen for child cancellation signals upon startup. Once a signal is received, these threads are immediately awakened and execute cleanup logic. Cleanup logic includes forcibly terminating the virtual machine by calling virtualization interfaces (such as `virsh destroy`), deleting the virtual machine definition, and deleting overlay files. Since the cleanup operations for each virtual machine are independent, the system uses multi-threading or asynchronous I / O to execute these cleanup tasks in parallel.

[0155] This invention, through a cascading cancellation mechanism based on AbortSignal, ensures that no orphaned virtual machines or dangling disk images are left even in abnormal situations, thus guaranteeing the long-term stable operation of the host machine and reducing operational complexity.

[0156] It should be noted that there are alternative implementations of the technical solutions provided by this invention in certain technical details. These alternative solutions can also achieve the purpose of the invention, but there are certain differences in performance, complexity or applicable scenarios.

[0157] For example, because containers start up faster (in seconds), have lower resource overhead, and can support higher concurrency, Linux containers (LXC) or LXD system containers can be used instead of KVM virtual machines. Isolation is achieved through Linux namespaces and control groups, and the container image also uses an overlay file system (OverlayFS) for copy-on-write. This alternative offers a lower level of isolation than a full virtual machine, and since containers share the host kernel, there is a risk of kernel vulnerabilities leading to container escape. Therefore, it is not suitable for scenarios requiring the compilation of kernel modules, replacement of system libraries, or testing of different kernel versions; it is suitable for scenarios with low security isolation requirements, where only user-space applications are built, and where extreme performance is prioritized.

[0158] For example, virtual machine creation and management can be migrated to cloud platforms (such as AWS EC2, Azure VMs, and Alibaba Cloud ECS). The orchestration controller dynamically creates and destroys virtual machine instances through cloud platform APIs, and shared storage is implemented through cloud storage services (such as NFS and object storage). This alternative has strong elastic scaling capabilities, dynamically adjusting the number of virtual machines according to task load, breaking through the resource limitations of a single physical host; and requires no maintenance of physical hardware. It is suitable for scenarios with large fluctuations in task load, the need for temporary large-scale concurrency, sufficient budget, and low data security requirements.

[0159] For example, AWS's open-source Firecracker microVM technology can be used. Firecracker is a lightweight virtual machine monitor optimized for serverless computing scenarios, with startup times in the millisecond range while providing complete virtualization isolation. This alternative offers extremely fast startup speeds (less than 125 milliseconds), low resource overhead, and security isolation levels comparable to KVM; it supports high-density deployment, with a single host capable of running thousands of microVM instances. It is suitable for scenarios requiring only a Linux environment, prioritizing extreme startup speed and resource density, and for which developers are willing to invest in custom image creation.

[0160] For example, the local Depot repository can be replaced with a distributed caching system (such as Redis or Memcached) or object storage (such as MinIO), with packages stored as key-value pairs. Virtual machines retrieve dependencies from the cache via an HTTP API. This alternative supports multiple physical hosts sharing the same dependency repository, offering better scalability; the caching system provides high availability and fault recovery capabilities. It is suitable for scenarios with multi-homed cluster deployments, high availability requirements, and existing mature distributed storage infrastructure.

[0161] For example, an SQLite database can be replaced with a distributed message queue (such as RabbitMQ or Kafka). Task status is managed through message passing, and the orchestration controller acts as a consumer, retrieving tasks from the queue and distributing them to virtual machines. Upon task completion, a result message is sent. This alternative inherently supports distributed deployment; multiple orchestration controller instances can consume the task queue in parallel, improving system availability. The message queue provides persistence and retry mechanisms. It is suitable for scenarios requiring high availability and horizontal scalability, with existing message queue infrastructure, and extremely high task volumes.

[0162] Based on any of the above embodiments, this invention provides a distributed task scheduling system for one-time virtual machines. It employs a separate architecture of an intelligent orchestration controller and a stateless task agent, combining core technologies such as QCOW2 copy-on-write technology, pre-computed dependency graphs, and local dependency repositories to achieve completely isolated, deterministic, and repeatable distributed task execution. The system's components, their interconnections, and the detailed process of the method will be described in detail below.

[0163] Figure 2 This is a schematic diagram of the structure of the distributed task scheduling system based on a one-time virtual machine provided by the present invention, as shown below. Figure 2 As shown, the system includes: The orchestration controller 210 is configured to run on the host machine to execute the distributed task scheduling method based on a one-time virtual machine as described in any of the above embodiments; A one-time virtual machine cluster 220 consists of multiple temporarily created virtual machine instances, each of which is used to receive instructions from the orchestration controller and execute tasks; The shared storage layer 230 includes a base image repository, an overlay image directory, and a local dependency repository, used to connect the orchestration controller and the one-off virtual machine cluster.

[0164] Specifically, the system consists of three main parts: an orchestration controller, a one-time virtual machine cluster, and a shared storage layer. They work together through network communication protocols and file system mounting. Figure 2 The arrows shown indicate the data flow. The orchestration controller sends commands to the virtual machine via the HTTP protocol (as shown by the solid arrows), and the virtual machine returns data via tar streams and log streams (as shown by the dashed arrows). Both the orchestration controller and the virtual machine access the shared storage layer via NFS mounts.

[0165] The orchestration controller is a stateful central control component running on the physical host machine, responsible for core business logic such as task scheduling, virtual machine management, and dependency computation. The orchestration controller comprises seven sub-modules: task scheduling, database, virtual machine driver, proxy client, dependency resolution, backend abstraction layer construction, and cancellation signal controller. These modules collaborate through internal interfaces, with the orchestration controller handling the centralized processing of all business logic.

[0166] The task scheduling module is used to calculate the hierarchical execution order of tasks based on a topological sorting engine using a directed acyclic graph (DAG). This module employs a variant of the Kahn algorithm, and the specific process is as follows: First, a task dependency graph is constructed, where nodes represent tasks to be executed and directed edges represent dependencies. Then, the in-degree of each node (i.e., the number of direct predecessor tasks that the node depends on) is calculated. Nodes with an in-degree of zero are added to the queue as first-level executable tasks. After executing all tasks in the current level, the in-degree of subsequent nodes is updated, and newly generated nodes with an in-degree of zero are added to the next level. This process is repeated until all nodes have been processed. This module can automatically identify tasks that can be executed in parallel, maximizing resource utilization.

[0167] The database module is used to persistently store metadata such as task status, dependencies, and execution history based on an embedded SQLite database. The database table structure includes a task table (recording task ID, package name, status, start time, end time, exit code, etc.), a dependency table (recording dependencies between tasks), and a virtual machine table (recording virtual machine ID, overlay path, associated tasks, etc.). This module ensures the atomicity of status updates through a transaction mechanism and supports state recovery after system crashes.

[0168] The virtual machine driver module is responsible for the complete lifecycle management of virtual machines, including overlay creation, virtual machine startup, resource monitoring, and virtual machine destruction. The core innovation of this module lies in two virtual machine startup paths: (1) the fast path, which directly generates an XML configuration file and calls the virsh define and virsh start commands, skipping the redundant operations of virt-install; (2) the traditional path, which uses the virt-install tool, which has better compatibility but is slower to start. The fast path avoids querying the operating system information database (osinfo) at runtime by pre-generating an XML template containing complete hardware configurations such as CPU, memory, disk, and network, thus reducing the virtual machine startup time from several minutes to less than 10 seconds. The virtual machine driver module also uses the virt-sysprep tool to clean up the machine ID, DHCP lease, SSH key, and other status information in the QCOW2 overlay, ensuring that multiple concurrently started virtual machines will not fail to obtain IP addresses due to MAC address or machine ID conflicts.

[0169] The agent client module serves as the communication interface between the orchestration controller and the task agent within the virtual machine, implementing remote procedure calls based on the HTTP protocol. This module provides three core APIs: (1) the copyIn interface, which packages host files or directories into a tar.gz stream and uploads it to the specified path in the virtual machine via HTTP POST; (2) the execStream interface, which sends shell commands to the virtual machine, receives streaming log output in NDJSON format, and supports real-time transmission of stdout, stderr, and heartbeat events; and (3) the copyOut interface, which downloads files or directories from the specified path in the virtual machine, and the virtual machine packages them into a tar.gz stream and returns it. This module adopts an atomic write strategy during file transmission, that is, it first decompresses the tar stream to a temporary directory, and then replaces the target path through an atomic renaming operation after successful decompression, avoiding partial write state caused by transmission interruption.

[0170] The dependency resolution module is used to quickly query the transitive dependencies of software packages based on a pre-computed dependency database. This module scans all package metadata files (such as Arch Linux's .SRCINFO) offline, extracting information such as package names, dependency lists, and sub-package relationships, constructing a complete dependency graph, and serializing it into a JSON file (such as archdb.json). At runtime, the module loads the JSON file into memory and uses a hash table to achieve dependency queries with O(1) time complexity. The module also handles the mapping relationship between package groups and sub-packages. For example, if a package depends on libfoo, but libfoo is a sub-package of the foo package group, the module can automatically reverse the lookup to find that the package group to be built is foo. For circular dependencies (such as A depending on B and B depending on A), the module checks visited nodes during depth-first traversal, silently breaking the circular edges to avoid infinite recursion.

[0171] A backend abstraction layer is constructed, providing a unified interface to adapt to different build systems (such as makepkg, vcpkg, dpkg, etc.). This module defines standard interfaces including: resolveBuildDir (calculates the build working directory path), resolveDepotDir (calculates the local repository path), resolveTransitiveDependencies (resolves transitive dependencies), build (executes build commands), and generateVMXML (generates virtual machine configurations). Each build backend implements these interfaces, and the orchestration controller executes specific operations through polymorphic calls. This design allows the system to be easily extended to support new build tools without modifying the core scheduling logic.

[0172] The signal controller is disabled for cascading resource cleanup based on the AbortSignal mechanism. The orchestration controller creates a root-level AbortController at startup, and all virtual machines derive child AbortSignals from this root controller upon creation. When a user sends an interrupt signal (such as Ctrl+C) or the system exits abnormally, the root controller calls the abort() method, and the signal automatically propagates to all child tasks. The virtual machine driver module listens for the aborted event of the AbortSignal. Upon receiving the signal, it immediately calls virsh destroy to forcibly terminate the virtual machine, virsh undefine to delete the virtual machine definition, and finally deletes the QCOW2 overlay file. This mechanism ensures that no orphaned virtual machines or dangling disk images are left even in abnormal situations.

[0173] A one-time virtual machine cluster consists of multiple virtual machine instances with temporary lifecycles, such as... Figure 2 As shown, multiple virtual machine instances, including task agents VM-1, VM-2, and VM-N, are displayed. Each virtual machine instance is dedicated to executing a single build task. Each virtual machine instance contains two core components: a QCOW2 overlay and an HTTP proxy service. These are dynamically created on demand and destroyed after the task is completed; the overlay is automatically deleted. Key features of the virtual machines include: ① Single-use mode: Each virtual machine executes only one task from creation to destruction. Upon completion of the task, regardless of success or failure, the virtual machine is immediately destroyed, completely eliminating state pollution between tasks. The QCOW2 overlay is automatically deleted when the virtual machine is destroyed, while the base image remains unchanged and can be reused by subsequent virtual machines.

[0174] ②QCOW2 Overlay Technology: The virtual machine disk adopts a three-layer structure. The bottom layer is a read-only base image, containing a complete operating system and pre-installed toolchain; the middle layer is the QCOW2 overlay layer, created using the `qemu-img create -b` command, specifying the base image as the backing file; the top layer is the runtime state. When a virtual machine writes a file, the QCOW2 format's copy-on-write mechanism automatically writes the modified data blocks to the overlay layer, while the base image remains read-only. This allows 100 concurrent virtual machines to share the same 50GB base image, requiring only additional storage for the incremental data of their respective overlay layers (typically less than 5GB per task), reducing the total storage overhead from 5000GB to approximately 550GB.

[0175] ③ HTTP Proxy Service: A lightweight HTTP server (approximately 600 lines of code) runs within the virtual machine, listening on port 8080 and providing three REST API endpoints: POST / upload (receives a tar.gz file stream and decompresses it to a specified path), POST / download (packages a specified path into a tar.gz stream and returns it), and POST / exec (executes a shell command and streams logs). The proxy service starts automatically via systemd when the virtual machine starts, requiring no manual intervention. The proxy service implements a path whitelist mechanism; the copyOut operation only allows reading predefined safe paths (such as / tmp, / depot, / var / log) to prevent malicious build scripts from reading sensitive system files.

[0176] ④ Concurrency Management: The orchestration controller dynamically determines the number of concurrent virtual machines based on the number of CPU cores and memory capacity of the physical host. For example, for a host with 128GB of memory, if each virtual machine is allocated 4GB of memory, theoretically 32 virtual machines can run concurrently. However, considering the host machine overhead, the actual limit is 28. The scheduler maintains a counter for running virtual machines. When a virtual machine is destroyed, the counter is decremented by one, and tasks waiting in the queue can be scheduled to be executed in the newly created virtual machine.

[0177] The shared storage layer mounts the physical host's file system directory to all virtual machines via the NFS protocol, enabling data sharing and persistence. The shared storage layer comprises five components: a base image repository, an overlay image directory, a local dependency repository, a build working directory, and a log directory.

[0178] The base image repository (e.g., / mnt / smith / wm-builder-baseimages) stores QCOW2 base images for various types of virtual machines, such as builder-archlinux-base.qcow2 and builder-windows-base.qcow2. The base images are read-only, shared by all virtual machines, and referenced through QCOW2's backing file mechanism. The base images are pre-created during system initialization and contain a complete environment including the operating system, build toolchain, and runtime libraries.

[0179] The overlay image directory (e.g., / mnt / smith / wm-builder-workdir / overlays) is used to store temporary QCOW2 overlay files for each virtual machine, with filenames in the format <task ID>.qcow2. Overlays are generated when the virtual machine is created and deleted when it is destroyed. If the build fails, you can choose to retain the overlay for debugging. Administrators can use the guestfish tool to mount the overlay to view the file system status within the virtual machine.

[0180] Local dependency repositories (e.g., / mnt / smith / wm-builder-workdir / depot) store built package files, with a directory structure organized hierarchically by build backend, build configuration, and architecture. For example, the Arch Linux O2 optimization level x86_64 architecture package is stored in the directory depot / archlinux / O2 / x86_64 / , which also contains the pacman database file (wm-repo.db.tar.gz). The package manager within the virtual machine is configured to install dependencies from the local repository, with the configuration item Server = file: / / / mnt / smith / wm-builder-workdir / depot / archlinux / O2 / x86_64. After the build is complete, newly generated packages are written directly to this directory, and the repo-add command is called through a global locking mechanism to update the package database, ensuring that the database is not corrupted during concurrent writes.

[0181] The build working directory (e.g., / mnt / smith / wm-builder-workdir / builddirs) is used to store the source code, intermediate compilation files, build logs, etc., for each task. The directory structure retains the native path format of the build tool; for example, the build directory for makepkg is depot / archlinux / O2 / x86_64 / _build / <package name> / , ensuring that the debug symbol paths in the compilation artifacts are consistent with the actual paths, facilitating the loading of source code by debugging tools such as GDB.

[0182] The log directory (e.g., / mnt / smith / wm-builder-workdir / logs) stores the complete build logs for each task, organized by task ID. Log files include stdout.log (standard output), stderr.log (standard error), and events.ndjson (structured event stream). Logs are written in real-time during task execution and transmitted via the NDJSON streaming protocol. Each line is a JSON object containing fields such as timestamp, log level, and message content.

[0183] The workflow of the system provided in this embodiment of the invention is described below. The task scheduling and execution process of this system mainly includes six stages: task submission, dependency resolution, virtual machine creation, task execution, artifact collection, and resource cleanup. Step 1: Task Submission and Dependency Resolution Users submit build tasks to the orchestration controller via command-line tools, specifying a list of package names, a build backend (e.g., archlinux), and build configurations (e.g., O2 optimization level). Upon receiving the request, the orchestration controller calls the `resolveTransitiveDependencies` interface of the dependency resolution module to query the pre-computed dependency database (archdb.json) and obtain the complete transitive dependency list for each package. For example, if a user requests to build the openssl package, the dependency resolution module finds that openssl depends on zlib and gcc, zlib depends on glibc and linux-api, and gcc depends on glibc, linux-api, and tzdata. The module organizes these dependencies into a dependency graph: glibc, linux-api, and tzdata are the root nodes; zlib depends on glibc and linux-api; gcc depends on glibc, linux-api, and tzdata; and openssl depends on zlib and gcc. After the dependency graph is built, the `computeBuildOrder` interface of the task scheduling module is called to calculate the topology sorting order.

[0184] Step 2: Topology sorting and hierarchical scheduling The task scheduling module performs Kahn's algorithm topological sorting on the dependency graph: First, it calculates the in-degree of each node. Since glibc, linux-api, and dataset have an in-degree of 0, they are placed in layer 0. After processing tasks in layer 0, the in-degree of zlib and gcc is reduced to 0, and they are placed in layer 1. After processing tasks in layer 1, the in-degree of openssl is reduced to 0 (zlib and gcc have already been processed), and it is placed in layer 2. The final output layering result is: [[glibc, linux-api,tadata], [zlib, gcc], [openssl]]. The orchestration controller writes the layering result to the database, marking each task as pending. The scheduler schedules tasks for execution layer by layer, starting from layer 0.

[0185] Step 3: Virtual Machine Creation and Environment Preparation When a task is scheduled for execution (such as glibc), the virtual machine driver module first calls the `qemu-img create` command to create a QCOW2 overlay. The command is: `qemu-img create -f qcow2 -b / mnt / smith / wm-builder-baseimages / builder-archlinux-base.qcow2 -F qcow2 / mnt / smith / wm-builder-workdir / overlays / task-001.qcow2`. This command creates a new QCOW2 file, with the backing file pointing to the base image, and the format is QCOW2. Next, `virt-sysprep` is called to clean up the state information in the overlay. The command is: `virt-sysprep -a task-001.qcow2 --operations dhcp-client-state,machine-id,net-hwaddr`, which clears configurations that may cause concurrent conflicts, such as DHCP leases, machine IDs, and network card MAC addresses. Then, use the guestfish tool to inject the build agent into the overlay layer with the command: `guestfish -a task-001.qcow2 -icopy-in / path / to / agent / usr / local / bin / : chmod 0755 / usr / local / bin / agent`. Finally, generate the virtual machine XML configuration file, which contains the following key configurations: <memory> 4194304< / memory> (Allocate 4GB of memory) <vcpu> 4< / vcpu> (Allocate 4 virtual CPUs) <disk> <source file=" / mnt / smith / wm-builder-workdir / overlays / task-001.qcow2"> < / disk> (Mount the overlay as the boot disk)<filesystemtype='mount'> <source dir=" / mnt / smith / wm-builder-workdir"> <target dir="depot" / > (Shared storage is mounted via virtiofs). The virtual machine is registered using `virsh define task-001.xml`, and started using `virsh start task-001`. After the virtual machine starts, the driver module polls the HTTP health check interface (GET / health) for the proxy service within the virtual machine to become ready, with a timeout of 60 seconds.

[0186] Step 4: File Transfer and Task Execution After the virtual machine starts, the agent client module first calls the copyIn interface to upload the source code directory from the host machine to the virtual machine. The specific process is as follows: the host machine packages the source code directory into a tar.gz stream and sends it to the virtual machine's / upload endpoint via an HTTP POST request. The request header includes X-Target-Path: / build / glibc and X-Transfer-Type: directory; the virtual machine receives the tar stream and decompresses it to the temporary directory / tmp / upload- <uuid>After successful decompression, execute `mv / tmp / upload-` <uuid> / Atomic replacement is implemented in ` / build / glibc / `. Then, the `execStream` interface is called to execute the build command: `cd / build / glibc&&makepkg --noconfirm -s`. The virtual machine starts this command in a child process, redirecting stdout and stderr via a pipe. The main process reads the pipe content line by line, encapsulating it into an NDJSON format event object (e.g., `{"type":"stdout","timestamp":"2025-01-05T10:30:45Z","message":"==>Starting build..."}`), and sends it to the host machine as a chunked HTTP response. The host machine receives the NDJSON events in real time, parses them, writes them to a log file, and updates the task status (e.g., runtime, last active time). After the command execution is complete, the virtual machine returns an exit code (e.g., `{"type":"exit","code":0}`), and the host machine uses the exit code to determine whether the task succeeded or failed.

[0187] Step 5: Constructing Product Collection After successful task execution, the agent client module calls the copyOut interface to download the build artifacts from the virtual machine. The request parameters specify the source path as / depot / archlinux / O2 / x86_64 / glibc-2.38-1-x86_64.pkg.tar.zst (makepkg directly writes the artifacts to depot via the PKGDEST environment variable). The virtual machine verifies that the path is within the whitelist and packages the file into a tar.gz stream, returning it. Since the depot directory itself is mounted via NFS, the build artifacts are already in shared storage; the copyOut operation is mainly used to verify the existence and integrity of the artifacts. After the task is completed, the orchestration controller acquires a global lock (through file locks or database transactions), calls repo-add / mnt / smith / wm-builder-workdir / depot / archlinux / O2 / x86_64 / wm-repo.db.tar.gz glibc-2.38-1-x86_64.pkg.tar.zst to update the package database, and then releases the lock. After the database update is complete, subsequent tasks (such as zlib and gcc) can install glibc as a dependency from the depot repository.

[0188] Step 6: Virtual Machine Destruction and Resource Cleanup After a task is completed (successfully or unsuccessfully), the virtual machine driver module immediately executes a cleanup process: first, it calls `virsh destroy task-001` to forcibly shut down the virtual machine (equivalent to unplugging the power), then it calls `virsh undefine task-001` to delete the virtual machine definition, and finally it deletes the overlay file `rm / mnt / smith / wm-builder-workdir / overlays / task-001.qcow2`. The entire cleanup process takes approximately 2 seconds. The database module updates the task status to `completed` (successful) or `failed` (failed), recording the end time and exit code. The scheduler checks if there are any tasks at subsequent levels that can be executed: if all tasks at the current level are completed, it calculates the in-degree of the next level's tasks, marks tasks with an in-degree of 0 as ready, and adds them to the execution queue. Steps 3 through 6 are repeated until all levels of tasks have been executed. If a task fails, the scheduler marks all downstream tasks that depend on that task as blocked, and these tasks are no longer executed to avoid unnecessary computational overhead. Users can query failed tasks using command-line tools, fix the problem, and then retry the failed task individually. The system will automatically recalculate the status of downstream tasks and continue execution.

[0189] After introducing the system structure and workflow, the following section, with reference to the accompanying diagrams, further explains the specific details of the system's implementation.

[0190] Figure 3 This is a schematic diagram of the one-time virtual machine lifecycle management process provided by the present invention, as follows: Figure 3 As shown, it illustrates the complete lifecycle of a virtual machine from creation to destruction, divided into four stages: Phase 1, Creating the Overlay Layer: This includes three steps: creating the QCOW2 overlay layer with qemu-img create, cleaning up the state with virt-sysprep, and injecting the proxy service with guestfish.

[0191] Phase 2, starting the virtual machine: This includes three steps: virsh define to register the virtual machine definition, virsh start to start the virtual machine, and waitForBoot to wait for the agent service to be ready.

[0192] Phase three, task execution: This includes three steps: copyIn (uploading files), execStream (executing commands), and copyOut (downloading artifacts). This is the core working phase of the virtual machine.

[0193] Phase four, cleanup and destruction: This includes three steps: `virsh destroy` to force shutdown the virtual machine, `virsh undefine` to delete the virtual machine definition, and deleting overlay files. Destruction results in either a success or failure status.

[0194] Figure 3 The lower center section shows two technical details. The left side explains the QCOW2 overlay technology principle, which illustrates the three-layer structure of base image (read-only shared) → overlay (temporary write) → runtime state, and implements copy-on-write through backing and COW mechanisms. The right side shows the cancellation signal propagation mechanism, demonstrating how AbortSignal propagates at multiple checkpoints (such as after overlay creation and after VM startup), ultimately triggering the automatic cleanup process.

[0195] Figure 4 This is a flowchart of the DAG scheduling process based on a pre-computed dependency database provided by the present invention, such as... Figure 4 As shown, it illustrates the complete process of task dependency resolution and parallel scheduling: Dependency Graph Computation Engine: The left side displays the pre-computed dependency database (archdb.json, i.e., dependency data blocks), which contains the mapping relationship from pkgbase (package group) to subpackages; the right side shows the three core algorithms of the topology sorting engine (Kahn algorithm variant, circular dependency detection, and parallel batch generation). The output is a hierarchical task list, and tasks within each level can be executed in parallel.

[0196] Parallel Execution Engine: This demonstrates specific execution examples. Level 0 includes three dependency-free basic packages: glibc, linux-api, and tzdata, which can be built in parallel. Level 1 includes zlib and gcc, which depend on packages from Level 0. Level 2 includes openssl, which depends on packages from previous levels. Arrows indicate dependencies and execution order.

[0197] Local Dependency Repository - Dependency Freezing Mechanism: Demonstrates the core role of Depot, building sequential dependencies (the DAG topology order calculated by topology.ts) before runtime dependencies are built (provided by Depot through pacman / vcpkg), ensuring key invariants, that is, when building B that depends on A, A must already be in Depot (topology order guaranteed).

[0198] Failure propagation mechanism: When task A fails, tasks B, C, and D that depend on A are marked as blocked, do not execute, and fail quickly; tasks that do not depend on A continue to execute normally; the system supports retrying failed task A separately.

[0199] Figure 5 This is a schematic diagram of the bidirectional file transfer protocol based on HTTP tar streams provided by the present invention, as shown below. Figure 5 As shown, this illustrates the file transfer protocol design between the orchestration controller and the virtual machine. The orchestration controller-host on the left includes a VMInstance client, copyIn and copyOut interfaces, and a security whitelist verification module. The task agent-virtual machine on the right includes an HTTP Agent service, / upload and / download endpoints, and a tar decompression / compression module.

[0200] The middle arrow shows the bidirectional data stream, with the upper one representing the copyIn upload protocol (POST / upload, tar.gz stream from left to right) and the lower one representing the copyOut download protocol (POST / download, tar.gz stream from right to left).

[0201] The specific formats of the two protocols are explained in detail below. The request header of the upload protocol (copyIn) includes X-Target-Path, X-Transfer-Type (directory or file), and Content-Type (application / gzip); the request body is a tar.gz binary stream; the virtual machine uses a temporary directory decompression + atomic renaming writing strategy to ensure that no part of the file is left behind in the event of a transmission failure.

[0202] The copyOut request header includes X-Source-Path; the response header includes X-Transfer-Type and Content-Type; the response body is a tar.gz stream; the proxy server performs security checks, and the target path must be on a whitelist to prevent malicious reading of sensitive system files.

[0203] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects: (1) Achieving a completely isolated and hygienic build environment ensures the determinism and repeatability of build results. By creating an independent virtual machine instance for each task, this invention achieves complete isolation based on hardware virtualization. Each virtual machine has an independent operating system kernel, file system, and network stack, with an isolation level far exceeding that of container solutions. The virtual machine and overlay are destroyed immediately after the task is completed, ensuring that the next task is executed in a completely clean environment, thus completely eliminating environmental contamination issues. In practical applications, the binary hash values ​​of the output results are completely consistent when the build task is repeatedly executed under the same input conditions, verifying the determinism of the build. This effect solves the state residue problem caused by Runner reuse in traditional CI systems, as well as the security risks caused by container kernel sharing.

[0204] (2) Significantly reduces storage overhead and virtual machine creation time, making virtual machine isolation technology applicable to large-scale concurrent scenarios. Using QCOW2 copy-on-write technology, 100 concurrent virtual machines share the same 50GB read-only base image. Each virtual machine only needs to store its own incremental overlay data (typically less than 5GB), reducing total storage overhead from 5000GB in traditional solutions to approximately 550GB, saving up to 90% of storage space. Overlay creation time is less than 1 second, hundreds of times faster than full image copying (several minutes). Combined with the virsh fast startup path, the total time from virtual machine creation to agent service readiness is optimized to less than 10 seconds, approximately 20 times faster than the traditional virt-install solution (several minutes). This makes virtual machine isolation technology practical for the first time in large-scale concurrent short-lifecycle task scenarios. A single physical host with 128GB of memory can run 28 build virtual machines concurrently, achieving performance close to container solutions but with stronger isolation.

[0205] (3) Eliminating the instability of network dependencies and achieving fully offline deterministic builds. Through a dual mechanism of a pre-computed dependency database and a local Depot repository, this invention achieves complete dependency version freezing. The pre-computed archdb.json file is only 16MB, covering over 14,000 packages for Arch Linux. After being loaded into memory, dependency lookup time is less than 1 millisecond, far faster than runtime dependency resolution using external tools. The package manager within the virtual machine is configured to install dependencies only from the local Depot repository, accessing NFS shared directories via the file: / / protocol, eliminating the need for an HTTP server and resulting in an extremely simple architecture. The build process is completely offline, unaffected by upstream service availability, network latency, or version changes. In actual testing, a complete Linux distribution containing hundreds of packages was successfully built in a completely offline environment, verifying the system's offline capabilities and stability.

[0206] (4) Automatically derives task dependencies to maximize parallel scheduling efficiency. Through a hierarchical topology sorting algorithm, this invention can automatically construct a dependency graph and calculate the optimal execution order based on package metadata, eliminating the need for users to manually write complex pipeline configuration files. Tasks without dependencies within the same level are automatically executed in parallel, fully utilizing the CPU and memory resources of the physical host. In practical applications, constructing a task set containing 500 packages traditionally takes about 50 hours in serial execution. This invention reduces the total time to about 3 hours through 28-way concurrency, achieving a speedup of 16 times (considering the dependencies between tasks, the theoretical 28-fold speedup cannot be achieved). The failure propagation mechanism ensures that downstream tasks are immediately marked as blocked instead of waiting for timeouts when dependencies fail, and the fast failure strategy saves about 30% of invalid computation time.

[0207] (5) Reduce migration costs and maintain compatibility with the existing package ecosystem. This invention does not require modification of existing build scripts and toolchains, and is directly compatible with mainstream package definition formats such as Arch Linux's PKGBUILD, vcpkg's portfile, and Debian's debian / rules. By building a backend abstraction layer, only an adapter (usually less than 500 lines of code) needs to be written for the new build tool, allowing the entire scheduling and isolation framework to be reused. This enables organizations with a large number of package definitions to migrate smoothly to this system without investing manpower to rewrite the build definitions. In actual deployment, the Arch Linux official repository containing more than 14,000 PKGBUILDs was successfully migrated to this system. The migration cost was only the writing of a makepkg adapter and the configuration of the depot path, which is far lower than the cost of migrating to functional package managers such as Nix (which requires rewriting all package definitions).

[0208] (6) Achieve fully automated resource lifecycle management and improve system reliability. Through the AbortSignal cascading cleanup mechanism, when the orchestration controller exits abnormally, a cancellation signal is automatically propagated to all running virtual machines. Upon receiving the signal, the virtual machines immediately terminate and clean up the overlay layer, requiring no manual intervention. Combined with the persistent task state of the SQLite database, tasks in the running state are automatically reset to pending after a system crash and restart, resuming scheduled execution. In actual operation and maintenance, multiple simulations of abnormal orchestration controller exit (e.g., kill -9) were performed. After system restart, no orphaned virtual machines or dangling disk images were found, and the task state was automatically restored, verifying the reliability of resource management. This effect solves the operational pain points of traditional solutions, such as container leakage, workspace disk space occupation, and inconsistent task states, reducing the manpower cost of system maintenance.

[0209] (7) Supports heterogeneous execution environments and has strong scalability. Thanks to the separation of responsibilities architecture between the orchestration controller and the build agent, and the lightweight communication protocol based on HTTP + tar stream, this invention can be easily extended to heterogeneous execution environments. The agent protocol is simple and universal, relying only on an HTTP server and the tar command, and can be ported to Windows virtual machines, ARM architecture virtual machines, and even embedded devices. The build backend abstraction layer supports different build systems through interface polymorphism, and adding new build tools does not require modification of the core scheduling logic. In actual verification, Linux x86_64 virtual machines (used for makepkg builds) and Windows virtual machines (used for vcpkg builds) are managed simultaneously under the same orchestration controller. The two types of virtual machines share the same scheduler and storage layer, demonstrating the system's cross-platform capability and scalability.

[0210] (8) Provides real-time streaming logs and complete debugging capabilities. Through the NDJSON streaming protocol, stdout, stderr, and heartbeat events during the build process are transmitted in real-time to the orchestration controller and persisted to log files. Users can view the task execution progress in real-time via command-line tools and quickly locate build errors. For failed tasks, the system can choose to retain the virtual machine overlay. Administrators can use the guestfish tool to mount the overlay and directly view the file system status within the virtual machine to analyze the cause of failure. In actual debugging, by viewing the / var / log and / tmp directories in the overlay, multiple build failures caused by missing dependencies and compiler version mismatches were successfully located. The debugging efficiency is significantly higher than traditional remote login or log collection methods.

[0211] Although this invention is described in detail with software package construction as its main application scenario, its core technical solution has good versatility and can be widely applied to the following scenarios: Scenario 1: Machine Learning Experiment Management Machine learning research and development requires numerous model training experiments, each potentially using different hyperparameters, dataset versions, model architectures, or dependency library versions. Dependencies exist between experiments; for example, preprocessing tasks must be completed before training, and multiple ablation experiments can be executed in parallel.

[0212] Specifically, in terms of technology adaptation, machine learning training scripts can be used as task definitions, and deep learning frameworks (such as PyTorch and TensorFlow) and GPU drivers can be pre-installed in the virtual machine base image; the dependency parsing module can be extended to parse Python's requirements.txt or conda environment.yaml files; the local dependency repository can be extended to store resources such as pre-trained model weights, dataset snapshots, and training checkpoints; each experiment is executed in an isolated virtual machine to avoid library version conflicts and GPU memory residue between different experiments; experimental results and logs are persisted through a shared storage layer for easy subsequent analysis and reproduction.

[0213] The technological advantages applied to this scenario are that it can ensure the reproducibility of experiments, support large-scale parallel experiments to accelerate hyperparameter search, and completely isolate the experimental environment, allowing for simultaneous testing of different versions of the framework and its dependencies.

[0214] Scenario 2: Security Testing and Vulnerability Scanning Enterprise security teams need to perform automated security testing on internal software, including static code analysis, dynamic fuzzing, dependency vulnerability scanning, and penetration testing. These testing tasks may execute untrusted code or trigger malicious behavior, requiring strict environmental isolation.

[0215] Specifically, in terms of technology adaptation, the virtual machine provides complete kernel-level isolation, so even if a test triggers a kernel vulnerability, it will not affect the host machine; each test uses a brand new virtual machine, and all traces are destroyed after the test is completed to prevent malicious code persistence; network isolation configuration can prevent the virtual machine from accessing the external network to prevent data leakage; the backend abstraction layer is built to support various security scanning tools (such as SonarQube, OWASP ZAP, AFL); the scan results are aggregated through the shared storage layer to generate a unified security report.

[0216] The technological advantages applied to this scenario are: the highest level of security isolation protects the host machine and other tasks; disposable virtual machines ensure the cleanliness of the test environment; and support for parallel execution of multiple scanning tasks accelerates the security assessment cycle.

[0217] Scenario 3: Multi-platform compatibility testing scenario Before software release, compatibility testing needs to be performed on various operating systems and hardware platforms, such as different distributions of Windows, macOS, and Linux, as well as different architectures such as x86 and ARM. The testing tasks are independent of each other and can be performed in parallel on a large scale.

[0218] Specifically, in terms of technology adaptation, the base image repository stores QCOW2 images for various operating systems (Windows Server, Ubuntu, CentOS, Debian, etc.); the backend abstraction layer is expanded to support test executors for different platforms (MSTest, pytest, Jest, etc.); each test task specifies a target platform, and the virtual machine driver module automatically selects the corresponding base image; ARM architecture is supported through QEMU user-space simulation or cross-architecture virtualization; test results are summarized by platform to generate a compatibility matrix report.

[0219] The technological advantages of this application are that a single system can manage multiple heterogeneous environments; there is no need to maintain a physical test equipment cluster; the test environment is completely consistent and repeatable; and it supports the rapid addition of new test platforms.

[0220] Scenario 4: Teaching and Training Environment Universities or training institutions provide practical environments for courses such as programming and system administration. Each student needs an independent experimental environment, which needs to be pre-configured according to the course content. All student environments need to be cleaned up after the course ends.

[0221] Specifically, in terms of technology adaptation, a pre-configured base image is prepared for each course, including the necessary development tools, sample code, and documentation; personal virtual machines are dynamically created when students log in, and QCOW2 overlays are used to save storage space; students' experimental data are stored in the overlays and can be retained or destroyed; teachers can define assignments and automatic evaluation scripts, and the system automatically evaluates all student submissions in parallel; all student virtual machines and overlays are destroyed in batches after the course ends.

[0222] The technological advantages applied to this scenario are: rapid creation of a large number of isolated experimental environments; high storage efficiency; hundreds of students sharing the same base image; students cannot interfere with other people's environments; and the environment can be reset to its initial state at any time.

[0223] Scenario 5: Data Processing Pipeline Big data processing typically includes multiple stages such as data collection, cleaning, transformation, aggregation, and analysis. These stages are interdependent and need to be executed sequentially. Multiple data shards within the same stage can be processed in parallel.

[0224] Specifically, in terms of technology adaptation, the data processing task is defined as a DAG structure, relying on the parsing module to support parsing the data flow definition file; the virtual machine base image is pre-installed with data processing tools (such as Spark, Pandas, Flink); a shared storage layer is used to store intermediate data and final results; each processing step is executed in an independent virtual machine to avoid memory leaks and state residues affecting subsequent steps; a checkpoint mechanism is supported, which can recover from intermediate states in case of failure.

[0225] The technological advantages applied to this scenario are: a completely isolated execution environment ensures determinism in data processing; automatic parallelization improves processing efficiency; a failure propagation mechanism quickly identifies problem nodes; and compatibility with existing data processing tools.

[0226] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a distributed task scheduling method based on a one-time virtual machine. This method includes: receiving a task request submitted by a user; performing dependency resolution on the task request based on a pre-built dependency database to obtain a task dependency graph required to complete the task request; determining the execution order of each task to be executed according to the task dependency graph; creating an independent one-time virtual machine instance for the current task to be executed according to the execution order; sending the execution data required by the current task to be executed to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the current task to be executed in an isolated environment using the execution data; obtaining the task artifacts generated by the one-time virtual machine instance; storing the task artifacts in a shared storage layer for subsequent task invocation; and reclaiming resources from the one-time virtual machine instance after the current task to be executed is completed, continuing to execute the next task to be executed, until all tasks in the task dependency graph are completed.

[0227] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0228] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the distributed task scheduling method based on a one-time virtual machine provided by the above methods. The method includes: receiving a task request submitted by a user; performing dependency resolution on the task request based on a pre-built dependency database to obtain a task dependency graph required to complete the task request; determining the execution order of each task to be executed according to the task dependency graph; creating an independent one-time virtual machine instance for the current task to be executed according to the execution order; sending the execution data required by the current task to be executed to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the current task to be executed in an isolated environment using the execution data; obtaining the task artifacts generated by the one-time virtual machine instance; storing the task artifacts in a shared storage layer, whereby the task artifacts are used for subsequent task invocation; and reclaiming resources from the one-time virtual machine instance after the current task to be executed is completed, and continuing to execute the next task to be executed until all tasks in the task dependency graph are completed.

[0229] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the distributed task scheduling method based on a one-time virtual machine provided by the above methods. The method includes: receiving a task request submitted by a user; performing dependency resolution on the task request based on a pre-built dependency database to obtain a task dependency graph required to complete the task request; determining the execution order of each task to be executed according to the task dependency graph; creating an independent one-time virtual machine instance for the current task to be executed according to the execution order; sending the execution data required by the current task to be executed to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the current task to be executed in an isolated environment using the execution data; obtaining the task artifacts generated by the one-time virtual machine instance; storing the task artifacts in a shared storage layer, whereby the task artifacts are used for subsequent task invocation; and reclaiming resources from the one-time virtual machine instance after the current task to be executed is completed, and continuing to execute the next task to be executed, until all tasks in the task dependency graph are completed.

[0230] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / uuid> < / uuid> < / uuid>

Claims

1. A distributed task scheduling method based on a one-time virtual machine, characterized in that, include: Receive a task request submitted by a user, and based on a pre-built dependency database, perform dependency parsing on the task request to obtain the task dependency graph required to complete the task request; The execution order of each task to be executed is determined according to the task dependency graph, and an independent one-time virtual machine instance is created for the current task to be executed according to the execution order. Send the execution data required for the currently pending task to the one-time virtual machine instance, instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data; Obtain the task artifacts generated by the one-time virtual machine instance, store the task artifacts in the shared storage layer, and use the task artifacts for subsequent tasks to call; After the current task to be executed is completed, the resources of the one-time virtual machine instance are reclaimed, and the next task to be executed is continued until all tasks in the task dependency graph are completed.

2. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, The step of creating an independent, one-time virtual machine instance for the currently pending task according to the execution order includes: Based on a pre-made read-only base image containing the operating system and runtime environment, an incremental overlay image is generated for the currently pending task. The state information of the incremental overlay image is cleaned up, and a task execution proxy service is injected into the cleaned incremental overlay image to obtain the processed incremental overlay image. Based on the preset hardware configuration template, the read-only base image, and the processed incremental overlay image, a virtual machine configuration file is generated. The virtual machine configuration file is used to create and quickly start the one-time virtual machine instance, thereby skipping the redundant steps of querying operating system information through the system installation tool. After the one-time virtual machine instance starts up, the task execution agent service is polled through its health check interface to wait for it to be ready in order to establish a communication connection with the one-time virtual machine instance.

3. The distributed task scheduling method based on a one-time virtual machine according to claim 2, characterized in that, The resource reclamation of the one-time virtual machine instance includes: Send a forced shutdown command to the one-time virtual machine instance; Delete the definition information of the one-time virtual machine instance and the files corresponding to the incremental overlay image.

4. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, The steps for constructing the dependency database include: Offline scan of the metadata files of all candidate task objects, and extract the object identifier, dependency list, and mapping relationship between package group and sub-package of the candidate task object from the metadata files; Based on the object identifiers and dependency lists of each candidate task object, an initial dependency graph is constructed, and a depth-first traversal is performed on the initial dependency graph. During the traversal, visited nodes are detected to identify circular dependencies. If a circular dependency is detected, the corresponding circular edge is silently disconnected to obtain an acyclic task dependency graph. Based on the extracted mapping relationship between the package group and the sub-package, a forward mapping from the package base to the sub-package and a reverse mapping from the sub-package to the package base are established. The acyclic task dependency graph, along with the established forward and reverse mappings, are serialized and stored to obtain the dependency database.

5. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, Sending the execution data required for the currently pending task to the one-time virtual machine instance includes: The execution data is packaged into a binary stream in archive format; The binary stream is transmitted to the task execution agent service in the one-time virtual machine instance via a network transmission protocol. The task execution agent service is used to decompress the binary stream to a temporary directory and move the decompressed content in the temporary directory to the target execution path through an atomic renaming operation.

6. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, The instruction to the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data includes: The one-time virtual machine instance is controlled to access the local dependency repository in the shared storage layer through the file system protocol. The local dependency repository stores the task artifacts generated by the previous task and the dependency resources of the current task to be executed. By polymorphically calling the standard interface corresponding to the currently pending task, a task execution instruction is issued to the one-time virtual machine instance. The task execution instruction is used to trigger the one-time virtual machine instance to perform closed computation in an isolated environment where external network connections are prohibited, using the task artifacts and dependency resources in the local dependency repository and the execution data. The standard interface is provided through a preset build backend abstraction layer, which is statically configured with a standard interface that is uniformly adapted to different build systems.

7. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, Following the step of instructing the one-time virtual machine instance to execute the currently pending task in an isolated environment using the execution data, the method further includes: Receive the task execution logs transmitted back in real time by the one-time virtual machine instance via a streaming protocol; The task execution log is parsed to update the running status of the currently pending task in real time based on the parsing results.

8. The distributed task scheduling method based on a one-time virtual machine according to claim 1, characterized in that, The step of obtaining the task artifacts generated by the one-time virtual machine instance and storing the task artifacts in the shared storage layer includes: A request to download the task artifacts is sent to the one-time virtual machine instance. The one-time virtual machine instance is used to verify whether the source path specified in the request is within a preset security path whitelist. If it is, the generated task artifacts are packaged and sent back. The system receives the task artifacts returned by the one-time virtual machine instance, writes the task artifacts into the local dependency repository in the shared storage layer, and calls an update command through a global lock mechanism to update the package database maintained in the local dependency repository.

9. The distributed task scheduling method based on a one-time virtual machine according to any one of claims 1 to 8, characterized in that, After the step of determining the execution order of each task to be executed based on the task dependency graph, the method further includes: The task dependency graph is divided into multiple execution levels using a topological sorting algorithm, where tasks within the same execution level are independent of each other. Accordingly, the step of creating an independent, one-time virtual machine instance for the currently pending task according to the execution order includes: According to the order of the execution levels, multiple one-time virtual machine instances are created concurrently for multiple tasks to be executed in the current execution level.

10. The distributed task scheduling method based on a one-time virtual machine according to claim 9, characterized in that, The process of dividing the task dependency graph into multiple execution levels using a topological sorting algorithm includes: Calculate the in-degree of each task node in the task dependency graph, and place the task node with an in-degree of zero into the first execution level. The in-degree represents the number of direct predecessor tasks that the task node depends on. After processing the task nodes in the current execution level, update the in-degree of subsequent task nodes, and put the task nodes with an in-degree of zero after the update into the next execution level; Repeat the steps of updating the in-degree of subsequent task nodes and placing task nodes with an in-degree of zero after the update into the next execution level, until all task nodes in the task dependency graph are placed into the corresponding execution level.

11. The distributed task scheduling method based on a one-time virtual machine according to any one of claims 1 to 8, characterized in that, Also includes: Monitor the execution status of each pending task; If the execution status of any pending task is detected as failed, all downstream tasks that depend on any pending task are retrieved according to the task dependency graph. Mark each downstream task as blocked to cancel the scheduled execution of each downstream task.

12. The distributed task scheduling method based on a one-time virtual machine according to any one of claims 1 to 8, characterized in that, Also includes: When creating each one-time virtual machine instance, a corresponding child cancellation signal is derived from the root-level cancellation signal controller, which is created during system startup. Upon receiving an external interrupt command or a system abnormal exit signal, the root-level cancellation signal controller is activated to propagate the cancellation signal to all child-level cancellation signals. In response to the child-level cancellation signal, concurrent resource reclamation operations are performed on all one-time virtual machine instances.

13. A distributed task scheduling system based on a one-time virtual machine, characterized in that, include: An orchestration controller is configured to run on a host machine for executing the distributed task scheduling method based on a one-off virtual machine as described in any one of claims 1 to 12; A one-time virtual machine cluster consists of multiple temporarily created virtual machine instances, each of which is used to receive instructions from the orchestration controller and execute tasks. A shared storage layer, including a base image repository, an overlay image directory, and a local dependency repository, is used to connect the orchestration controller and the one-off virtual machine cluster.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the distributed task scheduling method based on a one-time virtual machine as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed task scheduling method based on a one-time virtual machine as described in any one of claims 1 to 12.

Images