An application-driven job scheduling method and system based on supercomputing internet
Patent Information
- Application Number
- CN202610735115.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-26
- Publication Date
- 2026-08-18
AI Technical Summary
[0003]现有技术中的元调度器或网格调度虽然尝试在多个集群间进行任务分发，但往往面临侵入性过强的问题；这类系统通常要求在超算中心内部安装特定的监控组件或修改调度器核心代码，这直接威胁到了超算中心生产环境的安全性与稳定性
1、通过“全局调度中心-边缘代理节点”的拓扑结构，在完全不侵入超算中心内部系统的前提下，实现异构调度协议的标准化封装；同时基于边缘代理节点的非侵入式协议转换机制，设计了一种在超算中心 DMZ 区部署边缘代理节点的方法，该方法通过标准SSH/REST接口封装底层 Slurm/LSF 指令，不触动超算核心内核，实现了调度指令的标准化映射；由于不依赖超算中心修改调度系统代码，任何具备SSH访问权限的超算中心均可快速加入调度中心，边缘代理节点的轻量化设计大大降低了接入成本；实现了高扩展性与零侵入性。
Smart Images
Figure CN122593940A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to an application-driven job scheduling method and system based on a supercomputing internet. Background Technology
[0002] In the current field of high-performance computing, the utilization model of computing resources is undergoing a strategic transformation from isolated "supercomputing centers" to an integrated "supercomputing internet." Traditional supercomputing service models primarily rely on the independent operation of individual computing centers, each with its own resource management and job scheduling systems, such as Slurm, LSF, and PBS. This discrete distribution pattern leads to a severe "computing power island" phenomenon. When performing large-scale scientific computing or engineering simulations, users must first apply for accounts at each center separately, manually evaluate the hardware architecture, software stack, queuing times, and billing standards of different centers, and then submit jobs through complex SSH commands and scripts.
[0003] While existing meta-schedulers or grid schedulers attempt to distribute tasks across multiple clusters, they often face the problem of being too invasive. These systems typically require the installation of specific monitoring components or modification of the scheduler's core code within the supercomputing center, which directly threatens the security and stability of the supercomputing center's production environment. Summary of the Invention
[0004] The main purpose of the invention is to overcome the shortcomings of the existing technology and provide an application-driven job scheduling method and system based on supercomputing internet, so as to solve the problem of excessive intrusion of meta-schedulers or grid scheduling in the existing technology.
[0005] To achieve the above objectives, this invention proposes an application-driven job scheduling method and system based on supercomputing internet.
[0006] In a first aspect, the present invention provides an application-driven job scheduling method based on a supercomputing internet, comprising: S101: Users register accounts through the global scheduling center; users log in and submit scheduling job requests through the account management module; S102: The scheduling instructions for scheduling jobs are standardized and mapped through a topology consisting of a global scheduling center and edge agent nodes deployed in various supercomputing centers. S103: Based on the user's job requirements and identity information, a preference vector is set through the global scheduling center; the user submits the scheduling job to the target supercomputing center through the comprehensive evaluation and decision module; S104: Dynamically monitor and adjust the execution position and execution order of scheduled jobs through the job release module; Among them, the execution of S102-S104 includes S201: S201: Manage the entire lifecycle of scheduled jobs through the state machine synchronization module.
[0007] A further improvement of this invention is that the account management module includes a mapping module; the mapping module includes an identity relay engine, a ternary mapping matrix, a task packaging proxy, and a virtual root directory redirection technology module; the identity relay engine solves the account binding problem in the supercomputing internet scenario through the ternary mapping matrix; The ternary mapping matrix includes a global identity identifier, a local physical account identifier, and a virtual space identifier; the ternary mapping matrix solves the account binding problem. The global identity identifier is represented as a unified user ID at the scheduling platform level; the local physical account is represented as the original account managed internally by each supercomputing center; and the virtual space identifier is represented as a unique hash label generated by the combination of each global identity identifier and local physical account identifier.
[0008] A further improvement of the present invention is that the identity relay engine realizes account switching through a task packaging proxy; the task packaging proxy includes identity redirection and dynamic credential injection; The identity redirection includes: when the global identity identifier initiates a scheduling job request, the identity relay engine retrieves the ternary mapping matrix to obtain the local physical account and the associated virtual space identifier of the target supercomputing center; The dynamic credential injection includes: the identity relay engine calling the pre-stored authorization credentials of the local physical quantity account identifier.
[0009] A further improvement of the present invention is that the virtual root directory redirection technology module realizes data isolation when multiple global identity identifiers share a single local physical account identifier; The virtual root directory redirection technology module includes a virtual root directory, a sandboxed environment, and permission writeback; The virtual root directory includes: creating an independent subdirectory for each of the global identity identifiers within the storage space of the local physical account; The sandboxed environment includes: before the script for the scheduled job is submitted, the identity relay engine automatically inserts an environment initialization command; The permission write-back includes: the output data after the scheduling job is completed is uniformly transmitted back or persisted by the identity relay engine according to the virtual space identifier.
[0010] A further improvement of the present invention is that the comprehensive evaluation decision module includes evaluation factors and a scoring function; the evaluation factors include application adaptation factors, resource satisfaction and queuing prediction factors, data gravity and migration cost factors, computing efficiency and running speed factors, billing economy factors, and historical success rate and stability factors. The application compatibility factor represents the degree of support the supercomputing center provides for application software; the resource satisfaction and queuing prediction factors are used to predict the queuing time of scheduled jobs; the data gravity and migration cost factors are used to solve the scheduling of large-scale datasets; the computing performance and running speed factors ensure that computationally intensive tasks are preferentially allocated to supercomputing centers with more suitable architectures, higher clock speeds, or stronger acceleration capabilities; the billing economic factor evaluates the core time cost consumed by scheduled jobs; and the historical success rate and stability factors ensure the robustness of scheduling. The scoring function determines the target supercomputing center to which the scheduling job should be submitted.
[0011] A further improvement of the present invention is that the comprehensive evaluation and decision-making module further includes a data-aware scheduling module; the data-aware scheduling module selects the optimal decision for data processing based on the dataset size of the scheduling job. The data-aware scheduling includes computation local priority and computation and transmission overlap; The local priority calculation includes: the target supercomputing center already has a mirror image of the scheduling job dataset; the scheduling job will preferentially select the target supercomputing center; The overlap of computation and transmission includes: in a sharded computation scenario, the edge proxy node submits a scheduling job immediately after the first batch of data arrives, and subsequent data is transmitted in parallel during the computation process.
[0012] A further improvement of the present invention is that the job release module includes a global secondary queue, a temporary storage stage, dynamic reordering logic, a precise distribution strategy, and a preemption and withdrawal module; The job temporary storage phase includes: user-submitted scheduling jobs entering the global secondary queue of the global scheduling center; the global scheduling center determining the optimal release window for scheduling jobs based on the real-time resource level and queuing prediction of the supercomputing center; The dynamic reordering logic includes: the global scheduling center dynamically adjusts the order of scheduling jobs that have not yet been issued within the global secondary queue according to rules such as "emergency task insertion" and "fair sharing attenuation coefficient"; The precise delivery strategy includes: the edge agent node will push the "scheduling slot" information of the local scheduling system to the global scheduling center in real time; The preemption and withdrawal module includes: scheduling jobs waiting in the queue of the global secondary queue; when a higher-priority scheduling job arrives at the queue of the global secondary queue, the global scheduling center can adjust the execution order of the scheduling jobs.
[0013] Secondly, the present invention provides an application-driven job scheduling system based on a supercomputing internet, the system comprising an access layer, a decision layer, and an execution agent layer; The access layer is used to collect location information of user application requirements and scheduling job data; The decision-making layer reduces the average latency cost of cross-domain fiber optic communication; the decision-making layer includes a global scheduling center; the global scheduling center is used for major functions such as global resource discovery, multi-objective scheduling decision-making, global secondary queue maintenance, and billing auditing. The execution proxy layer interacts with the internal network of the supercomputing center through lightweight edge proxy nodes; the execution proxy layer includes edge proxy nodes; the edge proxy nodes are used for main functions such as protocol conversion, local resource polling, job lifecycle monitoring, and data transfer triggering; The unified scheduling protocol runs through the access layer, the decision layer, and the execution agent layer. The unified scheduling protocol is used to standardize key functions such as job description language, state synchronization primitives, and authentication token transmission.
[0014] Compared with the prior art, the present invention has at least the following beneficial effects: 1. By adopting a topology of "global scheduling center - edge agent node", a standardized encapsulation of heterogeneous scheduling protocols is achieved without any intrusion into the internal system of the supercomputing center. Simultaneously, based on the non-intrusive protocol conversion mechanism of the edge agent node, a method for deploying edge agent nodes in the DMZ zone of the supercomputing center is designed. This method encapsulates the underlying Slurm / LSF instructions through standard SSH / REST interfaces, without touching the supercomputing core kernel, thus achieving standardized mapping of scheduling instructions. Since it does not rely on the supercomputing center to modify the scheduling system code, any supercomputing center with SSH access can quickly join the scheduling center. The lightweight design of the edge agent node greatly reduces access costs, achieving high scalability and zero intrusion.
[0015] 2. The comprehensive evaluation and decision-making module based on multi-objective optimization designs a comprehensive scoring model and calculation method that integrates application adaptability, dynamic queuing prediction, data gravity migration cost, billing economy, and historical success rate. It can achieve dynamic trade-offs of multiple objectives based on user preference vectors. Compared with manual selection of the center by the user, this module can perform calculations based on real-time telemetry data from the entire network. The selected execution center is often at the Pareto optimal boundary among "time-cost-success rate", which significantly improves resource utilization efficiency and achieves global optimality of scheduling decisions.
[0016] 3. The job release module adopts a global two-level queue job timing control technology and designs a virtual buffer and reordering algorithm built at the front end of the local scheduling system. This technology solves the problem that once a scheduled job is enqueued, it cannot be rescheduled across centers through "controlled release". The global two-level queue mechanism makes the scheduling job more flexible. When a sudden congestion is found in the queue of supercomputing center A, the scheduled job can be quickly switched to supercomputing center B before enqueuing (or even in the early stage after enqueuing), which greatly shortens the average turnaround time and realizes the flexibility of job queuing.
[0017] 4. Simultaneously, the mapping module designs a cross-domain identity relay method through virtual identity mapping and application layer isolation logic, supporting dynamic binding of the upper-layer main account and multiple lower-layer physical accounts, and includes a software-defined data isolation technology that automatically generates isolation directories and permission templates through edge proxy nodes; the mapping module of the account management module solves the compliance issues of cross-center settlement and multi-level management, enabling users to maintain a single entry point experience while securely and controllably mobilizing network resources and achieving smooth support for complex identity scenarios.
[0018] 5. A data-aware scheduling strategy was designed to dynamically adjust scheduling based on data scale. The weighting and triggering mechanism of the cross-domain data synchronization engine realizes the optimal decision logic of "computation moving to data" under large-scale scientific computing; through accurate modeling of data gravity, the system can effectively avoid unnecessary cross-center massive data flow, alleviate the bandwidth pressure of the national backbone network, and save a lot of cloud storage and transmission costs; thus achieving the optimization of data transfer costs.
[0019] In summary, the method and system provided by this invention have high scalability and zero intrusion compared to existing technologies. They use a data-aware scheduling strategy to accurately control the scale of scheduling job data, avoiding the problems of long overall scheduling job cycles and poor economy when processing large-scale scheduling job data. At the same time, it supports dynamic binding of upper-level master accounts to multiple lower-level physical accounts, and also allows multiple upper-level master accounts to correspond to multiple lower-level physical accounts, as well as multiple upper-level master accounts to correspond to one lower-level physical account. Data isolation with multiple upper-level master accounts corresponding to one lower-level physical account makes the account management mechanism more flexible and meets the needs of various complex use cases. Attached Figure Description
[0020] Figure 1 This is a flowchart of an application-driven job scheduling method based on a supercomputing internet according to the present invention.
[0021] Figure 2 This is a diagram of the architecture components of an application-driven job scheduling system based on a supercomputing internet according to the present invention.
[0022] Figure 3 This is a schematic diagram of the system structure of an application-driven job scheduling system based on supercomputing internet according to the present invention. Detailed Implementation
[0023] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.
[0024] Example 1
[0025] In the current field of high-performance computing, the utilization model of computing resources is undergoing a strategic transformation from isolated "supercomputing centers" to an integrated "supercomputing internet." Traditional supercomputing service models primarily rely on the independent operation of individual computing centers, each with its own resource management and job scheduling systems, such as Slurm, LSF, and PBS. This discrete distribution pattern leads to a severe "computing power island" phenomenon. When performing large-scale scientific computing or engineering simulations, users must first apply for accounts at each center separately, manually evaluate the hardware architecture, software stack, queuing times, and billing standards of different centers, and then submit jobs through complex SSH commands and scripts.
[0026] While existing meta-schedulers or grid schedulers attempt to distribute tasks across multiple clusters, they often suffer from excessive invasiveness. These systems typically require the installation of specific monitoring components or modification of the scheduler's core code within the supercomputing center, directly threatening the security and stability of the supercomputing center's production environment.
[0027] Furthermore, existing cross-center scheduling schemes exhibit a significant lack of awareness of "data gravity" when processing large-scale datasets. When the datasets required for a job reach TB or even PB levels, the selection of computing resources without considering the time and bandwidth costs of data migration will result in a lengthy overall job cycle and extremely poor economic efficiency.
[0028] In a multi-center federated environment, the rigidity of account management mechanisms is another major pain point restricting resource sharing. Currently, most systems only support one-to-one local account mapping, which cannot meet the needs of complex scientific research collaboration scenarios. For example, a research consortium composed of multiple upper-level institutions may need to share multiple computing accounts, or a top-level user may need to dynamically select different lower-level identities in real time based on quotas to submit jobs. Existing federated identity authentication (AAI) often relies on complex CA certificate systems, which are difficult to smoothly integrate with the traditional file system permission model based on Linux UID / GID within supercomputing centers.
[0029] To address the shortcomings of the existing technologies, the present invention aims to provide a non-intrusive hierarchical job scheduling model based on supercomputing internet.
[0030] like Figures 1-3 As shown Firstly, this embodiment 1 provides an application-driven job scheduling method based on a supercomputing internet, the method comprising: S101: Users register accounts through the global scheduling center; users log in and submit scheduling job requests through the account management module; S102: The scheduling instructions for scheduling jobs are standardized and mapped through a topology consisting of a global scheduling center and edge agent nodes deployed in various supercomputing centers. S103: Based on the user's job requirements and identity information, a preference vector is set through the global scheduling center; the user submits the scheduling job to the target supercomputing center through the comprehensive evaluation and decision module; S104: Dynamically monitor and adjust the execution position and execution order of scheduled jobs through the job release module; Among them, the execution of S102-S104 includes S201: S201: Manage the entire lifecycle of scheduled jobs through the state machine synchronization module.
[0031] When executing step S101, the account management module includes a mapping module; the mapping module includes an identity relay engine, a ternary mapping matrix, a task packaging proxy, and a virtual root directory redirection technology module; the identity relay engine solves the account binding problem in the supercomputing internet scenario through the ternary mapping matrix; The ternary mapping matrix includes a global identity identifier, a local physical account identifier, and a virtual space identifier; the ternary mapping matrix solves the account binding problem. The global identity identifier is represented by a unified user ID at the scheduling platform level; the local physical account is represented by the original account managed internally by each supercomputing center; and the virtual space identifier is represented by a unique hash label generated by the combination of each global identity identifier and local physical account identifier.
[0032] This invention constructs an "identity relay engine" that solves the account binding problem in the supercomputing internet scenario by maintaining a ternary mapping matrix of "global identity identifier - local physical account identifier - virtual space identifier" without changing the underlying authentication mechanism of the supercomputing center.
[0033] The mapping module pre-defines three logical entities: (1) Global Identity Identifier (hereinafter referred to as GID): A unified user ID at the scheduling platform level.
[0034] (2) Local Physical Account Identifier (hereinafter referred to as LAID): The original account managed internally by each supercomputing center.
[0035] (3) Virtual Space Identifier (hereinafter referred to as VID): A unique hash label generated for each (GID, LAID) combination, used to identify the user's private data range under this physical account.
[0036] The mapping relationship of the ternary mapping matrix satisfies the function f(GID) n Site_ID n )→{LAID n VID n}, where Site_ID n This represents the supercomputing center ID.
[0037] The above mapping relationship supports: (1) 1:N mapping: One GID corresponds to multiple LAIDs under Site_ID in the mapping table.
[0038] (2) M:1 mapping: Multiple GIDs point to the same LAID under the same Site_ID in the mapping table, but are assigned different VIDs.
[0039] (3) M:N mapping: Allows users to dynamically select specific items in the mapping table through parameterized instructions when submitting a job.
[0040] When a user wants to switch accounts, the identity relay engine enables the account switching through a task packaging proxy; the identity relay engine enables account switching through a task packaging proxy; the task packaging proxy includes identity redirection and dynamic credential injection; Identity redirection includes: when a global identity initiates a scheduling job request, the identity relay engine retrieves the ternary mapping matrix to obtain the local physical account and associated virtual space identifier of the target supercomputing center; Dynamic credential injection includes: the identity relay engine calling the pre-stored local physical quantity account identifier's authorization credentials.
[0041] Therefore, its identity relay engine achieves account switching through a "task packaging proxy" method: (1) Identity Redirection: When GID initiates a scheduling job request, the identity relay engine retrieves the ternary mapping matrix to obtain the LAID and associated VID of the target supercomputing center.
[0042] (2) Dynamic credential injection: The identity relay engine calls the pre-stored LAID authorization credentials (such as SSH Key or APIToken) to complete the underlying authentication interface with the supercomputing center on behalf of the user.
[0043] In the M:1 mapping mode, data isolation exists. Thus, the virtual root directory redirection technology module achieves data isolation when multiple global identity identifiers share a single local physical account identifier. The virtual root directory redirection technology module enables data isolation when multiple global identities share a single local physical account identity. The virtual root directory redirection technology module includes virtual root directory, sandboxed environment, and permission writeback; The virtual root directory includes: creating a separate subdirectory for each global identity within the storage space of the local physical account; The sandboxed environment includes: before the script for the scheduled job is submitted, the identity relay engine automatically inserts environment initialization instructions; Permission write-back includes: the output data after the scheduled job is completed is uniformly returned or persisted by the identity relay engine according to the virtual space identifier.
[0044] To address the scenario where multiple global users share a single underlying supercomputing center account, this invention employs a virtual root directory redirection technology module to achieve data isolation: (1) Virtual root directory: Within the storage space of LAID, the system automatically creates an independent subdirectory for each GID, for example, the path format is / home / LAID / v_space / {VID} / .
[0045] (2) Sandboxed environment: Before the job script is submitted, the identity relay engine automatically inserts environment initialization instructions: ① Lock the job's runtime environment to the corresponding VID directory by modifying environment variables (such as HOME, TMPDIR).
[0046] ② Utilize the Linux system's File Access Control List (FACL) to dynamically grant the job process read and write permissions to a specific VID directory, and perform masking operations on other VID directories.
[0047] (3) Permission write-back: The output data after the job is completed is uniformly returned or persisted by the identity relay engine according to the VID, ensuring that the data of different GIDs belong to the same LAID user at the physical layer, but are completely isolated at the logical layer.
[0048] Meanwhile, the mapping module designs a cross-domain identity relay method through virtual identity mapping and application layer isolation logic, which supports dynamic binding of the upper-layer main account and multiple physical accounts at the bottom layer, and includes a software-defined data isolation technology that automatically generates isolation directories and permission templates through edge proxy nodes. The mapping module in the account management module solves the compliance issues of cross-center settlement and multi-level management, enabling users to securely and controllably mobilize resources across the entire network while maintaining a single entry point experience, and achieving smooth support for complex identity scenarios.
[0049] During the execution of step S102, the global scheduling center is located in the computer room of the regional center, and each edge agent node is located in the DMZ area of each supercomputing center. In this system, the edge proxy node (hereinafter referred to as AN) acts as a "protocol translator." AN does not directly modify the database of the supercomputing center's internal scheduler (such as Slurmctld). Instead, it interacts with the logged-in node by simulating user behavior through pre-defined high-privilege accounts. AN utilizes command-line tools (such as sbatch, bjobs, sinfo) or REST API interfaces provided by the local scheduling system to obtain real-time node and job status. This design ensures low coupling between the scheduling model and the underlying supercomputing system; even if the scheduling center fails, the normal operation of tasks within the supercomputing center will not be affected. Simultaneously, AN employs a primary-backup redundancy configuration, using a combination of high-frequency polling and event triggering to map the state machine transitions of the local scheduler in real time.
[0050] By employing a "global scheduling center - edge agent node" topology, standardized encapsulation of heterogeneous scheduling protocols is achieved without any intrusion into the supercomputing center's internal system. Simultaneously, based on the non-intrusive protocol conversion mechanism of the edge agent nodes, a method for deploying AN in the supercomputing center's DMZ is designed. This method encapsulates underlying Slurm / LSF instructions through standard SSH / REST interfaces, without affecting the supercomputing core kernel, thus achieving standardized mapping of scheduling instructions. Since it does not rely on the supercomputing center to modify the scheduling system code, any supercomputing center with SSH access can quickly join the scheduling center. AN's lightweight design significantly reduces access costs, achieving high scalability and zero intrusion.
[0051] When executing step S103, the comprehensive evaluation decision module includes evaluation factors and scoring functions; the evaluation factors include application adaptation factors, resource satisfaction and queuing prediction factors, data gravity and migration cost factors, computing efficiency and running speed factors, billing economy factors, and historical success rate and stability factors. Application suitability factor indicates the degree of support a supercomputing center provides for application software; resource sufficiency and queuing prediction factors are used to predict the queuing time of scheduled jobs; data gravity and migration cost factors are used to address the scheduling of large-scale datasets; computing performance and running speed factors ensure that computationally intensive tasks are prioritized for allocation to supercomputing centers with more suitable architectures, higher clock speeds, or stronger acceleration capabilities; billing economics factors assess the core time cost consumed by scheduled jobs; historical success rate and stability factors ensure the robustness of scheduling. The scoring function determines the target supercomputing center to which the scheduled job should be submitted.
[0052] The core of this invention lies in a comprehensive evaluation and decision-making module based on multi-objective optimization (MOO). This model transforms the complex scheduling problem into a candidate center scoring and ranking problem by quantifying various evaluation factors. Let the job to be scheduled be J, and the set of candidate supercomputing centers be S = {s1, s2, ..., s}. n For any center s i ∈S, define the comprehensive evaluation score function F(s) i The evaluation factors are as follows: (1) Application of adaptation factor ( ) This factor characterizes the supercomputing center Homework Determine the level of support from the required application software. Consider version number, compilation environment, and library dependencies.
[0053] in The function performs hard filtering based on the software list reported by AN. This reflects the application's historical compatibility score on the corresponding platform. If the application is completely unusable, then... This allows for a veto.
[0054] (2) Resource satisfaction and queuing predictor ( ) Evaluate Number of currently available nodes With job requirements The relationship between the two, combined with historical data, is used to predict queuing times. .
[0055] in This is the logistic regression activation function, used to smooth resource boundary pressure. Queue sensitivity coefficient; The latest delivery time expected by the user.
[0056] (3) Data gravity and migration cost factors ( ) This is the core factor in solving large-scale dataset scheduling. Let the job dataset size be... The current location and center of the dataset The real-time link bandwidth between them is The decay coefficients for the time dimension and the cost dimension are respectively and .
[0057] here The theoretical time cost of data transmission was calculated. Cost per unit of data transfer. If the dataset already exists... Internal storage, then Take the maximum value.
[0058] (4) Computational efficiency and operating speed factor ( ) according to CPU / GPU computational efficiency (FLOPS) predicts actual job runtime .
[0059] This factor ensures that compute-intensive tasks are preferentially assigned to centers with better architecture, higher clock speeds, or stronger acceleration capabilities.
[0060] (5) Billing economic factors ( ) Assessment of the estimated nuclear time cost of the operation .
[0061] Pareto frontier normalization is used to convert costs into revenue values between 0 and 1; the lower the cost, the higher the score.
[0062] (6) Historical success rate and stability factor ( ) Statistical data on job completion status based on long-term monitoring by AN.
[0063] This factor incorporates weights for network jitter and system failure rate, ensuring the robustness of scheduling.
[0064] The global scheduling center uses the user-defined preference vector. Calculate the final weighted score: The scheduling model will select The center with the highest value is submitted as the target site. This modeling approach supports different user priorities: if a user prioritizes efficiency, then increase the value... and If cost is a priority, then increase... .
[0065] The comprehensive evaluation and decision-making module based on multi-objective optimization designs a comprehensive scoring model and calculation method that integrates application adaptability, dynamic queuing prediction, data gravity migration cost, billing economy, and historical success rate. It can achieve dynamic trade-offs among multiple objectives based on user preference vectors. Compared to manual selection by users, this module can perform calculations based on real-time telemetry data from the entire network. The selected execution center is often at the Pareto optimal boundary among "time-cost-success rate", which significantly improves resource utilization efficiency and achieves global optimality in scheduling decisions.
[0066] When scheduling jobs to process large-scale datasets, the comprehensive evaluation and decision-making module also includes a data-aware scheduling module; the data-aware scheduling module selects the optimal decision for data processing based on the size of the dataset of the scheduling job. Data-aware scheduling includes computation-local priority and computation-transmission overlap; The local preference calculation includes: the target supercomputing center already has a mirror of the dataset for the scheduled job; the scheduled job will preferentially select the target supercomputing center; Overlapping computation and transmission includes: in a sharded computation scenario, the edge agent node submits a scheduling job immediately after the first batch of data arrives, and subsequent data is transmitted in parallel during the computation process.
[0067] For large-scale datasets, the model introduces "data gravity" weights. If Then the data migration cost factor will be included in the evaluation function. weight Increased to above 0.7.
[0068] Localization priority in computation: if the target center A mirror of the dataset already exists. The model will ignore the negative impact of the queuing factor and prioritize this center to avoid the cost and delay caused by transmitting massive amounts of data across the backbone network.
[0069] Overlapping computation and transmission: In MapReduce or sharded computing scenarios, AN starts the "pipeline transmission" mode, that is, the job is submitted immediately after the first batch of data arrives, and subsequent data is transmitted in parallel during the computation process, realizing the overlap of computation and communication.
[0070] By employing a data-aware scheduling strategy, a method for dynamically adjusting scheduling based on data scale was designed. The weighting and triggering of the cross-domain data synchronization engine linkage mechanism realizes the optimal decision logic of "moving computation to data" under large-scale scientific computing; By accurately modeling the gravity of data, the system can effectively avoid unnecessary cross-center massive data flows, alleviate the bandwidth pressure on the national backbone network, and save a lot of cloud storage and transmission costs; thus achieving the optimization of data transfer costs.
[0071] When executing step S104, the job release module includes a global secondary queue, a temporary storage stage, dynamic reordering logic, a precise distribution strategy, and a preemption and withdrawal module. The job storage phase includes: user-submitted scheduling jobs entering the global secondary queue of the global scheduling center; the global scheduling center determining the optimal release window for scheduling jobs based on the real-time resource level and queuing prediction of the supercomputing center; The dynamic reordering logic includes: the global scheduling center dynamically adjusts the order of scheduling jobs that have not yet been issued within the global secondary queue according to rules such as "emergency task insertion" and "fair sharing of attenuation coefficient"; The precise distribution strategy includes: edge agent nodes will push the "scheduling slot" information of the local scheduling system to the global scheduling center in real time; The preemption and withdrawal module includes: scheduled jobs waiting in the global secondary queue; when a higher-priority scheduled job arrives in the global secondary queue, the global scheduling center can adjust the execution order of scheduled jobs.
[0072] To address the issue that once a queue enters a task within a supercomputing center, its order cannot be flexibly adjusted, this invention introduces a global secondary queue (GSQ) for the job release module. (1) Job storage stage: The scheduling jobs submitted by users enter the GSQ of the global scheduling center; the global scheduling center calculates the best release window for the job based on the real-time resource level and queuing prediction of each supercomputing center.
[0073] (2) Dynamic reordering logic: In GSQ, the global scheduling center can dynamically adjust the order of scheduling jobs that have not yet been issued according to rules such as "emergency task insertion" and "fair-share attenuation coefficient".
[0074] (3) Precise dispatch strategy: AN will push the "scheduling slot" information of the local scheduling system to the global scheduling center in real time; when AN detects that there are about to be idle resources or the queuing environment is better than the predicted value, the global scheduling center will officially dispatch the scheduling job to AN; at this time, the scheduling job will be officially submitted by AN to the local Slurm / LSF.
[0075] (4) Preemption and withdrawal mechanism: If a higher priority task arrives while the scheduled job is waiting in GSQ, the model can adjust the order; if the scheduled job has been issued to AN but the local queuing time is abnormal (monitored by AN), the global scheduling center can order AN to cancel the scheduled job and put it back into GSQ for cross-center rescheduling.
[0076] The job release module adopts the job timing control technology of Global Secondary Queue (GSQ) and designs a virtual buffer and reordering algorithm built in the front end of the local scheduling system. This technology solves the problem that once a scheduled job is enqueued, it cannot be rescheduled across centers through "controlled release". The global two-level queue mechanism makes job scheduling more flexible; when a sudden congestion is detected in the queue of supercomputing center A, the scheduled job can be quickly switched to supercomputing center B before (or even early after) entering the queue, which greatly shortens the average turnaround time and realizes the flexibility of job queuing.
[0077] During the execution of S101-S104, the scheduling model uses a state machine synchronization module to shield the command differences between the underlying Slurm, LSF, and PBS; (1) Resource selection: Score ranking based on evaluation function.
[0078] (2) Data copy: If it is determined that data needs to be migrated, AN triggers a cross-center high-speed transmission service (such as GridFTP or parallel FASP protocol), and the scheduling center monitors the transmission progress.
[0079] (3) Job submission: AN generates a local scheduling script and executes the submission command.
[0080] (4) Status query: AN continuously polls the local queue, maps the statuses such as PEND, RUN, and DONE to standard statuses, and records the job execution details.
[0081] (5) Work intervention: ① Pause: If the local scheduler / application software supports pausing jobs, AN will perform the pause job operation according to the standard protocol.
[0082] ②Resume: Perform a resume operation on a paused job, allowing it to continue running at the nearest save point.
[0083] ③Termination: The job will be terminated and cannot be resumed.
[0084] Results collection: After the job is completed, AN automatically pulls the output file back to the specified storage node or pushes it to the user portal according to the user configuration.
[0085] In summary, the purpose of this invention is to provide a non-intrusive hierarchical job scheduling method based on a supercomputing internet. First, an "application-oriented" transparent service model is constructed, allowing users to define only the application type of their computing tasks, with the system automatically matching resources based on multiple evaluation factors. Second, through a "global scheduling center - edge agent node" topology, standardized encapsulation of heterogeneous scheduling protocols is achieved without intruding on the internal systems of the supercomputing center. Third, a global secondary queue (GSQ) is designed to achieve highly flexible priority reordering even when local queuing sequences are irreversible. Finally, a mapping module supporting many-to-many mappings is established, supplemented by software-level data isolation technology, thereby providing efficient, secure, and economical computing power scheduling services in a supercomputing internet environment.
[0086] Secondly, this embodiment 1 provides an application-driven job scheduling system based on a supercomputing internet, the system including an access layer, a decision layer and an execution agent layer; The access layer is used to collect user application requirements and location information for scheduling job data; The decision-making layer reduces the average delay cost of cross-domain fiber optic communication; the decision-making layer includes a global scheduling center; the global scheduling center is responsible for key functions such as global resource discovery, multi-objective scheduling decision-making, global secondary queue maintenance, and billing auditing. The execution agent layer interacts with the internal network of the supercomputing center through lightweight edge agent nodes; the execution agent layer includes edge agent nodes; the edge agent nodes are used for main functions such as protocol conversion, local resource polling, full lifecycle monitoring of jobs, and data transfer triggering; The unified scheduling protocol runs through the access layer, decision layer, and execution agent layer. It is used to standardize key functions such as job description language, state synchronization primitives, and authentication token transmission.
[0087] The scheduling system model proposed in this invention adopts a distributed architecture that combines physical centralization and logical edge computing; The core logic of the system is distributed across three layers: the access layer, the decision layer, and the execution agent layer. The access layer is responsible for collecting user application requirements and data location information. The decision layer (scheduling center) is located at the geographical center of the network topology to reduce the average latency cost of cross-domain fiber optic communication. The execution agent layer interacts with the supercomputing internal network through lightweight agent nodes.
[0088] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. An application-driven job scheduling method based on supercomputing Internet, characterized by, The method includes: S101: Users register accounts through the global scheduling center; users log in and submit scheduling job requests through the account management module; S102: The scheduling instructions for scheduling jobs are standardized and mapped through a topology consisting of a global scheduling center and edge agent nodes deployed in various supercomputing centers. S103: Based on the user's job requirements and identity information, a preference vector is set through the global scheduling center; the user submits the scheduling job to the target supercomputing center through the comprehensive evaluation and decision module; S104: Dynamically monitor and adjust the execution position and execution order of scheduled jobs through the job release module; Among them, the execution of S102-S104 includes S201: S201: Manage the entire lifecycle of scheduled jobs through the state machine synchronization module.
2. The application-driven job scheduling method based on supercomputing internet according to claim 1, characterized in that, The account management module includes a mapping module; the mapping module includes an identity relay engine, a ternary mapping matrix, a task packaging proxy, and a virtual root directory redirection technology module; the identity relay engine solves the account binding problem in the supercomputing internet scenario through the ternary mapping matrix; The ternary mapping matrix includes a global identity identifier, a local physical account identifier, and a virtual space identifier; the ternary mapping matrix solves the account binding problem. The global identity identifier is represented as a unified user ID at the scheduling platform level; the local physical account is represented as the original account managed internally by each supercomputing center; and the virtual space identifier is represented as a unique hash label generated by the combination of each global identity identifier and local physical account identifier.
3. The application-driven job scheduling method based on supercomputing internet according to claim 2, characterized in that, The identity relay engine achieves account switching through task packaging agents; The task packaging agent includes identity redirection and dynamic credential injection; The identity redirection includes: when the global identity identifier initiates a scheduling job request, the identity relay engine retrieves the ternary mapping matrix to obtain the local physical account and the associated virtual space identifier of the target supercomputing center; The dynamic credential injection includes: the identity relay engine calling the pre-stored authorization credentials of the local physical quantity account identifier.
4. The application-driven job scheduling method based on supercomputing internet according to claim 2, characterized in that, The virtual root directory redirection technology module enables data isolation when multiple global identity identifiers share a single local physical account identifier. The virtual root directory redirection technology module includes a virtual root directory, a sandboxed environment, and permission writeback; The virtual root directory includes: creating an independent subdirectory for each of the global identity identifiers within the storage space of the local physical account; The sandboxed environment includes: before the script for the scheduled job is submitted, the identity relay engine automatically inserts an environment initialization command; The permission write-back includes: the output data after the scheduling job is completed is uniformly transmitted back or persisted by the identity relay engine according to the virtual space identifier.
5. The application-driven job scheduling method based on supercomputing internet according to claim 1, characterized in that, The comprehensive evaluation and decision-making module includes evaluation factors and scoring functions; the evaluation factors include application adaptation factors, resource satisfaction and queuing prediction factors, data gravity and migration cost factors, computing efficiency and running speed factors, billing economy factors, and historical success rate and stability factors. The application adaptation factor represents the degree of support the supercomputing center provides for application software; The resource satisfaction and queuing prediction factor are used to predict the queuing time of scheduled jobs. The data gravity and migration cost factors are used to solve the scheduling of large-scale datasets; the computing performance and running speed factors ensure that computationally intensive tasks are preferentially allocated to supercomputing centers with more suitable architectures, higher clock speeds, or stronger acceleration capabilities; the billing economy factor evaluates the core time cost consumed by the scheduled jobs; and the historical success rate and stability factors ensure the robustness of the scheduling. The scoring function determines the target supercomputing center to which the scheduling job should be submitted.
6. The application-driven job scheduling method based on supercomputing internet according to claim 5, characterized in that, The comprehensive evaluation and decision-making module also includes a data-aware scheduling module; the data-aware scheduling module selects the optimal decision for data processing based on the dataset size of the scheduling job. The data-aware scheduling includes computation local priority and computation and transmission overlap; The local priority calculation includes: the target supercomputing center already has a mirror image of the scheduling job dataset; the scheduling job will preferentially select the target supercomputing center; The overlap of computation and transmission includes: in a sharded computation scenario, the edge proxy node submits a scheduling job immediately after the first batch of data arrives, and subsequent data is transmitted in parallel during the computation process.
7. The application-driven job scheduling method based on supercomputing internet according to claim 1, characterized in that, The job release module includes a global secondary queue, a temporary storage stage, dynamic reordering logic, a precise distribution strategy, and a preemption and withdrawal module. The job temporary storage phase includes: user-submitted scheduling jobs entering the global secondary queue of the global scheduling center; the global scheduling center determining the optimal release window for scheduling jobs based on the real-time resource level and queuing prediction of the supercomputing center; The dynamic reordering logic includes: the global scheduling center dynamically adjusts the order of scheduling jobs that have not yet been issued within the global secondary queue according to rules such as "emergency task insertion" and "fair sharing attenuation coefficient"; The precise distribution strategy includes: the edge agent node will push the "scheduling slot" information of the local scheduling system to the global scheduling center in real time; The preemption and withdrawal module includes: scheduling jobs waiting in the queue of the global secondary queue; when a higher-priority scheduling job arrives at the queue of the global secondary queue, the global scheduling center can adjust the execution order of the scheduling jobs.
8. An application-driven job scheduling system based on a supercomputing internet, characterized in that, The system is used to execute the application-driven job scheduling method based on supercomputing internet as described in any one of claims 1 to 7, and the system includes an access layer, a decision layer, and an execution agent layer; The access layer is used to collect location information of user application requirements and scheduling job data; The decision-making layer reduces the average latency cost of cross-domain fiber optic communication; the decision-making layer includes a global scheduling center; the global scheduling center is used for major functions such as global resource discovery, multi-objective scheduling decision-making, global secondary queue maintenance, and billing auditing. The execution agent layer interacts with the internal network of the supercomputing center through lightweight edge agent nodes; the execution agent layer includes edge agent nodes; the edge agent nodes are used for main functions such as protocol conversion, local resource polling, full lifecycle monitoring of jobs, and data transfer triggering; the unified scheduling protocol runs through the access layer, the decision layer and the execution agent layer, and is used for main functions such as standardizing job description language, state synchronization primitives and authentication token transmission.