Slurm scheduling specification integration method and system

Abstract

The invention relates to the field of high-performance computing cluster resource scheduling management, and discloses a Slurm scheduling specification integration method and system, and the method comprises the following steps: 1, analyzing a heterogeneous job description file, and extracting a resource demand parameter and a dependency relationship through a regular expression rule base; 2, based on the extracted original parameters, converting the original parameters into SLURM standard parameters through a preset mapping rule; step 3, according to the converted standard parameters; 4, calling a Slurm interface command to submit a script; and step 5, monitoring the execution state of the submitted job, and triggering a re-submission process for the abnormal job with the resource overrun or dependency missing. Through a multi-level analysis architecture and a regular expression rule base, job description files in different formats are effectively compatible, the problem that analysis of a non-standardized input format by a traditional method fails is solved, unified processing of cross-platform job definition is achieved, and the manual adaptation cost is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of high-performance computing cluster resource scheduling management, and in particular to a Slurm scheduling specification integration method and system. Background Art

[0002] The Slurm scheduling specification integration method and system is a core technology for high-performance computing (HPC) and heterogeneous computing environments. It aims to convert job description files from different platforms (such as Kubernetes batch processing jobs and traditional HPC job management systems) into scripts that conform to the Slurm scheduler specification, and implements automated management of the entire process, from job submission and status monitoring to resource auditing. Its technical value lies in solving the problem of unified scheduling of multi-source heterogeneous jobs in Slurm clusters through standardized conversion and intelligent scheduling, thereby improving the utilization efficiency of large-scale computing resources.

[0003] Existing technologies primarily rely on manually written scripts or static template-based parameter replacement for Slurm jobs. These methods impose strict constraints on the format of job description files and parameter definition methods, making them difficult to adapt to platform differences in resource parameter naming conventions (e.g., "CPU_Cores" versus "vCPUs"), unit representation (e.g., "GB" versus "GiB"), and dependency declarations (explicit dependency chains versus implicit workflows). For example, resource configuration descriptions for Kubernetes jobs are typically defined in YAML format, while traditional HPC jobs often use key-value pairs or custom configuration files. Existing technologies lack a unified parsing rule base and dynamic adaptation mechanisms, resulting in extensive manual adaptation required for cross-platform job migration. Furthermore, existing solutions rely on manual intervention for exception handling (e.g., resource limit overruns, dependency failures), lack closed-loop control logic, and struggle to meet the high availability requirements of enterprise-level clusters.

[0004] Existing technologies lack the ability to parse non-standardized input formats, the parameter conversion process is susceptible to unit differences and semantic ambiguity, and dependency management logic is rigid. These issues lead to increased job submission failure rates, frequent debugging intervention by operations and maintenance personnel, and severely restrict the automated scheduling capabilities of large-scale heterogeneous computing clusters. Summary of the Invention

[0005] The purpose of the present invention is to provide a Slurm scheduling specification integration method and system, which solves the problems in the prior art of high job submission failure rate and high operation and maintenance cost caused by format differences, parameter mapping errors and lack of exception handling.

[0006] To achieve the above objectives, the present invention is implemented by the following technical solutions: The present invention is implemented by the following technical solutions: A Slurm scheduling specification integration method includes the following steps: Step 1: Parse the heterogeneous job description file and extract resource requirement parameters and dependencies using the regular expression rule library; Step 2: Based on the extracted original parameters, convert them into SLURM standard parameters through preset mapping rules, including base unit conversion of numerical parameters and keyword mapping of enumeration parameters; Step 3: Based on the converted specification parameters, use the template engine to generate a SLURM script containing resource quotas, dependency instructions, and environment variables. The dependency instructions are generated by concatenating the dependency job IDs. Step 4: Call the Slurm interface command to submit the script and parse the output log using regular expressions to obtain the job ID; Step 5: Monitor the execution status of submitted jobs and trigger the resubmission process for abnormal jobs with resource excess or missing dependencies.

[0007] Preferably, the step 1 includes: Match parameter key-value pairs using a predefined regular expression rule library. The rules include the CPU_Cores=(\d+) and Memory_Req=(\d+[GM]B) patterns, and record exception logs for unmatched parameters. Verify the integrity of required parameters and trigger parsing interruption for description files with missing job names or resource quotas. Extract dependency declarations, identify dependency type identifiers and combine them into --dependency= <type>:<job_id_list> instruction.

[0008] Preferably, the step 1 further comprises: Adopting a multi-level parser architecture: First level analysis of basic resource parameters; The second level parses the extended configuration parameter environment variables, input and output paths; The third level verifies the logical consistency of parameters; Trigger an alert for detected circular dependencies and terminate the process.

[0009] Preferably, the step 2 includes: Input dependency: the original parameter set parsed based on step 1; Numeric parameter conversion: Memory parameters are converted from GB / MB units to SLURM standard MB units; The number of GPUs is directly mapped to the --gres=gpu:N command; Enumeration parameter mapping: Priority parameter; Queue type parameter; Exception handling: For parameters that cannot match the mapping rules, the error code ERR_UNKNOWN_PARAM is triggered and the process is terminated.

[0010] Preferably, the step three includes: Template input: receives the set of standard parameters output in step 2; Dynamic Rendering: Fill the --cpus-per-task, --mem, and --gres resource parameters into the script header instructions; Concatenate the list of dependent job IDs with colon separators as follows: --dependency=afterany: <job1:job2>instruction; Convert the environment variable key-value pair into exportKEY=VALUE statement and insert it into the script execution area; Syntax check: Execute sbatch --test-only pre-submission verification on the generated script to intercept syntax errors.

[0011] Preferably, the step 4 includes: Submit the script by executing the sbatch[script_path] command through the system call; Use regular expressions to extract job IDs from command output and detect script syntax errors if matching fails. Categorize and record Slurm service exceptions and script syntax errors.

[0012] Preferably, the step five includes: For jobs with resource overruns, increase the resource quota by the preset ratio and resubmit; For jobs with missing dependencies, resolve the missing dependent job IDs and notify the upstream system; Set a maximum retry threshold. If the threshold is exceeded, the job is terminated and an alarm is generated.

[0013] Preferably, the method further comprises: Define the maximum retry threshold in the job description file; When the number of job failures exceeds the threshold, the process is automatically terminated and an error report is generated; Archives metadata for terminated jobs to a history database.

[0014] Preferably, the monitoring in step five includes: Periodically poll Slurm's squeue interface to obtain job status; Triggering a priority increase mechanism for jobs that have been in the PENDING state for a long time; Automatically capture logs and tag abnormally terminated jobs with error types.

[0015] The present invention also provides a Slurm scheduling specification integration system, comprising: Parameter parsing engine: includes regular expression matcher and parameter validator; Script generator: dynamically builds SLURM scripts, integrating resource quotas, dependency instructions, and environment variables; Job submission agent: encapsulates sbatch command calls across languages, supporting Python and Java environments; Status monitor: real-time tracking of job lifecycles, with a built-in exception handling strategy library; Audit module: records job metadata and generates multi-dimensional analysis reports on resource utilization.

[0016] In summary, the present invention includes at least one of the following beneficial technical effects: 1. The present invention uses a multi-level parsing architecture and a regular expression rule library to effectively accommodate job description files of different formats, solving the problem of traditional methods failing to parse non-standardized input formats, achieving unified processing of cross-platform job definitions, and significantly reducing manual adaptation costs.

[0017] 2. Based on preset mapping rules and a dynamic unit conversion mechanism, the present invention eliminates configuration errors caused by inconsistent resource parameter units and keyword differences, ensures that the generated Slurm instructions strictly comply with the scheduler specifications, and avoids job queuing anomalies or resource allocation conflicts caused by parameter mismatches.

[0018] 3. This invention uses a template engine and pre-verification mechanism to dynamically generate grammatically compliant Slurm scripts, combines intelligent splicing of dependencies and secure injection of environment variables, intercepts script errors at the source, reduces invalid job submissions, and improves cluster resource utilization.

[0019] 4. This invention uses an error classification system and an adaptive retry strategy to implement closed-loop processing for typical abnormal scenarios such as resource overruns and missing dependencies, reducing the frequency of manual intervention, ensuring the continuous execution of complex workflows, and improving system fault tolerance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is one of the flow charts of the method of the present invention; Figure 2 This is the second flow chart of the method of the present invention; Figure 3 This is the third flow chart of the method of the present invention; Figure 4 This is the fourth flow chart of the method of the present invention; Figure 5 This is the fifth flow chart of the method of the present invention; Figure 6 This is the sixth flow chart of the method of the present invention; Figure 7 This is the seventh flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The following is combined with Figure 1 -Attached Figure 7 , the present invention is described in further detail.

[0022] The present invention provides a Slurm scheduling specification integration method. Through a multi-level parsing architecture and a regular expression rule library, it is effectively compatible with job description files of different formats, solves the problem that traditional methods fail to parse non-standardized input formats, realizes unified processing of cross-platform job definitions, and significantly reduces manual adaptation costs.

[0023] like Figure 1 As shown, the Slurm scheduling specification integration method may include the following steps: Step 1: Parse the heterogeneous job description file and extract resource requirement parameters and dependencies using the regular expression rule library; Step 2: Based on the extracted original parameters, convert them into SLURM standard parameters through preset mapping rules, including base unit conversion of numerical parameters and keyword mapping of enumeration parameters; Step 3: Based on the converted specification parameters, use the template engine to generate a SLURM script containing resource quotas, dependency instructions, and environment variables. The dependency instructions are generated by concatenating the dependency job IDs. Step 4: Call the Slurm interface command to submit the script and parse the output log using regular expressions to obtain the job ID; Step 5: Monitor the execution status of submitted jobs and trigger the resubmission process for abnormal jobs with resource excess or missing dependencies.

[0024] In this embodiment, step 1 implements parameter extraction and verification of heterogeneous job description files through a multi-level parsing architecture, specifically including the following technical contents: The parsing process for heterogeneous job description files begins by matching parameter key-value pairs using a predefined regular expression rule library. This regular expression rule library contains matching patterns for different parameter types, such as the CPU_Cores=(\d+) pattern for extracting the number of CPU cores, the Memory_Req=(\d+[GM]B) pattern for matching memory requirements, and the Depends_On=([\w,:]+) pattern for identifying dependencies. For parameter items that do not match any rules, the system records a parsing exception log containing the parameter name and original content and continues the subsequent parsing process to avoid single points of failure.

[0025] During the parameter integrity check phase, the system checks whether the job description file contains the required parameter set. Required parameters include at least the job name, number of CPU cores, and memory requirements. If a required parameter is missing, the system triggers a parsing abort mechanism and generates an error report containing a list of missing parameters. This abort mechanism is implemented by throwing a structured exception to ensure resource release and transaction rollback.

[0026] For dependency declarations, the system further identifies dependency type identifiers, including afterany, afterok, and afternotok. The dependency type identifier and the list of dependent job IDs are concatenated to generate a Slurm-compliant instruction. Specifically, the dependent job IDs are concatenated with a colon separator and the type prefix is ​​added to form --dependency= <type>:<job_id_1> :<job_id_2> For example, the original dependency description depends_on=afterany:1001,1002 will be converted to --dependency=afterany:1001:1002.

[0027] Furthermore, the parsing process employs a multi-level parser architecture to improve processing efficiency and scalability. The first-level parser focuses on extracting basic resource parameters, including the number of CPU cores, memory requirements, and the number of GPUs. The second-level parser processes extended configuration parameters, including environment variable key-value pairs, input and output file paths, and job priority settings. The third-level parser performs parameter logical consistency verification, for example, verifying whether the ratio of memory size to the number of CPU cores meets preset constraints. The preset constraints are defined by inequalities, where is the memory size (in MB), is the number of CPU cores, and is a configurable proportional coefficient. In dependency processing, the system detects circular dependency scenarios by constructing a directed graph model. Specifically, this is achieved by traversing the graph structure using a depth-first search algorithm, using dependent job IDs as nodes and dependency relationships as directed edges. When a loop is detected, the system triggers an alarm and terminates the current parsing process, while outputting a detailed error message containing the loop path. Preferably, the regular expression rule library utilizes a dynamic loading mechanism, supporting the addition or modification of matching rules through configuration files to accommodate the expansion requirements of different job description formats. A data pipeline transmission mechanism is adopted between the levels of the multi-level parser. The output of the previous level parser is used as the input of the next level parser, and an intermediate result cache is set between the levels to support the breakpoint-resume parsing function.

[0028] In this embodiment, step 2 realizes the conversion of heterogeneous parameters to Slurm standard parameters through preset mapping rules, which specifically includes the following technical contents: The parameter conversion process takes the original parameter set parsed in step 1 as input and processes both numeric and enumerated parameters using a hierarchical conversion strategy. For numeric parameters, the system performs a baseline unit normalization operation, for example, converting memory parameters to the MB units required by the Slurm scheduler. This conversion is achieved by parsing the unit identifiers of the parameter values ​​(e.g., "GB" or "MB") and applying a linear conversion formula. Specifically, when the parameter value contains the "GB" unit, the operation is performed, where is the converted memory value and is the original value; when the parameter value contains the "MB" unit, the value is directly extracted.

[0029] The system maps the GPU count parameter to a generic Slurm resource request. Specifically, when the GPU_Count parameter is detected, a command of the form --gres=gpu:N is generated, where N is the parsed GPU count. This mapping ignores differences in GPU models and focuses solely on the number dimension to satisfy the scheduler's basic resource allocation requirements.

[0030] When processing enumeration parameters, the system implements semantic conversion through a predefined key-value mapping table. For example, the job priority parameter Priority=High is converted to the Slurm directive --qos=high, and the queue type parameter Queue=debug is mapped to --partition=debug. The mapping table is stored in an extensible JSON structure and supports dynamic loading and updating. Its entries contain the correspondence between the original parameter name, the target directive name, and the enumeration value. For enumeration values ​​not defined in the mapping table, the system triggers the ERR_UNKNOWN_PARAM error code and terminates the process.

[0031] Preferably, an exception detection mechanism is incorporated into the unit conversion process. When a parameter value contains an unrecognized unit identifier (such as "TB" or an invalid suffix), the system logs the original parameter content and the error type and throws a structured exception to notify upstream systems for intervention. The exception information includes the error code, parameter location, and recommended corrective actions, such as "Memory units must be GB or MB."

[0032] During the parameter conversion process, the system maintains an intermediate state cache to support incremental updates. If a partial parameter conversion failure is detected, the system retains the successfully converted parameter set and ensures data consistency through transaction rollback. The cache uses a key-value storage structure, where the key is the parameter name and the value is the converted canonical value or an error status flag.

[0033] Furthermore, the system provides fuzzy matching for enumeration parameter mapping. When an input parameter value partially matches a pre-set enumeration value (e.g., due to capitalization differences or abbreviations), the system selects the closest enumeration item by calculating a string similarity threshold. This similarity is calculated based on an edit distance algorithm, and mapping is performed when the threshold is met, with a configurable threshold parameter.

[0034] The error code system is preferably designed in a hierarchical structure, where ERR_UNKNOWN_PARAM belongs to the parameter semantic error category and is associated with a specific handling strategy, such as automatically ignoring non-critical parameters or triggering a manual review process. The error code information is returned in a standardized JSON format, including metadata such as the error level, the scope of impact, and the recommended repair.

[0035] In this embodiment, step three uses template engine technology to achieve dynamic generation and pre-verification of Slurm scripts, which specifically includes the following technical contents: The script generation process uses the standardized parameter set output in step 2 as input and employs a layered rendering strategy to construct a job script that conforms to the Slurm syntax. The template engine divides the content into a command area, an environment configuration area, and an execution logic area based on parameter type. The command area carries resource quotas and dependency declarations, the environment configuration area injects user-defined environment variables, and the execution logic area encapsulates the actual job launch commands.

[0036] During the resource parameter filling phase, the system inserts the converted standard parameters into template placeholders according to preset rules. For example, the CPU core count parameter is replaced by the {{cpu_cores}} placeholder to generate the #SBATCH --cpus-per-task=8 directive, and the memory parameter is replaced by the {{memory_mb}} placeholder to generate the #SBATCH --mem=64000 directive. For GPU resource configuration, when the --gres=gpu:N parameter is detected, the system automatically activates the corresponding GPU directive block in the template to avoid generating redundant configuration items.

[0037] Dependency directives are generated by concatenating the dependent job ID lists. This concatenation operation uses a colon delimiter to concatenate multiple job IDs into a single string, which is then combined with the dependency type identifier to form a complete directive. For example, the dependent job ID list [1001,1002] combined with the type afterany generates --dependency=afterany:1001:1002. A null value filter is introduced during the concatenation process to automatically remove invalid or duplicate job IDs, ensuring that the generated directive conforms to Slurm's syntax constraints.

[0038] For environment variable injection, the system converts the key-value pair parameters into a sequence of export statements, sorts them in lexicographical order, and inserts them into the script execution area. For example, the environment variables ENV=prod and LOG_LEVEL=debug will be rendered as: exportENV=prod; exportLOG_LEVEL=debug; Preferably, the environment variable injection process supports value escape processing. When it is detected that the value contains special characters (such as spaces or quotation marks), quotation marks are automatically added to avoid parsing errors.

[0039] After script generation is complete, the system performs syntax pre-verification to prevent potential errors. This verification is implemented by calling the --test-only parameter of the Slurm scheduler. Specifically, the generated script file is submitted to the local environment and the output log is captured. If an ERROR-level log is detected, the system parses the error type and associates it with the specific script line number, generating a structured report containing a description of the error and recommended fixes.

[0040] The template engine preferably supports dynamic loading of multiple template versions. The system selects the appropriate template file based on the job type or user group identifier. For example, a dedicated template containing MPI configuration is loaded for a high-performance computing job, while a basic template is used for a batch job. Template files are stored in a versioned directory structure, and hash checksums are used to ensure the integrity of the template content during the loading process.

[0041] Furthermore, the system incorporates a context-aware mechanism during the syntax verification phase. If a dependent job ID is detected as nonexistent or in an abnormal state, the system automatically queries the job status database and updates the dependency list, preventing submission failures caused by outdated dependency references. The status database maintains data freshness by periodically synchronizing information with the Slurm scheduler's job queue.

[0042] In this embodiment, step 4 implements the secure submission of Slurm jobs by encapsulating the system-level command call and output parsing mechanism, which specifically includes the following technical contents: The job submission process creates a child process that calls the Slurm scheduler's sbatch command to execute the script file. The child process's standard output and error streams are redirected to the parsing module through pipes. The system adds a timeout threshold parameter to the command call. If the command execution time exceeds the preset threshold, the process is forcibly terminated and marked with the ERR_TIMEOUT error type. This prevents system resource consumption caused by network delays or unresponsive services.

[0043] During the output parsing phase, the system matches job IDs using a precompiled regular expression pattern. The regular expression pattern, defined as r'Submitted\s+batch\s+job\s+(\d+)', extracts the numeric job identifier using capture groups. If a match fails, the system initiates an error tracing process: first, it scans the error stream for keywords (such as "Invalidpartition" or "Malformedscript"), then traces back to the original script file, checking each line of the #SBATCH directive for syntactical compliance by comparing it to the official Slurm syntax specification document.

[0044] For exception classification and handling, the system maps error types to handling strategies. When a script syntax error is detected, the ERR_SCRIPT_SYNTAX error code is returned, along with the specific line number. When the Slurm service becomes unavailable (for example, when the slurmctld process becomes unresponsive), the ERR_SLURM_SERVICE error code is generated, triggering a service health check. The error code system employs a hierarchical design, with the first level identifying the source of the error (e.g., configuration error, runtime error), and the second level detailing the specific error scenario.

[0045] Optimally, contextual correlation analysis is incorporated into the error tracing process. When a dependent job failure is detected, the system automatically queries the historical status records of the dependent jobs and locates the root cause by analyzing the job lifecycle logs (submission time, runtime, and exit code). Correlation analysis, based on a directed acyclic graph model, propagates status changes of job nodes in the dependency chain to the current job error context.

[0046] In the failure retry mechanism, the system designs an adaptive retry strategy for transient network errors (such as ERRNETWORK). The strategy dynamically adjusts the retry interval based on the number of failures and uses an exponential backoff algorithm to calculate the waiting time. The specific formula is, where is the waiting time (seconds), is the current number of retries, and is a configurable basic time unit. The atomicity of the submitted transaction is maintained during the retry process, ensuring that the status of the job description file is reset before each retry. Furthermore, the system provides a cross-platform compatibility layer for the submission process. For the Python environment, the subprocess module is used to encapsulate command calls; for the Java environment, the ProcessBuilder class is used to implement the same functionality. The compatibility layer abstracts a unified submission interface, accepts the script path, timeout period, and environment variables as input, and returns a standardized response object containing the job ID, error code, and structured data of the original log.

[0047] In this embodiment, step five implements job exception management through real-time monitoring and adaptive processing mechanisms, specifically including the following technical content: Job status monitoring is performed by periodically calling the Slurm scheduler's squeue command to obtain job queue information. The command output is parsed using regular expressions to extract key fields such as the job ID, status code, and run time. The system maintains a state machine model for each job, defining state transition rules, such as the legal transition from PENDING to RUNNING and the abnormal transition from RUNNING to FAILED. When an illegal state transition is detected, the system triggers a root cause analysis process, correlating job logs with resource usage data for comprehensive diagnosis.

[0048] In resource overage scenarios, the system parses the output of the Slurm seff command to obtain actual resource consumption data for the job. When memory usage exceeds a preset threshold relative to the requested value, the system automatically calculates an adjusted resource quota using a linear backoff strategy, using the formula: where is the backoff factor (default 1.2), is the actual consumption, and is the original requested value. The adjusted parameter set re-triggers the script generation and submission process starting in step 3. To handle missing dependencies, the system parses the DependencyNeverSatisfied error message in the job failure log to extract a list of missing dependent job IDs. Dependencies are verified for validity by querying the job metadata database. If a dependent job has been marked as failed, the system sends a notification message to the upstream workflow engine, triggering the rebuild or resubmission of the dependent job. This notification message is encapsulated in a standardized format and includes the missing job ID, failure timestamp, and recommended action. Preferably, a dynamic priority adjustment mechanism is implemented in the status monitoring process. For jobs that remain in the PENDING state for an extended period, the system increases the scheduling priority by modifying the –qos parameter or calling the scontrolupdatejobid=priority= command. The priority adjustment strategy is calculated based on the job waiting time and user authority level. The priority weight calculation formula is, where is the waiting time, is the user authority coefficient, and is a configurable weight parameter.

[0049] For abnormally terminated jobs, the system automatically captures and archives logs. This log collection is implemented using the slurm_job_log_collector tool, which pulls standard output, error logs, and performance metrics files (such as cpu_mem_usage.csv) from compute nodes and stores them in a distributed file system. The log analysis module identifies error patterns based on a predefined set of rules, such as matching line-level error information with the regular expression "ERROR\s+#\d+\s+in\s+.+?\.cpp," and associates the analysis results with the job lifecycle record.

[0050] During retry policy implementation, the system maintains retry counters and context snapshots. When the number of job failures reaches the configured maximum threshold, the system terminates the retry process and generates a diagnostic report containing the failure cause, resource usage history, and dependency links. This report uses visual charts to display resource consumption trends and state transition paths, helping operations personnel quickly identify bottlenecks.

[0051] Furthermore, the audit trail module records events throughout the job lifecycle, including submission time, execution start time, end time, and resource utilization metrics. This data is stored in a time-series database, enabling the generation of aggregated analysis reports by job queue, user group, or resource type. The report generation process utilizes dynamic SQL template technology to automatically construct queries based on filter criteria, such as those analyzing CPU core hour consumption or memory usage efficiency distribution.

[0052] The Slurm scheduling specification integration method described below and the Slurm scheduling specification integration system described above can be referenced to each other.

[0053] Slurm scheduling specification integration system, including: Parameter parsing engine: includes regular expression matcher and parameter validator; Script generator: dynamically builds SLURM scripts, integrating resource quotas, dependency instructions, and environment variables; Job submission agent: encapsulates sbatch command calls across languages, supporting Python and Java environments; Status monitor: real-time tracking of job lifecycles, with a built-in exception handling strategy library; Audit module: records job metadata and generates multi-dimensional analysis reports on resource utilization.

[0054] The system of this embodiment can be used to execute the above method embodiments, and its principles and technical effects are similar, so they will not be repeated here.

[0055] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.< / type> < / type>

Claims

1. Slurm scheduling specification integration method, characterized by, The following steps are involved: Step 1: Parse the heterogeneous job description file and extract resource requirement parameters and dependencies using the regular expression rule library; Step 2: Based on the extracted original parameters, convert them into SLURM standard parameters through preset mapping rules, including base unit conversion of numerical parameters and keyword mapping of enumeration parameters; Step 3: Based on the converted specification parameters, use the template engine to generate a SLURM script containing resource quotas, dependency instructions, and environment variables. The dependency instructions are generated by concatenating the dependency job IDs. Step 4: Call the Slurm interface command to submit the script and parse the output log using regular expressions to obtain the job ID; Step 5: Monitor the execution status of submitted jobs and trigger the resubmission process for abnormal jobs with resource excess or missing dependencies.

2. The Slurm scheduling specification integration method according to claim 1, characterized in that The step one comprises: Match parameter key-value pairs using a predefined regular expression rule library. The rules include the CPU_Cores=(\d+) and Memory_Req=(\d+[GM]B) patterns, and record exception logs for unmatched parameters. Verify the integrity of required parameters and trigger parsing interruption for description files with missing job names or resource quotas. Extract dependency declarations, identify dependency type identifiers and combine them into --dependency= <type> :<job_id_list> instruction.< / type> 3. The Slurm scheduling specification integration method according to claim 1, characterized in that The step one further comprises: Adopting a multi-level parser architecture: First level analysis of basic resource parameters; The second level parses the extended configuration parameter environment variables, input and output paths; The third level verifies the logical consistency of parameters; Trigger an alert for detected circular dependencies and terminate the process.

4. The Slurm scheduling specification integration method according to claim 1, characterized in that The second step includes: Input dependency: the original parameter set parsed based on step 1; Numeric parameter conversion: Memory parameters are converted from GB / MB units to SLURM standard MB units; The number of GPUs is directly mapped to the --gres=gpu:N command; Enumeration parameter mapping: Priority parameter; Queue type parameter; Exception handling: For parameters that cannot match the mapping rules, the error code ERR_UNKNOWN_PARAM is triggered and the process is terminated.

5. The Slurm scheduling specification integration method according to claim 1, characterized in that The step three includes: Template input: receives the set of standard parameters output in step 2; Dynamic Rendering: Fill the --cpus-per-task, --mem, and --gres resource parameters into the script header instructions; Concatenate the list of dependent job IDs with colon separators as follows: --dependency=afterany: <job1:job2> instruction; Convert the environment variable key-value pair into exportKEY=VALUE statement and insert it into the script execution area; Syntax check: Execute sbatch --test-only pre-submission verification on the generated script to intercept syntax errors.

6. The Slurm scheduling specification integration method according to claim 1, characterized in that: The fourth step includes: Submit the script by executing the sbatch[script_path] command through the system call; Use regular expressions to extract job IDs from command output and detect script syntax errors if matching fails. Categorize and record Slurm service exceptions and script syntax errors.

7. The Slurm scheduling specification integration method according to claim 1, characterized in that: The step five includes: For jobs with resource overruns, increase the resource quota by the preset ratio and resubmit; For jobs with missing dependencies, resolve the missing dependent job IDs and notify the upstream system; Set a maximum retry threshold. If the threshold is exceeded, the job is terminated and an alarm is generated.

8. The Slurm scheduling specification integration method according to claim 1, characterized in that: The method further comprises: Define the maximum retry threshold in the job description file; When the number of job failures exceeds the threshold, the process is automatically terminated and an error report is generated; Archives metadata for terminated jobs to a history database.

9. The Slurm scheduling specification integration method according to claim 1, characterized in that: The monitoring of step 5 includes: Periodically poll Slurm's squeue interface to obtain job status; Triggering a priority increase mechanism for jobs that have been in the PENDING state for a long time; Automatically capture logs and tag abnormally terminated jobs with error types.

10. A Slurm scheduling specification integration system, according to the Slurm scheduling specification integration method according to any one of claims 1 to 9, characterized in that: include: Parameter parsing engine: includes regular expression matcher and parameter validator; Script generator: dynamically builds SLURM scripts, integrating resource quotas, dependency instructions, and environment variables; Job submission agent: encapsulates sbatch command calls across languages, supporting Python and Java environments; Status monitor: real-time tracking of job lifecycles, with a built-in exception handling strategy library; Audit module: records job metadata and generates multi-dimensional analysis reports on resource utilization.

Images