Memory resource adaptive management method and storage medium

Abstract

The present application relates to the technical field of computer resource management, and especially relates to a memory resource adaptive management method and a storage medium, the present application generates stable retrieval keys by command semantic analysis, recalls similar historical jobs, adopts a combination of historical sample retrieval and reasoning, can output reliable prediction values even in a sample insufficient or cold start scene, and guarantees the applicability of the scheme; double quantile values cooperate with layered risk rules to make resource decisions, can select rewriting, conservative release or original value reservation according to the confidence level, and balances between reducing resource waste and guaranteeing job operation safety; the resource value is dynamically corrected according to the updated prediction result during queuing, which can effectively utilize the waiting window before job start; the real peak value is recovered after the job is completed and the historical sample library is rewritten, forming a closed-loop data flow of submission, operation and recovery, so that the prediction model can be continuously optimized, and the long-term adaptability and stability of the scheme are improved.

Description

Technical Field

[0001] This invention relates to the field of computer resource management technology, and in particular to an adaptive memory resource management method and storage medium. Background Technology

[0002] High-performance computing (HPC) and electronic design automation (EDA) clusters widely use job scheduling systems (such as IBM LSF) to manage and schedule computing tasks such as simulation, synthesis, placement and routing in chip design.

[0003] When users submit jobs, they must explicitly declare memory resources in the submission script or command line. This typically includes parameters for scheduling and accounting, as well as the -M parameter for the job's hard limit. The scheduler determines the job's queuing position, node allocation, and memory constraints during execution based on these declared values, thereby ensuring the overall stability and fairness of the cluster. This model has long been a standard practice in the industry. However, with the continuous increase in chip design complexity, the growing diversity of job types, and significant fluctuations in memory requirements, the inherent shortcomings of this model are becoming increasingly prominent.

[0004] However, in the aforementioned practices, to avoid jobs triggering forced termination due to insufficient memory (Out-Of-Memory, OOM), users often rely on experience to significantly increase the requested value, resulting in a large amount of book resources being occupied while physical memory remains idle, leading to low cluster resource utilization. Conversely, if the requested value is too small, the job is very likely to trigger OOM, causing requeuing and rerunning, extending the project cycle. Although there are estimation methods based on fixed rules or historical averages in existing technologies, their granularity is relatively coarse, making it difficult to accurately reflect the memory differences of the same tool under different command semantics. Furthermore, resource values ​​are determined once at the time of job submission, making it impossible to utilize the correction window during queuing, further exacerbating resource waste or OOM risks. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide an adaptive memory resource management method and a storage medium.

[0006] In a first aspect, embodiments of the present invention provide a memory resource adaptive management method, the method comprising: Obtain the job submission request and collect the submission data to generate a raw input object containing the job's unique identifier; Semantic parsing is performed on the submission commands in the original input object to generate stable retrieval keys, resulting in semantic feature records that uniquely identify associated jobs; The historical sample library is retrieved using the stable retrieval key in the semantic feature record. The memory requirement of the current job is predicted by reasoning, and the prediction result with the unique identifier of the associated job is obtained. The prediction result includes at least the predicted memory value of the first quantile and the predicted memory value of the second quantile, as well as the corresponding confidence level. Based on the prediction results, the user's original memory resource request value in the original input object, and the preset risk rules, resource decisions are made to obtain a resource decision record with a unique identifier for the associated job; During the job queuing period, the resource decision record is associated with the unique job identifier and the job status is continuously monitored. When the preset trigger conditions are met, the resource value is adjusted through the online modification command of the scheduling system according to the updated prediction results, so as to obtain the final resource record associated with the unique job identifier. After the job is completed, the final resource record is associated with the job's unique identifier, the actual peak memory of the job is collected, and the actual result record is generated and written back to the historical sample database.

[0007] In conjunction with the first aspect, the steps of obtaining the job submission request, collecting the submission data, and generating the original input object containing the job's unique identifier include: Intercept the job submission request to obtain the source of the raw data to be collected. The source of the raw data includes at least two of the following: bsub submission command, job script file, scheduling system parameter line in the script, user field, queue field, project field and submission time provided by the scheduling system. The submission command content, script path, queue name, username, project name, and the user's original memory resource request value are read from the original data source to obtain a multi-source submission data set; When both the bsub submission command and the job script file contain memory resource request parameters, the user's original memory resource request value is determined according to the priority rules that actually take effect in the scheduling system, and the final effective original memory request value is obtained. The job identifier returned by the scheduling system for the job corresponding to the job submission request is used as the unique identifier of the job; if the job identifier has not been obtained before submission, a temporary unique identifier is generated as the unique identifier of the job, and after successful submission, the temporary unique identifier is bound to the job identifier returned by the scheduling system to obtain the final unique identifier of the job. Associate the final job unique identifier and output the original input object containing the job unique identifier, submission command content, script path, queue name, username, project name, final effective original memory allocation value, and submission time.

[0008] In conjunction with the first aspect, the steps of semantically parsing the submission commands in the original input object and generating stable retrieval keys to obtain semantic feature records that uniquely identify associated jobs include: Read the original submission command and script file from the original input object to obtain the command content to be parsed; Extract at least one non-executable content from the interpreter declaration line, scheduling system parameter line, and environment variable setting statement from the command content to be parsed to obtain the valid command content; When the command header of a valid command contains a scripting language wrapper, the script or parameters following the wrapper are read recursively until the actual business execution program is located, and the actual program path is obtained. The tool identification result is obtained by identifying the tool name and tool version corresponding to the real program path based on at least one of the tool whitelist, path keywords or command words. Replace at least one variable parameter among random numbers, timestamps, batch numbers, or counters in the valid command content with a uniform placeholder, and extract and normalize at least one key parameter among compilation mode, test mode, coverage switch, waveform switch, debug database switch, number of threads, input path, and project name to obtain a standardized command and feature word set. A stable search key is generated based on the tool name, the job type in the normalization command, and the combination of normalized key parameters. Associate the unique identifier of the job and output a semantic feature record containing the unique identifier of the job, tool name, tool version, normalized command, feature term set, stable search key and job type.

[0009] In conjunction with the first aspect, the steps of retrieving historical sample databases using stable retrieval keys from semantic feature records, predicting the memory requirements of the current job through multi-level inference, and obtaining a prediction result that uniquely identifies the associated job include: The historical sample database is retrieved using a stable search key as an index, and it is determined whether the number of hit historical samples has reached a preset threshold. If so, directly perform quantile statistics based on the actual peak memory of each hit historical sample to obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile. If not, based on at least one of the job type, tool name, or queue information in the semantic feature record, the system routes to the corresponding sub-model for memory prediction, thereby obtaining the predicted memory value of the first quantile and the predicted memory value of the second quantile.

[0010] In conjunction with the first aspect, the step of routing to the corresponding sub-model for memory prediction based on at least one of the job type, tool name, or queue information in the semantic feature record to obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile further includes: When the sub-model coverage condition is not met, the global fallback model is called to perform memory prediction, and the predicted memory value of the first quantile and the predicted memory value of the second quantile are obtained. The sub-model coverage conditions include: the existence of a trained sub-model corresponding to the tool name and job type of the current job; the completeness of the input features of the current job; the number of training samples of the trained sub-model reaching a preset sample number threshold; and the prediction error of the trained sub-model being lower than a preset error threshold.

[0011] In conjunction with the first aspect, before the step of retrieving the historical sample database using a stable retrieval key from the semantic feature record, predicting the memory requirements of the current job through inference, and obtaining the predicted result of the unique identifier of the associated job, the method further includes: Obtain the historical sample library, which contains the semantic features of recycled historical jobs, the actual peak memory, the running status information, the corresponding stable retrieval keys, and the model training labels; Logarithmically transform the true peak memory of each historical sample in the historical sample library, and use the logarithmically transformed memory value as the model training label to obtain the transformed historical sample library; Based on the job type, tool name, and queue information of each historical sample in the historical sample library, train at least one sub-model and one global fallback model to obtain a set of trained prediction models.

[0012] In conjunction with the first aspect, the steps of making resource decisions based on the prediction results, the user's original memory resource requests in the original input objects, and preset risk rules to obtain a resource decision record with a unique identifier for the associated job include: Obtain the confidence level, first quantile, and second quantile from the prediction results. The second quantile is higher than the first quantile. The confidence level is calculated by weighting the number of hit historical samples, the error of the last N backtests, and the recent volatility, and then normalized to between 0 and 1. When the confidence level reaches the first confidence threshold and the high-risk exclusion rule is not hit, the first quantile value multiplied by the first coefficient is used as the scheduling accounting memory value, and the second quantile value multiplied by the second coefficient is used as the job hard upper limit value, generating a resource decision record containing an automatic rewrite flag; wherein, the second coefficient is greater than the first coefficient, and the first and second coefficients are determined based on at least one of the tool volatility, queue stability, and historical underestimation rate; When the confidence level is lower than the first confidence level threshold but reaches the second confidence level threshold, the predicted memory value in the prediction result is recorded but the resource request value of the current job is not modified, and a resource decision record containing a conservative release flag is generated. When the confidence level is lower than the second confidence level threshold or a high-risk exclusion rule is hit, the user's original memory resource request value remains unchanged, and a resource decision record containing a retain original value identifier is generated.

[0013] In conjunction with the first aspect, the steps of adjusting resource values ​​based on the updated prediction results using online modification instructions from the scheduling system to obtain the final resource record with a unique identifier for the associated job include: The queuing status of a task is monitored using its unique identifier to obtain the current queuing status information. When the current queuing status information indicates that the job is still in the queuing state and the preset triggering conditions are met, the historical sample library is retrieved using the stable retrieval key, and the memory requirement of the current job is predicted by reasoning to obtain the updated prediction result. Resource decisions are made based on the updated forecast results, resulting in updated resource decision records. Compare the scheduling accounting memory value in the updated resource decision record with the currently effective scheduling accounting memory value, and the job hard limit value in the updated resource decision record with the currently effective job hard limit value to obtain the resource value difference judgment result; When the resource value difference judgment result indicates that there is a difference, the scheduling accounting memory value and job hard limit value in the updated resource decision record are written into the job's resource configuration through the online modification command of the scheduling system, and the rewrite timestamp and correction log are recorded to generate the final resource record with a unique identifier associated with the job. When the resource value difference judgment result indicates that there is no difference, the currently effective resource value remains unchanged, and a final resource record with a unique identifier for the associated job is generated.

[0014] In conjunction with the first aspect, the method also includes: Retrieve resource decision records, which include rewritten decisions; Extract resource recommendation values ​​from the resource decision records to obtain recommendation value records; keep the resource recommendation values ​​from being written into the job's resource configuration to obtain simulation operation records; After the operation is completed, the suggested resource values ​​are compared with the actual resource consumption of the operation and the deviation is calculated to obtain the simulation verification results. When the simulation verification results meet the preset safety conditions, the rewrite decision is enabled for the operation within the preset range, and a restricted enable record is obtained; Monitor the running status of jobs that have enabled rewriting decisions and obtain limited verification results; When the restricted verification results meet the preset security conditions, the rewrite decision in the resource decision record is enabled for all applicable operations, resulting in a fully enabled record.

[0015] Secondly, this application provides a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0016] The embodiments of the present invention bring the following beneficial effects: This application provides a memory resource adaptive management method and storage medium. The method includes: obtaining a job submission request and collecting submission data to generate an original input object containing a unique job identifier; performing semantic parsing on the submission command in the original input object and generating a stable retrieval key to obtain a semantic feature record associated with the unique job identifier; retrieving a historical sample library using the stable retrieval key in the semantic feature record, predicting the memory requirement of the current job through reasoning to obtain a prediction result associated with the unique job identifier, wherein the prediction result includes at least the predicted memory value of the first quantile and the predicted memory value of the second quantile and the corresponding confidence level; making resource decisions based on the prediction result, the user's original memory resource request value in the original input object, and preset risk rules to obtain a resource decision record associated with the unique job identifier; during job queuing, associating the resource decision record with the unique job identifier and continuously monitoring the job status, and when a preset trigger condition is met, adjusting the resource value according to the updated prediction result through the online modification instruction of the scheduling system to obtain a final resource record associated with the unique job identifier; after the job ends, associating the final resource record with the unique job identifier, collecting the actual peak memory of the job and generating an actual result record, and writing the actual result record back to the historical sample library.

[0017] This invention generates stable retrieval keys through command semantic parsing to recall similar historical jobs. It then employs a combination of historical sample retrieval and inference to output reliable predicted values ​​even in scenarios with insufficient samples or cold starts, ensuring the broad applicability of the solution. Biquantile values, combined with hierarchical risk rules, enable resource decision-making, allowing for rewriting, conservative release, or retention of original values ​​based on confidence levels, achieving a balance between reducing resource waste and ensuring job operation safety. During queuing, resource values ​​are dynamically corrected based on updated prediction results, effectively utilizing the waiting window before job initiation. After job completion, the actual peak value is collected and written back to the historical sample database, forming a closed-loop data flow of submission, execution, and collection. This allows the prediction model to be continuously optimized, improving the long-term adaptability and stability of the solution.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an adaptive memory resource management method provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To facilitate understanding of this embodiment, the technical terms used in this application will be briefly introduced below.

[0024] Job Unique Identifier: A global identifier used to uniquely identify a job. It serves as the primary key for data association throughout the entire process of submission, queuing, execution, and recycling. In the LSF scheduling system, the job unique identifier preferentially uses the job_id returned by the scheduling system; if the job_id is not yet obtained before submission, a temporary unique identifier (submit_uuid) is generated first, and then bound to the job_id after successful submission.

[0025] Stable search key: A string generated by combining the tool name, job type, and normalized key parameters, used to accurately retrieve historical samples that are semantically similar to the current job from the historical sample library.

[0026] Semantic parsing: The process of performing structured parsing on the original submitted command, including stripping non-executable content (interpreter declaration line, scheduling system parameter line, environment variable setting statement), recursively locating the actual business execution program, replacing volatile parameters such as random numbers / timestamps / batch numbers with uniform placeholders, extracting key parameters such as compilation mode / test mode / coverage switch / waveform switch, and finally generating a standardized command and feature lexical set.

[0027] True peak memory: The maximum amount of physical memory actually consumed by the job during its execution. It is continuously sampled and recorded by the scheduling system or operating system kernel during the job's execution and its maximum value is retained at the end of the job, serving as training labels for evaluating prediction accuracy and writing back to the historical sample library.

[0028] Feature lexicon set: A set of structured parameters extracted and normalized from the content of valid commands, used to characterize the computational features of the job.

[0029] Tool variability: refers to the degree of variation in memory consumption of the same tool across different job instances. Tool variability reflects the stability and predictability of the tool's memory consumption behavior.

[0030] Queue stability refers to the regularity and consistency of historical job execution within a job queue. Queue stability reflects the predictability of the job execution environment within the queue.

[0031] Historical underestimation rate: refers to the proportion of samples in historical prediction records where the predicted memory value is lower than the actual peak memory of the job, out of the total number of prediction samples.

[0032] After introducing the technical terms used in this application, the application scenarios and design concepts of the embodiments of this application will be briefly described below.

[0033] In high-performance computing clusters, users need to manually enter memory resource request values ​​for jobs. This often results in over-requesting due to inaccurate estimations, leading to severe resource waste; or under-requesting, triggering Out of Memory (OOM) errors and causing job failures, impacting productivity. Existing estimation methods are too coarse-grained and cannot be dynamically adjusted, making it difficult to achieve on-demand resource allocation.

[0034] Based on this, this application provides a memory resource adaptive management method and storage medium.

[0035] Example 1 This application provides an adaptive memory resource management method, combined with Figure 1 As shown, the method includes: S110: Obtain the job submission request and collect the submission data to generate a raw input object containing the job's unique identifier.

[0036] S120: Semantically parse the submission commands in the original input object and generate stable retrieval keys to obtain semantic feature records that uniquely identify the associated jobs.

[0037] S130, retrieve the historical sample library using the stable retrieval key in the semantic feature record, predict the memory requirement of the current job through reasoning, and obtain the prediction result of the unique identifier of the associated job. The prediction result includes at least the predicted memory value of the first quantile and the predicted memory value of the second quantile and the corresponding confidence level.

[0038] S140, based on the prediction results, the user's original memory resource request value in the original input object, and the preset risk rules, make resource decisions and obtain resource decision records with unique identifiers of associated jobs.

[0039] S150: During the job queuing period, resource decision records are associated with the unique job identifier and the job status is continuously monitored. When the preset trigger conditions are met, the resource value is adjusted through the online modification command of the scheduling system according to the updated prediction results, so as to obtain the final resource record associated with the unique job identifier.

[0040] S160: After the job is completed, associate the final resource record with the job's unique identifier, collect the job's actual peak memory and generate an actual result record, and write the actual result record back to the historical sample database.

[0041] In this invention, step S110 intercepts job submission requests and collects multi-source submission data to generate an original input object with a unique job identifier as the primary key; step S120 performs semantic parsing on the submission commands to generate a stable retrieval key, enabling semantically similar jobs to be accurately identified and recalled; step S130 retrieves the historical sample library using the stable retrieval key, and outputs a bisector prediction value with confidence through three levels of inference: historical table lookup, sub-model, and global fallback model, solving the problem that existing technologies cannot distinguish memory differences of the same tool under different command semantics; step S140 performs hierarchical resource decision-making based on confidence and risk rules, achieving a quantitatively controllable balance between reducing resource waste and avoiding OOM; step S150 dynamically corrects biases during job queuing based on model updates or new samples, overcoming the defect that resource values ​​are fixed and cannot be adjusted once submitted; and step S160 recovers the true peak value and writes it back to the historical sample library, forming a closed-loop data flow to achieve continuous iterative optimization of the model.

[0042] In conjunction with the first aspect, step S110 includes: S111, intercept the job submission request and obtain the source of the raw data to be collected. The source of the raw data includes at least two of the following: bsub submission command, job script file, scheduling system parameter line in the script, user field, queue field, project field and submission time provided by the scheduling system.

[0043] S112: Read the submission command content, script path, queue name, user name, project name, and the user's original memory resource request value from the original data source to obtain a multi-source submission data set.

[0044] S113, when both the bsub submission command and the job script file contain memory resource request parameters, the user's original memory resource request value is determined according to the priority rules that actually take effect in the scheduling system, and the final effective original memory request value is obtained.

[0045] S114, use the job identifier returned by the scheduling system for the job corresponding to the job submission request as the unique identifier of the job; when the job identifier has not been obtained before submission, generate a temporary unique identifier as the unique identifier of the job, and bind the temporary unique identifier with the job identifier returned by the scheduling system after successful submission to obtain the final unique identifier of the job.

[0046] S115, associated with the final job unique identifier, outputs the original input object containing the job unique identifier, submission command content, script path, queue name, user name, project name, final effective original requested memory value, and submission time.

[0047] Step S110 intercepts the user's job submission request by an interceptor deployed at the job submission entry point (such as the esub mechanism in the LSF system), collects submission data from multiple data sources, and generates an original input object with the job's unique identifier as the primary key, providing a complete and traceable data foundation for subsequent semantic parsing, prediction, decision-making, and recycling processes.

[0048] When a user submits a job via the bsub command, the submission interceptor intercepts the submission request before the job enters the scheduling system queue. The original data sources include at least two of the following: the bsub submission command itself, the job script file, the scheduling system parameter line within the script file (i.e., the #BSUB parameter line), and user fields, queue fields, project fields, and submission time provided by the scheduling system (LSF). This multi-source collection ensures that the subsequently extracted information, such as the submission command content, script path, queue name, user name, project name, and the user's original memory resource request value, is comprehensive and traceable. This multi-source collection mechanism allows the system to obtain complete information regardless of whether the user places the resource request parameters in the command line or a script.

[0049] After collecting data from various sources, the system parses and extracts key fields: it reads the complete submission command (raw_command) and the storage path of the job script file (script_path) from the bsub command line; it obtains the queue name (queue), submitting user name (user), and project name (project) from the context information provided by the scheduling system; and it extracts the user's original memory resource request values ​​(request_rusage and request_M) from command line parameters (such as -M and -R rusage[mem=...]) and the #BSUB parameter line in the script file. This multi-source data is then aggregated to form a structured multi-source submission data set, which is used in subsequent steps to determine the final effective resource request value.

[0050] In a real production environment, users may specify memory resources in the bsub command line using `-M` and `-R rusage[mem=...]`, or in the script file using the `#BSUB` parameter line, or even both with inconsistent values. To accurately reflect the resource value ultimately adopted by the scheduling system, this step makes a decision according to the priority rules of the LSF scheduling system—when both command-line and script parameters exist, the configuration adopted first during actual scheduling is used to determine the final effective value. This ensures that the collected user's original memory resource request value is completely consistent with the value used by the scheduling system for actual accounting and enforcement limits, providing an accurate comparison benchmark.

[0051] To ensure data traceability for the same job across submission, queuing, execution, and recycling stages, a globally unique identifier needs to be established for each job. Specifically, the job_id returned by the LSF scheduling system is preferentially used as the unique identifier. However, the esub interceptor is executed before the job is actually successfully submitted, at which point it may not have yet obtained the job_id assigned by LSF. To address this, this step adopts a two-stage identifier strategy: when the job_id has not yet been obtained before submission, the system first generates a temporary unique identifier (submit_uuid) as the unique identifier for the job, used to associate various data objects during the submission stage; after the job is successfully submitted and the scheduling system returns the official job_id, the temporary identifier is bound to the job_id to form the final unique identifier for the job, thereby ensuring that even data generated in the pre-submission stage can be associated with data in subsequent queuing, execution, and recycling stages through the same primary key.

[0052] Subsequently, the final unique job identifier obtained in S114 is used as the primary key and associated with the multi-source submission data collected in S112 and the final effective original memory request value determined in S113 to form a structured raw input object (JobSubmitRecord). This object contains at least the following fields: job unique identifier (job_id), raw submission command (raw_command), script path (script_path), queue name (queue), user name (user), project name (project), scheduling accounting request value (request_rusage), job hard cap request value (request_M), and submission time (submit_time). This raw input object, as the sole output of this step, is passed to step S120 for command semantic parsing. The unique job identifier within this object will be used throughout the entire method flow to ensure that data records generated in each step can be traced back to the same job.

[0053] Step S110 intercepts the request at the job submission entry point through the submission interceptor esub, obtains the submission command, script path, queue, user, project and the user's original memory resource value through multi-source data collection, handles multi-source resource parameter conflicts with the priority rules that are actually effective in LSF, and addresses the situation where the job_id has not been obtained before submission through a two-stage binding mechanism of submit_uuid and job_id, and finally generates a complete original input object with job_id as the primary key.

[0054] In conjunction with the first aspect, step S120 includes: S121, Read the original submission command and script file from the original input object to obtain the command content to be parsed.

[0055] S122, extract at least one non-executable content from the interpreter declaration line, scheduling system parameter line, and environment variable setting statement from the command content to be parsed to obtain the valid command content.

[0056] S123, when the command header of the valid command content contains a script language wrapper, recursively read the script or parameters after the wrapper until the actual business execution program is located, and obtain the actual program path.

[0057] S124. Based on at least one of the tool whitelist, path keywords, or command words, identify the tool name and tool version corresponding to the real program path, and obtain the tool identification result.

[0058] S125 replaces at least one variable parameter among the random number, timestamp, batch number or counter in the valid command content with a uniform placeholder, and extracts and normalizes at least one key parameter among the compilation mode, test mode, coverage switch, waveform switch, debug database switch, number of threads, input path, and project name to obtain a set of normalized commands and feature terms.

[0059] S126. Generate a stable search key based on the tool name, the job type in the normalization command, and the combination of normalized key parameters.

[0060] S127, Associated Job Unique Identifier, Outputs semantic feature records containing job unique identifier, tool name, tool version, normalized command, feature term set, stable search key, and job type.

[0061] After generating the original input object in S110, step S120 performs deep semantic analysis on the submission commands in the object. By stripping away non-executable content, locating the real business program, normalizing volatile parameters, extracting key features, and generating stable retrieval keys, semantically similar jobs can be accurately identified and recalled, thus solving the core problem that existing technologies cannot distinguish memory differences of the same tool under different command semantics. Specifically: Step S121 reads the raw submission command (raw_command) and script path (script_path) from the raw input object output in step S110, and obtains their complete content by accessing the script file corresponding to the script path. The raw submission command contains all the instructions entered by the user in the bsub command line, while the script file contains the complete command sequence during the actual execution of the job. The two are merged or correlated to obtain the command content to be parsed, which serves as the raw input for subsequent semantic parsing. The key to this step is obtaining content from both the command line and the script simultaneously, because the actual business execution program may be directly embedded in the bsub command or encapsulated within a script file. Only by completely obtaining information from both sources can we ensure that the actual business execution program is not overlooked during subsequent location.

[0062] The command content to be parsed typically contains a large amount of non-executable semantic noise, which needs to be stripped before tool identification. Specifically, step S122 performs at least one of the following stripping operations: stripping interpreter declaration lines (such as #! / bin / bash, #! / usr / bin / python, etc.), which are only used to specify the script interpreter and do not participate in business logic; stripping scheduling system parameter lines (i.e., #BSUB parameter lines within the script), which have already been used in step S110 to read the resource parameters originally requested by the user, but retaining them during the semantic parsing stage would introduce noise. For example, -M in #BSUB -M 8G might be misidentified as a parameter of a business tool, thus interfering with tool name identification and key parameter extraction; stripping environment variable setting statements (such as export, setenv, etc.), which are used to set the runtime environment and do not reflect the computational characteristics of the job. After the above stripping operations, a valid command content containing only the actual business execution logic is obtained.

[0063] In real-world production environments, EDA jobs are typically launched using scripting language wrappers (such as python, python3, csh, bash, sh, etc.). In this case, the command header of the valid command content only displays the wrapper name, not the actual business execution program. Directly using the wrapper as the tool for identification would cause all different services launched through the same wrapper (such as Python scripts launching simulations, tests, coverage collection, etc.) to be categorized together, losing semantic distinguishability. To address this issue, step S123 employs a recursive parsing strategy: when a scripting language wrapper is detected in the command header of the valid command content, the script file path or parameters specified after the wrapper are read, and the contents of the script file are recursively opened and read. The contents are checked to see if there are deeper wrapper calls or actual business execution program paths within the script file, until the final, real business execution program is located. For example, for the submission command `python3 sim.py -t regress_a`, the first `python3` is a wrapper. Steps S123 recursively read the contents of the `sim.py` script, parse its core call logic, and ultimately locate the actual simulation business execution program. For `csh run.csh`, it recursively reads `run.csh` until it locates the underlying EDA tool call. Through the above recursive parsing, the actual program path is obtained, providing an accurate basis for subsequent tool name identification.

[0064] Next, step S124 performs tool identification on the path and its corresponding command tokens. Specifically, at least one of the following methods can be used for identification: based on tool whitelist matching, a whitelist of EDA tool names (such as vcs, irun, xrun, dc_shell, icc2_shell, etc.) is pre-built, and the real program path or command tokens are precisely matched with the whitelist; based on path keyword identification, tool-related keywords are extracted from the real program path (such as the path containing patterns like / tools / vcs / , / synopsys / dc / , etc.); based on command token identification, after segmenting the valid command content into tokens, it is matched with known tool call patterns.

[0065] The tool name (tool_name, such as vcs, irun) and tool version (tool_version, such as 2021.12-SP1) are identified using the above method, and the tool identification result is obtained.

[0066] The actual submission command in step S125 contains a large number of volatile parameters that change with the job instance. If the complete command is used directly as the matching key, jobs with the same semantics but different volatile parameter values ​​will not be associated. Specifically, two aspects of processing are performed: The first aspect involves normalizing volatile parameters. Specifically, at least one of the following volatile parameters in the valid command content is replaced with a standardized placeholder: random number (e.g., 1234 in seed=1234 is replaced with {SEED}), timestamp (e.g., timestamp=202608251200 is replaced with {TIMESTAMP}), batch number (e.g., batch_id=0728 is replaced with {BATCH_ID}), and counter (e.g., iter=0001 is replaced with {ITER}). Furthermore, random numbers of four digits or more should also be uniformly replaced. This normalization operation ensures that different running instances of the same business logic have the same standardized expression at the command level.

[0067] The second aspect involves extracting and normalizing key parameters. Specifically, from the effective command content, at least one of the following key parameters that significantly impact memory consumption is extracted and their values ​​are normalized: compilation mode (e.g., build, comp), test mode (e.g., test), coverage switch (e.g., +define +COVERAGE), waveform switch (e.g., +define +WAVEFORM), debug database switch (e.g., KDB), number of threads (e.g., 8 in -j8), input path (e.g., -f filelist.f), project name (e.g., -proj project_a), etc. For switch-type parameters, they are standardized into a key-value format; for numerical parameters, their values ​​are extracted and standardized into a uniform representation.

[0068] After the above processing, we obtain the normalized command (normalized_cmd) and the feature token set (feature_tokens). The normalized command is used to generate stable search keys, and the feature token set is used to input the prediction model into S130.

[0069] Based on the tool name obtained in S124 and the normalized command and feature term set obtained in S125, a stable retrieval key (semantic_key) for searching the historical sample database is generated in step S126. This stable retrieval key is not a newly added manual label, but a composite key automatically generated by the system. Its composition includes at least the tool name (tool_name), the job type parsed from the normalized command (job_type, such as build, test, cov, waveform), and the core dimensions from the normalized set of key parameters. For example, for the command `python3 sim.py -t regress_a -b build +define+SYN=1 seed=1234`, after processing in steps S121-S125, the generated stable retrieval key is `sim.py|build|regress_a|SYN=1`.

[0070] Finally, the unique job identifier determined in S114 is used as the primary key and associated with the tool name and version obtained in S124, the normalized command and feature token set obtained in S125, and the stable search key and job type obtained in S126 to form a structured semantic feature record (SemanticFeatureRecord). This record contains at least the following fields: unique job identifier (job_id), tool name (tool_name), tool version (tool_version), normalized command (normalized_cmd), feature token set (feature_tokens), stable search key (semantic_key), and job type (job_type).

[0071] The stable retrieval key (semantic_key) is used to retrieve semantically similar historical samples from the historical sample library, while the feature tokens (feature_tokens) or feature vectors are used for in-memory prediction in the second-level sub-model and the third-level global fallback model. Thus, the original natural language submission command has been transformed into a structured, computable semantic feature representation.

[0072] The goal of this stable search key is to establish precise semantic associations between jobs using the same tool, job type, and combination of key parameters. This allows for accurate retrieval of historical samples that are semantically similar to the current job when searching in the historical sample database. It also prevents similar jobs from being scattered under different search keys due to differences in variable parameters, and avoids incorrectly classifying jobs with significant semantic differences into one category by using only coarse-grained information (such as only the tool name).

[0073] After S120 completes command semantic parsing and generates semantic feature records, step S130 uses the stable retrieval key (semantic_key) in the record as an index to retrieve the historical sample library, predicts the memory requirements of the current job through a three-level inference mechanism, and outputs a bisector prediction value with confidence. This step solves the problem in the prior art that fixed rules or historical mean estimations have coarse granularity and cannot distinguish the memory differences of the same tool under different command semantics. Specifically, S130 further includes an inference pre-preparation stage and an inference execution stage.

[0074] In conjunction with the first aspect, prior to step S130, the following also includes: S1301, Obtain the historical sample library, which contains the semantic features, actual peak memory, running status information, corresponding stable retrieval keys, and model training labels of the historical jobs that have been recycled.

[0075] S1302, perform a logarithmic transformation on the true peak memory of each historical sample in the historical sample library, and use the logarithmically transformed memory value as the model training label to obtain the transformed historical sample library.

[0076] S1303, based on the job type, tool name and queue information of each historical sample in the historical sample library, train at least one sub-model and a global fallback model to obtain a set of trained prediction models.

[0077] Upon first execution of this method or upon triggering periodic retraining, the system acquires the HistorySampleStore. This HistorySampleStore is continuously accumulated through epilog mechanisms and serves as a core data asset carrying real-world in-memory data from the production environment.

[0078] Each sample record in the historical sample database contains at least the following fields: semantic features of the historical job (such as tool name, job type, normalization command, etc.), peak memory (peak_mem, obtained by S160), runtime status information (such as exit_code, oom_flag), the corresponding stable search key (semantic_key), and model training labels. The stable search key is the core index field for subsequent S131 retrieval operations, enabling precise association of semantically similar historical jobs; the model training labels are used in the model training processes of S1302 and S1303.

[0079] In real-world production environments, EDA jobs exhibit vastly different memory footprints, ranging from small-scale compilation jobs in the MB range to large-scale simulation or placement and routing jobs in the hundreds of GB range. Directly using raw memory values ​​as model training labels would result in the prediction errors of large memory jobs carrying excessive weight in the loss function. The model would prioritize optimizing the prediction accuracy of large memory jobs while neglecting the distribution characteristics of numerous small memory jobs, leading to a significant increase in prediction bias for small memory jobs. To address this issue, step S1302 performs a logarithmic transformation (log transformation) on the true peak memory values ​​of each historical sample in the historical sample library. This compresses the memory values ​​spanning multiple orders of magnitude into a relatively balanced range, allowing the model to pay equal attention to the memory distribution characteristics of jobs of different sizes during training, thus preventing large memory jobs from dominating the loss function.

[0080] After the logarithmic transformation, the logarithmically transformed memory values ​​are used as the model training labels for each historical sample. These labels are used for supervised learning in subsequent step S1303 when training the sub-model and the global fallback model. The model learns the mapping relationship between input features (semantic features, feature word sets, etc.) and logarithmic domain memory values. After these steps, a transformed historical sample library is obtained, where the label for each sample is the logarithmically transformed memory value, rather than the original peak memory value.

[0081] After completing the logarithmic transformation in S1302, this step trains a set of prediction models based on the transformed historical sample database. Specifically, based on the job type (job_type), tool name (tool_name), and queue information of each historical sample in the database, the historical samples are divided into multiple categories, and a sub-model is trained for each category. The sub-model is a specialized model trained for a specific job category (such as "build type job of the VCS tool" or "regression test job of the q_dv_regress_night queue"). Its input is a set of feature terms or feature vectors from the semantic feature records, and its output is the logarithmically transformed predicted memory value. When the number of historical samples for a certain category reaches a preset training sample threshold, the sub-model corresponding to that category can be used for subsequent second-level inference.

[0082] Simultaneously, a global fallback model is trained using all historical samples. This global fallback model does not distinguish between job type, tool name, or queue information; it is trained using all available samples, and its input and output are consistent with the sub-models. The global fallback model's role is to cover cold-start or long-tail scenarios that the sub-models cannot handle. When a job category cannot train an effective sub-model due to insufficient historical samples, or when the prediction error of the trained sub-model exceeds a preset error threshold, the global fallback model provides predictions, ensuring that usable prediction results can be obtained for any job.

[0083] Each sub-model and the global fallback model use logarithmic domain memory values ​​as labels and outputs during training and prediction. The model itself only learns the mapping relationship in the logarithmic domain. In the subsequent inference step S133, the logarithmic domain predicted values ​​output by the model are restored to the original memory values ​​in terms of dimensions through an inverse exponential transformation. After training in this step, a set of trained prediction models is obtained, including at least one sub-model and one global fallback model, for use in the subsequent inference execution stage.

[0084] In summary, S130 completes all preparatory work before inference by acquiring the historical sample library in S1301, performing logarithmic transformation on the historical true peak values ​​in S1302, and training the sub-model and global fallback model based on job type / tool ​​name / queue information in S1303.

[0085] In conjunction with the first aspect, step S130 includes: S131, use the stable search key as an index to search the historical sample database, and determine whether the number of hit historical samples has reached the preset threshold.

[0086] If yes, proceed to step S132; otherwise, proceed to step S133.

[0087] S132 directly performs quantile statistics based on the actual peak memory of each hit historical sample to obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile.

[0088] S133, based on at least one of the job type, tool name or queue information in the semantic feature record, route to the corresponding sub-model to perform memory prediction, and obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile.

[0089] Step S131 uses a stable search key (semantic_key) as an index to retrieve historical samples in the historical sample database that are semantically similar to the current job. The stable search key is generated by combining the tool name, job type, and normalized key parameters. Therefore, the matched historical samples have the exact same tool, job type, and key parameter combination as the current job, which constitutes a semantic exact match. It then determines whether the number of matched historical samples (sample_count) reaches a preset threshold (e.g., 30 samples). This preset threshold is the minimum sample size pre-configured by the system to support reliable quantile statistics.

[0090] If the number of hits reaches the preset threshold, it means that there are enough semantically exact matching samples in the historical sample library. At this time, S132 is executed to enter the first level of reasoning (historical table lookup). If the number of hits does not reach the preset threshold, it means that there are not enough semantically exact matching samples. It is necessary to use more generalized category information for prediction. At this time, S133 is executed to enter the second level of reasoning (sub-model).

[0091] When step S131 determines that the number of hits has reached a preset threshold, step S132 is executed, corresponding to the first level of inference: historical table lookup. Under the condition of sufficient historical samples and precise semantic matching, step S132 directly performs quantile statistics based on the true peak memory (peak_mem) of the hit historical samples. The true peak memory of each hit historical sample is arranged in ascending order, and its empirical distribution function is calculated. The values ​​of two target quantiles (e.g., P80 and P92) are extracted as the predicted memory values ​​for the first and second quantiles of the current job. The first quantile (e.g., P80) serves as a more robust scheduling and accounting reference, while the second quantile (e.g., P92 or P95) serves as a more conservative hard upper limit reference for the job. The advantages of this method are: it is entirely based on statistical analysis of real production data, without introducing model inference errors; and precise semantic matching (same tool, same job type, same key parameter combination) ensures that the memory consumption distribution of historical samples best represents the real needs of the current job, resulting in the highest reliability of the prediction results. While outputting the predicted value, step S132 also calculates the confidence level based on the number of hit samples and the dispersion of their memory values. The more hit samples and the lower the dispersion, the higher the confidence level.

[0092] When step S131 determines that the number of hits has not reached the preset threshold, step S133 is executed, corresponding to the second-level inference: the sub-model. When there are insufficient samples for semantic exact matching, the retrieval conditions need to be appropriately relaxed, and more generalized category information is used for prediction. This step routes the current job to the corresponding sub-model based on at least one of the job type (job_type), tool name (tool_name), or queue information (queue) in the semantic feature record. The sub-model is a specialized model trained in the pre-preparation stage of S130 based on the job type, tool name, and queue information of each historical sample in the historical sample library. Its input is the feature token set (feature_tokens) or feature vector in the semantic feature record, and its output is the predicted memory value of the first quantile and the predicted memory value of the second quantile after logarithmic transformation and restoration to the original dimension. Compared with the first-level inference, the semantic matching granularity of the sub-model is coarser (belonging to the same type of job rather than completely identical semantic keys), but its advantage lies in its ability to reuse the memory distribution features of the same type of job, and it can still provide effective prediction when there are insufficient samples for semantic exact matching.

[0093] In conjunction with the first aspect, after step S133, the following also includes: S134, when the sub-model coverage condition is not met, call the global fallback model to perform memory prediction and obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile. The sub-model coverage conditions include: the existence of a trained sub-model corresponding to the tool name and job type of the current job; the completeness of the input features of the current job; the number of training samples of the trained sub-model reaching a preset sample number threshold; and the prediction error of the trained sub-model being lower than a preset error threshold.

[0094] When the sub-model coverage condition is not met, execute the third-level inference: call the global fallback model for in-memory prediction. The sub-model coverage condition not being met refers to at least one of the following situations: there is no trained sub-model corresponding to the tool name and job type of the current job; the input features of the current job are incomplete; the number of training samples for the trained sub-model has not reached the preset sample number threshold; or the prediction error of the trained sub-model is higher than the preset error threshold.

[0095] When any of the above conditions are met, the system determines that the sub-model cannot reliably provide predictions for the current job. In this case, it calls the global fallback model trained with the full sample set to perform memory prediction, ensuring that cold start or long-tail jobs can also obtain usable prediction results.

[0096] In the three-level inference process of steps S131-S134 above, the confidence score is calculated and normalized to between 0 and 1 by weighting the number of hit historical samples (sample_count), the error of the last N backtests, and the recent volatility, and is output together with the quantile prediction value. For the first-level inference (historical lookup table), the confidence score mainly depends on the number of hit samples and the dispersion of its memory values; for the second and third-level inference (sub-model or global fallback model), the confidence score also takes into account the backtesting error of the model on the recent validation set and the volatility of the prediction results.

[0097] After inference is completed, the unique job identifier is associated, and the output (PredictionRecord) is generated, containing the job identifier (job_id), stable retrieval key (semantic_key), first quantile, second quantile, prediction source (model_source, indicating whether it comes from the first-level historical lookup table, the second-level sub-model, or the third-level global fallback model), number of hit samples (sample_count), and confidence. This prediction result is then passed to S140 for resource decision-making.

[0098] Step S131, the first-level processing, directly performs quantile statistics based on historical true peak values ​​when there are sufficient semantically accurate matching samples. This avoids introducing model inference errors and achieves the highest semantic alignment, ensuring prediction accuracy in the optimal scenario. Step S133, the second-level processing, generalizes the search conditions to similar tasks when samples are insufficient. The sub-model reuses the memory distribution features of similar tasks, achieving a balance between accuracy and coverage. Step S134, the third-level processing, provides predictions from a global fallback model when the sub-model cannot cover the full scope, ensuring the solution remains usable in cold-start or long-tail scenarios. This three-level inference, progressing from precision to generalization, achieves the goal of outputting confident biquantile predictions for all types of tasks. The first and second quantile values ​​in this prediction provide a quantitative basis for subsequent resource decisions in S140, while the confidence level provides a quantitative foundation for risk stratification control.

[0099] In conjunction with the first aspect, step S140 includes: S141, obtain the confidence level, first quantile, and second quantile from the prediction results. The second quantile is higher than the first quantile. The confidence level is calculated by weighting the number of hit historical samples, the error of the last N backtests, and the recent volatility, and then normalized to between 0 and 1.

[0100] S142, when the confidence level reaches the first confidence level threshold and the high-risk exclusion rule is not hit, the first quantile value multiplied by the first coefficient is used as the scheduling accounting memory value, and the second quantile value multiplied by the second coefficient is used as the job hard upper limit value, and a resource decision record containing an automatic rewrite identifier is generated; wherein, the second coefficient is greater than the first coefficient, and the first coefficient and the second coefficient are determined based on at least one of the tool volatility, queue stability and historical underestimation rate.

[0101] S143, when the confidence level is lower than the first confidence level threshold but reaches the second confidence level threshold, record the predicted memory value in the prediction result but do not modify the resource request value of the current job, and generate a resource decision record containing a conservative release flag.

[0102] S144: When the confidence level is lower than the second confidence level threshold or a high-risk exclusion rule is hit, keep the user's original memory resource request value unchanged and generate a resource decision record containing the identifier of retaining the original value.

[0103] After completing the inference prediction and outputting the prediction result in step S130, step S140 uses the first quantile, second quantile, and confidence level in the prediction result as the core basis, combined with the user's original memory resource request value and preset risk rules, to make a comprehensive resource decision and determine the optimal resource strategy for the current job. This decision-making step aims to achieve the following three objectives: first, to minimize memory resource waste and make the resource request value closer to the actual demand; second, to avoid the job being forcibly terminated due to insufficient memory because of underestimation in the prediction; and third, to dynamically adjust the decision intensity according to the confidence level of the prediction—automatically rewriting when the confidence level is high, conservatively allowing only the suggested value when the confidence level is medium, and completely retaining the user's original value when the confidence level is low or there is a high risk. Specifically, S140 further includes: The steps extract three key parameters from the prediction results (PredictionRecord) output by S130: confidence, the first quantile, and the second quantile. The first quantile (e.g., P80, where 80% of historical samples show a true peak value lower than this) serves as a robust scheduling accounting reference, used to calculate the rusage[mem] value upon which the scheduling system bases its resource allocation and accounting calculations. The second quantile (e.g., P92 or P95) serves as a more conservative hard upper limit reference for jobs, used to calculate the mandatory memory limit -M value during job execution. Since the quantile corresponding to the second quantile is higher than the first quantile, the second quantile must be greater than the first quantile. This ensures that the job hard upper limit is never lower than the scheduling accounting value, complying with the scheduling system's configuration constraints.

[0104] The confidence score is calculated as follows: it is weighted and normalized to between 0 and 1 based on three dimensions: the number of hit historical samples (sample_count), the error of the last N backtests, and recent volatility. The number of hit historical samples reflects the robustness of the statistical foundation; a larger sample size indicates higher reliability of quantile statistics or model predictions. The error of the last N backtests reflects the historical performance of the prediction model on the recent validation set; a lower backtesting error indicates higher prediction accuracy. Recent volatility reflects the stability of memory consumption for similar tasks; lower volatility indicates more stable memory requirements and higher reliability of the prediction results. The three dimensions are weighted, summed, and normalized to obtain the final confidence score. A confidence score closer to 1 indicates a more reliable prediction result; a confidence score closer to 0 indicates a less reliable prediction result, requiring more conservative handling.

[0105] When the confidence level obtained in step S141 reaches the first confidence threshold (e.g., 0.85), and the current job does not hit the high-risk exclusion rule, the system executes the automatic rewriting strategy. The first confidence threshold is a system-preset high-confidence threshold; only when the prediction result reaches this threshold is it considered to have sufficient credibility to directly modify the user-submitted original resource request value. The high-risk exclusion rule is used to identify special jobs that are not suitable for automatic rewriting, including but not limited to: the tool to which the job belongs is marked as a high-fluctuation tool (extremely unstable memory consumption), the queue to which the job belongs is marked as a high-risk queue (high historical OOM rate), and the job type belongs to a special type (such as a critical task that is extremely sensitive to memory). Only when the confidence level is high enough and the above-mentioned high-risk factors do not exist will the automatic rewriting branch be entered.

[0106] In the automatic rewrite branch, the system multiplies the first quantile value predicted in step S130 by the first coefficient (coefficient A) to obtain the scheduling accounting memory value (i.e., the rusage[mem] value in LSF); and multiplies the second quantile value by the second coefficient (coefficient B) to obtain the job hard limit value (i.e., the -M value in LSF). The second coefficient is greater than the first coefficient to ensure that the job hard limit is always not lower than the scheduling accounting value.

[0107] The values ​​of the first coefficient A and the second coefficient B are not fixed constants, but are dynamically determined based on at least one of the following factors: the degree of instrument volatility, the stability of the queue, and the historical undervaluation rate. Specifically: For stable tools (such as compilation tools with relatively fixed memory consumption patterns), the value range of A is 1.05 to 1.20, and the value range of B is 1.20 to 1.50. Smaller coefficients mean that the predicted value itself is highly reliable, and only a small safety margin is needed to cover normal fluctuations. For highly volatile tools (such as simulation tools where memory consumption varies drastically with design size), the value of A ranges from 1.20 to 1.50, and the value of B ranges from 1.50 to 2.00. The larger coefficient provides a more adequate safety buffer for high uncertainty.

[0108] The stability of the queue and the historical underestimation rate further participate in the dynamic adjustment of the coefficients: if the historical OOM rate of the current queue is high, the base coefficient is appropriately increased; if the historical underestimation rate is low (i.e., the model's predictions for this type of operation are consistently accurate), the coefficient can be appropriately decreased. Through the above dynamic coefficient adjustment mechanism, the coefficient values ​​achieve a quantitative balance between reducing resource waste (lower coefficients) and ensuring operational safety (higher coefficients).

[0109] Furthermore, in the automatic rewrite branch, the system also needs to enforce capping constraints: the calculated scheduling accounting memory value and job hard limit value are compared with the user's original memory resource request value. If either value is greater than the user's original request value, the user's original request value is used as the final adopted value for that corresponding item, ensuring that the rewritten resource value will not exceed the user's initially declared resource limit, and avoiding the infinite amplification of resource requests due to abnormally high predicted values. Finally, the unique job identifier is associated to generate a resource decision record containing the automatic rewrite flag (rewrite_flag=auto_rewrite), which at least contains the final adopted scheduling accounting memory value (new_rusage) and job hard limit value (new_M).

[0110] When the confidence level obtained in step S141 is lower than the first confidence threshold but reaches the second confidence threshold (e.g., 0.60), the confidence level of the prediction result is at a moderate level. Although it is insufficient to support automatic rewriting, it still has some reference value, and the system executes a conservative release strategy. The core operation of the conservative release strategy is to record the predicted memory value (including the first and second quantile values) in the prediction result of S130 as a suggested value and write it into the corresponding field of the resource decision record. However, the resource request value of the job is not actually modified through the online modification command of the scheduling system, and the job still runs with the user's original request value. This strategy does not directly intervene in the resource allocation of the job, but it accumulates important process data for subsequent analysis and decision-making. If the actual peak memory after the job is significantly lower than the predicted suggested value, it indicates that the model prediction is too high, and the coefficients can be adjusted or the model optimized accordingly. If the actual peak memory after the job is close to or exceeds the predicted suggested value, it indicates that the model prediction is accurate, and the job could have reduced resource requests within a safe range, providing evidence to support raising the confidence level of this job type to the automatic rewrite threshold. By continuously collecting statistical data on the deviation between the predicted suggested value and the actual peak value under the conservative release strategy, the model's performance under conditions where the automatic rewrite confidence level has not been reached can be quantitatively evaluated. When the statistical results show that the model performs stably within the confidence level range, the first confidence threshold can be appropriately lowered to include more jobs in the automatic rewrite scope. A unique job identifier is associated to generate a resource decision record containing a conservative release flag (rewrite_flag=guarded). This record must contain at least the predicted suggested value and the decision reason (confidence lower than the first threshold but reaching the second threshold).

[0111] When the confidence level obtained in step S141 is lower than the second confidence threshold (i.e., the confidence level of the prediction result is too low, and even the conservative release condition is not met), or when the job hits the high-risk exclusion rule mentioned in step S142 (such as the job belonging to a high-volatility tool, a high-risk queue, or a special type of critical task), the system executes the original value retention policy. The original value retention policy is the most conservative decision branch. It does not adopt any prediction results or prediction suggestions, does not modify the resource request value of the job, and completely keeps the user's original memory resource request value (i.e., the final effective original memory request value collected in step S110) unchanged, allowing the job to run with the original resource value declared by the user when submitting.

[0112] Under conditions of insufficient prediction reliability or high-risk characteristics of the job itself, any modification based on the prediction may bring unforeseen risks. Underestimating the risk may lead to forced termination of the job, while overestimating, although not causing functional failure, will waste resources. Therefore, retaining the original user value is the safest choice in this situation. The original user value is judged based on user experience, and although it may be too large, it at least ensures that the job runs successfully and will not trigger OOM forced termination due to insufficient resources. Associate the job's unique identifier and generate a resource decision record containing the original value retention flag (rewrite_flag=keep_original). This record must at least contain the user's original memory resource request value and the decision reason (confidence below the second threshold or hitting the high-risk exclusion rule).

[0113] In summary, a three-tiered risk-stratified decision-making mechanism—step S141 (obtaining confidence and quantile values), step S142 (automatic rewriting, high confidence and no high risk), step S143 (conservative release, medium confidence, no rewriting), and step S144 (retaining the original value, low confidence or high risk)—achieves a quantitative balance between reducing resource waste and ensuring operational safety. Specifically, the automatic rewriting branch further ensures the safety and controllability of the rewriting operation through dual coefficients A / B (stable tools: A=1.05~1.20, B=1.20~1.50; high volatility tools: A=1.20~1.50, B=1.50~2.00) and a capping constraint (the final value does not exceed the user's original application value). The conservative release branch accumulates auditable process data for subsequent model optimization and threshold adjustment by recording the predicted suggested value without actually modifying the resource application. The retaining the original value branch maintains a completely conservative stance when the prediction is unreliable or carries high risk, prioritizing the successful operation. The above three-tiered strategy, using confidence level as a quantitative benchmark and risk rules as a safety boundary, together constitute a complete decision-making spectrum from active optimization to conservative observation, and then to complete retention.

[0114] In conjunction with the first aspect, step S150 includes: S151 uses the unique identifier of the job to monitor the queuing status of the job and obtain the current queuing status information of the job.

[0115] S152, when the current queuing status information indicates that the job is still in a queuing state and the preset triggering conditions are met, the historical sample library is retrieved using the stable retrieval key, and the memory requirements of the current job are predicted by reasoning to obtain the updated prediction result.

[0116] S153, make resource decisions based on the updated prediction results, and obtain updated resource decision records.

[0117] S154. Compare the scheduling accounting memory value in the updated resource decision record with the currently effective scheduling accounting memory value, and the job hard limit value in the updated resource decision record with the currently effective job hard limit value to obtain the resource value difference judgment result.

[0118] S155, when the resource value difference judgment result indicates that there is a difference, the scheduling accounting memory value and job hard limit value in the updated resource decision record are written into the job's resource configuration through the online modification command of the scheduling system, and the rewrite timestamp and correction log are recorded to generate the final resource record with the unique identifier of the associated job.

[0119] S156, When the resource value difference judgment result indicates that there is no difference, keep the currently effective resource value unchanged and generate the final resource record with the unique identifier of the associated job.

[0120] After step S130 completes the prediction and outputs the prediction result, and step S140 completes the resource decision and outputs the resource decision record, the job enters the scheduling system's queue to await resource allocation. However, there is an uncertain time window between job submission and actual start of execution, ranging from a few seconds to several hours or even days (depending on cluster load and queue priority). During this queuing period, the following three situations may occur: First, the prediction model used for inference in step S130 has been updated to a new version due to periodic retraining. Second, in step S160, other similar jobs successively ended and wrote back new real peak samples to the historical sample database, which changed the number of historical samples and statistical characteristics corresponding to the stable retrieval key of the current job. Third, the system maintenance personnel adjusted the relevant risk coefficients in the preset risk rules based on the overall operation status of the cluster.

[0121] The occurrence of any of the above situations means that the prediction results output in step S130 and the resource decisions made in step S140 may be outdated. Jobs that were previously predicted using sub-models or global fallback models due to insufficient samples may have met the conditions for first-level historical lookup during the queuing period as new samples return. Jobs that were previously conservatively released or retained due to insufficient confidence may have met the confidence requirements for automatic rewriting after the model is updated.

[0122] In step S151, using the unique job identifier (job_id) that runs throughout the entire process as an index, the system continuously queries the scheduling system (LSF) for the current status of the job to obtain the current queuing status information. The queuing status information at least indicates whether the job is currently in a PEND (waiting in the queue) or RUN (running) state. This monitoring is periodic; the system scans jobs still in the PEND state at preset time intervals (e.g., every 30 seconds or every minute) to check if any new triggering conditions are met. Monitoring continues until the job status changes from PEND to RUN or DONE / EXIT (abnormal exit). Step S151 is a prerequisite for subsequent dynamic correction: only when it is accurately known that the job is still in a queue can resource value rewriting be performed; if the job has already started running, the scheduling system generally does not allow or should not modify resource constraints, otherwise it may lead to abnormalities in the running job.

[0123] When the current queuing status information obtained in step S151 indicates that the job is still in the PEND state, this step further determines whether the preset triggering conditions are met. The preset triggering conditions include at least one of the following: the model version used to perform inference is updated (such as the sub-model or global fallback model in step S130 generating a new version due to periodic retraining); the number of newly added samples in the historical sample library corresponding to the stable retrieval key reaches the preset incremental threshold (such as more than 10 new true peak samples with the same semantic key are added, making the previously insufficient statistical quantiles sufficient); the risk coefficient in the preset risk rules changes (such as the operation and maintenance personnel adjusting the value of the first confidence threshold or the second confidence threshold).

[0124] When any of the above triggering conditions are met, the system determines that there is new information that is sufficient to change the previous prediction or decision result. At this time, the reasoning prediction step of step S130 is re-executed. The updated historical sample library is retrieved with the same stable retrieval key. The memory requirement of the current job is re-predicted through three levels of reasoning (step S131 first-level historical table lookup, step S133 second-level sub-model, step S134 third-level global fallback model) to obtain the updated prediction result (including the new first quantile value, second quantile value and confidence level).

[0125] Based on the updated prediction results obtained in step S152, step S153 performs resource decision-making (corresponding to the decision logic in S140): It obtains the confidence level, first quantile, and second quantile from the updated prediction results; based on the comparison results of the latest confidence level with the first and second confidence thresholds, and whether the high-risk exclusion rule is met, it re-determines the resource decision strategy: automatic rewriting, conservative approval, or retention of the original value; if the decision is automatic rewriting, the updated first quantile multiplied by the first coefficient is used as the new scheduling accounting memory value, and the updated second quantile multiplied by the second coefficient is used as the new job hard upper limit value, ensuring that the final value does not exceed the user's original request value under the capping constraint. Finally, an updated resource decision record is obtained, which includes the new scheduling accounting memory value and the new job hard upper limit value.

[0126] Subsequently, step S154 compares it with the resource values ​​currently in effect in the scheduling system: The job's current rusage[mem] value and -M value in the scheduling system (which may come from the user's original request value, or from a value rewritten during previous submission or the last queuing correction) are read and compared one by one with the updated decision value to obtain the resource value difference judgment result. This difference judgment result is used to indicate whether the new scheduling accounting memory value is consistent with the currently effective value, and whether the new job hard upper limit value is consistent with the currently effective value. Only when at least one value differs is it necessary to perform an actual online modification operation; if both are consistent, no modification is required to avoid unnecessary system calls and log recordings.

[0127] Subsequently, step S155 writes the updated resource values ​​into the job's resource configuration using online modification instructions provided by the scheduling system (such as the bmod instruction in the LSF system). The bmod instruction is an online job modification tool provided by LSF, which supports modifying the job's resource constraint parameters, including rusage[mem] (scheduling accounting memory value) and -M (job hard limit value), without interrupting the job queuing state. After executing the bmod instruction, the job resource configuration in the scheduling system is updated, and subsequent job scheduling will allocate resources and enforce restrictions according to the new resource values. After completing the online modification, step S155 records the modification timestamp (bmod_update_time) and correction log (correction_log). The correction log includes at least the triggering reason for the modification (such as model version update, new sample addition, or risk coefficient change), a comparison of resource values ​​before and after the modification, and the operation execution result. Finally, associate the job's unique identifier to generate the final resource record (FinalResourceRecord), which includes at least the job's unique identifier (job_id), the final effective scheduling accounting memory value (final_rusage), the final effective job hard cap value (final_M), the rewrite timestamp, the queuing status (pend_status), the model version number (model_version), and the correction log (correction_log).

[0128] When the resource value difference judgment result in step S154 indicates that there is no difference between the new scheduling accounting memory value and the new job hard limit value and the currently effective value, step S156 does not perform any online modification operation. Instead, it directly uses the currently effective resource value as the final resource value, associates it with the job unique identifier, and generates a final resource record (FinalResourceRecord). This record includes at least the job unique identifier (job_id), the currently effective scheduling accounting memory value, the currently effective job hard limit value, the current queuing status, and the model version number. This final resource record has the same data structure as the final resource record generated in S155, the only difference being whether it includes a rewrite timestamp and a correction log (these fields are empty or the record is "no change" when there is no rewrite).

[0129] In step S150, a complete closed loop is established, encompassing queuing status monitoring, trigger condition judgment, re-prediction, re-decision, difference comparison, and online modification / maintaining the status quo, enabling dynamic resource correction of the job during the queuing period. This mechanism effectively overcomes the shortcomings of traditional schemes where resource values ​​are fixed once submitted and cannot be corrected using the latest models and samples during the queuing period. It extends the correctable time window from being determined once at submission to being continuously adjustable during the queuing period, ensuring that the job always starts running with the current optimal resource values. Simultaneously, a gating mechanism with preset trigger conditions avoids unnecessary load on the scheduling system caused by excessively frequent re-prediction and rewriting. Finally, the resource record forms a complete data closed loop through the job's unique identifier, the original input object from the submission stage, semantic feature records, prediction results, resource decision records, and the actual result records from the subsequent S160 steps, supporting traceable auditing throughout the entire lifecycle.

[0130] Subsequently, when the job finishes execution in step S160, the scheduling system (LSF) automatically triggers the execution of the epilog script. Epilog is one of the job lifecycle hook mechanisms provided by LSF. It runs after the job exits but before the scheduling system releases the job context, making it an ideal time to collect actual job execution data. This step uses the job's unique identifier (job_id) as the primary key, linking it to the original input object generated in S110, the semantic feature record generated in S120, the prediction result generated in S130, the resource decision record generated in S140, and the final resource record generated in S150, ensuring that the collected data can be completely traced back to the entire lifecycle of the job.

[0131] Specifically, the epilog module collects the following key information from the scheduling system records and job execution logs: Peak memory (peak_mem), the maximum amount of physical memory actually consumed by the job during execution. This value is continuously sampled and recorded by the scheduling system or operating system kernel during job execution, and its maximum value is retained at the end of the job; Exit code (exit_code), the exit status code of the job process, used to determine whether the job completed normally or exited abnormally; OOM flag (oom_flag), indicating whether the job was forcibly terminated by the system due to insufficient memory. This flag reflects whether the rewritten resource request value is lower than the job's actual memory requirement and is a core indicator for assessing whether the prediction is underestimated; Runtime, the actual time taken for the job from start to finish, providing auxiliary information for subsequent analysis of the correlation between memory consumption and runtime; and the final effective scheduling accounting memory value (final_rusage) and job hard limit value (final_M), i.e., the resource constraint values ​​used by the job during actual execution as determined in S150. The above information is aggregated to form the TruthRecord.

[0132] Subsequently, step S160 writes the actual results back to the history sample store. During the write-back process, the system uses the stable retrieval key (semantic_key) as the index key of the sample and stores the semantic features, peak memory (peak_mem), exit code (exit_code), OOM flag (oom_flag), runtime (runtime), and training version number (training_version) of the job as a complete historical sample record in the history sample store.

[0133] Semantic features enable subsequent similar tasks to accurately recall the sample using a stable retrieval key; the true peak memory serves as a model training label and is the core data foundation for quantile statistics and sub-model training in step S130; the OOM flag and exit code are used to evaluate the safety and effectiveness of the prediction. If the OOM flag for a sample is true, it indicates that the task was forcibly terminated due to insufficient memory. This sample has important warning value for subsequent predictions and can be treated as a risk sample in model training; the training version number is used to identify which model version period the sample was generated in, facilitating subsequent performance comparison analysis between model versions.

[0134] It should be noted that the writing back of the historical sample library is a continuous accumulation process. One new sample is added after each job is completed, and the size of the historical sample library grows continuously as production operation time increases. When the prediction model in step S130 needs periodic retraining, the system retrains the sub-model and the global fallback model based on the incremental samples accumulated in the historical sample library, and updates the model version number, so that newly submitted jobs during the queuing period can obtain updated prediction results based on the latest model version. During this process, the OOM flag and exit code in each sample written back in step S160 serve two purposes during model training. First, they act as a basis for data quality screening. OOM samples, because they did not capture the true peak value (actual demand exceeds the hard upper limit, and the captured peak value is only the upper limit value, not the true demand), are removed or specially labeled before being used for model training to avoid training label distortion. Second, the OOM flag also serves as a core input for underestimating the rate of error. The system quantitatively evaluates the model's safety by statistically analyzing the proportion of OOM occurrences in rewritten jobs. When the OOM rate exceeds a preset threshold, an alarm is triggered, and further expansion of the automatic rewriting strategy is restricted. Through this mechanism, this method achieves data-driven continuous iteration. Each job operation contributes a piece of real data to subsequent predictions, and the prediction accuracy continuously improves with the accumulation of samples, while the utilization rate of cluster resources is also continuously optimized.

[0135] As an example, taking a nighttime DV regression job as an example, the user submits the command via LSF: bsub -q q_dv_regress_night -M 8G -R rusage[mem=8G] python3 sim.py -t regress_a -b build +define+SYN=1 seed=1234.

[0136] In S110, the esub interceptor intercepts the job submission request, reads the original submission command content (raw_command="python3 sim.py -t regress_a -b build +define+SYN=1 seed=1234"), queue name (q_dv_regress_night), the user's original requested scheduling and accounting memory value (request_rusage=8G), and the job hard limit value (request_M=8G) from the submission command, generates a temporary unique identifier as the job's unique identifier, and binds it to the job_id returned by LSF after successful submission to obtain the final job_id. Since the user directly specified the resource parameters in the command line and did not use a script file, the final effective original memory allocation value of 8G is determined based on the command line parameters.

[0137] In S120, the system performs semantic parsing on the submitted command: it removes volatile random parameters such as seed=1234 and replaces them with uniform placeholders, extracts key parameters -t regress_a (test case), -b build (compilation mode), and +define+SYN=1 (comprehensive switch), identifies the real business execution program as sim.py, and generates a stable search key sim.py|build|regress_a|SYN=1.

[0138] In S130, the system uses the stable search key `sim.py|build|regress_a|SYN=1` to search the historical sample database, finding 38 semantically similar historical samples, reaching the preset threshold (30 samples). The first-level historical lookup inference is then directly executed. The peak memory usage of the 38 samples is statistically analyzed using quantiles, yielding the first quantile (P80) = 720MB and the second quantile (P92) = 880MB. The confidence score is calculated by weighting the number of historical samples (38 samples), the error of the last N backtests, and recent volatility, resulting in a confidence score of 0.91.

[0139] In S140, the system obtains the confidence level (0.91) and quantile values ​​(P80=720MB, P92=880MB). If the confidence level reaches the first confidence threshold (0.85) and the high-risk exclusion rule is not hit, the automatic rewrite strategy is executed. The tool used for this job is a stable tool; taking the first coefficient A=1.10 and the second coefficient B=1.20, the scheduling accounting memory value is calculated to be 720×1.10≈792MB, and the job hard limit value is 880×1.20≈1056MB. The capping constraint is checked: 792MB<8G and 1056MB<8G, both of which do not exceed the user's original request value; the calculated value is used as the final adopted value. The unique identifier of the associated job is used to generate a resource decision record containing the automatic rewrite identifier.

[0140] In S150, during job queuing, the system continuously monitors the job status using the job's unique identifier. If the job is still in the queue and the model version is updated or the risk factor changes, the resource values ​​are rewritten to rusage[mem]=792MB and -M=1056MB using the LSF's bmod instruction. If the job has already started running, the rewriting is not performed, and the job runs with the resource values ​​that were rewritten at the time of submission.

[0141] In S160, after the job is completed, the epilog module collects the actual peak memory (peak_mem=710MB), exit code (exit_code=0), and OOM flag (oom_flag=false), generates a record of the actual results, and writes it back to the historical sample library with the stable search key sim.py|build|regress_a|SYN=1 as the index, so as to provide new training samples for the prediction of subsequent similar jobs.

[0142] This embodiment fully demonstrates that the method of this application uses the unique job identifier (job_id) as the primary key to concatenate the submitted input (S110), semantic features (S120), prediction results (S130), resource decisions (S140), final resource values ​​(S150), and actual peak values ​​(S160) into a traceable closed-loop data stream. After prediction and rewriting, the user's original 8G request was reduced to rusage≈792MB and -M≈1056MB, while the actual peak value was only 710MB. Without triggering OOM, the memory waste rate was reduced from approximately 67.5% (original 8G relative to 710MB) to approximately 15% (1056MB relative to 710MB).

[0143] The method described in this application has been validated in a production LSF cluster with approximately 2300 cores, primarily for nighttime regression queues such as q_dv_regress_night. Based on monitoring statistics, the actual results are as follows: (1) Memory waste rate is significantly reduced. Before the modification, the waste rate of the user's original application relative to the actual peak was about 67.5%, and the waste rate after rewriting using the application method in this paper was reduced to 15.6%; among which, the waste rate of the q_dv_regress_night queue was about 15.2%, and the memory resource utilization efficiency was significantly improved.

[0144] (2) Resource requests are closer to actual consumption. 60,820 reconciliation jobs have been completed, accounting for 89.1% of the 68,286 rewrite jobs; the cumulative recoverable request amount is about 511,292 GB, with an average recoverable amount of about 7.3 GB per job.

[0145] (3) The safety margin is well maintained. The underestimation rate in the monitoring statistics is about 4.1%, and the OOM risk rate is about 0.97%, which indicates that the method of this application is not simply to lower the application value, but to optimize resources under the constraints of confidence level and risk stratification - to prioritize the successful operation of the operation and then consider resource saving.

[0146] (4) The ability to continuously iterate has been verified. Training labels are formed through the epilog truth value collection mechanism of S160, and the model is continuously updated with production data; through the phased deployment strategy of the simulation verification stage of S210 and the restricted effective stage of S240, the model can be verified and scaled up in the production environment in stages, effectively reducing the operational risks caused by direct rewriting.

[0147] (5) It has the value of cross-queue and cross-project reuse. The method of this application is based on command semantics and historical truth value modeling, does not rely on manual rules of a single user, and can be migrated to other EDA tools, projects or queues for adaptation and application.

[0148] In conjunction with the first aspect, the method also includes: S210, Obtain resource decision records, which include rewrite decisions.

[0149] S220: Extract resource recommendation values ​​from the resource decision record to obtain the recommendation value record; keep the resource recommendation values ​​from being written into the job's resource configuration to obtain the simulation operation record.

[0150] S230: After the operation is completed, the recommended resource values ​​are compared with the actual resource consumption of the operation and the deviation is statistically analyzed to obtain the simulation verification results.

[0151] S240, when the simulation verification results meet the preset safety conditions, the rewrite decision is enabled for the operation within the preset range, and a restricted enable record is obtained.

[0152] S250 monitors the running status of jobs that have enabled rewriting decisions and obtains limited verification results.

[0153] S260, when the restricted verification result meets the preset security conditions, the rewrite decision in the resource decision record is enabled for all applicable operations, resulting in a fully enabled record.

[0154] After resource decision-making is completed in step S140, for strategies with automatic rewriting as the decision result, this application also provides a progressive deployment mechanism. Through three progressive stages—simulation verification, limited implementation, and full implementation—the deployment of the automatic rewriting strategy is transformed from direct full implementation to a gradual verification, batch rollout, and rollback-enabled secure release process. This preserves the integrity of the S140 decision logic while providing a phased verification security guarantee for the actual implementation of the decision result. Specifically: In step S210, the resource decision record is generated by step S140. Its content includes at least the automatic rewrite flag (rewrite_flag=auto_rewrite) and the calculated scheduling accounting memory value and job hard limit value. After obtaining this record, step S210 uses it as input for the subsequent simulation verification phase, rather than making it effective directly. In other words, step S210 and the automatic rewrite branch in step S140 are completely identical in terms of decision content; the difference is that the resource decision record obtained in step S210 enters the gradual deployment process rather than taking effect directly in the production environment.

[0155] Subsequently, the calculated scheduling accounting memory value and job hard limit value are extracted from the resource decision record generated in step S210 as resource suggestion values, forming a suggestion value record. Then, the system performs a crucial control operation: keeping this resource suggestion value from being written into the job's resource configuration. That is, although the resource decision has been completed, the system does not write the resource suggestion value into the job's actual resource configuration through online modification instructions from the scheduling system (such as the bmod instruction in LSF). The job still runs with the memory resource value originally requested by the user. This operation corresponds to the core feature of the simulation verification phase: "record only, do not take effect." This means fully executing the resource decision logic to generate the suggestion value and recording its complete decision chain, but prohibiting the suggestion value from having any impact on the actual operation of the job, ensuring that the job's actual operating environment is completely consistent with when the automatic rewriting strategy is not enabled. Through this operation, the system obtains a simulation operation record, which at least includes the unique identifier of the job in this simulation operation, the resource suggestion value, and the "not effective" status indicator.

[0156] After the job execution is completed and the actual peak memory reclamation in step S160 is finished, this step associates the suggested value record from step S220 and the actual result record from step S160 with the job's unique identifier. It compares the suggested resource values ​​(including the scheduling accounting memory suggested value and the job hard limit suggested value) with the job's actual resource consumption (i.e., the actual peak memory) one by one, calculating the deviation rate (e.g., the percentage by which the suggested value is overestimated or underestimated compared to the actual peak). Through the cumulative statistics of a large number of jobs, the system summarizes the key safety indicators for the simulation verification phase, including at least: underestimation rate (the proportion of jobs with suggested resource values ​​lower than the actual peak to the total number of simulated jobs); and OOM rate (the proportion of jobs forcibly terminated due to insufficient memory to the total number of simulated jobs). The above indicators are summarized to form the simulation verification results. These results are used to assess whether there are systemic safety risks in the current model and decision-making logic. If the underestimation rate or OOM rate exceeds a preset safety threshold, it indicates that the model has a systemic underestimation tendency and should not proceed to the next stage.

[0157] When the underestimation rate in the simulation verification result of step S230 is lower than the preset underestimation rate threshold (e.g., 5%) and the OOM rate is lower than the preset OOM rate threshold (e.g., 1%), the system determines that the simulation verification has passed and enters the restricted activation stage. The restricted activation stage is characterized by the fact that the system does not enable rewrite decisions for all jobs, but only for jobs within a preset range. This preset range can be a preset proportion of jobs (e.g., enabling rewrite only for 10% of jobs, subsequently expanding to 30%, 50% based on the effect), a preset portion of queues (e.g., enabling rewrite only for test queues or non-core business queues), or other system-preset activation range limitations. By limiting the activation range, even if the rewrite decision exposes problems not discovered in the simulation verification stage after actual activation, its impact is controlled within a limited scope, reducing production risks. This step outputs a restricted activation record, which at least includes the unique identifier of the job for this restricted activation, the activation range configuration, and the activation time.

[0158] During the restricted activation phase, the system continuously monitors the operational status of jobs with rewrite decisions enabled, using unique job identifiers. It focuses on tracking the following metrics: 1) the underestimation rate (the proportion of jobs whose actual peak value after rewrite exceeds the hard limit); 2) the OOM rate (the proportion of jobs forcibly terminated due to insufficient memory); and 3) the success rate of the rewrite operation itself. After sufficient restricted activation phase operation (e.g., several days or a certain number of rewritten job samples), the system summarizes the above metrics to form a restricted verification result. This result is used to evaluate the security and stability of the automatic rewrite strategy in a real production environment, providing a basis for deciding whether to extend the rewrite strategy to the entire system.

[0159] When the underestimation rate in the restricted verification result of step S250 is lower than the preset underestimation rate threshold and the OOM rate is lower than the preset OOM rate threshold, the system determines that the restricted verification has passed and enters the full implementation phase. The full implementation phase marks the official full deployment of the automatic rewriting strategy. The system enables the rewriting decision for all applicable jobs (i.e., jobs that meet the automatic rewriting conditions, such as jobs with a confidence level reaching the first confidence threshold and not hitting the high-risk exclusion rule): For newly submitted jobs that meet the conditions, the system executes a complete resource decision-making process to generate a rewriting decision, and actually modifies the job's resource configuration through the online modification instructions of the scheduling system, so that the job runs with optimized memory resource values. This step outputs a full implementation record, which at least includes a timestamp of full implementation and a description of the applicable scope.

[0160] Subsequently, as the job continues to run, the epilog mechanism in step S160 continuously collects the true peak values ​​and writes them back to the historical sample library, and the prediction model in step S130 is continuously optimized through periodic retraining, forming a continuous iterative cycle of prediction, rewriting, collection, and retraining.

[0161] In summary, the three-stage progressive deployment process (steps S210 to S260) – simulation verification, limited implementation, and full implementation – addresses the production risk of forced job termination due to underestimation when the predictive strategy is implemented directly to the full extent. The simulation verification stage (steps S210-S230) verifies the effectiveness of the decision recommendation value without affecting production; the limited implementation stage (steps S240-S250) verifies the safety and stability of the actual rewriting within a limited production flow; and the full implementation stage (step S260) expands the rewriting strategy to all applicable jobs after sufficient verification. This phased verification and rollout mechanism, together with the confidence gating and risk rules in step S140, constitutes a multi-level safety control system. Confidence gating controls whether a rewrite occurs at the single-job level (micro-control), while progressive deployment controls the scope and pace of rewriting effectiveness at the strategy deployment level (macro-control). The synergistic effect of these two mechanisms achieves the safety control objective of auditable and phased deployment.

[0162] Secondly, embodiments of this application provide a readable storage medium storing computer program instructions, which are read and executed by a processor to perform the above-described method.

[0163] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0164] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0165] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0167] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for adaptive memory resource management, characterized in that, The method includes: Obtain the job submission request and collect the submission data to generate a raw input object containing the job's unique identifier; Semantic parsing is performed on the submission commands in the original input object to generate stable retrieval keys, thereby obtaining semantic feature records associated with the unique identifier of the job; The historical sample library is retrieved using the stable retrieval key in the semantic feature record. The memory requirement of the current job is predicted by reasoning, and a prediction result associated with the unique identifier of the job is obtained. The prediction result includes at least the predicted memory value of the first quantile and the predicted memory value of the second quantile, as well as the corresponding confidence level. Based on the prediction results, the user's original memory resource request value in the original input object, and the preset risk rules, resource decisions are made to obtain resource decision records associated with the unique identifier of the job; During the job queuing period, the resource decision record is associated with the unique job identifier and the job status is continuously monitored. When the preset trigger condition is met, the resource value is adjusted according to the updated prediction result through the online modification instruction of the scheduling system to obtain the final resource record associated with the unique job identifier. After the job is completed, the final resource record is associated with the unique identifier of the job, the actual peak memory of the job is collected and the actual result record is generated, and the actual result record is written back to the historical sample library.

2. The method according to claim 1, characterized in that, The steps for obtaining job submission requests, collecting submission data, and generating raw input objects containing unique job identifiers include: The job submission request is intercepted to obtain the source of the raw data to be collected. The source of the raw data includes at least two of the following: bsub submission command, job script file, scheduling system parameter line in the script, user field, queue field, project field and submission time provided by the scheduling system. The submission command content, script path, queue name, user name, project name, and the user's original memory resource request value are read from the original data source to obtain a multi-source submission data set; When both the bsub submission command and the job script file contain memory resource request parameters, the user's original memory resource request value is determined according to the priority rules that are actually effective in the scheduling system, and the final effective original memory request value is obtained. The job identifier returned by the scheduling system for the job corresponding to the job submission request is used as the unique identifier of the job; when the job identifier has not been obtained before submission, a temporary unique identifier is generated as the unique identifier of the job, and after successful submission, the temporary unique identifier is bound to the job identifier returned by the scheduling system to obtain the final unique identifier of the job. Associate the final job unique identifier and output an original input object containing the job unique identifier, the submission command content, the script path, the queue name, the user name, the project name, the final effective original memory allocation value, and the submission time.

3. The method according to claim 1, characterized in that, The steps of semantically parsing the submission commands in the original input object and generating stable retrieval keys to obtain semantic feature records associated with the unique identifier of the job include: Read the original submission command and script file from the original input object to obtain the command content to be parsed; Extract at least one non-executable content from the interpreter declaration line, scheduling system parameter line, and environment variable setting statement from the command content to be parsed to obtain the valid command content; When the command header of the valid command content contains a script language wrapper, the script or parameters after the wrapper are read recursively until the actual business execution program is located, and the actual program path is obtained. The tool name and tool version corresponding to the real program path are identified based on at least one of the tool whitelist, path keywords or command words to obtain the tool identification result; Replace at least one variable parameter among the random number, timestamp, batch number or counter in the effective command content with a uniform placeholder, and extract and normalize at least one key parameter among the compilation mode, test mode, coverage switch, waveform switch, debug database switch, number of threads, input path, and project name to obtain a standardized command and feature word set. A stable search key is generated based on the tool name, the job type in the normalization command, and the normalized combination of the key parameters. Associate the job's unique identifier and output a semantic feature record that includes the job's unique identifier, the tool name, the tool version, the normalized command, the feature term set, the stable search key, and the job type.

4. The method according to claim 1, characterized in that, The steps of retrieving a historical sample database using a stable retrieval key from the semantic feature record, predicting the memory requirement of the current job through multi-level inference, and obtaining a prediction result associated with the unique identifier of the job include: The historical sample database is retrieved using the stable retrieval key as an index, and it is determined whether the number of hit historical samples reaches a preset threshold. If so, perform quantile statistics directly based on the actual peak memory of each of the historical samples that were hit, and obtain the predicted memory value of the first quantile and the predicted memory value of the second quantile. If not, based on at least one of the job type, tool name, or queue information in the semantic feature record, the system routes to the corresponding sub-model for memory prediction, thereby obtaining the predicted memory value of the first quantile and the predicted memory value of the second quantile.

5. The method according to claim 4, characterized in that, The step of routing to the corresponding sub-model for memory prediction based on at least one of the job type, tool name, or queue information in the semantic feature record, and obtaining the predicted memory value of the first quantile and the predicted memory value of the second quantile, further includes: When the sub-model coverage condition is not met, the global fallback model is called to perform memory prediction, and the predicted memory value of the first quantile and the predicted memory value of the second quantile are obtained. The sub-model coverage conditions include: the existence of a trained sub-model corresponding to the tool name and job type of the current job; the completeness of the input features of the current job; the number of training samples of the trained sub-model reaching a preset sample number threshold; and the prediction error of the trained sub-model being lower than a preset error threshold.

6. The method according to claim 1, characterized in that, Before the steps of retrieving the historical sample library using the stable retrieval key in the semantic feature record, predicting the memory requirement of the current job through inference, and obtaining the prediction result associated with the unique identifier of the job, the method further includes: Obtain the historical sample library, which contains the semantic features of the collected historical jobs, the actual peak memory, the running status information, the corresponding stable retrieval keys, and the model training labels; Logarithmically transform the true peak memory of each historical sample in the historical sample library, and use the logarithmically transformed memory value as the training label of the model to obtain the transformed historical sample library. Based on the job type, tool name, and queue information of each historical sample in the historical sample library, at least one sub-model and one global fallback model are trained to obtain a set of trained prediction models.

7. The method according to claim 1, characterized in that, The step of making resource decisions based on the prediction results, the user's original memory resource request value in the original input object, and preset risk rules to obtain a resource decision record associated with the unique identifier of the job includes: The confidence level, the first quantile, and the second quantile are obtained from the prediction result, wherein the second quantile is higher than the first quantile; the confidence level is calculated by weighting the number of hit historical samples, the error of the last N backtests, and the recent volatility and normalized to between 0 and 1; When the confidence level reaches the first confidence level threshold and the high-risk exclusion rule is not hit, the first quantile value multiplied by the first coefficient is used as the scheduling accounting memory value, and the second quantile value multiplied by the second coefficient is used as the job hard upper limit value, generating a resource decision record containing an automatic rewrite identifier; wherein, the second coefficient is greater than the first coefficient, and the first coefficient and the second coefficient are determined based on at least one of the tool volatility, queue stability and historical underestimation rate; When the confidence level is lower than the first confidence level threshold but reaches the second confidence level threshold, the predicted memory value in the prediction result is recorded but the resource request value of the current job is not modified, and a resource decision record containing a conservative release flag is generated. When the confidence level is lower than the second confidence level threshold or a high-risk exclusion rule is hit, the user's original memory resource request value remains unchanged, and a resource decision record containing a retain original value identifier is generated.

8. The method according to claim 1, characterized in that, The steps of adjusting resource values ​​based on the updated prediction results using online modification instructions from the scheduling system to obtain the final resource record associated with the unique identifier of the job include: The queuing status of a task is monitored using its unique identifier to obtain the current queuing status information of the task. When the current queuing status information indicates that the job is still in a queuing state and the preset triggering condition is met, the historical sample library is retrieved using the stable retrieval key, and the memory requirement of the current job is predicted by reasoning to obtain the updated prediction result; Based on the updated prediction results, resource decisions are made to obtain updated resource decision records; By comparing the scheduling accounting memory value in the updated resource decision record with the currently effective scheduling accounting memory value, and the job hard limit value in the updated resource decision record with the currently effective job hard limit value, a resource value difference judgment result is obtained. When the resource value difference judgment result indicates that there is a difference, the scheduling accounting memory value and job hard limit value in the updated resource decision record are written into the job's resource configuration through the online modification command of the scheduling system, and the rewrite timestamp and correction log are recorded to generate the final resource record associated with the unique identifier of the job. When the resource value difference judgment result indicates that there is no difference, the currently effective resource value remains unchanged, and a final resource record associated with the unique identifier of the job is generated.

9. The method according to claim 7, characterized in that, The method further includes: Obtain the resource decision record, which includes the rewrite decision; Extract resource suggestion values ​​from the resource decision records to obtain suggestion value records; keep the resource suggestion values ​​from being written into the job's resource configuration to obtain simulation operation records; After the operation is completed, the suggested resource values ​​are compared with the actual resource consumption of the operation and the deviation is statistically analyzed to obtain the simulation verification results. When the simulation verification results meet the preset security conditions, the rewrite decision is enabled for operations within the preset range, resulting in a restricted enable record; Monitor the running status of jobs that have enabled the rewrite decision, and obtain limited verification results; When the restricted verification result meets the preset security conditions, the rewrite decision in the resource decision record is enabled for all applicable jobs, resulting in a fully enabled record.

10. A storage medium, characterized in that, The storage medium stores computer program instructions, which, when read and executed by a processor, perform the method described in any one of claims 1 to 9.

Images