A simulation job scheduling method and system based on a hybrid expert model and a medium

Abstract

The application discloses a simulation job scheduling method and system based on a hybrid expert model and a medium, and relates to the field of data processing.In the method, the job parameters of a simulation job to be scheduled and host state information of computing nodes in a cluster are acquired; a feature vector representing job features and cluster states is generated based on the job parameters and the host state information; the feature vector is input into a routing module in the hybrid expert model to obtain the probability and probability distribution of the simulation job relative to a plurality of preset subject experts; the information entropy of the probability distribution is calculated, and the information entropy is compared with a preset information entropy threshold; if it is determined that the information entropy is less than the preset information entropy threshold, a target subject expert with the highest probability is determined from the plurality of preset subject experts to obtain a scheduling decision; and the scheduling decision is sent to a job executor on a target host, so that the job executor executes the simulation job according to the scheduling decision.Implementation of the technical solution provided by the application improves the accuracy of the scheduling decision.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a simulation job scheduling method, system, and medium based on a hybrid expert model. Background Technology

[0002] With the rapid development of scientific computing and engineering simulation, high-performance computing clusters have become a key infrastructure supporting cutting-edge scientific research and industrial innovation. In these cluster environments, a large number of simulation jobs of different types are submitted concurrently. How to efficiently schedule these jobs to maximize resource utilization and shorten user waiting time is a core challenge for cluster management systems. Traditional job scheduling methods typically rely on manually specified resource requirements from users. However, users often request resources far exceeding actual needs due to conservative estimates, or underestimate resource requirements due to a lack of experience, leading to job failures. This scheduling method, based on static resource requirements and user experience estimations, cannot perceive the actual operational characteristics of jobs, resulting in rigid resource occupancy on computing nodes, significant resource waste, and low overall scheduling efficiency.

[0003] To address the efficiency issues of traditional scheduling methods, a predictive scheduling scheme based on machine learning has been proposed in existing technologies. This scheme trains a general machine learning model, such as a deep neural network, by collecting historical job execution data. When scheduling new jobs, this model can predict their potential resource requirements and execution time based on the new job's parameters. The scheduler can then utilize this predictive information to formulate more refined and intelligent scheduling strategies, such as inserting jobs predicted to have short execution times into the idle time slices of larger jobs, thereby improving cluster resource utilization and job turnaround efficiency to some extent.

[0004] However, the aforementioned approach using a general model still faces technical challenges in practical applications. Simulation jobs running in high-performance computing clusters typically originate from diverse disciplines, such as computational fluid dynamics, molecular dynamics, weather forecasting, and financial modeling. These different disciplines exhibit significant differences in computational patterns, memory access behaviors, and dependence on network communication, resulting in marked heterogeneity in their operational characteristics and resource requirements. Using a single, general-purpose machine learning model to learn the complex patterns of all job types can lead to the model sacrificing prediction accuracy in specific domains to accommodate all job types, resulting in lower accuracy in scheduling decisions. Summary of the Invention

[0005] This application provides a simulation job scheduling method, system, and medium based on a hybrid expert model, which improves the accuracy of scheduling decisions.

[0006] A first aspect of this application provides a simulation job scheduling method based on a hybrid expert model. The method includes: acquiring job parameters of a simulation job to be scheduled and host status information of computing nodes in a cluster; extracting application features of the simulation job based on the job parameters and the host status information, and generating a feature vector characterizing the simulation job and the cluster status; inputting the feature vector into a routing module in a hybrid expert model, and having the routing module output the probability of the simulation job relative to multiple preset subject experts, and determining the probability distribution of the simulation job relative to the multiple preset subject experts based on each probability, wherein the preset subject experts are those in the hybrid expert model that are relevant to various types of simulations. The deep learning model constructed for the task; the information entropy of the probability distribution is calculated and compared with a preset information entropy threshold; if the information entropy is determined to be less than the preset information entropy threshold, the target subject expert with the highest probability is determined from multiple preset subject experts; the subject history feature vector corresponding to the target subject expert is obtained, and the feature vector and the subject history feature vector are input to the target subject expert to obtain a scheduling decision, the scheduling decision including the target host for executing the simulation task, the resource allocation scheme, and the task execution time; the scheduling decision is sent to the task executor on the target host, so that the task executor executes the simulation task according to the scheduling decision.

[0007] By employing the above technical solutions, the simulation job's parameters and the host status information of the computing nodes are obtained. Application characteristics of the simulation job are extracted, and feature vectors are generated, comprehensively characterizing the resource requirements and computational features of the simulation job. Utilizing the routing module in the hybrid expert model, the probability of the simulation job relative to each preset subject expert is calculated, forming a probability distribution. This allows for precise matching of the simulation job with the subject expert most proficient in handling that type of job, improving the professionalism and relevance of scheduling decisions. By calculating the information entropy of the probability distribution and comparing it with a preset threshold, it can be determined whether the current job falls within the knowledge blind spots of existing subject experts. For routine jobs with low entropy values, selecting the subject expert with the highest probability for scheduling decision generation fully leverages the expert's experience in that specific domain, improving resource utilization efficiency and job performance. For new domain jobs with high entropy values, a new expert discovery mechanism needs to be activated to dynamically expand the knowledge base of the hybrid expert model to address unknown job domains. Finally, the scheduling decision is sent to the job executor on the target host, ensuring that the simulation job is executed strictly according to the optimized scheduling scheme, fully utilizing the computing power of the distributed environment. The aforementioned technical solution introduces a dynamic routing mechanism based on a hybrid expert model, combining job feature extraction, information entropy control, and subject-specific expert model selection to accurately match the resource requirements and computing node states of heterogeneous simulation jobs. This solves the problems of insufficient prediction accuracy, resource waste, low scheduling efficiency, and poor system adaptability inherent in traditional general-purpose models. The technical effect is a significant improvement in job scheduling accuracy, resource utilization, and system adaptability, while reducing resource fragmentation and user intervention requirements, thereby achieving efficient and intelligent cluster management.

[0008] Optionally, the step of extracting application features of the simulation job based on the job parameters and the host status information, and generating a feature vector representing the simulation job and cluster status, specifically includes: parsing the job parameters to obtain user-specified basic resource requests, and extracting subject keywords representing the computation mode of the simulation job from the command line parameters of the simulation job through keyword matching to form an initial subject identifier. The basic resource requests include the number of CPU cores, the amount of memory, the number of GPU cards, and the amount of GPU memory. The initial subject identifier is input into a preset word embedding model to obtain a subject identifier embedding vector. Based on the initial subject identifier, a set of historical jobs matching the simulation job is retrieved from the historical job log database, and the resource consumption data of the historical job set is statistically aggregated to generate historical log features. The historical log features include average iteration time, peak memory usage, and total inter-node communication traffic. The basic resource requests, the subject identifier embedding vector, and the historical log features are vectorized and concatenated to obtain the feature vector.

[0009] By employing the aforementioned technical solutions, the system parses job parameters to obtain user-specified resource requirements and extracts keywords related to the calculation mode from command-line parameters to form initial subject identifiers. This allows for direct retrieval of the subject affiliation information of the simulation job from explicit job attributes. Using a word embedding model to map the subject identifiers into dense vectors transforms textual information into a numerical form easily processed by neural networks, while also uncovering semantic correlations between identifiers. By retrieving historical jobs similar to the current job and aggregating their resource consumption data to form statistical features, the system can extract patterns in resource requirements for this type of job from historical experience data, compensating for discrepancies between user-requested resources and actual needs. Finally, concatenating these features into a unified vector representation creates a comprehensive and refined simulation job profile, providing a reliable basis for subsequent precise scheduling. Furthermore, this automated feature engineering process significantly improves the usability of the scheduling system, allowing users to achieve better scheduling performance without manually setting various tuning parameters.

[0010] Optionally, the step of inputting the feature vector into the routing module of the hybrid expert model and having the routing module output the probability of the simulation task relative to multiple preset subject experts specifically includes: inputting the feature vector into a multilayer perceptron network built into the routing module, and performing a nonlinear transformation through at least one hidden layer using the ReLU activation function to generate an intermediate hidden state vector, the intermediate hidden state vector including higher-order combination features of the feature vector; inputting the intermediate hidden state vector into the Gumbel-Softmax layer of the multilayer perceptron network to obtain a logistic value vector, the dimension of the logistic value vector being a preset number, the preset number being the number of preset subject experts; performing a preset transformation on the logistic value vector to obtain a weight vector, the dimension of the weight vector being the preset number, the sum of the values ​​of each dimension of the weight vector being one, the preset transformation including a preset noise transformation and a normalized exponential function transformation; and using the dimension values ​​of the weight vector as the probability of the preset subject expert.

[0011] By employing the above technical solution and using a multilayer perceptron network in the routing module, nonlinear transformation of feature vectors can be achieved, extracting high-order interaction patterns inherent in the original features and enhancing feature representation capabilities. Specifically, the ReLU activation function preserves feature diversity while preventing gradient vanishing, accelerating network training convergence. The Gumbel-Softmax layer transforms continuous hidden states into discrete weight distributions, ensuring that this distribution is differentiable to support end-to-end gradient backpropagation training. Through pre-defined noise transformation and normalized exponential transformation, certain perturbation factors are introduced while ensuring the legitimacy of the weight distribution, preventing the routing strategy from prematurely converging to suboptimal local solutions. The final output probability vector intuitively presents the differences in the suitability of experts across disciplines, providing a quantitative indicator for determining the domain affiliation of simulation tasks. Furthermore, this carefully designed weight vector possesses a clear probabilistic meaning and can be seamlessly integrated with information entropy measurement to construct novel routing optimization objectives.

[0012] Optionally, the step of inputting the feature vector and the subject history feature vector to the target subject expert to obtain a scheduling decision specifically includes: concatenating the feature vector and the subject history feature vector to form a combined input vector; inputting the combined input vector into a deep neural network built into the target subject expert, the deep neural network containing at least one fully connected hidden layer; performing a nonlinear transformation on the combined input vector through the fully connected hidden layer of the deep neural network to obtain a high-dimensional feature representation; and determining the scheduling decision based on the high-dimensional feature representation through a multi-task learning output layer built into the deep neural network, the multi-task learning output layer including a node selection head, a resource allocation head, and a runtime estimation head.

[0013] By employing the aforementioned technical solutions, deep neural networks are used in target subject experts for scheduling decision generation, enabling the modeling of complex nonlinear mapping relationships between job characteristics and scheduling strategies. The use of multiple fully connected hidden layers allows for hierarchical representation learning of combined input features, capturing key information within the scheduling patterns. The introduction of a multi-task learning paradigm simultaneously optimizes sub-tasks such as node selection, resource allocation, and runtime estimation. Knowledge transfer and sharing among sub-tasks significantly improve the overall generalization performance of the task. Specifically, the node selection head rationally allocates the optimal target host to the job based on job characteristics and node status, fully considering the differences in computing power between nodes in heterogeneous environments. The resource allocation head meticulously plans the allocation of CPU, memory, and other resources for the job, improving the utilization efficiency of shared resources while meeting the actual needs of the job. The runtime estimation head estimates the time required to complete the job, providing a longer-term optimization target for job scheduling. By integrating collaborative decision-making across multiple expert sub-tasks, the execution quality of the job can be improved from multiple perspectives while ensuring scheduling feasibility.

[0014] Optionally, the scheduling decision includes the target host for executing the simulation job, the resource allocation scheme, and the job execution time. The step of determining the scheduling decision based on the high-dimensional feature representation and through the multi-task learning output layer built into the deep neural network specifically includes: outputting the normalized allocation probability of the simulation job on each computing node through the node selection head, and selecting the computing node with the highest allocation probability as the target host; outputting the type and quantity of resources allocated to the simulation job through the resource allocation head to obtain the resource allocation scheme; and outputting the estimated execution time of the simulation job through the execution time prediction head to obtain the job execution time.

[0015] By adopting the above technical solutions, and outputting the normalized allocation probability of each computing node through the node selection header, the target node with the best overall performance can be selected based on a quantitative evaluation of the relative merits of each node, avoiding resource waste and uneven load caused by blind random selection. The resource allocation header outputs a quantitative allocation scheme for resources such as CPU and memory, which can maximize the satisfaction of the job's demand for critical resources and reconcile resource competition conflicts between different jobs. Simultaneously, fine-grained customized resource allocation can reduce fragmented resources on nodes and improve resource utilization. The runtime estimation header can consider multiple factors such as job complexity, resource requirements, and node capabilities to provide a realistic runtime estimate, helping the scheduling system better control job completion time and rationally plan scheduling queues and node resource usage. Organically combining the various decisions of the multi-task learning output layer can form a globally coordinated scheduling scheme that maximizes job performance while considering different scheduling objectives, ensuring that the workload is completed on time and with quality.

[0016] Optionally, after sending the scheduling decision to the job executor on the target host, causing the job executor to execute the simulation job according to the scheduling decision, the method further includes: after the simulation job is completed, obtaining the actual runtime and actual resource consumption of the simulation job; comparing the actual runtime with the expected runtime in the scheduling decision, and calculating the prediction error; based on the prediction error, calculating the scheduling loss through a preset loss function, and using the scheduling loss to perform gradient updates on the network parameters of the target subject expert through a backpropagation algorithm.

[0017] By adopting the above technical solutions, the actual runtime and resource consumption data obtained after job execution can be used to objectively evaluate the actual effectiveness of scheduling decisions. Quantifying the deviation between the predicted and actual values ​​as prediction error can measure the current predictive capabilities of subject matter experts. Constructing a reasonable loss function based on the prediction error can guide subject matter experts to optimize their internal prediction models and minimize errors. Utilizing the backpropagation algorithm to update the scheduling loss with the gradient of network parameters allows the subject matter expert model to adapt online to new workload characteristics, continuously learning and improving from practice. Simultaneously, this feedback optimization process effectively bridges the gap between offline training data and the distribution of actual online jobs, improving the model's robustness to dynamic environmental changes. Timely transmission of execution feedback information to the upper-level scheduler enables a closed-loop self-optimization process through reinforcement learning, continuously improving the scheduling performance of the hybrid expert model. Furthermore, summarizing real-time feedback data into the model training set can continuously expand the model's knowledge capacity and optimize long-term scheduling benefits.

[0018] Optionally, after comparing the information entropy with a preset information entropy threshold, the method further includes: if it is determined that the information entropy is greater than or equal to the preset information entropy threshold, then the simulation job is determined to be a new domain job; a preset default expert template is obtained, and the network topology and initial weight parameters of the preset default expert template are copied to construct a new expert instance in the hybrid expert model; the new domain job is assigned to the new expert instance, an exploration scheduling decision is generated, and an incremental learning data queue is generated for the new expert instance, and the job feature vector of the new domain job is stored in the incremental learning data queue as the first sample; after the simulation job is completed according to the exploration scheduling decision, the corresponding execution performance index is obtained, and the execution performance index is associated with the job feature vector to construct a training sample, which is stored in the incremental learning data queue; when it is determined that the number of training samples in the incremental learning data queue reaches a preset number, the incremental learning process is started, and the network parameters of the new expert instance are iteratively optimized based on the incremental learning data queue.

[0019] By employing the aforementioned technical solution, and by determining whether the information entropy of the probability distribution exceeds a preset threshold, the current task can be automatically identified as belonging to a completely new and unknown domain. For this new domain task, a new subject expert is copied and instantiated from the preset default expert template. This allows for the rapid expansion of the hybrid expert model to cover the new task domain without affecting existing experts. The new expert instance is responsible for generating exploratory scheduling decisions. Although this inevitably involves some degree of randomness, it serves as a beneficial attempt, accumulating a data foundation for subsequent optimization. Simultaneously, the timely creation of an incremental learning data queue for the new expert and the recording of feedback on the effectiveness of the exploratory decisions ensure that the expert can subsequently improve the scheduling strategy in a targeted manner. Once sufficient new domain samples have been accumulated, incremental learning is initiated, enabling the new expert to quickly learn the scheduling characteristics of the new domain based on the default template, completing the transition from novice to expert. The entire adaptive learning process continuously expands the hybrid expert model's coverage of unknown tasks and, through the mechanism of selectively activating experts, balances exploration and utilization. This steadily improves the scheduling level of existing domains while also providing strong support for exploring new domains.

[0020] 8. Optionally, the method further includes the step of constructing the hybrid expert model, wherein constructing the hybrid expert model specifically includes: obtaining a historical job dataset, wherein each job sample in the historical job dataset contains job parameters, job category, and actual resource consumption and actual runtime of executing the job sample; based on the job category, dividing the historical job dataset into multiple subject subsets corresponding to the preset subject experts, with one preset subject expert corresponding to one subject subset; for each subject subset, training a deep neural network as the corresponding preset subject expert; after training, fixing the network parameters of all preset subject experts, and training the routing module using the historical job dataset; and combining the trained routing module with the trained preset subject experts to form the hybrid expert model.

[0021] By adopting the above technical solution and building an offline training process for a hybrid expert model based on historical job datasets, prior scheduling experience can be fully utilized to provide a reliable initial strategy for online scheduling. Dividing the training set by job category yields homogeneous subsets corresponding to experts in each discipline, reflecting the personalized characteristics of different experts' areas of expertise and improving the consistency of internal expert decisions. Training corresponding expert networks for each discipline subset encourages experts to form relatively complete knowledge representations in their respective fields, avoiding knowledge confusion and conflicts in optimization objectives between different specialties. After all experts have been trained, the routing module is trained using the global job distribution, allowing the routing mechanism to fully adapt to the relative weight of different jobs and balance the overall activation probability of each expert. Simultaneously, fixing expert parameters and individually optimizing the routing strategy enables knowledge distillation of the expert network, condensing key information for judging jobs in each domain into the routing module. Combining the trained routing module and discipline experts into a complete hybrid expert model forms a domain-adaptive intelligent decision-making framework for simulation job scheduling, providing a solid knowledge base and flexible strategy optimization capabilities for online scheduling.

[0022] Secondly, embodiments of this application provide a simulation job scheduling system based on a hybrid expert model. The simulation job scheduling system based on a hybrid expert model includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the simulation job scheduling system based on the hybrid expert model to perform the method described in the first aspect and any possible implementation thereof.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a simulation job scheduling system based on a hybrid expert model, cause the simulation job scheduling system based on a hybrid expert model to perform the method described in the first aspect and any possible implementation thereof.

[0024] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a simulation job scheduling system based on a hybrid expert model, cause the simulation job scheduling system based on a hybrid expert model to execute the method described in the first aspect and any possible implementation thereof.

[0025] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages:

[0026] 1. By introducing a dynamic routing mechanism based on a hybrid expert model, combined with job feature extraction, information entropy control, and subject-specific expert model selection, the resource requirements and computing node states of heterogeneous simulation jobs are accurately matched. This solves the problems of insufficient prediction accuracy, resource waste, low scheduling efficiency, and poor system adaptability inherent in traditional single models. The technical effect is a significant improvement in job scheduling accuracy, resource utilization, and system adaptability, while reducing resource fragmentation and user intervention requirements, thereby achieving efficient and intelligent cluster management. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a simulation job scheduling method based on a hybrid expert model disclosed in an embodiment of this application;

[0028] Figure 2 This is another flowchart illustrating a simulation job scheduling method based on a hybrid expert model disclosed in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the structure of a simulation job scheduling system based on a hybrid expert model provided in an embodiment of this application.

[0030] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0032] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0033] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0034] This application provides a simulation job scheduling method based on a hybrid expert model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a simulation job scheduling method based on a hybrid expert model, provided in an embodiment of this application. The method is applied to a system, which can execute a simulation job scheduling program based on a hybrid expert model. The method includes steps S101 to S107, as follows:

[0035] Step S101: Obtain the job parameters of the simulation job to be scheduled and the host status information of the computing nodes in the cluster.

[0036] In step S101, a simulation job refers to a specific task that requires computation to simulate real-world processes, such as a structural simulation task to analyze the stress on a bridge during an earthquake. Job parameters represent data describing the specific requirements and attributes of the simulation job, indicating its computational characteristics, such as the number of processor cores requested, the required memory size, the estimated runtime, and its application domain, such as structural mechanics. A computing node refers to each independent server in a high-performance computing cluster that actually performs the computational task, providing physical resources such as processors and memory. Host status information represents data on the resource usage and workload of each computing node at a specific moment, indicating the node's real-time availability, such as the node's CPU utilization, remaining available memory, current network throughput, and disk I / O load.

[0037] Specifically, when a simulation job enters the scheduling system via a user submission request, the job management service on the server immediately performs a data acquisition operation. This service parses the description file submitted with the simulation job, precisely extracting all job parameters. For example, the system identifies this as a job named Bridge_Seismic_Test, with an application type of structural mechanics, requesting 128 processor cores and 512 gigabytes of memory. Simultaneously, the scheduling service on the server accesses a pre-configured system storage. This system storage holds host status information for all compute nodes, periodically reported by an independent monitoring service and accompanied by timestamps. The scheduling service queries this system storage, retrieving the record with the timestamp closest to the current scheduling time for each compute node in the cluster, such as Node-01 and Node-02. In this way, the system obtains the latest status of Node-01 as 80% CPU utilization and 100 gigabytes of available memory; and the latest status of Node-02 as 5% CPU utilization and 900 gigabytes of available memory. The server will aggregate all collected job parameters and host status information of each node to form a raw data set for the next step of processing.

[0038] Step S102: Based on job parameters and host status information, extract application features of the simulation job and generate feature vectors representing the simulation job and cluster status.

[0039] In step S102, application features refer to the digital information extracted from the original job parameters that can be understood and processed by the model, reflecting the inherent attributes of the job at a deeper level. A feature vector is a structured dataset composed of multiple numerical values, combining the application features of the simulation job with the overall cluster status information to form a unified feature vector usable by the machine learning model. Cluster status refers to a global view describing the current resource status of the entire computing environment, composed of the host status information of all computing nodes in the cluster.

[0040] Specifically, after acquiring the raw data, the system initiates a feature engineering process to generate a feature vector. First, the system processes the non-numerical information in the job parameters. For example, for the application type of structural mechanics, the system uses an encoding method, such as one-hot encoding, to convert it into a numerical vector, for example, [0, 1, 0], where each bit corresponds to a specific subject area. Then, the system concatenates this converted application feature with other numerical parameters of the job, such as the requested 128 cores and 512 gigabytes of memory, and host status information collected from all computing nodes, such as the CPU utilization and available memory of each node, in a predefined order. To eliminate the influence of different units and magnitudes between different features, the system also normalizes the concatenated long vector, for example, scaling all values ​​to the range of 0 to 1. After this series of processing steps, the system finally generates a high-dimensional feature vector that contains both a detailed profile of the job to be scheduled and a real-time snapshot of the entire cluster.

[0041] In one possible implementation, based on job parameters and host status information, application features of the simulation job are extracted to generate feature vectors representing the simulation job and cluster status. Specifically, this includes steps S1021-S1024, as follows:

[0042] Step S1021: Parse the job parameters to obtain the user-specified basic resource requests, and extract subject keywords representing the computing mode of the simulation job from the command line parameters of the simulation job through keyword matching to form an initial subject identifier. The basic resource requests include the number of CPU cores, the amount of memory, the GPU card, and the amount of GPU memory.

[0043] In step S1021, the job parameters represent a set of complete information describing the simulation job, provided by the user when submitting the job. Basic resource requests refer to the most basic hardware resource specifications required for the job to run; this is a core subset of the job parameters, representing the job's basic requirements for computing node hardware. For example, a job might explicitly require 16 CPU cores, 64 gigabytes of memory, one graphics processor, and 2 gigabytes of graphics processor memory. Command line parameters are the complete instruction strings entered by the user to start the simulation program. They include the path to the executable file, runtime flags, and the names of input files, serving as an important basis for analyzing the specific behavior of the job. Keyword matching is a text analysis technique that identifies and extracts specific information by searching a predefined list of keywords within a text string. Computational mode refers to the typical computational characteristics exhibited by the simulation job during runtime, such as whether it is computationally intensive, memory-intensive, or communication-intensive, determining which hardware configuration the job is more sensitive to. Subject-specific keywords are specific terms that directly or indirectly indicate the scientific or engineering field to which the simulation job belongs; these are usually the names of commonly used simulation software or abbreviations of specific algorithms. The initial subject identifier refers to the raw subject keywords extracted directly from the command line parameters through keyword matching, without any conversion, which serves as the starting point for subsequent processing.

[0044] Specifically, when a simulation job is submitted to the system, the system first launches a parsing program to process the submission script. This program scans the script line by line, locating the section declaring resources. The system precisely extracts the values ​​to form a basic resource request: 32 CPU cores, 128 gigabytes of memory, 2 GPUs, and 4 gigabytes of graphics card memory. Simultaneously, the system finds the line of code in the script that actually executes the simulation program—the command-line arguments. Next, the system loads a pre-configured subject-specific keyword dictionary, which associates keywords with subject areas; for example, `vasp` corresponds to materials science. The system compares each keyword in this dictionary with the command-line arguments. When it finds that the string `vasp_std` contains the keyword `vasp` in the command-line arguments, a match is found. The system then records this found keyword `vasp` as the initial subject identifier for the job.

[0045] Step S1022: Input the initial subject identifier into the preset word embedding model to obtain the subject identifier embedding vector.

[0046] In step S1022, the initial subject identifier refers to the original text keywords extracted from the command-line parameters in the previous step, representing the application domain of the assignment. The pre-trained word embedding model is a pre-trained deep learning model that converts words from their textual form into a dense numerical vector. Its core function is to capture the semantic relationships between words; for example, a good model will map similar software names like ANSYS and Abaqus to similar positions in the vector space. The subject identifier embedding vector is a numerical vector generated by the word embedding model to represent the initial subject identifier in a multi-dimensional space. This vector encodes the computational characteristics of the subject domain in a machine-understandable way.

[0047] Specifically, after obtaining the initial subject identifier (the text string "vasp") generated in the previous step, the system invokes an internal vectorization module. This module loads a pre-defined word embedding model from disk. This model is likely trained on a dataset containing a large amount of scientific computing literature, software manuals, and historical assignment scripts, thus understanding the deeper meanings of words like "vasp," "gromacs," and "OpenFOAM." The system takes the string "vasp" as input and passes it to this loaded word embedding model. The neural network inside the model processes this input through a series of complex matrix multiplications and table lookup operations, ultimately outputting a fixed-length array of floating-point numbers. For example, if the model's embedding dimension is set to 128, the output subject identifier embedding vector will be a vector containing 128 values, such as [0.23, -0.11, 0.87, ..., -0.45]. This vector now becomes a quantified and abstract mathematical expression of the computational patterns in the field of materials science.

[0048] Step S1023: Based on the initial subject identifier, retrieve the set of historical assignments that match the simulation assignments from the historical assignment log database, and statistically aggregate the resource consumption data of the historical assignment set to generate historical log features. The historical log features include average iteration time, peak memory usage, and total communication traffic between nodes.

[0049] In step S1023, the initial subject identifier is used as an index key for a database query to locate similar historical jobs. The historical job log database is a long-running and maintained structured database storing detailed records of all completed jobs on the cluster, providing a data foundation for predicting future job behavior. The historical job set is a subset of job records retrieved from the historical job log database using the initial subject identifier as a filtering criterion; all jobs in this set are considered similar to the currently scheduled job in terms of computational patterns. Resource consumption data refers to performance metrics actually measured and recorded by the monitoring system during the execution of historical jobs, reflecting the actual resource usage of the jobs. Statistical aggregation refers to summarizing and calculating a set of numerical data to derive one or a few statistics that represent the overall characteristics of the data set, such as calculating the average, maximum, or median. Historical log features are a set of numerical values ​​generated by statistically aggregating the resource consumption data of the historical job set, quantifying the possible resource consumption behavior of the current job from a historical experience perspective. Average iteration time, peak memory usage, and total inter-node communication traffic are specific metrics that constitute the characteristics of historical logs, representing the job's computation speed, memory requirement limit, and parallel communication overhead, respectively.

[0050] Specifically, the system uses the initial subject identifier `vasp` determined in the previous step to initiate a query request to the historical job log database. The logic of this request is to return all job records in the database whose command-line arguments contain 'vasp'. After the database executes the query, it returns a set containing hundreds or even thousands of historical job records; this is the historical job set. The system then iterates through each record in this set. For each record, the system extracts the resource consumption data stored within, specifically including the average time spent per iteration, the maximum memory usage reached during the entire run, and the total amount of data transferred between nodes during multi-node runtime. After iterating through all matching historical jobs and collecting these three metrics for each, the system initiates statistical aggregation calculations. It sums all collected iteration times and divides by the number of jobs to obtain the average iteration time. It finds the largest of all collected peak memory usages as the final peak memory usage. It averages all collected inter-node communication traffic to obtain a representative total communication traffic. Ultimately, the system generates a historical log feature consisting of these three statistical values, such as [250.5 seconds, 210.3 gigabytes, 450.7 gigabytes].

[0051] Step S1024: Vectorize and concatenate the basic resource request, subject identifier embedding vector, and historical log features to obtain the feature vector.

[0052] In step S1024, the basic resource request refers to the quantity of hardware resources required directly parsed from the job parameters submitted by the user. The subject identifier embedding vector refers to a numerical vector representing the subject area of ​​the job, generated through a word embedding model. The historical log feature refers to the predicted quantity of resources when performance is good, calculated based on statistical data from similar historical jobs. Vectorized concatenation refers to the operation of combining multiple independent numerical values ​​or numerical vectors, following a predetermined fixed order, to form a longer single vector. The feature vector is the final product generated after vectorized concatenation; it is a high-dimensional numerical vector that comprehensively and structurally encapsulates all known information about the current simulation job to be scheduled, and can be directly used as input to a machine learning model.

[0053] Specifically, at this stage, the system summarizes the outputs of all previous steps. It has obtained three key data fragments: First, the basic resource request from step S1021, which is a short vector [32, 128, 2, 4] representing the number of CPU cores, memory, GPU cards, and GPU memory. Second, the subject identifier embedding vector from step S1022, which is a 128-dimensional long vector [0.23, -0.11, ..., -0.45]. Third, the historical log features from step S1023, which is another short vector [250.5, 210.3, 450.7, 224.3] representing the estimated performance metrics. The system then performs a vectorized concatenation operation. It combines these data in a fixed order determined during model design. For example, the order is defined as basic resource requests first, historical log features in the middle, and subject identifier embedding vector last. The system creates a new vector, first filling in [32, 128, 2, 4], then filling in [250.5, 210.3, 450.7, 224.3], and finally appending all 128 subject identifiers embedded in the vector [0.23, -0.11, ..., -0.45] to the end. This forms a single long vector with a total dimension of 4 + 4 + 128 = 134. This final generated long vector is the feature vector representing the simulation job and the current cluster state, and it will be fed into the hybrid expert model for subsequent scheduling decisions.

[0054] Step S103: Input the feature vector into the routing module in the hybrid expert model, and have the routing module output the probability of the simulation job relative to multiple preset subject experts, and determine the probability distribution of the simulation job relative to multiple preset subject experts based on each probability.

[0055] In step S103, the hybrid expert model refers to an advanced neural network structure that includes a routing module and multiple expert modules. Its design philosophy is to allow different experts to focus on solving different types of problems, thereby improving the overall decision-making quality. The routing module acts as the scheduler in the hybrid expert model, typically a gating network. Its responsibility is to analyze the input data and determine which expert(s) to assign the task to. Preset subject experts are deep learning models built for various simulation tasks within the hybrid expert model. These preset subject experts are multiple independent sub-networks, each optimized for a specific subject-matter simulation task, such as a fluid mechanics expert and a structural mechanics expert. The probability represents a score calculated by the routing module for each preset subject expert, reflecting the expert's suitability for handling the current simulation task. The probability distribution is the set of probabilities for all preset subject experts. These probabilities are usually processed and summed to form a probability distribution, visually indicating the system's preference for expert selection.

[0056] Specifically, the system inputs the feature vector generated in the previous step into the hybrid expert model. This vector is first sent to the routing module. The internal structure of the routing module is a neural network that receives the feature vector and performs a series of matrix operations and nonlinear transformations. The number of neurons in its output layer corresponds exactly to the number of preset subject experts. The output of the routing module is processed by a Softmax function, which converts the original output values ​​into a set of probability values ​​that sum to one, i.e., a probability distribution. For example, for a feature vector of a structural mechanics task, the routing module might output a probability distribution such as [0.05, 0.94, 0.01], corresponding to fluid mechanics experts, structural mechanics experts, and electromagnetics experts, respectively. This indicates that the routing module believes with a 94% confidence level that the task should be handled by a structural mechanics expert.

[0057] In one possible implementation, the feature vector is input into the routing module of the hybrid expert model, and the routing module outputs the probability of the simulation job relative to multiple preset subject experts, specifically including steps S1031-S1034, as follows:

[0058] Step S1031: Input the feature vector into the multilayer perceptron network built into the routing module, and perform a nonlinear transformation through at least one hidden layer using the ReLU activation function to generate an intermediate hidden state vector, which includes a high-order combination feature of the feature vector.

[0059] In step S1031, a multilayer perceptron network refers to a feedforward artificial neural network composed of multiple sequentially connected layers of neurons. The ReLU activation function is a modified linear unit nonlinear function that outputs zero for negative inputs and preserves the original output for positive inputs. A hidden layer refers to all the neuron layers between the input and output layers of the multilayer perceptron network. Nonlinear transformation refers to the process of processing the weighted summation of the neuron layers using the activation function. An intermediate hidden state vector is the output vector obtained after the feature vector has been processed by at least one hidden layer in the multilayer perceptron network. Higher-order combined features refer to the complex nonlinear combinations of the original input features learned by the network, represented by the intermediate hidden state vectors.

[0060] Specifically, the system takes the 136-dimensional feature vector generated in the preceding steps as input and feeds it into the input layer of a multilayer perceptron network within the routing module. This input layer has 136 neurons, corresponding one-to-one with the dimension of the feature vector. Next, the data flows to the first hidden layer, assuming it contains 64 neurons. Each element of the input vector is multiplied by the connection weights leading to these 64 neurons, and a weighted sum is performed within each neuron, with a bias term added. This weighted sum is then input into the ReLU activation function. If the result is greater than zero, the function outputs the result itself; if the result is less than or equal to zero, the function outputs zero. The outputs of these 64 neurons collectively form a new 64-dimensional vector. This process can be repeated multiple times, meaning the data can continue to flow to the second and third hidden layers for more rounds of nonlinear transformation. The output of the final hidden layer yields the final intermediate hidden state vector. This vector may have a different dimension than the original input, but it contains high-order combined features learned by the network from the original features, providing more discriminative information for subsequent expert selection.

[0061] Step S1032: Input the intermediate hidden state vector into the Gumbel-Softmax layer of the multilayer perceptron network to obtain the logical value vector. The dimension of the logical value vector is a preset number, which is the preset number of subject experts.

[0062] In step S1032, the Gumbel-Softmax layer refers to the output layer in the multilayer perceptron network used to generate the raw scores for expert selection. The logistic vector refers to the unnormalized raw numerical vector ultimately output by the multilayer perceptron network. Dimension refers to the number of numerical elements contained in the vector. The preset number refers to the total number of predefined expert models in the system.

[0063] Specifically, the system inputs the intermediate hidden state vector generated in the previous step, for example, a 64-dimensional vector, into the last layer of the multilayer perceptron network. This layer is a standard linear fully connected layer that does not use any activation function. The number of neurons in this layer is set to be equal to the number of preset subject matter experts. Assuming the system presets 8 subject matter experts, then this output layer contains 8 neurons. The 64 elements of the intermediate hidden state vector are multiplied by the weight matrix connected to these 8 neurons, and the corresponding bias is added. The result of the operation is a vector containing 8 original values, which is the logistic vector. For example, the system might output a logistic vector such as [1.2, -3.4, 9.8, 0.5, -1.1, 2.3, 4.5, -0.8]. The value of 9.8 in the third dimension of this vector is much higher than the other dimensions, initially indicating that the system considers the third expert, possibly representing materials science, to be the most suitable choice for handling the current simulation task.

[0064] Step S1033: Perform a preset transformation on the logical value vector to obtain a weight vector. The weight vector has a preset number of dimensions, and the sum of the values ​​of each dimension of the weight vector is one. The preset transformation includes a preset noise transformation and a normalized exponential function transformation.

[0065] In step S1033, the preset transformation refers to a specific data transformation process that converts the original logical value vector into an effective probability distribution. The weight vector is the final vector obtained after the complete preset transformation, where each dimension value represents the weight of the corresponding expert being activated. The preset noise transformation refers to adding random noise sampled from the Gumbąl distribution to each element of the logical value vector in this process. The normalized exponential function transformation refers to a mathematical function called Softmax, which can transform any real number vector into a vector whose element-wise sum is one.

[0066] Specifically, the system first obtains the logistic value vector generated in the previous step, for example, [1.2, -3.4, 9.8, 0.5, -1.1, 2.3, 4.5, -0.8]. Next, the system performs a preset noise transformation, which generates another noise vector with 8 dimensions, where each value is independently and randomly drawn from a standard Gumbel distribution. The system adds this noise vector element-wise to the logistic value vector, resulting in a new logistic value vector with added random perturbation. Subsequently, the system applies a normalized exponential function transformation to this new vector. For each element in the vector, the system calculates the natural constant e raised to the power of that element, and then divides this result by the sum of the powers of e of all eight elements.

[0067] Step S1034: Use the dimension value of the weight vector as the probability of the preset subject expert.

[0068] In step S1034, the dimension value refers to a single numerical element in the weight vector.

[0069] Specifically, the system performs a direct assignment operation. It obtains the 8-dimensional weight vector generated in the previous step, namely [0.01, 0.00, 0.95, 0.01, 0.00, 0.01, 0.02, 0.00]. The system assigns the first dimension value of this vector, 0.01, as the probability of the first preset subject matter expert. Similarly, it assigns the second dimension value, 0.00, as the probability of the second expert, the third dimension value, 0.95, as the probability of the third expert, and so on, until the eighth dimension value, 0.00, is assigned as the probability of the eighth expert. At this point, the routing process is complete. The system calculates a specific probability for each preset subject matter expert. These probabilities guide the system in selecting the most suitable preset subject matter expert to handle the current simulation task and provide a basis for subsequent allocation of computing resources and task execution.

[0070] Step S104: Calculate the information entropy of the probability distribution and compare the information entropy with a preset information entropy threshold.

[0071] In step S104, information entropy, a metric derived from information theory, is used here to quantify the uncertainty of the probability distribution. A low information entropy value means that the probability is highly concentrated on a particular expert, indicating that the routing module's decision is very clear; conversely, a high information entropy value means that the weight distribution is relatively uniform, indicating that the routing module is uncertain about which expert to choose. The preset information entropy threshold is a pre-defined value that serves as a criterion to distinguish whether the routing module's decision is confident or ambiguous.

[0072] Specifically, after obtaining the probability distribution, the system immediately invokes a built-in calculation module to calculate its information entropy. This module performs logarithmic and multiplicative operations on each weight value in the probability distribution according to the mathematical formula for information entropy, then sums all the results and takes a negative value. For example, for the distribution [0.05, 0.94, 0.01], the calculated information entropy will be a very small positive number. After the calculation is complete, the system reads a preset information entropy threshold from its configuration file. This threshold may be set by the system administrator based on historical scheduling data and experience, for example, set to 0.6. Then, the system compares the calculated information entropy value with this threshold to determine the degree of certainty in the decision.

[0073] In one possible implementation, after comparing the information entropy with a preset information entropy threshold, the method further includes steps S1041-S1045, as follows:

[0074] Step S1041: If the information entropy is determined to be greater than or equal to the preset information entropy threshold, then the simulation job is determined to be a new domain job.

[0075] In step S1041, information entropy, in information theory, is a metric used to measure the uncertainty or information content of a random variable. In this scenario, it quantifies the degree of confusion the gating network in the hybrid expert model feels regarding which existing expert to assign the current simulation task to. The preset information entropy threshold is a pre-defined value that serves as a criterion to distinguish whether the output of the gating network is explicit or fuzzy. A simulation task refers to a computational task submitted by the user that needs to be performed. A novel domain task refers to a simulation task identified by the system based on its characteristics that is novel and does not match the domains of expertise of any existing experts.

[0076] Specifically, after receiving a simulation task and obtaining the selection probability distribution for each expert from the gating network, the system calculates an information entropy based on these probability values. For example, if the gating network provides nearly equal selection probabilities for two experts, such as 0.51 and 0.49, the calculated information entropy will be very high, close to its maximum value. The system compares this calculated information entropy value with an internally configured preset information entropy threshold. If the calculated information entropy is greater than or equal to this threshold, the system determines that the gating network itself cannot confidently decide which expert should handle the task, strongly suggesting that the task may not belong to a domain familiar to any existing expert. Therefore, the system officially marks the simulation task as a new domain task to initiate subsequent special processing procedures.

[0077] Step S1042: Obtain the preset default expert template, and copy the network topology and initial weight parameters of the preset default expert template to construct a new expert instance in the hybrid expert model.

[0078] In step S1042, the preset default expert template refers to a standardized, predefined neural network model blueprint, which includes a validated network structure and a set of common initial parameters as the basis for creating new experts. The network topology refers to the overall design of the neural network, including the number of layers, the number of neurons in each layer, the connection methods between layers, and the type of activation function. Initial weight parameters refer to the initial values ​​assigned to the connection weights when the network is not trained; good initial values ​​help the model converge faster and better. A hybrid expert model refers to an ensemble learning model composed of a gating network and multiple expert networks, capable of handling diverse tasks. A new expert instance refers to a completely new, independent expert network dynamically created based on the default template, ready to learn new domain knowledge.

[0079] Specifically, once the system determines that it needs to handle a new domain task, it accesses its internal model library and loads a resource called a preset default expert template. This template specifies the complete architecture of an expert network, for example, it contains three hidden layers, each with 256 neurons, and uses ReLU as the activation function. The template also includes a set of initial weight parameters, either pre-trained on general data or generated using a specific initialization strategy. The system then allocates new resources in memory for the hybrid expert model and precisely replicates the network topology and initial weight parameters of this template, thus constructing a fully functional new expert instance that has not yet been trained on domain-specific data. This new expert instance is logically added to the expert list of the hybrid expert model, ready to receive and learn tasks from the new domain.

[0080] Step S1043: Assign the new domain job to the new expert instance, generate an exploration scheduling decision, generate an incremental learning data queue for the new expert instance, and store the job feature vector of the new domain job as the first sample in the incremental learning data queue.

[0081] In step S1043, a new domain job refers to a new type of job identified in step S1041. A new expert instance refers to the expert network just created in step S1042. Assignment refers to giving the scheduling rights of a job to a specific expert. Exploratory scheduling decision refers to a tentative scheduling arrangement made by an untrained expert, the purpose of which is not only to execute the job but also to collect feedback data. Incremental learning data queue refers to a data storage structure specifically created for new expert instances, used to collect and store samples for their subsequent training in a sequential manner. Job feature vector refers to a set of numerical representations of various attributes of the simulation job, such as software type, input file size, required computational precision, etc.

[0082] Specifically, the system explicitly assigns the scheduling task of this new domain job to the newly created expert instance. Since the weight parameters of this new expert instance are initialized, its output—the scheduling decision—based on the input job feature vector lacks empirical basis and is essentially an exploratory behavior, hence the term "exploratory scheduling decision." This decision might be one or more randomly selected available nodes, or a conservative choice based on general rules, such as maximizing available computing resources or ensuring that the computing resources of several nodes reach their resource utilization limits. Simultaneously with making the decision, the system initializes a first-in, first-out (FIFO) data queue and binds it to the new expert instance. Then, the system stores the current job feature vector of this new domain job as the first data entry in this newly created incremental learning data queue, marking the official start of the data collection process customized for the new expert.

[0083] Step S1044: After the simulation job is completed based on the exploration scheduling decision, the corresponding execution performance index is obtained, and the execution performance index is associated with the job feature vector to construct a training sample, which is then stored in the incremental learning data queue.

[0084] In step S1044, exploratory scheduling decision refers to the tentative scheduling scheme generated for the new domain job in the previous step. "Execution complete" refers to the state where the job has finished running and exited on the computing node. Execution performance metrics refer to the actual performance data collected after the job runs, such as actual runtime and actual peak memory usage. The job feature vector is a numerical vector describing the job's attributes that was previously stored in the queue. "Association" refers to pairing the job's features with its execution results to form a complete data point. A training sample is a data pair containing input features (X) and corresponding true labels (Y), which is the basic unit for the machine learning model to learn. The incremental learning data queue is a dedicated data buffer prepared for new experts.

[0085] Specifically, the system submits new domain jobs to the computing cluster for execution based on exploration scheduling decisions and continuously monitors their status. Once a job is completed, the system, like with regular jobs, collects key performance metrics from the monitoring system; for example, if the job actually ran for 3 hours and consumed 64GB of memory. The system then retrieves the job feature vector previously stored in the incremental learning data queue associated with the new expert. Next, the system pairs this job feature vector with the newly acquired performance metrics, combining them into a structured data record. This record is a complete training sample suitable for supervised learning, where the job feature vector is the input and the performance metrics are the desired output or label. Finally, the system stores this newly constructed training sample in the same incremental learning data queue for later use.

[0086] Step S1045: When it is determined that the number of training samples in the incremental learning data queue has reached the preset number, the incremental learning process is started, and the network parameters of the new expert instance are iteratively optimized based on the incremental learning data queue.

[0087] In step S1045, the incremental learning data queue refers to the dedicated queue that continuously collects training samples for new domain assignments. The number of training samples refers to the number of complete training samples currently stored in the queue. The preset quantity refers to a pre-defined batch size, such as 1000; when the number of samples in the queue reaches this value, a training session is triggered. The incremental learning process refers to an online learning paradigm where the model does not see all the data at once, but rather trains and updates in batches as new data arrives. The new expert instance is the target model for this learning process. The network parameters are the model weights and biases that need to be optimized through learning. Iterative optimization refers to the process of gradually improving model performance by repeatedly performing a forward propagation-loss calculation-backpropagation-parameter update loop.

[0088] Specifically, whenever training samples for a new domain task are added to the incremental learning data queue, the system checks the current number of training samples in that queue. The system has a preset number, such as 1000. When the system detects that the number of samples in the queue has reached or exceeded 1000 for the first time, it triggers and starts the incremental learning process. The system retrieves these 1000 training samples from the queue, grouping them into a small batch. Then, using this batch of data as input, it performs one or more complete training iterations on the new expert instance. In each iteration, the model makes predictions based on the input features, compares the prediction results with the actual performance metrics in the samples to calculate the loss, calculates the gradient using the backpropagation algorithm, and finally uses the optimizer to update its network parameters. This process is called iterative optimization. After optimization, these used samples can be removed from the queue to make room for subsequent new samples. Through this periodic incremental learning, the capabilities of the new expert instance will gradually improve as new domain tasks are continuously executed.

[0089] Step S105: If the information entropy is determined to be less than the preset information entropy threshold, then the target subject expert with the highest probability is determined from multiple preset subject experts.

[0090] In step S105, the target subject expert refers to the expert module that has the highest probability among all preset subject experts and is ultimately selected to generate a specific scheduling scheme.

[0091] Specifically, the system determines the subsequent operation path based on the comparison results of the previous step. In this method, since the calculated information entropy is determined to be less than the preset information entropy threshold, it indicates that the routing module's decision is reliable. Therefore, the system will execute the process under certain conditions. The system first checks the probability distribution, finds the maximum value (0.94), and determines the corresponding expert, namely the structural mechanics expert. This expert is then designated as the target subject expert for this scheduling. Next, the system packages the complete feature vector generated in step S102 and the original host state information collected in step S101 together as input and sends it to the selected structural mechanics expert module. This expert module is itself a deep neural network. It uses its specialized knowledge accumulated in processing structural mechanics simulation job scheduling to perform in-depth analysis of the input information, comprehensively consider the job requirements and the refined state of each node, and finally outputs a clear scheduling decision. For example, the decision result is the unique identifier of computing node Node-02.

[0092] Step S106: Obtain the subject history feature vector corresponding to the target subject expert, and input the feature vector and the subject history feature vector into the target subject expert to obtain a scheduling decision. The scheduling decision includes the target host for executing the simulation job, the resource allocation scheme, and the job running time.

[0093] In step S106, the subject history feature vector refers to a numerical vector extracted from historical data and associated with the target subject expert, representing the typical resource consumption pattern of that subject area. The scheduling decision refers to the specific instructions ultimately generated by the target subject expert, guiding job placement and resource allocation. The target host refers to the computing node ultimately selected to run the current simulation job according to the scheduling decision. The resource allocation scheme refers to the specific quantities of various resources allocated to the job as specified in the scheduling decision. The job execution time refers to the time estimated in the scheduling decision for the job to complete.

[0094] Specifically, the server loads the subject-specific historical feature vector corresponding to the target subject expert determined in the previous step from storage. Then, the server concatenates the feature vector generated in step S102 with this subject-specific historical feature vector to form a new input vector. The server passes this new input vector to the neural network corresponding to the target subject expert. This network performs a forward propagation computation, and its multi-task output layer simultaneously generates three sets of information: one set is the allocation probability of all computing nodes, one set is the recommended number of CPU cores and memory, and the other set is the estimated runtime in seconds. The server selects the node with the highest allocation probability as the target host, formats the recommended resource quantity into a resource allocation scheme, and converts the estimated runtime in seconds into job execution time. These three sets of information together constitute the final scheduling decision.

[0095] Please refer to Figure 2 In one possible implementation, the feature vector and the subject history feature vector are input to the target subject expert to obtain a scheduling decision, specifically including steps S201-S207, as follows:

[0096] Step S201: Concatenate the feature vector with the subject history feature vector to form a combined input vector.

[0097] In step S201, the combined input vector refers to the final vector formed by concatenating the feature vector and the historical feature vector of the corresponding discipline, which is used as input to the neural network.

[0098] Specifically, the server first locates and loads the matching subject history feature vector from a pre-built subject knowledge base based on the application type parameter of the current simulation job, such as structural mechanics. Then, the server concatenates the feature vector representing the current job and cluster state generated in step S102 with the newly loaded subject history feature vector according to a preset dimensional order to generate a combined input vector.

[0099] Step S202: Input the combined input vector into the deep neural network built into the target subject expert, the deep neural network containing at least one fully connected hidden layer.

[0100] In step S202, a deep neural network refers to a machine learning model composed of multiple interconnected computational layers, capable of learning complex patterns in data. A fully connected hidden layer is the core computational layer in a deep neural network used to learn the inherent complex relationships within the data.

[0101] Specifically, the server takes the combined input vector generated in the previous step as input data and feeds it into the target subject, namely structural mechanics, which is pre-trained and loaded into the target subject expert deep neural network model in memory. This operation is the starting point of the entire inference process, serving as the entry point for data into the decision model.

[0102] Step S203: The combined input vector is nonlinearly transformed through the fully connected hidden layer of the deep neural network to obtain a high-dimensional feature representation.

[0103] In step S203, nonlinear transformation refers to processing the computation results of the fully connected hidden layer through an activation function, enabling the neural network to learn and represent nonlinear relationships. High-dimensional feature representation refers to the internal data representation generated after the combined input vectors undergo nonlinear transformation through the fully connected hidden layer, containing deeper and more abstract information.

[0104] Specifically, the combined input vector flows through at least one fully connected hidden layer within the deep neural network. In each fully connected hidden layer, the input data vector is multiplied by the layer's weight matrix, and a bias vector is added. The result is then passed through a predefined non-linear activation function, such as the ReLU family of functions, for a non-linear transformation. This process progresses layer by layer, mapping the original input to a higher-dimensional feature space, ultimately generating a high-dimensional feature representation that contains deeply abstract information.

[0105] Step S204: Based on the high-dimensional feature representation, the scheduling decision is determined through the multi-task learning output layer built into the deep neural network. The multi-task learning output layer includes a node selection head, a resource allocation head, and a runtime prediction head.

[0106] In step S204, the multi-task learning output layer refers to the last layer of the deep neural network, which is designed to simultaneously output multiple different but related predictions. The scheduling decision refers to a complete job deployment instruction that integrates node selection, resource allocation, and runtime estimation. The node selection header, part of the multi-task learning output layer, is specifically used to output the probability of assigning the job to each computing node. The resource allocation header, another part of the multi-task learning output layer, is specifically used to determine the specific amount of resources that should be allocated to the job. The runtime estimation header, a third part of the multi-task learning output layer, is specifically used to predict the execution time of the job under a given allocation scheme.

[0107] Specifically, the high-dimensional feature representation generated in the previous step is used as input and simultaneously passed to the multi-task learning output layer at the end of the deep neural network. This output layer is structurally parallel and contains three functionally independent heads: a node selection head, a resource allocation head, and a runtime prediction head. The high-dimensional feature representation is shared by these three heads to simultaneously perform different prediction tasks and ultimately determine the complete scheduling decision.

[0108] Step S205: Output the normalized allocation probability of the simulation job on each computing node through the node selection head, and select the computing node with the highest allocation probability as the target host.

[0109] In step S205, the normalized allocation probability represents the likelihood of the job being assigned to each computing node, with the sum of the probabilities across all nodes being one. A computing node refers to an independent server unit in the cluster that can be used to execute computing tasks. The target host refers to the computing node selected from all computing nodes based on the normalized allocation probability as the most suitable for running the current simulation job.

[0110] Specifically, the node selection head receives the high-dimensional feature representation, performs calculations through its own network layer structure, and finally outputs a vector. The dimension of this vector is equal to the total number of computing nodes in the cluster, and each element in the vector represents the normalized allocation probability of assigning the simulation job to the corresponding computing node. The server then iterates through this probability vector, selects the computing node corresponding to the element with the largest value, and determines that computing node as the target host for this scheduling.

[0111] Step S206: Output the types and quantities of resources allocated to the simulation job through the resource allocation header to obtain the resource allocation scheme.

[0112] In step S206, "resource type" refers to the type of computing resources allocated to the job, such as CPU cores. "Resource quantity" refers to the specific amount of each type of resource allocated to the job, such as the number of cores or memory capacity. "Resource allocation scheme" refers to the specific configuration, such as the number of CPU cores and memory size, determined for the simulation job in the scheduling decision.

[0113] Specifically, the resource allocation header also receives and calculates high-dimensional feature representations. The output of this header directly determines the specific types and quantities of resources for the simulation job; for example, it might output two values ​​representing 128 processor cores and 512 gigabytes of memory to be allocated. These values ​​together constitute a detailed resource allocation scheme.

[0114] Step S207: Output the estimated runtime of the simulation job through the runtime estimation header to obtain the job runtime.

[0115] In step S207, the runtime estimate represents the scheduling system's prediction of the time required for a simulation job to complete under a specific resource configuration. The job runtime refers to a specific time metric determined based on the value output by the runtime estimate header, for example, in seconds.

[0116] Specifically, the runtime prediction head uses the same high-dimensional feature representation for calculation and outputs a single numerical value. This value represents the estimated runtime required for the current simulation job under the target host and resource allocation scheme determined in steps S205 and S206, i.e., the job runtime. This result provides an important reference for the performance prediction of the scheduling system.

[0117] Step S107: Send the scheduling decision to the job executor on the target host, so that the job executor executes the simulation job according to the scheduling decision.

[0118] In step S107, the scheduling decision refers to the specific instructions generated by the target subject experts to guide job placement, clearly indicating which computing node the simulation job should be assigned to. The target host refers to the computing node ultimately selected to run the current simulation job based on the scheduling decision. The job executor refers to a background service program pre-installed on each computing node, responsible for receiving commands from the main scheduling system and managing the entire lifecycle of the job on its node, including startup, monitoring, and termination.

[0119] Specifically, after receiving scheduling decisions from experts in the target discipline, such as the node identifier Node-02, the system enters the final execution phase of scheduling. The system generates a standardized execution instruction containing the job name, all runtime parameters, and the address of the target host Node-02. Then, the system securely sends this instruction to the job executor running on the target host Node-02 via an internal message queue or a direct network connection. Upon receiving the instruction, the job executor on Node-02 first verifies its validity, then parses its content, creates a runtime environment for the job on its local operating system, allocates the required 128 processor cores and 512 gigabytes of memory, and finally initiates the execution of the simulation program. After the job begins running, the job executor continuously reports its running status to the main scheduling system until the job is completed.

[0120] In one possible implementation, after the scheduling decision is sent to the job executor on the target host, and the job executor executes the simulation job according to the scheduling decision, the method further includes steps S108-S110, as follows:

[0121] Step S108: After the simulation job is completed, obtain the actual runtime and actual resource consumption of the simulation job.

[0122] In step S108, "simulation job completion" means that the job executor has completed all computational tasks on the target host and exited normally or abnormally. Actual runtime refers to the real physical time elapsed from the moment the simulation job officially begins execution on the computing node until its complete termination; this is a value precisely measured using the system clock. Actual resource consumption refers to the statistical data on the actual computing resources used by the process during the entire job's execution, typically including key performance indicators such as total CPU time and peak or average memory usage.

[0123] Specifically, when the job executor deployed on the target host detects that the simulation job process it manages has terminated, it immediately starts a data acquisition program. This program first records the job's end timestamp and then subtracts the start timestamp recorded when the job began execution from the end timestamp to calculate the precise actual runtime in seconds. Simultaneously, the system collects and organizes resource usage records from the underlying monitoring agent or operating system kernel during the job's execution. For example, the system calculates the number of CPU cores consumed by all threads of the job throughout its entire lifecycle and identifies the maximum memory usage across all sampling points as the peak memory consumption. This collected actual runtime and resource consumption data will be stored as a true label or standard answer for subsequent model evaluation and optimization.

[0124] Step S109: Compare the actual runtime with the expected runtime in the scheduling decision and calculate the prediction error.

[0125] In step S109, the actual runtime refers to the time actually taken for the job to be executed, as obtained in the previous step. The estimated runtime in the scheduling decision refers to the job execution time predicted by the runtime estimator of the target subject expert before the job is scheduled. Comparison refers to performing a mathematical comparison operation between these two time values ​​to quantify the difference between them. Prediction error refers to a numerical value obtained through comparison that reflects the accuracy of the model's prediction; it represents the gap between the predicted value and the actual value.

[0126] Specifically, the system extracts two key pieces of information from the data storage corresponding to the simulation job that has just been completed: the first is the actual runtime collected in step S107, for example, 16200 seconds, or 4.5 hours; the second is the estimated runtime generated by the model at the beginning of scheduling, for example, 14400 seconds, or 4 hours. The system performs a subtraction operation, subtracting the estimated runtime from the actual runtime to calculate the prediction error. In this example, the prediction error is 16200 minus 14400, which equals 1800 seconds. This positive error indicates that the model's prediction is too low, that is, it underestimates the job's running time. This prediction error value, representing the model's performance in this prediction, will be used as the core input and passed to the next step of loss calculation.

[0127] Step S110: Based on the prediction error, calculate the scheduling loss using a preset loss function, and use the scheduling loss to update the network parameters of the target subject expert using the backpropagation algorithm.

[0128] In step S110, the prediction error refers to the difference between the model's predicted value calculated in the previous step and the actual value. The preset loss function is a mathematical function defined during the model design phase. It is used to convert the prediction error into a non-negative scalar value representing the degree of error made by the model, such as the mean squared error function. The scheduling loss is the specific value calculated by substituting the prediction error into the loss function; the higher this value, the worse the model's performance in a single prediction.

[0129] Specifically, the system inputs the 1800-second prediction error calculated in the previous step into a preset loss function, such as the mean squared error loss function. This function squares the error value, resulting in a large scheduling loss value: 1800 squared equals 3,240,000. This scheduling loss value signifies the penalty for this prediction. Subsequently, the system initiates the backpropagation algorithm. Starting from the output layer, the algorithm propagates this loss value back to every layer of the network, using the chain rule of calculus to precisely calculate the gradients of the thousands of network parameters within the target subject expert model. The gradient indicates the direction in which parameters are adjusted to minimize the loss as quickly as possible. Finally, the system performs gradient updates, fine-tuning each network parameter along the opposite direction of the gradient based on a hyperparameter called the learning rate. For example, if the gradient of a weight parameter is positive, the system will decrease that weight slightly. Through this complete process of calculating loss, backpropagation, and gradient updates, the network parameters of the target subject expert are optimized, making future scheduling decisions more accurate.

[0130] In one possible implementation, the method further includes the step of constructing a hybrid expert model, specifically comprising steps S111-S115, as follows:

[0131] Step S111: Obtain the historical job dataset. Each job sample in the historical job dataset contains job parameters, job category, actual resource consumption and actual runtime of executing the job sample.

[0132] In step S111, the historical job dataset refers to a collection containing a large number of previously completed simulation job records, serving as the foundation for the model's learning experience. A job sample refers to a complete record in the dataset representing a single job. Job parameters refer to input features describing the inherent attributes and requirements of a job, such as the software name used, input file size, and computational precision settings; these parameters are converted into numerical forms for model processing. The job category refers to a label that categorizes the job by domain, such as computational fluid dynamics or molecular dynamics, indicating the professional field to which the job belongs. Actual resource consumption and actual runtime refer to the peak computational resources consumed by the job during actual execution, such as maximum memory usage, and the total time consumed from start to finish; these are the targets that the model needs to learn and predict.

[0133] Specifically, the system first connects to the historical databases or log files of the job scheduling system and performance monitoring system. The system extracts and integrates information from these data sources to build a structured job sample for each completed job. This process includes parsing the job submission script to obtain job parameters, querying records in the job management system to determine the job category, and retrieving the peak resource consumption and total runtime of the job during execution from the performance monitoring logs. The system integrates all this information to form a massive historical job dataset, where each row represents a complete job sample, providing rich and realistically labeled data for subsequent model training.

[0134] Step S112: Based on the assignment category, divide the historical assignment dataset into multiple subject subsets corresponding to preset subject experts, with one preset subject expert corresponding to one subject subset.

[0135] In step S112, the job category refers to the label obtained in the previous step used to distinguish jobs from different domains. The historical job dataset refers to the complete dataset containing jobs from all domains. Partitioning refers to the process of dividing a large dataset into multiple smaller datasets according to a certain criterion. The subject subset refers to the data set selected from the complete dataset that contains only samples of a specific job category. The pre-defined subject experts refer to the specialized models to be trained for a specific subject domain in the plan. Each expert will have its own exclusive training data, which is the corresponding subject subset.

[0136] Specifically, the system iterates through every single task sample in the entire historical task dataset. For each sample, the system reads the value of its task category field. Then, based on this category value, the system places the task sample into the corresponding subject subset. For example, if a task sample's category is weather forecast simulation, the system adds it to the subject subset specifically for weather forecast simulation tasks. This process continues until all samples in the historical task dataset have been assigned. Ultimately, the system obtains multiple independent subject subsets, each containing highly homogeneous tasks; for example, one subset might consist entirely of weather forecast tasks, while another might consist entirely of materials science calculation tasks. This partitioning ensures that each pre-defined subject expert can focus their training on highly relevant data.

[0137] Step S113: For each subject subset, train a deep neural network as the corresponding preset subject expert.

[0138] In step S113, the subject subset refers to the dataset divided in the previous step that contains job samples for a specific domain. Separate training means executing the model training process independently and without interference for each subject subset. A deep neural network is a complex machine learning model with multi-layered nonlinear transformations, capable of learning deep-level abstract features in data; here, it is used as the specific implementation for each expert. The corresponding preset subject expert refers to the deep neural network model obtained after training on a specific subject subset, which can accurately predict the resource consumption of jobs in that domain.

[0139] Specifically, the system initiates an independent training task for each subject subset. Taking the weather forecasting simulation subject subset as an example, the system constructs a deep neural network model architecture. Then, the system uses the job parameters from this subset as the network input, and the corresponding actual resource consumption and actual runtime as the network's prediction targets or labels. The training process is conducted through multiple iterations. In each iteration, the model makes predictions based on the input, calculates the error between the predicted value and the true label, and then uses the backpropagation algorithm to adjust the connection weights within the network to reduce the error of the next prediction. This training process continues until the model's performance on the validation data no longer shows significant improvement. The system repeats this process for each subject subset, ultimately training a pre-defined subject expert model proficient in the job characteristics of each pre-defined subject domain.

[0140] Step S114: After training is completed, fix the network parameters of all preset subject experts and train the routing module using the historical job dataset.

[0141] In step S114, fixing the network parameters of all preset subject experts means setting the weights of the already trained expert models to a non-updateable state, so that they only serve as feature extractors or predictors in subsequent training, and no longer learn themselves. The historical assignment dataset refers to a complete dataset containing all categories of assignments, which is used to train the routing module.

[0142] Specifically, the system first loads all the pre-set subject expert models trained in step S112. Then, the system freezes all the parameters of these models to ensure their values ​​will not change during subsequent training. Next, the system constructs a new neural network, namely the routing module. The system uses a complete historical job dataset to train this routing module. For each job sample in the dataset, its job category is the correct target that the routing module needs to learn. For example, for a weather forecast simulation job, the training objective of the routing module is to output the highest probability of pointing to a weather forecast expert. In this way, the routing module learns how to identify different job categories and make the correct routing decisions.

[0143] Step S115: Combine the trained routing module with the trained preset subject experts to form a hybrid expert model.

[0144] In step S115, the trained routing module refers to the gating network that has learned how to classify tasks and assign experts in the previous step. The trained preset subject experts refer to multiple expert networks that have been independently trained in step S112 and are proficient in specific domain knowledge. Combination refers to the architectural connection of these independent, trained modules to form a collaborative whole. The hybrid expert model refers to the final integrated model that consists of one routing module and multiple expert modules and is capable of handling diverse tasks.

[0145] Specifically, the system executes the final step of model building, which is a purely structural assembly without involving any parameter training or updates. The system integrates the trained routing module with all trained preset subject matter experts to construct a hybrid expert model. When the model makes scheduling decisions, job parameters are first sent to the routing module as input. The routing module analyzes the input and outputs a probability distribution; the preset subject matter expert corresponding to the highest probability value in this distribution is selected as the activated expert. Subsequently, the same job parameter input is passed to this activated expert. This activated expert, combining its inherent subject matter knowledge, independently performs reasoning and generates a complete scheduling decision including the target host, resource allocation scheme, and job execution time. This scheduling decision is the final output of the hybrid expert model.

[0146] The following describes a simulation job scheduling system based on a hybrid expert model from the perspective of hardware processing. Please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of the structure of a simulation job scheduling system based on a hybrid expert model in an embodiment of this application.

[0147] It should be noted that, Figure 3 The structure of a simulation job scheduling system based on a hybrid expert model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0148] like Figure 3 As shown, a simulation job scheduling system based on a hybrid expert model includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage section 308 into Random Access Memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0149] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0150] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0151] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0153] Specifically, a simulation job scheduling system based on a hybrid expert model in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the simulation job scheduling method based on a hybrid expert model provided in the above embodiment.

[0154] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the hybrid expert model-based simulation job scheduling system described in the above embodiments; or it may exist independently and not incorporated into the hybrid expert model-based simulation job scheduling system. The storage medium carries one or more computer programs, which, when executed by a processor of the hybrid expert model-based simulation job scheduling system, enable the hybrid expert model-based simulation job scheduling system to implement the hybrid expert model-based simulation job scheduling method for IoT data encryption transmission provided in the above embodiments.

Claims

1. A simulation job scheduling method based on a hybrid expert model, characterized in that, The method includes: Obtain the job parameters of the simulation job to be scheduled and the host status information of the computing nodes in the cluster; Based on the job parameters and the host status information, the application features of the simulation job are extracted, and feature vectors representing the simulation job and cluster status are generated. The feature vector is input into the routing module of the hybrid expert model, and the routing module outputs the probability of the simulation task relative to multiple preset subject experts. The probability distribution of the simulation task relative to multiple preset subject experts is determined according to each probability. The preset subject experts are the deep learning models constructed in the hybrid expert model for various types of simulation tasks. Calculate the information entropy of the probability distribution and compare the information entropy with a preset information entropy threshold; If it is determined that the information entropy is less than a preset information entropy threshold, then the target subject expert with the highest probability is determined from among the multiple preset subject experts; Obtain the subject history feature vector corresponding to the target subject expert, and input the feature vector and the subject history feature vector into the target subject expert to obtain a scheduling decision. The scheduling decision includes the target host for executing the simulation job, the resource allocation scheme, and the job execution time. The scheduling decision is sent to the job executor on the target host, so that the job executor executes the simulation job according to the scheduling decision.

2. The method according to claim 1, characterized in that, The step of extracting application features of the simulation job based on the job parameters and the host status information, and generating feature vectors characterizing the simulation job and cluster status, specifically includes: The job parameters are parsed to obtain the user-specified basic resource requests, and subject keywords representing the computation mode of the simulation job are extracted from the command line parameters of the simulation job through keyword matching to form an initial subject identifier. The basic resource requests include the number of CPU cores, the amount of memory, the GPU card, and the amount of GPU memory. The initial subject identifier is input into a preset word embedding model to obtain the subject identifier embedding vector; Based on the initial subject identifier, a set of historical assignments matching the simulation assignment is retrieved from the historical assignment log database, and the resource consumption data of the historical assignment set is statistically aggregated to generate historical log features, which include average iteration time, peak memory usage, and total communication traffic between nodes. The basic resource request, the subject identifier embedding vector, and the historical log features are vectorized and concatenated to obtain the feature vector.

3. The method according to claim 1, characterized in that, The step of inputting the feature vector into the routing module of the hybrid expert model, and having the routing module output the probability of the simulation task relative to multiple preset subject experts, specifically includes: The feature vector is input into the multilayer perceptron network built into the routing module, and a nonlinear transformation is performed through at least one hidden layer using the ReLU activation function to generate an intermediate hidden state vector, which includes higher-order combined features of the feature vector. The intermediate hidden state vector is input into the Gumbel-Softmax layer of the multilayer perceptron network to obtain a logical value vector. The dimension of the logical value vector is a preset number, which is the preset number of subject experts. The logical value vector is subjected to a preset transformation to obtain a weight vector. The dimension of the weight vector is the preset number, and the sum of the values ​​of each dimension of the weight vector is one. The preset transformation includes a preset noise transformation and a normalized exponential function transformation. The dimension value of the weight vector is used as the probability of the preset subject expert.

4. The method according to claim 1, characterized in that, The step of inputting the feature vector and the discipline historical feature vector into the target discipline expert to obtain a scheduling decision specifically includes: The feature vector is concatenated with the discipline history feature vector to form a combined input vector; The combined input vector is input into the deep neural network built into the target subject expert, the deep neural network containing at least one fully connected hidden layer; The combined input vector is nonlinearly transformed through the fully connected hidden layer of the deep neural network to obtain a high-dimensional feature representation; Based on the high-dimensional feature representation, the scheduling decision is determined through the multi-task learning output layer built into the deep neural network. The multi-task learning output layer includes a node selection head, a resource allocation head, and a runtime prediction head.

5. The method according to claim 4, characterized in that, The process of determining the scheduling decision based on the high-dimensional feature representation and through the multi-task learning output layer built into the deep neural network specifically includes: The node selection head outputs the normalized allocation probability of the simulation job on each computing node, and selects the computing node with the highest allocation probability as the target host. The resource allocation scheme is obtained by outputting the types and quantities of resources allocated to the simulation job through the resource allocation header; The runtime estimate of the simulation job is output through the runtime estimate header to obtain the job runtime.

6. The method according to claim 1, characterized in that, After sending the scheduling decision to the job executor on the target host, causing the job executor to execute the simulation job according to the scheduling decision, the method further includes: After the simulation job is completed, obtain the actual runtime and actual resource consumption of the simulation job; The actual runtime is compared with the estimated runtime in the scheduling decision, and the prediction error is calculated. Based on the prediction error, a scheduling loss is calculated using a preset loss function, and the scheduling loss is used to update the network parameters of the target subject expert using a backpropagation algorithm.

7. The method according to claim 1, characterized in that, After comparing the information entropy with a preset information entropy threshold, the method further includes: If the information entropy is determined to be greater than or equal to a preset information entropy threshold, then the simulation task is determined to be a new domain task. Obtain a preset default expert template, and copy the network topology and initial weight parameters of the preset default expert template to construct a new expert instance in the hybrid expert model; The new domain task is assigned to the new expert instance, an exploration scheduling decision is generated, an incremental learning data queue is generated for the new expert instance, and the task feature vector of the new domain task is stored in the incremental learning data queue as the first sample. After the simulation job is completed according to the exploration scheduling decision, the corresponding execution performance index is obtained, and the execution performance index is associated with the job feature vector to construct a training sample, which is then stored in the incremental learning data queue. When the number of training samples in the incremental learning data queue reaches a preset number, the incremental learning process is started, and the network parameters of the new expert instance are iteratively optimized based on the incremental learning data queue.

8. The method according to claim 1, characterized in that, The method further includes the step of constructing the hybrid expert model, wherein constructing the hybrid expert model specifically includes: Obtain a historical job dataset, wherein each job sample in the historical job dataset contains job parameters, job category, and actual resource consumption and actual runtime of executing the job sample; Based on the job category, the historical job dataset is divided into multiple subject subsets corresponding to the preset subject experts, with one preset subject expert corresponding to one subject subset; For each of the aforementioned subject subsets, a deep neural network is trained to serve as the corresponding preset subject expert; After training is completed, the network parameters of all preset subject experts are fixed, and the routing module is trained using the historical job dataset. The trained routing module is combined with the trained preset subject experts to form the hybrid expert model.

9. A simulation job scheduling system based on a hybrid expert model, characterized in that, The simulation job scheduling system based on a hybrid expert model includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the simulation job scheduling system based on a hybrid expert model to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is run on a simulation job scheduling system based on a hybrid expert model, the simulation job scheduling system based on a hybrid expert model performs the method as described in any one of claims 1-7.

Images