Heterogeneous resource collaborative scheduling optimization method and system based on dynamic memory and reinforcement learning

Abstract

The invention provides a heterogeneous resource collaborative scheduling optimization method and system based on dynamic memory and reinforcement learning, and relates to the technical field of resource scheduling. The method comprises the following steps: constructing a job waiting queue and a node matrix by acquiring job information and resource node state information, and generating a scheduling state containing job characteristics, job waiting time and a resource state; an SE-Channel-Temporal layer and a gating Transform-XL (Extensible Library) module are adopted to realize feature enhancement and dynamic memory updating; outputting a scheduling action in the adjustable waiting queue window based on the reinforcement learning strategy network and executing resource allocation or reservation; and a scheduling strategy is updated in combination with a multi-target reward mechanism and a near-end strategy optimization algorithm, so that self-adaptive and efficient scheduling in a heterogeneous resource environment is realized.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, and in particular to a method and system for optimizing the collaborative scheduling of heterogeneous resources based on dynamic memory and reinforcement learning. Background Technology

[0002] In high-performance computing (HPC) environments, jobs have data dependencies, control dependencies, and resource contention dependencies. Different types of jobs have significantly different requirements for computing resources: compute-intensive jobs rely more on the computing power of the central processing unit (CPU) or graphics processing unit (GPU), while data-intensive jobs rely more on storage capacity and network bandwidth. At the same time, the system is developing towards heterogeneity, and the types of resources are becoming increasingly diverse. The difficulty of adapting and allocating jobs to heterogeneous resources has increased significantly. Job scheduling needs to coordinate resource allocation and execution order under complex dependencies and heterogeneous resource constraints.

[0003] Rule-based classical strategies, deterministic algorithms, heuristics, and metaheuristics are widely used in job scheduling. However, in real-time dynamic environments, they rely heavily on prior knowledge and are not adaptable enough to arrival fluctuations and changes in demand. This has prompted the industry to introduce Deep Reinforcement Learning (DRL) to model the scheduling process as a Markov Decision Process (MDP), which learns strategies through interaction with the environment to maximize long-term cumulative rewards. At the same time, the Multi-Head Attention (MHA) mechanism of the Transformer model is used to perform deeper sequence modeling of job submission and completion data to improve the quality of scheduling decisions.

[0004] Traditional scheduling methods often rely on static models and deterministic assumptions, making them difficult to work stably under uncertain and dynamically changing conditions. First Come First Served (FCFS), Shortest Job First (SJF), and priority scheduling tend to overlook dependency characteristics and differences in resource requirements in complex environments, making it difficult to dynamically adapt to changes in job and resource dependencies, and the rules are costly to formulate and maintain. On the other hand, long queue scheduling has the characteristic of long sequence dependencies. Traditional Transformer models are limited by the fixed length of the context window and computational complexity, making it difficult to effectively handle long sequence scheduling data, resulting in insufficient ability to capture long-term dependencies. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for collaborative scheduling optimization of heterogeneous resources based on dynamic memory and reinforcement learning. By combining the dynamic memory mechanism with the reinforcement learning scheduling strategy, a scheduling decision model that integrates current state and historical information is constructed, thereby realizing adaptive scheduling for complex operation characteristics and dynamic changes in heterogeneous resource environments.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning includes:

[0008] Obtain job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node;

[0009] A state vector is constructed based on the job information and the resource node status information, and the state vector and historical information cache are used together to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status;

[0010] The state vector is input into the SE-Channel-Temporal layer for feature enhancement, generating a first channel weight and a second channel weight respectively. The first channel weight and the second channel weight are then fused to obtain the final channel weight. Based on the final channel weight, the channel weighted feature is output.

[0011] The channel weighted features are input into the gated Transformer-XL module. The local memory is updated and the memory cache is maintained using a sliding window mechanism. The historical scheduling features corresponding to the historical information cache are fused with the current attention features output by the gated Transformer-XL module through the GRU gated unit to obtain the updated historical information cache.

[0012] Based on the scheduling status, the policy network outputs a scheduling action; the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, and allocates or reserves idle resource nodes in the node matrix to the job to be scheduled, so as to update the job waiting queue, the node matrix and the scheduling status.

[0013] A multi-objective reward is generated based on resource utilization, job waiting time, system throughput, and job priority. The PPO algorithm based on near-end policy optimization is used to update the parameters of the policy network and value network using the multi-objective reward to obtain the scheduling policy.

[0014] Preferably, a state vector is constructed based on the job information and the resource node status information, and the state vector and historical information cache are used together to form the scheduling state, including:

[0015] The job features are extracted based on the job information; the job features include at least the number of resource nodes in the job request and the running time of the job request;

[0016] The job waiting time is determined based on the job information; the job waiting time is the difference between the current time and the job submission time.

[0017] The resource status is determined based on the resource node status information; the resource status includes at least the idle or occupied status of each resource node in the node matrix;

[0018] The state vector is obtained by combining the job characteristics, the job waiting time, and the resource status according to a preset concatenation rule.

[0019] The scheduling state is obtained by concatenating or aggregating the state vector with the historical information cache according to a preset combination rule.

[0020] Preferably, the state vector is input into the SE-Channel-Temporal layer for feature enhancement, generating first channel weights and second channel weights respectively. The first channel weights and second channel weights are then fused to obtain the final channel weights. Based on the final channel weights, channel-weighted features are output, including:

[0021] Temporal dimension features are extracted from the state vector to obtain temporal enhancement features;

[0022] The temporal enhancement features are modeled using the first path channel, and the weights of the first channel corresponding to each channel are output.

[0023] The temporal enhancement features are modeled using a second path channel, and the weights of the second channel corresponding to each channel are output.

[0024] The first channel weight and the second channel weight are fused to obtain the final channel weight that corresponds one-to-one with each channel;

[0025] Based on the final channel weights, the temporal enhancement features are weighted channel by channel, and the channel-weighted features are output.

[0026] Preferably, the channel-weighted features are input into the gated Transformer-XL module, a sliding window mechanism is used to update the local memory and maintain the memory cache, and the historical scheduling features corresponding to the historical information cache are fused with the current attention features output by the gated Transformer-XL module through the GRU gating unit to obtain the updated historical information cache, including:

[0027] The channel-weighted features are mapped to the input sequence of the gated Transformer-XL module and attention is calculated to output the current attention features;

[0028] The sliding window mechanism is used to read historical segments corresponding to the current input sequence from the memory cache as local memory, and the local memory is updated after the current attention calculation is completed;

[0029] The historical scheduling features and the current attention features in the historical information cache are input into the GRU gating unit, and the GRU gating unit outputs the fusion result;

[0030] The fusion result is written into the memory cache and the historical information cache is updated synchronously to obtain the updated historical information cache.

[0031] Preferably, a sliding window mechanism is used to update local memory and maintain the memory cache, including:

[0032] Set the sliding window length and update step size, and extract continuous historical segments from the memory cache based on the sliding window length as the local memory;

[0033] After each generation of the current attention feature, a new segment corresponding to the current input sequence is appended to the memory cache;

[0034] When the length of the memory cache exceeds the preset upper limit, the earliest written historical fragment is removed from the memory cache according to the update step size;

[0035] The local memory for the next time step is regenerated based on the updated memory cache.

[0036] Preferably, the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, including:

[0037] The window size of the waiting queue is determined, and the window size is dynamically adjusted according to the queue length of the job waiting queue or the resource availability status represented by the node matrix.

[0038] A waiting queue window is created at the head of the job waiting queue to obtain a set of candidate jobs within the window;

[0039] The policy network outputs the scheduling action within the waiting queue window, and the scheduling action is used to indicate the target job in the candidate job set;

[0040] The target job indicated by the scheduling action is identified as the job to be scheduled.

[0041] Preferably, allocating or reserving idle resource nodes in the node matrix for the job to be scheduled, in order to update the job waiting queue, the node matrix, and the scheduling status, includes:

[0042] The set of idle resource nodes for the scheduled job is determined based on the node matrix;

[0043] When the set of idle resource nodes meets the resource requirements of the job to be scheduled, the idle resource nodes are allocated to the job to be scheduled, and the available status of the corresponding resource node in the node matrix is ​​updated to the occupied status.

[0044] When the set of idle resource nodes does not meet the resource requirements of the job to be scheduled, the idle resource nodes are reserved for the job to be scheduled, and a reservation identifier is written into the node matrix for the reserved resource nodes.

[0045] When subsequent resource nodes are released, the resource requirements of the job to be scheduled are supplemented based on the reserved identifier, and the reserved resource nodes are allocated to the job to be scheduled.

[0046] After allocation or reservation is completed, the scheduled jobs that have started execution are removed from the job waiting queue, or the queue positions of the scheduled jobs that have not yet started execution in the job waiting queue are adjusted, and the scheduling status is updated based on the node matrix and the historical information cache.

[0047] Preferably, a multi-objective reward is generated based on resource utilization, job waiting time, system throughput, and job priority, including:

[0048] The resource utilization rate is calculated based on the node matrix, and the resource utilization rate is the ratio of the number of resource nodes in an occupied state to the total number of resource nodes.

[0049] Generate a waiting time reward component or a waiting time penalty component based on the job waiting time;

[0050] The system throughput is generated based on the number of jobs completed per unit time, and a throughput reward component is generated based on the system throughput.

[0051] A priority reward component is generated based on the job priority;

[0052] The multi-objective reward is obtained by combining the reward component corresponding to the resource utilization rate, the reward component or penalty component corresponding to the job waiting time, the reward component corresponding to the system throughput, and the reward component corresponding to the job priority.

[0053] Preferably, the Proximity-Based Policy Optimization (PPO) algorithm updates the parameters of the policy network and value network using the multi-objective reward to obtain a scheduling policy, including:

[0054] The advantage estimate is calculated based on the multi-objective reward, and the advantage estimate is used as the optimization basis for updating the policy network;

[0055] The pruning target of the PPO algorithm is optimized using a near-end strategy to limit the policy update magnitude, so as to update the policy network;

[0056] The value loss is calculated based on the difference between the multi-objective reward and the state value output by the value network, and the value network parameters are updated with the value loss.

[0057] When updating the policy network, an entropy regularization term is introduced to improve the exploration capability, resulting in the scheduling policy.

[0058] A heterogeneous resource collaborative scheduling optimization system based on dynamic memory and reinforcement learning includes:

[0059] The job and resource information acquisition unit is used to acquire job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node.

[0060] The scheduling state construction unit is used to construct a state vector based on the job information and the resource node status information, and to combine the state vector with historical information cache to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status;

[0061] The channel feature enhancement unit is used to input the state vector into the SE-Channel-Temporal layer for feature enhancement, generate a first channel weight and a second channel weight respectively, fuse the first channel weight and the second channel weight to obtain the final channel weight, and output the channel weighted feature based on the final channel weight;

[0062] The dynamic memory update unit is used to input the channel weighted features into the gated Transformer-XL module, update the local memory and maintain the memory cache using a sliding window mechanism, and fuse the historical scheduling features corresponding to the historical information cache with the current attention features output by the gated Transformer-XL module through the GRU gated unit to obtain the updated historical information cache.

[0063] The scheduling decision and resource allocation unit is used to output scheduling actions from the policy network based on the scheduling status; the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, and allocates or reserves idle resource nodes in the node matrix to the job to be scheduled, so as to update the job waiting queue, the node matrix and the scheduling status.

[0064] The strategy optimization and reward update unit is used to generate multi-objective rewards based on resource utilization, job waiting time, system throughput and job priority, and update the policy network and value network parameters using the multi-objective rewards based on the near-end policy optimization (PPO) algorithm to obtain the scheduling strategy.

[0065] The present invention discloses the following technical effects:

[0066] This invention addresses the shortcomings of existing heterogeneous resource scheduling methods, such as insufficient adaptability to dynamic environments and difficulty in balancing job dependence characteristics with resource heterogeneity. By constructing a state vector that includes job characteristics, job waiting time, and resource status, and combining this state vector with historical information cache to form a scheduling state, scheduling decisions not only reflect the current instantaneous state of the system but also explicitly integrate historical scheduling information. This effectively alleviates scheduling fluctuations caused by relying solely on static or short-term information for decision-making, and improves the stability and continuity of the scheduling process.

[0067] This invention enhances channel-level features by introducing an SE-Channel-Temporal layer onto the state vector, generating first and second channel weights and fusing them to obtain the final channel weights. This enables adaptive modeling of the importance of different feature channels, overcoming the shortcomings of existing technologies that treat job features, waiting time, and resource status equally and fail to highlight key influencing factors. This allows the scheduling model to dynamically focus on feature information that has a more significant impact on scheduling results based on the system's operating status, thereby improving the precision of scheduling decisions.

[0068] This invention employs a gated Transformer-XL module combined with a sliding window mechanism to update local memory. By fusing historical scheduling features with current attention features through GRU gated units, it solves the problem that traditional sequence modeling methods are difficult to capture long-term dependencies and are easily limited by the length of the context window in long job queue scenarios. This enables the scheduling model to effectively utilize scheduling history information over a long period of time while ensuring computational efficiency, thereby improving scheduling consistency and global coordination in long queue scenarios.

[0069] This invention introduces a dynamically adjustable waiting queue window during the scheduling decision-making stage. The policy network selects jobs to be scheduled from the waiting queue window and updates the job waiting queue and node matrix in conjunction with the allocation or reservation mechanism of idle resource nodes. This avoids the problem of long-term waiting or starvation of jobs caused by traditional first-come-first-served or fixed-priority strategies when resources are scarce, and achieves a better balance between fairness and resource utilization efficiency in the scheduling process.

[0070] This invention constructs a multi-objective reward system that simultaneously considers resource utilization, job waiting time, system throughput, and job priority. It also updates the policy network and value network based on the near-end policy optimization (PPO) algorithm. This enables the scheduling policy to adaptively balance multiple performance indicators, overcoming the shortcomings of scheduling result bias caused by a single optimization objective. Overall, it improves resource utilization efficiency, job completion efficiency, and system performance in heterogeneous resource environments. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of the system structure provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The purpose of this invention is to provide a method and system for collaborative scheduling optimization of heterogeneous resources based on dynamic memory and reinforcement learning. Through multi-layer feature enhancement, long sequence memory modeling and multi-objective reward optimization, resource utilization and overall system operating efficiency are significantly improved while ensuring scheduling fairness.

[0076] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning, including:

[0078] Step 100: Obtain job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node;

[0079] Step 200: Construct a state vector based on job information and resource node status information, and combine the state vector with historical information cache to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status;

[0080] Step 300: Input the state vector into the SE-Channel-Temporal layer for feature enhancement, generate the first channel weight and the second channel weight respectively, and fuse the first channel weight and the second channel weight to obtain the final channel weight. Output the channel weighted feature based on the final channel weight.

[0081] Step 400: Input the channel weighted features into the gated Transformer-XL module, use the sliding window mechanism to update the local memory and maintain the memory cache, and use the GRU gated unit to fuse the historical scheduling features corresponding to the historical information cache with the current attention features output by the gated Transformer-XL module to obtain the updated historical information cache;

[0082] Step 500: Based on the scheduling status, the policy network outputs scheduling actions; the scheduling actions select jobs to be scheduled from the job waiting queue within the dynamically adjustable waiting queue window, and allocate or reserve idle resource nodes in the node matrix to the jobs to be scheduled, so as to update the job waiting queue, node matrix and scheduling status.

[0083] Step 600: Generate multi-objective rewards based on resource utilization, job waiting time, system throughput and job priority. Use the near-end policy optimization PPO algorithm to update the policy network and value network parameters using multi-objective rewards to obtain the scheduling policy.

[0084] Specifically, in step 100 of this embodiment, job information and resource node status information are acquired at the start of the scheduling process. The job information is formed from the submission records of jobs to be scheduled, and includes at least the job identifier, job submission time, number of resource nodes requested by the job, and the running time of the job request. The resource node status information is formed from the status records of resource nodes, and includes at least the resource node identifier, the current status of the resource node, and the remaining running time of the assigned jobs. Based on the job information, this embodiment uses the job submission time as the sorting criterion, sorts the jobs to be scheduled according to a first-in-first-out (FIFO) rule, and establishes a job waiting queue. The job waiting queue stores jobs that have not yet been scheduled, arranged sequentially according to their entry order, with the job at the head of the queue being the earliest submitted job that has not yet been scheduled. Within one scheduling cycle, this embodiment can acquire no fewer than 10 jobs to be scheduled and construct a job waiting queue with a queue length of 10. The scheduling cycle can be set to 1 second to limit the time granularity of job information acquisition and queue updates.

[0085] After obtaining the resource node status information, this embodiment further constructs a node matrix based on the resource node status information to characterize the availability status of each resource node. The "node matrix" is a data structure that collectively describes the resource node status. Its rows correspond to resource node identifiers, and its elements represent the status of the corresponding resource node within the current scheduling cycle. The status includes at least idle and occupied states, and may further include reserved states, indicating that the resource node has been allocated to a scheduled job but has not yet entered the execution phase. In this embodiment, the total number of resource nodes can be set to 64, and the node matrix is ​​updated in each scheduling cycle. When 8 resource nodes in the node matrix are identified as idle, it indicates that the number of currently available resource nodes is 8, and the remaining resource nodes are identified as occupied or reserved, thus providing a clear data basis for the allocation or reservation of idle resource nodes in subsequent steps.

[0086] Optionally, in step 200 of this embodiment, after completing the construction of the job waiting queue and node matrix, a state vector is constructed based on the job information and the resource node status information. Specifically, this embodiment first extracts job features based on the job information, wherein the "job features" are used to characterize the basic resource requirements of the job, including at least the number of resource nodes requested by the job and the running time of the job request; the number of resource nodes requested by the job indicates the number of resource nodes occupied by the job during execution, and the running time of the job request indicates the length of time the job continuously executes on the resource nodes. In this embodiment, the number of resource nodes requested by the job can be 1 to 8, and the running time of the job request can be 10 seconds to 300 seconds, thereby covering typical scheduling scenarios for short-term and medium-sized jobs.

[0087] This embodiment further determines the job waiting time based on the job information and the resource status based on the resource node status information. The "job waiting time" reflects the degree of waiting of a job in the job waiting queue, and its value is determined by the time difference between the current time and the job submission time. By introducing the job waiting time, the relative length of time a job stays in the queue can be reflected, which is used to characterize fairness in subsequent scheduling decisions. The "resource status" describes the availability of each resource node in the current scheduling period, including at least the idle or occupied status of each resource node in the node matrix. When a resource node is not occupied by any job, it is marked as idle; when a resource node has been assigned to a job and is in the execution phase, it is marked as occupied. With a total of 64 resource nodes, this embodiment can determine that 40 resource nodes are occupied and the remaining 24 resource nodes are idle within a certain scheduling period.

[0088] After obtaining the job characteristics, job waiting time, and resource status, this embodiment combines the job characteristics, job waiting time, and resource status according to a preset concatenation rule to obtain a state vector. The "preset concatenation rule" defines the order and correspondence of different types of information within the state vector to ensure that the state vector fully represents the current scheduling environment. Subsequently, this embodiment concatenates or aggregates the state vector with historical information cache according to a preset combination rule to form a scheduling state. The "historical information cache" stores scheduling state information formed within several historical scheduling cycles to reflect the temporal evolution characteristics of the scheduling process. In this embodiment, the historical information cache can store scheduling states formed within the last 20 scheduling cycles, thus enabling the current scheduling state to simultaneously reflect the current job and resource status as well as recent scheduling evolution trends.

[0089] As an optional implementation, this embodiment addresses the practical shortcomings of directly concatenating the job features, job waiting time, and resource status in a fixed order. These shortcomings include "feature dominance bias caused by differences in dimensions," "positional semantic drift caused by missing features or asynchronous updates," and "consistency in the concatenation order due to changes in key factors under different scheduling scenarios." Therefore, this embodiment defines the "preset concatenation rule" as an adaptive concatenation rule with semantic slot constraints. Here, a "semantic slot" refers to a fixed position segment reserved in the state vector for the job features, job waiting time, and resource status, respectively, to ensure that the same type of information occupies a consistent segment range in any scheduling cycle. Within a semantic slot, this embodiment allows adaptive sorting and encoding of sub-information within the slot, but does not allow cross-slot migration, thereby avoiding the situation where the same position carries different semantics in different scheduling cycles. The number of semantic slots is 3; the number of sub-information in each slot is not less than 2; and the scheduling cycle can be 1 second.

[0090] This embodiment further performs scale unification processing on the sub-information within each semantic slot before constructing the state vector, in order to eliminate the impact of scale differences between the job request runtime, the job waiting time, and the resource status statistics on the splicing result. Specifically, this embodiment uses a quantile-based normalization method to generate sub-information values ​​with a uniform scale for each sub-information, thus forming "scale-consistent sub-information":

[0091]

[0092] in, The values ​​of the sub-information to be scaled uniformly processed; The median of the sub-information within a preset statistical window; The interquartile range of the sub-information within the statistical window; To prevent positive numbers with a denominator of 0; The values ​​of the sub-information after scaling are used as input for subsequent concatenation. The statistical window length can be 20 scheduling cycles. The value can be 0.001; the runtime of the job request can be 10 to 300 seconds, and the job waiting time can be 1 to 60 seconds.

[0093] After completing the scaling process, this embodiment introduces a saliency score within each semantic slot to determine the order of sub-information within the slot. This allows the state vector to prioritize expressing sub-information that changes more drastically and is more sensitive to scheduling at different scheduling stages, overcoming the limitation of fixed-order concatenation failing to highlight key factors under dynamic loads. Specifically, this embodiment calculates the intensity of change of each sub-information between adjacent statistical windows and generates a saliency score accordingly.

[0094]

[0095] in, Score the saliency of the sub-information; The mean of the sub-information within the current statistical window; The mean of the sub-information within the previous statistical window; The standard deviation of the sub-information within the current statistical window; To prevent positive numbers with a denominator of 0; the significance score is used to sort multiple sub-information items within a slot, with a higher score indicating a more significant change and higher priority for that sub-information in the current stage. Both the current and previous statistical windows can be set to 10 scheduling cycles; when A value of 5 or higher is considered a significant change; A value of 0.001 is acceptable.

[0096] Based on the saliency score, this embodiment generates a slot-level sequence identifier within each semantic slot and uses this identifier to adaptively concatenate the state vector. Specifically, this embodiment arranges multiple scale-consistent sub-information within the same slot in descending order of saliency score to obtain the slot-level sequence identifier, and concatenates each slot in the slot order of job feature slot—job waiting time slot—resource status slot to obtain the state vector:

[0097]

[0098] in, The state vector; Concat This is an operation to splice the parts according to a predetermined sequence; These are the slot contents for the job characteristic slot, the job waiting time slot, and the resource status slot, respectively. These are the sequential identifiers within each slot, determined based on the salience score. Indicates the first The contents of each slot are rearranged according to the order within the slot. Among them, the job feature slot can contain 2 sub-information items; the job waiting time slot can contain 1 sub-information item; the resource status slot can contain at least 2 sub-information items (e.g., number of idle items, number of occupied items); the length of the concatenated status vector is not less than 5.

[0099] Finally, to avoid instability in the state vector structure caused by temporary absences or delayed updates of job information or resource node status information, this embodiment introduces missing placeholder constraints and historical information cache alignment constraints: When a sub-information in any slot is unavailable in the current scheduling cycle, this embodiment fills the corresponding position of the sub-information with a preset placeholder value and uses this placeholder value as a missing identifier in subsequent processing; simultaneously, when the state vector and the historical information cache together constitute the scheduling state, this embodiment aligns the historical scheduling features in the historical information cache with the same slot structure as the state vector, ensuring that the historical scheduling features are consistent with the current state vector at the slot level, thereby avoiding semantic mismatch between the historical information cache and the current state vector. The placeholder value can be 0; the placeholder strategy remains unchanged even when the number of consecutive missing occurrences reaches 3; the historical information cache can retain historical scheduling features from the most recent 20 scheduling cycles.

[0100] Optionally, in step 300 of this embodiment, after obtaining the state vector, the state vector is input into the SE-Channel-Temporal layer for feature enhancement processing. The SE-Channel-Temporal layer is a feature enhancement structure that characterizes channel correlation and extracts temporal dimension information from the input features. Here, "channel" represents the feature grouping corresponding to different information components formed according to a preset concatenation rule in the state vector, and "temporal dimension feature extraction" is used to extract time-varying correlation features from the state vector within a continuous scheduling period. In this embodiment, the temporal dimension feature extraction is performed on the state vector to obtain temporal enhanced features, so that subsequent channel modeling can simultaneously consider the current state information and its temporal changes.

[0101] This embodiment further performs first-path channel modeling and second-path channel modeling on the temporal enhancement features to generate first-channel weights and second-channel weights, respectively. The first-path channel modeling calculates the importance of each channel from a first-channel correlation perspective, outputting a first-channel weight corresponding to each channel. The second-path channel modeling calculates the importance of each channel from a second-channel correlation perspective, outputting a second-channel weight corresponding to each channel. By setting two channel modeling paths, this embodiment can characterize channel importance from different information aggregation methods, thereby avoiding feature enhancement bias caused by insufficient characterization of channel importance by a single path.

[0102] After obtaining the first channel weight and the second channel weight, this embodiment fuses the first channel weight and the second channel weight to obtain the final channel weight corresponding to each channel. Here, "fusion" is used to synthesize the channel importance results obtained from the two paths into a unified channel importance representation, forming a comprehensive weight for each channel. Subsequently, this embodiment performs channel-by-channel weighting on the time-series enhancement features based on the final channel weights to obtain channel-weighted features. Here, "channel-by-channel weighting" is used to amplify or suppress each channel feature according to the final channel weight, enhancing channel features that contribute more to scheduling decisions and weakening channel features that contribute less, thereby outputting the channel-weighted features for subsequent processing by the gated Transformer-XL module.

[0103] Further, in step 400 of this embodiment, after obtaining the channel-weighted features, the channel-weighted features are input into the gated Transformer-XL module for scheduling feature modeling. The gated Transformer-XL module is used to perform attention modeling on the input feature sequence and combine historical information to achieve feature association across scheduling cycles; wherein, the "input sequence" refers to the serialized representation composed of the channel-weighted features in a preset order, used to characterize the continuous change of the current scheduling state in the time dimension. This embodiment obtains the current attention feature used to characterize the current scheduling state by performing attention calculation on the input sequence, so as to reflect the degree of influence of features at different time positions on the current scheduling decision.

[0104] This embodiment further introduces a mechanism in the gated Transformer-XL module to update local memory and maintain the memory cache. The "memory cache" stores intermediate feature representations formed over multiple historical scheduling cycles, while the "local memory" selects partial historical information associated with the current input sequence from the memory cache. During current attention calculation, this embodiment uses the sliding window mechanism to read historical segments corresponding to the current input sequence from the memory cache as the local memory, and integrates the local memory with the current input sequence in the attention calculation. This ensures that the current attention feature simultaneously includes both historical and current scheduling information.

[0105] After completing the current attention calculation, this embodiment updates the local memory. Specifically, the sliding window mechanism is used to limit the range and order of reading and writing historical fragments from the memory cache, so that the local memory is updated as the scheduling process progresses. In this way, the local memory can continuously reflect the changes in scheduling state over the most recent scheduling cycles, avoiding the infinite accumulation of historical information that leads to memory redundancy, thereby maintaining the stability of scheduling feature modeling while ensuring the validity of historical information.

[0106] This embodiment further fuses the historical scheduling features corresponding to the historical information cache with the current attention features using a GRU gating unit. Here, "historical scheduling features" refers to the feature representations stored in the historical information cache, formed from previous scheduling cycles. The GRU gating unit is used to model the correlation between the historical scheduling features and the current attention features, and adjusts the proportion of the two types of features in the fusion result based on the gating mechanism. Through this fusion method, this embodiment can retain the validity of historical scheduling information while incorporating the latest changes in the current scheduling state, thereby forming a more continuous scheduling feature representation.

[0107] After obtaining the fusion result output by the GRU gating unit, this embodiment writes the fusion result into the memory cache and synchronously updates the historical information cache to form an updated historical information cache. The updated historical information cache is used to provide historical scheduling feature support for subsequent scheduling cycles, thereby enabling the scheduling process to have continuity and correlation in the time dimension. Through the above method, this embodiment implements a dynamic memory update mechanism based on the gated Transformer-XL module, enabling the scheduling model to effectively utilize historical information and continuously adjust the scheduling decision basis within multiple scheduling cycles.

[0108] As an optional implementation, in this embodiment, to overcome the shortcomings of the existing sliding window mechanism, which uses a fixed window length and causes local memory lag during load fluctuations and local memory redundancy during load stability, the sliding window mechanism is improved into an adaptive window length sliding window mechanism: within each scheduling cycle, based on the change in the queue length of the job waiting queue and the change in resource utilization reflected by the node matrix, a window adaptive factor is generated, and the window length corresponding to the local memory is determined accordingly. This allows the local memory to shorten the window when job arrival fluctuates greatly or resource status changes rapidly, and to extend the window when fluctuations are small, thereby enhancing the sensitivity of the local memory to recent changes and avoiding unnecessary accumulation of historical fragments.

[0109]

[0110] in, For this embodiment, in the first... The length of the sliding window, determined in each scheduling cycle, is used to extract historical fragments from the memory cache to form the local memory. This is the lower limit of the window length; This is the maximum window length. This is the window adaptation factor; For the first The length of the job waiting queue within each scheduling cycle. The queue length in the previous scheduling cycle; For the first The resource utilization rate for each scheduling cycle, wherein the resource utilization rate is determined by the ratio of the number of resource nodes in an occupied state to the total number of resource nodes in the node matrix. This represents the resource utilization rate of the previous scheduling cycle; This is the floor function. Specifically, The total number of resource nodes is 64. When the queue length changes from 30 to 45 and the resource utilization rate changes from 0.70 to 0.85, the window adaptation factor increases and causes the window length to decrease.

[0111] In this embodiment, to address the drawback of only extracting continuous historical segments while ignoring key historical segments more relevant to the current decision, the sliding window mechanism is further improved into a hierarchical local memory sliding window mechanism: when forming the local memory, in addition to extracting continuous historical segments determined by the sliding window length as recent segments, several historical segments with higher correlation to the current input sequence are selected from the memory cache as key segments, and the recent segments and the key segments together constitute the local memory, so as to improve the utilization efficiency of the gated Transformer-XL module for cross-cycle key state changes.

[0112]

[0113] in, For the first A score indicating the relevance of a historical segment to the current scheduling cycle; For this embodiment, in the first... The representation of the current attention features obtained by the gated Transformer-XL module for each scheduling cycle ↓ For the memory cache of the first The representation vector of each historical segment; is the inner product operation. It is the vector norm; This refers to the set of key segment indexes selected within the memory cache based on relevance scores; To select the top from the rating set Operations corresponding to each score index; It is a set of continuous historical segments extracted by the length of the sliding window; For the local memory; Concat This is an operation that merges the recent segments and the key segments in a predetermined order to form a local memory. Specifically, ; Continuous historical segments cover the most recent 12 scheduling cycles; The memory cache can store up to 200 historical segments.

[0114] In this embodiment, to avoid the defect of indiscriminately discarding critical historical information when the memory cache accumulates with the scheduling cycle and the cache length is limited, the sliding window mechanism is further improved into an update step size and eviction rule with importance constraints: after a new fragment is appended to the memory cache, the update step size is determined according to the memory cache capacity and the current resource utilization rate, and when the cache length exceeds the preset upper limit, historical fragments with lower relevance scores and outside the window are removed first, so that the memory cache maintenance process has a tendency to retain critical historical fragments.

[0115]

[0116] in, For the first The update step size for each scheduling cycle is used to limit the step size of each translation update when maintaining the memory cache in this embodiment; To update the lower limit of the step size, To update the step size limit; For the first Resource utilization rate for each scheduling cycle; This is for rounding up; The set of candidates to be eliminated consists of historical fragments that are outside the window; Score the relevance of historical fragments; The operation is to select the index corresponding to the lowest score; The index of the eliminated segment, determined in this embodiment, is used to remove the corresponding historical segment from the memory cache. Specifically, The maximum length of the memory cache is 200. When the resource utilization rate is 0.90, the update step size is taken to increase the update frequency. When the resource utilization rate is 0.40, the update step size is taken to extend the historical coverage.

[0117] Furthermore, in step 500 of this embodiment, after obtaining the scheduling state, a scheduling action is output by the policy network based on the scheduling state. The "scheduling action" is used to characterize the decision result of job selection and resource allocation in this embodiment within the current scheduling cycle; the "waiting queue window" is used to limit the range of jobs participating in the candidate selection in the job waiting queue in this embodiment, so as to avoid excessive dispersion of decision objects due to traversal selection across the entire queue. This embodiment first determines the window size of the waiting queue window, and dynamically adjusts the window size according to the queue length of the job waiting queue or the resource availability status represented by the node matrix; for example, when the queue length of the job waiting queue is 30 and the current number of idle resource nodes is 8, the window size of the waiting queue window is set to 10 to limit the size of the candidate job set.

[0118] In this embodiment, a waiting queue window is then established at the head of the job waiting queue, resulting in a set of candidate jobs within the window. The "candidate job set" represents the set of jobs located within the waiting queue window, with its members arranged in a first-in, first-out (FIFO) order according to the job waiting queue. The first job in the candidate job set is the earliest submitted job that has not yet been scheduled. As anchor data, with a window size of 10, the candidate job set contains 10 candidate jobs, with the first candidate job being the head job and the tenth candidate job being the end job, thus limiting the selection range of the policy network within the current scheduling cycle.

[0119] After obtaining the candidate job set, this embodiment outputs the scheduling action within the waiting queue window via the policy network. The scheduling action indicates the target job in the candidate job set, which is the job to be scheduled determined in the current scheduling cycle of this embodiment. To ensure the clear indication relationship of the scheduling action, this embodiment can set the scheduling action as an indication value for the job sequence number in the candidate job set; for example, when the scheduling action indication value is 3, it indicates that the 3rd candidate job in the candidate job set is determined as the job to be scheduled, thereby achieving a one-to-one correspondence between "scheduling action, target job, and job to be scheduled".

[0120] After determining the job to be scheduled, this embodiment determines a set of idle resource nodes for the job based on the node matrix. The "set of idle resource nodes" represents the set of resource nodes available for allocation within the current scheduling period, and its members are the resource nodes marked as idle in the node matrix. As anchor data, when the total number of resource nodes is 64 and 56 of them are marked as occupied, the set of idle resource nodes contains 8 idle resource nodes. This embodiment further reads the resource requirement information from the job characteristics corresponding to the job to be scheduled, for example, the job to be scheduled requests 4 resource nodes, for subsequent allocation or reservation determination.

[0121] When the set of idle resource nodes meets the resource requirements of the job to be scheduled, this embodiment allocates the idle resource nodes to the job to be scheduled and updates the available status of the corresponding resource nodes to the occupied status in the node matrix. For example, when the number of idle resource nodes is 8 and the number of resource nodes requested by the job to be scheduled is 4, 4 idle resource nodes are selected from the set of idle resource nodes and allocated to the job to be scheduled, and the status of the 4 resource nodes in the node matrix is ​​updated from idle to occupied. When the set of idle resource nodes does not meet the resource requirements of the job to be scheduled, this embodiment reserves the idle resource nodes for the job to be scheduled and writes a reservation identifier to the reserved resource nodes in the node matrix. For example, when the number of idle resource nodes is 2 and the number of resource nodes requested by the job to be scheduled is 4, the 2 idle resource nodes are reserved for the job to be scheduled, and a reservation identifier is written to the 2 resource nodes to limit their subsequent release to priority use by the job to be scheduled.

[0122] When subsequent resource nodes are released, this embodiment supplements the resource requirements of the scheduled job based on the reserved identifier and allocates the reserved resource nodes to the scheduled job. For example, when the number of subsequently released idle resource nodes reaches 2, combined with the 2 previously reserved resource nodes, the number of resource nodes available for the scheduled job reaches 4, thereby completing the supplementation of the resource requirements of the scheduled job and entering the allocation state. After the allocation or reservation is completed, this embodiment removes the scheduled job that has started execution from the job waiting queue. When the scheduled job has not yet started execution, this embodiment adjusts the queue position of the scheduled job in the job waiting queue and updates the scheduling status based on the node matrix and the historical information cache. As anchor data, this embodiment can complete one queue update and one scheduling status update at the end of each scheduling cycle, so that subsequent scheduling cycles can continue to output the scheduling action based on the updated job waiting queue, the node matrix, and the scheduling status.

[0123] Furthermore, in step 600 of this embodiment, after completing job selection and resource allocation or reservation, a multi-objective reward is generated based on resource utilization, job waiting time, system throughput, and job priority. First, the resource utilization is calculated based on the node matrix. The resource utilization is the ratio of the number of resource nodes in an occupied state to the total number of resource nodes; for example, when the total number of resource nodes is 64 and the number of resource nodes in an occupied state is 56, the ratio corresponding to the resource utilization is the ratio of 56 to 64. Through the resource utilization, this embodiment can characterize the impact of the current scheduling action on the resource occupancy level, so as to reflect the guidance on resource utilization efficiency in the reward.

[0124] This embodiment further generates a waiting time reward component or a waiting time penalty component based on the job waiting time, and generates the system throughput and throughput reward component based on the number of jobs completed per unit time. The waiting time reward component or waiting time penalty component reflects the waiting level of the jobs to be scheduled or jobs within the window. A waiting time reward component is generated when the job waiting time is short, and a waiting time penalty component is generated when the job waiting time is long; for example, a reward component is generated when the job waiting time is 3 seconds, and a penalty component is generated when the job waiting time is 30 seconds. The system throughput describes the number of jobs completed per unit time; for example, if 5 jobs are completed within 10 seconds, the system throughput is determined by the unit time length and the number of completed jobs, and a throughput reward component is generated based on the system throughput to encourage the completion of more jobs per unit time.

[0125] This embodiment also generates a priority reward component based on job priority, where job priority is used to characterize the relative urgency or scheduling importance of the job; for example, job priority can be set to an integer from 1 to 5, with a priority of 5 corresponding to a higher priority reward component, and a priority of 1 corresponding to a lower priority reward component. After obtaining the reward component corresponding to the resource utilization, the reward component or penalty component corresponding to the job waiting time, the reward component corresponding to the system throughput, and the reward component corresponding to the job priority, this embodiment combines the above components to obtain the multi-objective reward; the "combination" is used to form a unified reward result from multiple components, so that the scheduling strategy forms a comprehensive trade-off between resource utilization, waiting time, throughput, and priority.

[0126] After obtaining the multi-objective reward, this embodiment uses the Proximal Policy Optimization (PPO) algorithm to update the parameters of the policy network and value network, thereby obtaining a scheduling policy. Specifically, this embodiment calculates an advantage estimate based on the multi-objective reward and uses the advantage estimate as the optimization basis for updating the policy network; subsequently, pruning objectives are used to limit the policy update magnitude to update the policy network, ensuring that the policy update does not deviate too much from the existing policy. Simultaneously, this embodiment determines the value loss based on the difference between the multi-objective reward and the state value output by the value network, and updates the value network parameters with the value loss; and when updating the policy network, an entropy regularization term is introduced to improve exploration capability, for example, setting the coefficient of the entropy regularization term to 0.01, so that the policy network maintains the possibility of selecting different scheduling actions during training, thereby obtaining the scheduling policy for subsequent scheduling cycles.

[0127] As an optional implementation, this embodiment addresses the problems of existing near-end policy optimization algorithms in scheduling scenarios, such as the common use of a uniform update rhythm, difficulty in considering the differences in rewards for multiple objectives, and insufficient flexibility in responding to changes in environmental state. It improves the near-end policy optimization (PPO) algorithm to make it more suitable for heterogeneous resource collaborative scheduling scenarios. Specifically, this embodiment introduces an adaptive update control mechanism based on the degree of change in scheduling state during the policy update process. This mechanism increases the update frequency when the scheduling state changes drastically and decreases the update frequency when the scheduling state changes smoothly. This avoids policy adjustment lag when the scheduling environment changes rapidly, or unnecessary policy oscillations when the environment is stable.

[0128] This embodiment further addresses the issue of inconsistent impact of different reward components on scheduling performance in multi-objective rewards by improving the advantage estimation process of the near-end policy optimization (PPO) algorithm. When calculating the advantage estimate, this embodiment no longer simply treats multi-objective rewards as a single reward signal. Instead, it dynamically adjusts the contribution of each reward component to the advantage estimation based on the magnitude of change in each reward component within recent scheduling periods. This allows waiting time, resource utilization, system throughput, and job priority to have different impacts on policy updates at different scheduling stages, thereby making the policy network's update direction more aligned with the current scheduling objectives.

[0129] This embodiment also addresses the issue that traditional near-end policy optimization algorithms may limit the exploration of effective policies under fixed pruning constraints by improving the policy update constraint method. Specifically, this embodiment dynamically adjusts the constraint strength of policy updates based on the stability of the current scheduling state. When the scheduling state changes little, the policy update constraint is appropriately relaxed to enhance policy exploration capability; when the scheduling state changes significantly, the policy update constraint is tightened to prevent drastic policy fluctuations, thereby achieving a balance between policy stability and exploration capability.

[0130] Furthermore, this embodiment introduces a feedback adjustment mechanism based on historical scheduling performance during the policy update process. The scheduling performance trends reflected in the historical information cache are used as auxiliary information to participate in the policy network update decision. This ensures that the policy update not only depends on the reward result of the current scheduling cycle but also comprehensively considers the scheduling performance changes over multiple recent scheduling cycles. Through these improvements, this embodiment forms an improved near-end policy optimization algorithm for heterogeneous resource collaborative scheduling, enabling the scheduling policy to gradually evolve and continuously optimize in complex dynamic environments.

[0131] Corresponding to the above methods, such as Figure 2 As shown, this embodiment also provides a heterogeneous resource collaborative scheduling optimization system based on dynamic memory and reinforcement learning, including:

[0132] The job and resource information acquisition unit is used to acquire job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node.

[0133] The scheduling state construction unit is used to construct a state vector based on the job information and the resource node status information, and to combine the state vector with historical information cache to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status;

[0134] The channel feature enhancement unit is used to input the state vector into the SE-Channel-Temporal layer for feature enhancement, generate a first channel weight and a second channel weight respectively, fuse the first channel weight and the second channel weight to obtain the final channel weight, and output the channel weighted feature based on the final channel weight;

[0135] The dynamic memory update unit is used to input the channel weighted features into the gated Transformer-XL module, update the local memory and maintain the memory cache using a sliding window mechanism, and fuse the historical scheduling features corresponding to the historical information cache with the current attention features output by the gated Transformer-XL module through the GRU gated unit to obtain the updated historical information cache.

[0136] The scheduling decision and resource allocation unit is used to output scheduling actions from the policy network based on the scheduling status; the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, and allocates or reserves idle resource nodes in the node matrix to the job to be scheduled, so as to update the job waiting queue, the node matrix and the scheduling status.

[0137] The strategy optimization and reward update unit is used to generate multi-objective rewards based on resource utilization, job waiting time, system throughput and job priority, and update the policy network and value network parameters using the multi-objective rewards based on the near-end policy optimization (PPO) algorithm to obtain the scheduling strategy.

[0138] The beneficial effects of this invention are as follows:

[0139] (1) This invention introduces an adaptive splicing rule based on semantic slots, scale uniformity, saliency adaptive sorting and missing position constraints to structurally combine job features, job waiting time and resource status. This overcomes the problems of dimensional interference, semantic position drift and key feature submersion caused by fixed-order splicing in the prior art. It enables the state vector to maintain semantic consistency and expression integrity in different scheduling scenarios, thereby improving the accuracy of the scheduling state representation in depicting the actual scheduling environment.

[0140] (2) This invention improves the sliding window mechanism by introducing an adaptive window length, hierarchical local memory, and an update and elimination strategy for importance constraints. This enables the local memory to dynamically adjust the historical coverage as the job arrives and the resource status changes. This avoids the problem of memory lag when the load changes suddenly and memory redundancy when the load is stable in the fixed window mechanism, thereby enhancing the effectiveness and relevance of historical scheduling information in different operating stages.

[0141] (3) In the gated Transformer-XL module, the present invention combines the improved sliding window mechanism and the GRU gated fusion method to achieve dynamic balance fusion of historical scheduling features and current attention features, which solves the defects of long-term dependence and insufficient utilization efficiency of historical information in long job queue scenarios, enabling the scheduling model to perceive the scheduling evolution trend over a longer time scale, thereby improving the continuity of scheduling decisions and global coordination capabilities.

[0142] (4) By introducing a dynamically adjustable waiting queue window in the scheduling decision-making stage and combining it with the allocation and reservation mechanism of idle resource nodes, the present invention enables the policy network to make refined selections within a limited range of candidate jobs. At the same time, when resources are insufficient, available resources are locked in advance to avoid the problem of jobs waiting for a long time or being repeatedly skipped, thereby forming a better balance between scheduling efficiency and job fairness.

[0143] (5) This invention improves the near-end policy optimization (PPO) algorithm for scheduling scenarios by introducing a policy update adjustment mechanism based on scheduling state changes, multi-objective reward differences and historical scheduling performance. This makes the policy update process more in line with the dynamic characteristics of heterogeneous resource collaborative scheduling, avoids the problem of rigidity or oscillation in the update of traditional algorithms under complex multi-objective conditions, and improves the overall effect of scheduling strategy in terms of resource utilization, system throughput and job response performance.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0145] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning, characterized in that, include: Obtain job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node; A state vector is constructed based on the job information and the resource node status information, and the state vector and historical information cache are used together to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status; The state vector is input into the SE-Channel-Temporal layer for feature enhancement, generating a first channel weight and a second channel weight respectively. The first channel weight and the second channel weight are then fused to obtain the final channel weight. Based on the final channel weight, the channel weighted feature is output. The channel weighted features are input into the gated Transformer-XL module. The local memory is updated and the memory cache is maintained using a sliding window mechanism. The historical scheduling features corresponding to the historical information cache are fused with the current attention features output by the gated Transformer-XL module through the GRU gated unit to obtain the updated historical information cache. Based on the scheduling status, the policy network outputs a scheduling action; the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, and allocates or reserves idle resource nodes in the node matrix to the job to be scheduled, so as to update the job waiting queue, the node matrix and the scheduling status; A multi-objective reward is generated based on resource utilization, job waiting time, system throughput, and job priority. The PPO algorithm based on near-end policy optimization uses the multi-objective reward to update the parameters of the policy network and value network to obtain the scheduling policy.

2. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, A state vector is constructed based on the job information and the resource node status information, and the state vector, together with the historical information cache, constitutes the scheduling state, including: The job features are extracted based on the job information; the job features include at least the number of resource nodes in the job request and the running time of the job request; The job waiting time is determined based on the job information; the job waiting time is the difference between the current time and the job submission time. The resource status is determined based on the resource node status information; the resource status includes at least the idle or occupied status of each resource node in the node matrix; The state vector is obtained by combining the job characteristics, the job waiting time, and the resource status according to a preset concatenation rule. The scheduling state is obtained by concatenating or aggregating the state vector with the historical information cache according to a preset combination rule.

3. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, The state vector is input into the SE-Channel-Temporal layer for feature enhancement, generating first channel weights and second channel weights respectively. The first channel weights and second channel weights are then fused to obtain the final channel weights. Based on the final channel weights, channel-weighted features are output, including: Temporal dimension features are extracted from the state vector to obtain temporal enhancement features; The temporal enhancement features are modeled using the first path channel, and the weights of the first channel corresponding to each channel are output. The temporal enhancement features are modeled using a second path channel, and the weights of the second channel corresponding to each channel are output. The first channel weight and the second channel weight are fused to obtain the final channel weight that corresponds one-to-one with each channel; Based on the final channel weights, the temporal enhancement features are weighted channel by channel, and the channel-weighted features are output.

4. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, The channel-weighted features are input into the gated Transformer-XL module. A sliding window mechanism is used to update the local memory and maintain the memory cache. The historical scheduling features corresponding to the historical information cache are fused with the current attention features output by the gated Transformer-XL module using a GRU gating unit to obtain the updated historical information cache, including: The channel-weighted features are mapped to the input sequence of the gated Transformer-XL module and attention is calculated to output the current attention features; The sliding window mechanism is used to read historical segments corresponding to the current input sequence from the memory cache as local memory, and the local memory is updated after the current attention calculation is completed; The historical scheduling features and the current attention features in the historical information cache are input into the GRU gating unit, and the GRU gating unit outputs the fusion result; The fusion result is written into the memory cache and the historical information cache is updated synchronously to obtain the updated historical information cache.

5. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, A sliding window mechanism is used to update local memory and maintain the memory cache, including: Set the sliding window length and update step size, and extract continuous historical segments from the memory cache based on the sliding window length as the local memory; After each generation of the current attention feature, a new segment corresponding to the current input sequence is appended to the memory cache; When the length of the memory cache exceeds the preset upper limit, the earliest written historical fragment is removed from the memory cache according to the update step size; The local memory for the next time step is regenerated based on the updated memory cache.

6. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, The scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, including: The window size of the waiting queue is determined, and the window size is dynamically adjusted according to the queue length of the job waiting queue or the resource availability status represented by the node matrix. A waiting queue window is created at the head of the job waiting queue to obtain a set of candidate jobs within the window; The policy network outputs the scheduling action within the waiting queue window, and the scheduling action is used to indicate the target job in the candidate job set; The target job indicated by the scheduling action is identified as the job to be scheduled.

7. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, Allocating or reserving idle resource nodes in the node matrix for the scheduled job, in order to update the job waiting queue, the node matrix, and the scheduling status, includes: The set of idle resource nodes for the scheduled job is determined based on the node matrix; When the set of idle resource nodes meets the resource requirements of the job to be scheduled, the idle resource nodes are allocated to the job to be scheduled, and the available status of the corresponding resource node in the node matrix is ​​updated to the occupied status. When the set of idle resource nodes does not meet the resource requirements of the job to be scheduled, the idle resource nodes are reserved for the job to be scheduled, and a reservation identifier is written into the node matrix for the reserved resource nodes. When subsequent resource nodes are released, the resource requirements of the job to be scheduled are supplemented based on the reserved identifier, and the reserved resource nodes are allocated to the job to be scheduled. After allocation or reservation is completed, the scheduled jobs that have started execution are removed from the job waiting queue, or the queue positions of the scheduled jobs that have not yet started execution in the job waiting queue are adjusted, and the scheduling status is updated based on the node matrix and the historical information cache.

8. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, A multi-objective reward is generated based on resource utilization, job latency, system throughput, and job priority, including: The resource utilization rate is calculated based on the node matrix, and the resource utilization rate is the ratio of the number of resource nodes in an occupied state to the total number of resource nodes. Generate a waiting time reward component or a waiting time penalty component based on the job waiting time; The system throughput is generated based on the number of jobs completed per unit time, and a throughput reward component is generated based on the system throughput. A priority reward component is generated based on the job priority; The multi-objective reward is obtained by combining the reward component corresponding to the resource utilization rate, the reward component or penalty component corresponding to the job waiting time, the reward component corresponding to the system throughput, and the reward component corresponding to the job priority.

9. The heterogeneous resource collaborative scheduling optimization method based on dynamic memory and reinforcement learning according to claim 1, characterized in that, The Proximal Policy Optimization (PPO) algorithm updates the parameters of the policy network and value network using the multi-objective reward to obtain a scheduling policy, including: The advantage estimate is calculated based on the multi-objective reward, and the advantage estimate is used as the optimization basis for updating the policy network; The pruning target of the PPO algorithm is optimized using a near-end strategy to limit the policy update magnitude, so as to update the policy network; The value loss is calculated based on the difference between the multi-objective reward and the state value output by the value network, and the value network parameters are updated with the value loss. When updating the policy network, an entropy regularization term is introduced to improve the exploration capability, resulting in the scheduling policy.

10. A heterogeneous resource collaborative scheduling optimization system based on dynamic memory and reinforcement learning, characterized in that, include: The job and resource information acquisition unit is used to acquire job information and resource node status information, establish a job waiting queue based on the job information according to the first-in-first-out rule, and construct a node matrix based on the resource node status information to represent the availability status of each resource node. The scheduling state construction unit is used to construct a state vector based on the job information and the resource node status information, and to combine the state vector with historical information cache to form the scheduling state; wherein, the state vector includes job characteristics, job waiting time and resource status; The channel feature enhancement unit is used to input the state vector into the SE-Channel-Temporal layer for feature enhancement, generate a first channel weight and a second channel weight respectively, fuse the first channel weight and the second channel weight to obtain the final channel weight, and output the channel weighted feature based on the final channel weight; The dynamic memory update unit is used to input the channel weighted features into the gated Transformer-XL module, update the local memory and maintain the memory cache using a sliding window mechanism, and fuse the historical scheduling features corresponding to the historical information cache with the current attention features output by the gated Transformer-XL module through the GRU gated unit to obtain the updated historical information cache. The scheduling decision and resource allocation unit is used to output scheduling actions from the policy network based on the scheduling status; the scheduling action selects a job to be scheduled from the job waiting queue within a dynamically adjustable waiting queue window, and allocates or reserves idle resource nodes in the node matrix to the job to be scheduled, so as to update the job waiting queue, the node matrix and the scheduling status. The strategy optimization and reward update unit is used to generate multi-objective rewards based on resource utilization, job waiting time, system throughput and job priority, and update the policy network and value network parameters using the multi-objective rewards based on the near-end policy optimization (PPO) algorithm to obtain the scheduling strategy.

Images