Deep reinforcement learning task scheduling method fusing heterogeneous computing power

Abstract

The application discloses a kind of fusion isomorphic computing power deep reinforcement learning task scheduling method, belong to computer task scheduling technical field.The method includes: the initial isomorphic computing power platform state is extracted, including the real-time load of each computing unit, computing capacity index and energy consumption efficiency parameter;Deep learning task is analyzed, including the computing type of task, data dependency relationship and priority constraint;The output of feature extraction and task characteristic analysis is input, and the preset deep reinforcement learning model is executed task scheduling decision, and the mapping relationship of task to computing unit is output;According to mapping relationship, scheduling operation is executed, including allocating task to target computing unit, adjusting task execution order and configuring resource parameter;Real-time acquisition of the state data of computing platform after scheduling is fed back to deep reinforcement learning model to update model parameters.

Description

Technical Field

[0001] This invention relates to the field of computer task scheduling technology, specifically to a deep reinforcement learning task scheduling method that integrates heterogeneous computing power. Background Technology

[0002] Large-scale deep learning training tasks suffer from significant scheduling inefficiencies when running on heterogeneous computing platforms containing CPUs, GPUs, neural network processors, and FPGAs. Existing technologies typically rely on static scheduling strategies based on heuristic rules, allocating tasks by manually setting load thresholds or resource matching rules. These methods fail to recognize the data dependency graph within tasks and the instantaneous energy efficiency fluctuations of individual computing units when facing dynamically changing task flows and platform states. This leads to sub-task blocking on critical paths, idle heterogeneous computing resources, and increased overall power consumption. Some research has attempted to introduce traditional reinforcement learning methods for dynamic scheduling, but their state representation only aggregates a single dimension of load average or task number, losing architectural-level instruction set characteristics and memory access bandwidth differences. This makes it difficult for policy networks to distinguish the processing efficiency of different computing units for computationally intensive or memory-intensive tasks, limiting the accuracy of mapping decisions. Meanwhile, conventional reinforcement learning models often employ a single-step decision-making approach, outputting only the target unit of one task to be scheduled at a time. This fails to generate a comprehensive mapping scheme for a batch of subtasks in parallel. When there are strict priority constraints and complex dependencies between subtasks, this can easily lead to oscillations in the execution sequence, prolonging the overall task completion time. In the training feedback phase, existing methods only use the total task execution time as the reward signal, ignoring fine-grained environmental feedback such as peak resource usage and cache hit rate during execution. This results in slow model convergence and insensitivity to local platform overload. Therefore, how to extract multi-dimensional hierarchical state representations from heterogeneous computing platforms and construct a scheduling decision-making mechanism capable of parallel processing of task dependencies has become an urgent problem to be solved in improving the scheduling performance of complex deep learning tasks. Summary of the Invention

[0003] The purpose of this invention is to provide a deep reinforcement learning task scheduling method that integrates heterogeneous computing power, so as to solve the problems of loss of architectural difference information in state representation, inability of decision-making mechanism to process sub-task dependencies in parallel, and insufficient model generalization ability caused by coarse environmental feedback granularity in existing scheduling technologies.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a deep reinforcement learning task scheduling method that integrates heterogeneous computing power. By extracting multi-level features from the state of heterogeneous computing power platforms and combining the computational characteristics of deep learning tasks, the method uses a deep reinforcement learning model to generate a mapping decision from tasks to computing power units and dynamically adjusts the scheduling strategy. Thus, under the condition of satisfying task dependencies and priority constraints, the method achieves efficient utilization of computing power resources and reduces task execution latency.

[0005] As a technical solution of this invention, during the feature extraction process, real-time load characteristics, computing power index characteristics, energy efficiency parameter characteristics, and physical constraint characteristics are collected from the monitoring interfaces, hardware configuration registers, power management units, and operating system scheduling logs of computing power units such as the central processing unit, graphics processing unit, neural network processor, and field-programmable gate array. This multi-dimensional state characterization method enables the platform state information to truly reflect the immediate availability and operating environment of each computing power unit, providing accurate decision-making basis for reinforcement learning models.

[0006] As a further technical solution of this invention, when performing task characteristic analysis on deep learning tasks, historical task execution logs are obtained from the task queue of the heterogeneous cluster. Information such as operator type and input / output data tensor dimensions are extracted and matched with a preset algorithm library to identify the computationally intensive, memory-intensive, or communication-intensive type of the task. Simultaneously, a task dependency graph is obtained from the task dependency description file to clarify the data transfer direction and sequence constraints between subtasks. Based on this, subtasks are divided, and task feature vectors containing subtask identifiers, data size, and dependency depth are generated. This analysis process effectively characterizes the computational and communication characteristics of the task, helping the model perceive the differentiated requirements of different tasks for heterogeneous computing power.

[0007] The deep reinforcement learning model employs an asynchronous dominant actor-commentator network structure, comprising a parallel policy network and a value function network. After concatenating the task feature vector with the computing platform state data to form a joint state vector, it is input into the aforementioned network. The policy network outputs the mapping probability distribution from each subtask to each computing unit, and the value function network outputs the value estimate of the current state. A deterministic policy gradient algorithm is used to determine the mapping relationship from the mapping probability distribution, ensuring the stability and exploration efficiency of scheduling decisions in the continuous action space. Preferably, both the policy network and the value function network employ a recurrent neural network structure containing long short-term memory layers to process the temporal state information of heterogeneous computing platforms and task queues, enabling the model to capture the long-term impact of historical scheduling behavior on the current state.

[0008] When executing scheduling operations based on the mapping relationship output by the policy network, the system sends subtask execution instructions containing operator code, input data addresses, and expected output memory regions to the target computing unit. Simultaneously, based on the current task execution order and subtask priority constraints, the system adjusts the insertion position of subtasks in the task queue and sets configuration parameters such as memory allocation strategies and thread concurrency based on the hardware characteristics of the target computing unit. Through this process, the scheduling operation not only satisfies the direct assignment of tasks to computing units but also takes into account execution order optimization and underlying resource parameter adaptation, significantly reducing waiting between tasks and computing fragmentation.

[0009] The feedback loop of scheduling execution is a key feature of this invention. During subtask execution, real-time monitoring data such as task execution time, peak resource usage, and cache hit rate reported by each computing unit are collected. The time deviation value is calculated and combined with the peak resource usage and cache hit rate to form an environmental feedback vector, which is input to the value function network to calculate the temporal difference error. The model parameters are updated based on this temporal difference error. Specifically, the gradients of the loss functions of the policy network and the value function network are calculated. A near-end policy optimization pruning function is used to constrain the gradient of the policy network's loss function, generating a truncated gradient update. Then, the Adam optimizer asynchronously updates the weight parameters of the two networks, and the gradients from multiple parallel processes are accumulated and averaged. Preferably, the pruning threshold ε in the near-end policy optimization pruning function is set to 0.2, and the learning rate of the Adam optimizer is set to 0.001 to balance training stability and convergence speed. This feedback update mechanism enables the deep reinforcement learning model to adapt online to load changes and platform state fluctuations, continuously optimize the scheduling strategy, and constantly improve resource utilization and task execution efficiency.

[0010] In a preferred embodiment of the present invention, a task pre-allocation step is further included before scheduling decisions. Based on the dependency depth and parallelism of subtasks in the task dependency graph, the critical path of the task dependency graph is identified. The initial subtask on the critical path is pre-allocated to an idle computing unit that has a direct data channel with the computing unit executing the initial subtask. Input data for subsequent subtasks is pre-loaded into the data cache of the idle computing unit. By establishing data channels on the critical path in advance and completing data pre-fetching, the data transmission latency between heterogeneous computing powers is effectively hidden, further compressing the overall task completion time.

[0011] This invention organically integrates multi-dimensional state awareness, in-depth task characteristic analysis, asynchronous deep reinforcement learning decision-making, fine-grained scheduling execution, and online feedback updates based on near-end policy optimization to form an adaptive heterogeneous computing power task scheduling scheme. When faced with complex deep learning tasks and dynamically changing computing platform states, this scheme can generate scheduling decisions that take into account computation, memory access, and communication characteristics, significantly improving the overall throughput of heterogeneous computing resources while reducing average task waiting time and system energy consumption, demonstrating strong engineering practicality.

[0012] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Based on multi-level feature extraction of the heterogeneous computing platform state, real-time load rate, instruction set architecture information, peak floating-point operation, memory bandwidth parameters, energy consumption per unit time, cooling system temperature, task completion time, and context switching count are collected from monitoring interfaces, hardware configuration registers, power management units, and operating system scheduling logs, forming a joint state space that integrates real-time load, computing power, energy efficiency, and physical constraints. This joint state space injects the underlying architectural characteristics and transient physical state differences of each type of computing unit into the deep reinforcement learning model, enabling the policy network to accurately distinguish the large cache and low latency characteristics of the CPU and the high concurrency memory bandwidth advantage of the GPU based on the computational, memory access, or communication-intensive nature of the task, thereby eliminating scheduling mismatches caused by the flattening of state information. The data transmission direction and sequence constraints in the task dependency graph are transformed into subtask partitioning and priority ranking, generating a task feature vector containing subtask identifiers, data size, and dependency depth. This feature vector is then concatenated with the joint state vector and input into the asynchronous dominant actor-critic network. The policy network in this network processes temporal states using a recurrent structure containing long short-term memory (LSM) layers, outputting the probability distribution of each subtask's mapping to each computing unit in parallel. A deterministic policy gradient algorithm is then used to generate a one-to-one mapping between all subtasks and target computing units in a single step. This mapping mechanism avoids the fragmentation of dependency chains during sequential subtask-by-subtask decision-making, ensuring that subtasks on the critical path are assigned to computing units with direct data channels, while simultaneously reducing communication waiting overhead between subtasks from a global perspective. In the environment feedback loop, task execution time, peak resource usage, and cache hit rate are collected. The time deviation from the expected execution time is calculated and combined into an environment feedback vector, which is input to the value function network to calculate the temporal difference error. Based on this temporal difference error, a near-end policy optimization pruning function is used to truncate the gradient of the policy network's loss function. The Adam optimizer, combined with the cumulative averaging of gradients from multiple parallel processes, is used to perform asynchronous weight updates on the policy network and the value function network. Fine-grained environmental feedback provides real-time signals such as local overload and cache jitter to gradient updates, while pruning constraints prevent high-variance gradients from significantly perturbing the policy, enabling the model to continuously and stably approach the optimal scheduling policy under dynamic loads on heterogeneous platforms. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 This is a flowchart of a deep reinforcement learning task scheduling method that integrates heterogeneous computing power; Figure 2 This is a flowchart of the multi-dimensional feature acquisition process for heterogeneous computing resources; Figure 3 This is a flowchart of the task feature vector generation method; Figure 4 It is a flowchart of heterogeneous computing power task scheduling based on policy network and real-time feedback; Figure 5 This is a flowchart of heterogeneous computing power task scheduling based on critical path pre-allocation and asynchronous advantage actor-critic network. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] See Figure 1 This invention provides a deep reinforcement learning task scheduling method integrating heterogeneous computing power, comprising: extracting features from the initial heterogeneous computing power platform state, the initial heterogeneous computing power platform state including the real-time load, computing power indicators, and energy efficiency parameters of each computing power unit; performing task characteristic analysis on the deep learning task, the task characteristic analysis including the task's computation type, data dependencies, and priority constraints; taking the output of the feature extraction step and the output of the task characteristic analysis as input, and executing task scheduling decisions by a preset deep reinforcement learning model, the deep reinforcement learning model outputting the mapping relationship between tasks and computing power units; performing scheduling operations according to the mapping relationship, the scheduling operations including allocating tasks to target computing power units, adjusting the task execution order, and configuring resource parameters; and collecting the scheduling-post-scheduling computing power platform state data in real time, feeding the state data back to the deep reinforcement learning model, and updating the model parameters.

[0017] Example 1: In specific implementation, please refer to Figure 2The system collects real-time load rates, task queue lengths, and memory bandwidth utilization rates for each computing unit (CPU, GPU, neural network processor, and FPGA) from their monitoring interfaces. These data serve as real-time load characteristics. The CPU's monitoring interface can be the ` / proc / loadavg` file exported by the operating system kernel. Reading this file yields the 1-minute average load rate, which is then used as the CPU's real-time load rate. The GPU's monitoring interface can be a management library function provided by the GPU manufacturer. For example, calling the `nvmlDeviceGetUtilizationRates` function yields the GPU core utilization rate, which is then used as the GPU's real-time load rate. The neural network processor's monitoring interface can be a device status query interface provided by the neural network processor driver. Sending a status query command to this interface retrieves the percentage of computational cores used by the neural network processor, which is then used as the neural network processor's real-time load rate. The monitoring interface for the Field-Programmable Gate Array (FPGA) computing unit can be the board-level status register read interface provided by the FPGA board management controller. By reading the load monitoring register field in this board-level status register, the percentage of activity of the internal logic resources of the FPGA is obtained, serving as the real-time load rate of the FPGA computing unit. For task queue length acquisition, the count of pending tasks maintained by each computing unit's task scheduler is read. The task queue length of the CPU computing unit is obtained by reading the `nr_running` field of the run queue in the operating system scheduling entity; the task queue length of the GPU computing unit is obtained by calling the function in the GPU driver interface that retrieves the length of the command buffer to be executed; the task queue length of the neural network processor computing unit is obtained by reading the difference between the neural network processor task submission queue pointer and the completed task pointer; and the task queue length of the FPGA computing unit is obtained by reading the pending descriptor counter in the FPGA task management module. Memory bandwidth utilization is obtained through the hardware performance counters of each computing unit. The memory bandwidth utilization of the central processing unit (CPU) computing unit is obtained by reading the ratio of bus-occupied clock cycles to total clock cycles from the memory controller performance monitoring unit; the memory bandwidth utilization of the graphics processing unit (GPU) computing unit is obtained by reading the ratio of video memory read / write throughput to the theoretical peak bandwidth of video memory from the graphics processing unit performance monitor; the memory bandwidth utilization of the neural network processor (NN) computing unit is obtained by reading the memory interface data transfer utilization from the neural network processor on-chip network monitoring unit; and the memory bandwidth utilization of the field-programmable gate array (FPGA) computing unit is obtained by reading the bandwidth monitoring register from the FPGA memory interface generation module.

[0018] In practice, instruction set architecture information, peak floating-point performance, and video memory bandwidth parameters are read from the hardware configuration registers of each computing unit as indicators of computing power. Instruction set architecture information is read by accessing the feature identification registers of the processor cores of each computing unit. The CPU computing unit obtains the instruction set architecture type identifier, such as the AVX-512 identifier, by executing the CPUID instruction and parsing the return value of the EAX register; the graphics processing unit (GPU) computing unit obtains the major and minor version numbers of the instruction set architecture by reading the computeCapability field in the GPU device attribute structure; the neural network processor (NNF) computing unit obtains the instruction set architecture version by reading the ISA_version field in the NNF firmware information block; and the field-programmable gate array (FPGA) computing unit obtains the instruction set architecture information by parsing the architecture identifier field in the FPGA bitstream file header. The peak floating-point performance is calculated using the computing unit specifications and operating frequency recorded in the hardware configuration registers. The peak floating-point performance of the CPU (CPU) is obtained by multiplying the number of single-instruction multiple-data (SIMDD) floating-point operands per clock cycle, the number of cores, and the highest frequency. These parameters are obtained from the model-specific registers MSR_PLATFORM_INFO and MSR_TURBO_RATIO_LIMIT, respectively. The peak floating-point performance of the GPU (Graphics Processing Unit) is calculated by reading the multiprocessorCount, the number of CUDA cores in each multiprocessor, and the maximum operating frequency from the GPU length information structure. The peak floating-point performance of the Neural Network Processing Unit (NNPM) is calculated by reading the number of rows and columns of the NNPM's computing array and the clock cycle time; these parameters are read from the NNPM configuration register. The peak floating-point performance of the FPGA (Field-Programmable Gate Array) is calculated by reading the number of digital signal processing units (DSPs) and their operating frequencies within the FPGA, combined with the number of floating-point multiply-accumulate operations that each DSP can complete per cycle. Memory bandwidth parameters are read from the memory configuration registers or device attributes of each computing unit. The memory bandwidth parameters of the graphics processing unit (GPU) are calculated by reading the memoryClockRate and memoryBusWidth information in the GPU memory information structure to obtain the peak memory bandwidth. The memory bandwidth parameters of the neural network processor (NN) are obtained by reading the double data rate memory interface rate and data bus width register in the NN memory subsystem. For CPUs and FPGAs that are not equipped with independent memory, the memory bandwidth parameters are replaced by the system memory bandwidth that communicates with the main memory, and are calculated by reading the memory frequency and channel number information in the system management basic input / output system.

[0019] In practical implementation, the unit-time energy consumption value and the real-time temperature value of the cooling system are obtained from the power management unit of each computing unit as energy efficiency parameters. The unit-time energy consumption value is obtained by accessing the power monitoring circuit of each computing unit. The central processing unit (CPU) computing unit obtains the cumulative energy consumption by reading the PKG_ENERGY_STATUS model-specific register in the Running Average Power Limit (RAPL) interface, and obtains the unit-time energy consumption value by dividing the energy difference between adjacent sampling intervals by the time interval; the graphics processing unit (GPU) computing unit obtains the instantaneous power consumption value by calling the nvmlDeviceGetPowerUsage interface, which is the unit-time energy consumption value; the neural network processor (NN) computing unit calculates the instantaneous power consumption value as the unit-time energy consumption value by reading the output voltage of the NN processor current sensing amplifier and combining it with the board power supply voltage; the field-programmable gate array (FPGA) computing unit directly obtains the wattage value as the unit-time energy consumption value by reading the power register in the FPGA system monitoring module. The real-time temperature value of the cooling system is read by the on-chip temperature sensor of each computing unit. The central processing unit (CPU) computing unit obtains the chip junction temperature by reading the digital temperature readout value from the model-specific register IA32_THERM_STATUS; the graphics processing unit (GPU) computing unit obtains the real-time temperature value by calling the nvmlDeviceGetTemperature interface and specifying the temperature sensor type as the GPU core; the neural network processor (NN) computing unit obtains the chip temperature through the NN processor temperature sensor interface register; and the field-programmable gate array (FPGA) computing unit obtains the real-time temperature value by reading the temperature sensor channel data monitored by the FPGA system.

[0020] In practical implementation, the task completion time and context switch count recorded in the operating system scheduling logs of each computing unit are used as physical constraint characteristics. The task completion time is obtained as follows: when the task scheduler of each computing unit marks a task as completed, the system timestamp is captured, and the task completion time is obtained by subtracting the task submission timestamp from the system timestamp; the task completion time of the CPU computing unit is obtained by calculating the difference between the task's execution start time and execution end time in the scheduling entity; the task completion time of the GPU computing unit is obtained by parsing the timestamp attached to the GPU command buffer completion interrupt signal; the task completion time of the neural network processor computing unit is obtained by reading the completion timestamp field of the task completion flag register in the task management unit; and the task completion time of the FPGA computing unit is obtained by reading the timestamp of the FPGA task completion interrupt propagation. The context switch count is obtained from the context switch cumulative counter in the scheduling domain corresponding to each computing unit maintained by the operating system kernel; the context switch count of the central processing unit computing unit is obtained by reading the ctxt field of the corresponding logical core in the / proc / stat file; the context switch count of the graphics processing unit computing unit is obtained from the context switch count field in the sysfs node exposed by the graphics processing driver; the context switch count of the neural network processor computing unit is obtained by reading the context switch statistics interface of each device instance in the neural network processor driver; the context switch count of the field-programmable gate array computing unit, due to the use of hardware task switching, is obtained by reading the reconfigurable region state switch count register.

[0021] Example 2: In specific implementation, please refer to Figure 3The system collects historical task execution logs from the task queue of a heterogeneous cluster. These logs include the operator type, input data tensor dimension, and output data tensor dimension for each task. The task queue in the heterogeneous cluster can be the central task queue in a distributed task management system. The collection process is implemented by calling the log query interface of the task management system, which returns historical task execution logs organized in JSON format. The operator type is obtained by parsing the "operator_type" field in each task entry of the historical task execution log. The operator type value includes string identifiers such as convolution operator, fully connected operator, pooling operator, activation function operator, batch normalization operator, or custom operator. The input and output data tensor dimensions are obtained by parsing the "input_shape" and "output_shape" fields in each task entry of the historical task execution log. The input data tensor dimension is recorded as a four-dimensional integer array, representing the batch size, number of channels, height, and width, respectively. The output data tensor dimension is also recorded as a four-dimensional integer array with the same meaning as the input data tensor dimension.

[0022] In practice, operator types are matched against a pre-defined algorithm library to identify the computationally intensive, memory-intensive, and communication-intensive types of tasks. The pre-defined algorithm library is a collection of description files containing the computational characteristics of various deep learning operators. Each operator type in the collection corresponds to an entry, which records the ratio of the operator's floating-point operations to its data access volume. The matching process involves using the operator type string as the key to search a hash table constructed from the pre-defined algorithm library to obtain the corresponding operator's computation-to-memory ratio. The calculated memory-to-memory ratio is determined by the following formula:

[0023] in, This indicates the calculation of the memory access ratio. This indicates the total number of floating-point operations required for the operator to complete one forward computation. This indicates the total number of bytes that the operator needs to read from and write from memory to complete one forward computation. The value of is obtained by analyzing the computation graph of the operator, and its value is equal to the sum of all multiplication and addition operands inside the operator. The value is obtained by considering the total data volume of the input tensor, output tensor, and intermediate temporary tensor of the operator. The data volume of each tensor is obtained by multiplying the number of elements in the tensor by the number of bytes per element. When the computation-to-memory access ratio is greater than the preset upper threshold for dense types, the task corresponding to the operator is identified as a computationally intensive type. The upper threshold for dense types is set to 50.0, which is theoretically calculated and indicates that under mainstream hardware architectures, when the ratio of floating-point operations to memory access bytes exceeds 50.0, task performance is mainly constrained by computing power. When the computation-to-memory access ratio is less than the preset lower threshold for dense types, the task corresponding to the operator is identified as a memory-intensive type. The lower threshold for dense types is set to 5.0, which is theoretically calculated and indicates that when the computation-to-memory access ratio is less than 5.0, task performance is mainly constrained by memory bandwidth. When an operator type is marked as involving cross-device data transfer in the preset algorithm library, and the memory access ratio is between the two thresholds mentioned above, the task corresponding to the operator is identified as a communication-intensive type. The marking of cross-device data transfer is determined by the "requires_allreduce" boolean field in the algorithm library entry. When the "requires_allreduce" field is true, it indicates that the operator execution process includes cross-device communication operations such as gradient synchronization or feature map aggregation.

[0024] In practice, a task dependency graph is extracted from the dependency description file of each task. This graph contains the data transfer direction and sequence constraints between subtasks. The dependency description file is a protocol buffer file or structured text file generated by the deep learning framework when constructing the computation graph. Internally, it records the dependencies between nodes in the form of a list of directed edges. The process of extracting the task dependency graph involves reading the dependency description file and parsing all node definition statements and node connection statements. Each node definition statement corresponds to a subtask and contains a globally unique identifier for that subtask. Each node connection statement contains a source node identifier and a target node identifier, indicating that data is transferred from the source node to the target node, and implicitly implying that the target node can only begin execution after the source node has finished executing. Based on the parsed node definition statements and node connection statements, a directed acyclic graph (DAG) data structure is constructed. Vertices in the DAG correspond to subtasks, and directed edges correspond to the data transfer direction and sequence constraints, pointing from the source node to the target node.

[0025] In practice, based on the task dependency graph, each task is divided into subtasks and prioritized, generating a task feature vector containing subtask identifiers, data size, and dependency depth. Subtasks are divided by directly using each vertex in the task dependency graph as an independent subtask. Each subtask inherits the operator type, input tensor shape, and output tensor shape attributes of that vertex from the dependency description file. Prioritization is performed using a critical path-based list sorting algorithm to calculate the priority value of all subtasks in the task dependency graph; subtasks with higher priority values ​​are assigned to execution earlier. The priority value of a subtask is determined by the length of the longest path from the subtask vertex to the end vertex of the task dependency graph. This longest path length is measured by the cumulative computation time of all vertices along the path. The computation time of each vertex is obtained by querying an operator time database based on the operator type and input data size of the subtask corresponding to that vertex. The data size is calculated as follows: for each subtask, the total number of input tensor elements and the total number of output tensor elements of the subtask are added together, and then multiplied by the number of bytes occupied by a single data element. The number of bytes occupied by a single data element is determined according to the data type of the operator; single-precision floating-point data types occupy 4 bytes, and half-precision floating-point data types occupy 2 bytes. The dependency depth is calculated as follows: in the task dependency graph, the number of vertices contained in the longest directed path from the starting subtask vertex to the current subtask vertex is used as the dependency depth of the current subtask. The dependency depth of the starting subtask vertex itself is 1. The final generated task feature vector is a quadruple containing the subtask identifier, the type encoding (compute-intensive, memory-intensive, or communication-intensive), the data size, and the dependency depth. The subtask identifier uses a globally unique identifier string defined in the dependency description file. The type encoding encodes compute-intensive types as 0, memory-intensive types as 1, and communication-intensive types as 2. The data size is represented as a 64-bit unsigned integer value in bytes, and the dependency depth is represented as a 32-bit unsigned integer value.

[0026] Example 3: In its implementation, an asynchronous dominant actor-critic network is constructed. This network comprises a parallel policy network and a value function network. Both the policy network and the value function network employ a recurrent neural network structure with long short-term memory (LSM) layers to process temporal state information. The overall architecture of the asynchronous dominant actor-critic network consists of a global network and multiple parallel processes. Each parallel process contains an independent policy network instance and an independent value function network instance. All policy network instances share network weight parameters with the global policy network, and all value function network instances share network weight parameters with the global value function network. The internal structure of the policy network is as follows: input layer, first fully connected layer, LSM layer, second fully connected layer, and output layer. The dimension of the input layer is the same as the dimension of the joint state vector, and the input layer receives the joint state vector as input. The first fully connected layer contains 256 neurons, and the activation function is a rectified linear unit function. The number of hidden units in the LSM layer is set to 128. The LSM layer receives the output of the first fully connected layer as sequential input and maintains internal cell states and hidden states to capture the temporal dependencies of continuous decisions during task scheduling. The output of the Long Short-Term Memory (LSTM) network layer is passed to the second fully connected layer, which contains 128 neurons and uses the rectified linear unit function (RCU) as the activation function. The output layer contains neurons of the same size as the action space, which is equal to the product of the total number of currently scheduled subtasks and the total number of computing units. The output layer uses the softmax function to normalize the output of each neuron, generating a probability distribution mapping each subtask to each computing unit. The internal structure of the value function network is as follows: input layer, first fully connected layer, LSM network layer, second fully connected layer, and output layer. The dimension of the input layer of the value function network is also the same as the dimension of the joint state vector. The first fully connected layer contains 128 neurons and uses the RCU as the activation function. The number of hidden units in the LSM network layer is set to 64. The second fully connected layer contains 64 neurons and uses the RCU as the activation function. The output layer contains only one neuron, does not use an activation function, and outputs a scalar as a value estimate of the current state.

[0027] In practice, the task feature vector and the computing platform state data are concatenated into a joint state vector, which is then input into the policy network and the value function network. The task feature vector is composed of four sequentially concatenated parts: the one-hot encoded vector of the subtask identifier, the one-hot encoded vector of the data type encoding, the normalized value of the data size, and the normalized value of the dependency depth. The length of the one-hot encoded vector of the subtask identifier is equal to the total number of subtasks in the task dependency graph; the position corresponding to the current subtask is set to 1, and the remaining positions are set to 0. The unique encoded vector of the data type encoding has a length of 3, with the three positions corresponding to computationally intensive, memory-intensive, and communication-intensive types, respectively. The corresponding position is set to 1 based on the task's computationally intensive, memory-intensive, or communication-intensive type. The normalized value of the data size is obtained by normalizing the original data size in bytes using a maximum-minimum normalization method. The upper limit of normalization is the maximum data size of historically scheduled tasks in the heterogeneous cluster, and the lower limit is 0. The normalized value of dependency depth is obtained by normalizing the original dependency depth value using a maximum-minimum method. The upper limit of normalization is the maximum depth of the task dependency graph, and the lower limit is 0. The computing platform state data consists of a concatenated vector of real-time load rates for all computing units, a normalized concatenated vector of task queue lengths for all computing units, a concatenated vector of memory bandwidth utilization for all computing units, a normalized concatenated vector of peak floating-point operations for all computing units, a normalized concatenated vector of energy consumption per unit time for all computing units, and a normalized concatenated vector of real-time temperature values ​​of the cooling system for all computing units. Each concatenated vector is arranged in the order of CPU computing units, GPU computing units, neural network processor computing units, and FPGA computing units. The above task feature vectors are concatenated with the computing platform state data in the last dimension to form a joint state vector. At each scheduling decision time step, the joint state vector corresponding to one time step is simultaneously fed into the input layers of the policy network and the value function network.

[0028] In practice, the policy network outputs the mapping probability distribution from each subtask to each computing unit, while the value function network outputs a value estimate of the current state. After receiving the joint state vector, the policy network performs forward propagation calculations and generates the mapping probability distribution at the output layer using a softmax function. Each element in the mapping probability distribution represents the probability that a specific subtask is assigned to a specific computing unit. The probability values ​​range from 0 to 1, and the sum of all elements in the mapping probability distribution equals 1. After receiving the joint state vector, the value function network performs forward propagation calculations and directly outputs a scalar value at the output layer. This scalar value represents the estimated cumulative reward that can be obtained by executing the scheduling according to the current policy until the task is completed under the environmental state represented by the current joint state vector.

[0029] In practical implementation, based on the mapping probability distribution, a deterministic policy gradient algorithm is used to select mapping relationships from the mapping probability distribution as the mapping relationships from tasks to computing power units. Specifically, in the mapping probability distribution output by the policy network, the subtask index and computing power unit index corresponding to the largest probability element are found. The subtask identified by that subtask index is uniquely mapped to the target computing power unit identified by that computing power unit index, forming a deterministic mapping relationship. During model training, the parameter updates of the policy network use the deterministic policy gradient algorithm, and its gradient calculation formula is:

[0030] in, This represents the policy network loss function with respect to the policy network parameters. gradient, This represents the number of state transition samples taken from the experience replay pool. The value is set to 64. Indicates the first The joint state vector in each sampled state transition sample Represents action variables, This indicates that the value function network is in the input joint state vector. and actions The output value at that time The parameters of the value function network are represented. The value function network output represents the action. The gradient in the action Equals the policy network for the joint state vector Output The value at that location, Indicates the policy network in parameters The following is a joint state vector The output is a deterministic action. This indicates that the policy network output affects the policy network parameters. The gradient. Action The representation is a one-hot encoded vector with the same dimension as the mapping probability distribution, where the selected mapping position is set to 1, and the remaining positions are set to 0. The input to the value function network during the training phase is the joint state vector. and actions The concatenated vector, the concatenation method is to combine the action spliced ​​into the joint state vector Afterwards, the value function network receives the concatenated vector and outputs a scalar action value.

[0031] Example 4: In specific implementation, please refer to Figure 4The system retrieves the mapping relationship output by the policy network, which includes a one-to-one correspondence between subtasks and target computing units. After generating the mapping probability distribution, the scheduling decision module locates the index of the maximum probability value in a search space of the same dimension as the mapping probability distribution to obtain the subtask index and the computing unit index. The subtask index is represented by the sequential number of each subtask in the task dependency graph, and the computing unit index is represented by a unique numerical identifier pre-assigned to all heterogeneous computing units. This numerical identifier starts from 0 and increments continuously, with 0 corresponding to a central processing unit (CPU) computing unit, 1 to a graphics processing unit (GPU) computing unit, 2 to a neural network processor (NNPM) computing unit, and 3 to a field-programmable gate array (FPGA) computing unit. The one-to-one correspondence identifier is stored in a structure containing a subtask index field and a computing unit index field, which together serve as the unique identifier for the mapping relationship.

[0032] In practical implementation, subtask execution instructions are sent to the target computing unit by calling the resource scheduling interface of the computing unit. These subtask execution instructions include operator code, input data addresses, and expected output memory regions. For the CPU computing unit, the resource scheduling interface is the dynamic library loading and execution interface provided by the operating system. The subtask execution instructions are organized in the form of function calls. The operator code is the entry address of the corresponding function in the deep learning operator dynamic library. The input data address is the starting address of a pre-allocated contiguous virtual address segment in system memory. The expected output memory region is the buffer address allocated from the CPU's memory pool by calling the memory allocation interface beforehand. For the GPU computing unit, the resource scheduling interface is the task submission interface provided by the GPU computing driver. For example, operators are submitted to the GPU task flow by calling the `cuLaunchKernel` function. The operator code is a function handle containing GPU binary instructions. The input data address is a pointer to the filled device memory in the GPU's video memory. The expected output memory region is the video memory address allocated from the GPU's video memory pool. For the neural network processor computing unit, the resource scheduling interface is the command queue submission interface provided by the neural network processor runtime library. This is achieved by filling the command descriptor with the operator descriptor, the offset address of the input data tensor in the neural network processor's on-chip memory, and the pre-allocated memory offset address of the output data tensor, and then writing the command descriptor into the neural network processor's command queue. For the field-programmable gate array (FPGA) computing unit, the resource scheduling interface is the bitstream dynamic loading and data transfer interface provided by the FPGA management driver. The operator code is the bitstream configuration file of the corresponding computing module, the input data address is the physical address of the buffer in the FPGA's onboard dynamic random access memory (DRAM) moved from main memory via direct memory access, and the expected output memory area is the memory space reserved in the FPGA's onboard memory for storing computation results.

[0033] In practice, the current task execution order is obtained from the task scheduler of the computing unit. Based on this order and the priority constraints of the subtasks, the insertion position of the subtasks within the current execution order is adjusted. The task scheduler maintains a local task queue manager independently for each computing unit. This manager records the sequence of subtasks to be executed in a linked list and labels the priority value of each subtask. The current task execution order is obtained by calling the query interface exposed by each computing unit's scheduler, which returns a list of all subtasks in the unit's task queue, sorted by priority from highest to lowest. Each element in the list contains a subtask identifier and a priority value. The priority constraints of the subtasks are described in the task dependency graph, specifically meaning that any subtask can only begin execution after all its predecessor subtasks have been completed. When adjusting the insertion position, based on the current task execution order list of the target computing unit, the system traverses backward from the head of the list to find the first position in the list where the priority value is lower than that of the subtask to be inserted, and inserts the subtask to be inserted before that position. At the same time, the system checks the completion status of all predecessor subtasks of the subtask to be inserted on the target computing unit or other computing units. If there are any incomplete predecessor subtasks, the insertion position is postponed further until the first position that satisfies the priority order after all predecessor subtasks are completed, thereby ensuring that the scheduling operation does not violate the dependency relationship.

[0034] In practice, based on the hardware characteristics of the target computing unit, the memory allocation strategy and thread concurrency configuration parameters for subtasks are set. For CPU computing units, the memory allocation strategy is implemented by setting a memory binding strategy, which restricts the memory pages required by the subtask to memory nodes close to the currently executing core. The memory allocation strategy parameters include the memory node mask, and memory is allocated by calling the `numa_alloc_onnode` function and passing in the memory node number. The thread concurrency configuration parameter is set to twice the number of physical cores of the current CPU computing unit. This value is obtained by reading the system file ` / sys / devices / system / cpu / cpu* / topology / thread_siblings_list` to determine the total number of logical threads. For the graphics processing unit (GPU) computing unit, the memory allocation strategy migrates data memory pages to the GPU memory of the executing operator by setting a preferred location for a uniform memory address. The memory allocation strategy parameters include the device serial number of the specified GPU device, and the thread concurrency configuration parameter is set to the number of threads contained in each thread block. This thread number is the smaller value between the maximum number of threads per thread block returned by the GPU hardware query and the computational amount of the subtask. The maximum number of threads per thread block is obtained by calling the `cudaDeviceGetAttribute` function and passing the `cudaDevAttrMaxThreadsPerBlock` attribute. For the neural network processor (NN) computing unit, the memory allocation strategy is implemented by setting the storage layout format of tensor data. Subtask tensor data is preferentially allocated in the NN processor's on-chip high-speed scratchpad memory. When the on-chip high-speed scratchpad memory is insufficient, data is allocated in off-chip memory. The memory allocation strategy parameter is a Boolean flag indicating whether to enable preferential on-chip allocation. The thread concurrency configuration parameter is set to the number of parallel computing cores of the NN processor's computing array multiplied by the batch dimension parallelizability. The parallelizability is the ratio of the batch size of the subtask input tensor to the vector processing width supported by the NN processor. For field-programmable gate array (FPGA) computing units, the memory allocation strategy is implemented by directly specifying the base address and address offset in the onboard dynamic random access memory (DRAM). The memory allocation strategy parameters include the configuration values ​​of the base address offset register and address range register of the onboard DRAM. The thread concurrency configuration parameter is set to the number of FPGA computing data path instantiations, which is determined by the computing unit concurrency defined in the loaded bitstream configuration file.

[0035] In practice, the system receives real-time monitoring data reported by each computing unit during the execution of subtasks. This data includes task execution time, peak resource usage, and cache hit rate. Task execution time is obtained by calculating the timestamp returned by the task completion callback function of each computing unit after the subtask execution instruction is sent. The timestamp is a monotonically increasing system hardware counter value, and the task execution time equals the difference between the task completion timestamp and the task start timestamp. The peak resource usage is obtained as follows: during the subtask execution period, the device management module of each computing unit collects a sequence of computing resource utilization at a fixed sampling frequency, and takes the maximum value in the sequence as the peak resource usage. The peak resource usage of the CPU computing unit is the highest value of the utilization of all cores during the subtask execution, obtained by continuous sampling from the / proc / stat file. The peak resource usage of the GPU computing unit is the peak value of the GPU core utilization during the subtask execution, obtained by continuous sampling through calls to the nvmlDeviceGetUtilizationRates interface. The peak resource usage of the neural network processor computing unit is the peak value of the neural network processor computing array utilization during the subtask execution, obtained by continuous reading from the neural network processor performance monitoring register. The peak resource usage of the FPGA computing unit is the peak value of the activity of programmable logic resources during the subtask execution, obtained by sampling through the internal performance counter of the FPGA. The cache hit rate is obtained as follows: during the execution of subtasks, the number of cache hits and the total number of cache accesses are read from the hardware performance counters of each computing unit. The cache hit rate is equal to the number of cache hits divided by the total number of cache accesses. The CPU computing unit calculates the cache hit count and cache access count by reading the model-specific register MSR_PERF_STATUS. The GPU computing unit reads the l2_cache_hit_rate indicator from the GPU performance counter by calling cudaEventRecord in conjunction with event timing. The NN processor computing unit calculates the cache hit count register and access count register by reading the NN processor cache monitoring unit. The FPGA computing unit calculates the cache hit counter and access counter by reading the cache statistics module in the FPGA memory controller.

[0036] In practice, the time deviation is calculated by comparing the task execution time with the expected execution time in the subtask execution instructions. The expected execution time is estimated by the value assessment model based on the subtask operator type and data size before executing the subtask execution instructions. The estimation method involves querying the operator time consumption database to obtain the historical average time for that operator type when executing the same data size on the target computing unit. The time deviation is calculated using a signed difference form, and the formula is as follows:

[0037] in, This represents the time deviation value, in milliseconds. Indicates the actual task execution time. Indicates the expected execution time. The value is the task execution time extracted from the real-time monitoring data reported by the computing unit. The value is the estimated execution time obtained from the operator time database before the subtask execution instruction is issued. A positive value indicates that the actual execution time exceeded expectations. A negative value indicates that the actual execution time is shorter than expected.

[0038] In practice, the time deviation value, peak resource usage, and cache hit rate are combined into an environmental feedback vector, which is input into the value function network in the asynchronous advantage actor-critic network to calculate the time difference error. The environmental feedback vector is a three-dimensional real number vector. The first dimension stores the time deviation value after max-min normalization, with the upper limit of normalization being the absolute value of the largest time deviation in historical statistics and the lower limit being the negative value of that absolute value. The second dimension stores the peak resource usage, which is a normalized value between 0.0 and 1.0. The third dimension stores the cache hit rate, which is a normalized value between 0.0 and 1.0. When the environmental feedback vector is input to the value function network, it is concatenated with the current joint state vector before being fed into the value function network. During the temporal difference learning process, the value function network calculates the temporal difference error. The calculation of the temporal difference error is based on the Bellman equation, specifically the current immediate reward plus a discount factor multiplied by the next state value estimate and then subtracted from the current state value estimate. The immediate reward is obtained by weighted summation of the components of the environmental feedback vector. The weight coefficient of the time deviation value is set to -1.0 to penalize execution time exceeding the expected time, the weight coefficient of the peak resource consumption is set to -0.5 to encourage the reduction of peak resource consumption, the weight coefficient of the cache hit rate is set to 0.3 to encourage the improvement of cache utilization, and the discount factor is set to 0.9.

[0039] Example 5: In specific implementation, please refer to Figure 5Based on the dependency depth and parallelism of subtasks in the task dependency graph, the critical path in the task dependency graph is identified. Dependency depth is calculated when generating the task feature vector; the dependency depth of each subtask represents the number of vertices contained in the longest directed path from the starting subtask vertex to the current subtask vertex. Parallelism is determined by analyzing the number of subtasks that can be executed in parallel at the same level in the task dependency graph. The calculation method is to topologically sort the task dependency graph and divide it into layers, with each layer containing all vertices with an in-degree of zero; the number of vertices in each layer is the parallelism of that layer. The critical path identification process involves traversing every directed path from the starting subtask vertex to the ending subtask vertex, accumulating the estimated execution time of all subtasks on each directed path, and selecting the directed path with the longest total estimated execution time as the critical path. The estimated execution time of each subtask is obtained by querying the operator time consumption database, which stores the historical average execution time of each operator type on different computing power units. When there is no historical data for the estimated execution time, the total number of floating-point operations of the operator is divided by the peak floating-point operation of the target computing power unit for estimation.

[0040] In practice, starting with the initial subtask on the critical path, the initial subtask is pre-assigned to idle computing units with direct data channels to the computing units executing it. The determination of a direct data channel is based on the hardware interconnection topology between computing units in the heterogeneous computing platform. This topology is obtained by reading the physical link relationships recorded in the system peripheral component interconnection standard device tree and the interconnection management library lookup table. If there is a non-switched direct data transmission link between two computing units, then the two computing units are considered to have a direct data channel. Idle computing units are determined by the following criteria: the current task queue length of the computing unit is zero, and the real-time load rate of the computing unit is lower than a preset idle load threshold, which is set to 5%. The execution method of the initial subtask pre-allocation is as follows: before the task scheduling decision step is executed, the scheduling pre-allocation module traverses the list of all computing power units that have a direct data channel with the target computing power unit that executes the initial subtask, checks the task queue length and real-time load rate of each computing power unit in the list one by one, and when a computing power unit that meets the idle criterion is found, the pre-allocation information of the initial subtask is recorded in the pre-allocation mapping table. The pre-allocation mapping table uses the subtask identifier as the key and the pre-allocated target computing power unit identifier as the value.

[0041] In practical implementation, before the task scheduling decision step is executed, input data for subsequent subtasks on the critical path is preloaded into the data cache of the idle computing unit. The timing of this preloading is as follows: after the initial subtask begins execution, once the output data of the initial subtask is available on the source computing unit, the data transmission engine of the heterogeneous computing platform immediately uses the output data as input data for subsequent subtasks and transmits it along the direct data channel to the pre-allocated data cache of the idle computing unit, without waiting for the deep reinforcement learning model to make a complete scheduling decision. The data transmission engine is implemented by calling the memory copy interface of each computing unit. Data transmission from the CPU to the GPU is achieved by calling the cudaMemcpyAsync function and specifying the transmission direction as cudaMemcpyHostToDevice. Data transmission from the GPU to the neural network processor is achieved through the dma_copy_from_gpu interface provided by the neural network processor runtime library. Data transmission from the neural network processor to the field-programmable gate array (FPGA) is achieved through the pcie_fpga_dma_transfer interface provided by the FPGA driver. The memory region preloaded into the data cache is reserved in advance from the corresponding memory pool by the memory allocator of the target computing unit. The reserved size is equal to the total number of bytes of the input data tensors of subsequent subtasks.

[0042] In practical implementation, based on the temporal difference error, the gradients of the loss functions of the policy network and the value function network in the asynchronous dominant actor-commentator network are calculated. The temporal difference error is calculated by the value function network during the forward propagation process, and the calculation is based on the following formula:

[0043] in, Indicates timing difference error. Indicates an immediate reward. Indicates the discount factor. This represents the value estimate of the next state. This represents the estimated value of the current state. The environmental feedback vector is obtained by weighted summation of its components. The environmental feedback vector consists of time deviation, peak resource usage, and cache hit rate. The weight coefficient of time deviation is set to -1.0, the weight coefficient of peak resource usage is set to -0.5, and the weight coefficient of cache hit rate is set to 0.3. Set it to 0.9. The scalar value output by the value function network after receiving the joint state vector at the next time step. The scalar value output by the value function network after receiving the joint state vector at the current time step. The loss function of the value function network uses the mean squared error form, and its gradient is obtained by taking the partial derivative of the loss function with respect to all parameters of the value function network. The loss function of the policy network is constructed based on the near-end policy optimization algorithm, and its gradient is obtained by taking the partial derivative of the near-end policy optimization pruning objective with respect to all parameters of the policy network. Temporal difference error is used in the calculation. As an approximate estimate of the advantage function.

[0044] In practice, a proximal policy optimization pruning function is used to constrain the gradient of the loss function of the policy network, generating truncated gradient updates. The proximal policy optimization pruning function takes the form of limiting the ratio between the mapping probability distribution of the current output of the policy network and the mapping probability distribution of the old policy network to a certain value. Within the interval, Indicates the clipping threshold. Set it to 0.2. The rationale for setting it to 0.2 is that this value has been shown in the original paper on the proximal policy optimization algorithm to achieve a balance between policy update step size and learning stability. When the value is too small, the strategy updates slowly. When the value is too large, the policy may change drastically, leading to performance collapse. 0.2 is a universally accepted value that has been verified in numerous environments. The truncated gradient update amount limits the magnitude of each policy network update by setting the gradient components corresponding to the ratios that exceed the range to zero, and only retaining the gradient components corresponding to the ratios that are within the range.

[0045] In practice, based on the truncated gradient update, the Adam optimizer is used to asynchronously update the weight parameters of the policy network and value function network, and the gradients from multiple parallel processes are accumulated and averaged. The learning rate of the Adam optimizer is set to 0.001. This learning rate is commonly used as an initial learning rate in deep reinforcement learning, ensuring stable convergence of the optimization process in most environments. Internally, the Adam optimizer maintains estimates of the first and second moments of the gradient. When updating the weight parameters, the exponential moving average of the first and second moments of the truncated gradient update is first calculated. Then, the first and second moments are corrected for bias. Finally, the corrected first moment is divided by the sum of the square root of the corrected second moment and a small constant, and multiplied by the learning rate to obtain the final weight update. The asynchronous update process is as follows: after each parallel worker process completes a certain number of scheduling decision steps, it calculates the local gradient update amount of its own policy network and value function network, and asynchronously pushes the local gradient update amount to the global policy network and global value function network. After receiving the gradient update amounts from multiple parallel worker processes, the global network calculates the arithmetic mean of all the gradient update amounts accumulated within the same time window, and then applies the arithmetic mean gradient update amount to the global network parameters. The updated global network parameters are periodically synchronized back to each parallel worker process to maintain the consistency of the policy network and value function network parameters of the worker processes.

[0046] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A deep reinforcement learning task scheduling method integrating heterogeneous computing power, characterized in that, Includes the following steps: Feature extraction is performed on the initial heterogeneous computing platform state, which includes the real-time load, computing power index and energy efficiency parameters of each computing unit. The task characteristics analysis of deep learning tasks includes the task's computation type, data dependencies, and priority constraints. The output of the feature extraction step and the output of the task characteristic analysis are used as inputs, and a preset deep reinforcement learning model performs task scheduling decisions. The deep reinforcement learning model outputs the mapping relationship between tasks and computing units. The scheduling operation is performed according to the mapping relationship. The scheduling operation includes allocating tasks to the target computing unit, adjusting the task execution order, and configuring resource parameters. The status data of the computing platform after scheduling is collected in real time, and the status data is fed back to the deep reinforcement learning model to update the model parameters; The steps for executing task scheduling decisions by a pre-defined deep reinforcement learning model specifically include: Construct an asynchronous dominant actor-critic network, which includes a parallel policy network and a value function network; The task feature vector and the computing platform state data are concatenated into a joint state vector, which is then input into the policy network and the value function network. The policy network outputs the mapping probability distribution from each subtask to each computing unit in parallel, and the value function network outputs the value estimate of the current state. Based on the mapping probability distribution, a deterministic policy gradient algorithm is used to select a mapping relationship from the mapping probability distribution as the mapping relationship from task to computing power unit; wherein, the mapping relationship is a one-to-one correspondence between all subtasks and target computing power units generated at once by the deterministic policy gradient algorithm; The step of performing the scheduling operation according to the mapping relationship specifically includes: Obtain the mapping relationship output by the policy network, wherein the mapping relationship includes a one-to-one correspondence identifier between subtasks and target computing units; By calling the resource scheduling interface of the computing power unit, a subtask execution instruction is sent to the target computing power unit. The subtask execution instruction includes operator code, input data address and expected output memory area. The current task execution order is obtained from the task scheduler of the computing unit, and the insertion position of the subtask in the current task execution order is adjusted according to the task execution order and the priority constraints of the subtask. Based on the hardware characteristics of the target computing unit, set the memory allocation strategy and thread concurrency configuration parameters for the subtasks.

2. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 1, characterized in that, The step of extracting features from the initial heterogeneous computing power platform state includes: The real-time load rate, task queue length, and memory bandwidth utilization of each computing unit in the central processing unit, graphics processing unit, neural network processor, and field-programmable gate array are collected from the monitoring interface of each computing unit as real-time load characteristics. Instruction set architecture information, peak floating-point operation value, and video memory bandwidth parameters are read from the hardware configuration registers of each computing unit as indicators of computing power. The energy consumption per unit time and the real-time temperature of the heat dissipation system are obtained from the power management unit of each computing unit as energy efficiency parameter features; The task completion time and context switching count recorded in the operating system scheduling logs of each computing unit are used as physical constraint characteristics.

3. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 2, characterized in that, The steps for analyzing the characteristics of deep learning tasks include: Historical task execution logs are collected from the task queue of the heterogeneous cluster. The historical task execution logs include the operator type, input data tensor dimension, and output data tensor dimension of each task. The operator type is matched with a preset algorithm library to identify the computationally intensive, memory-intensive, and communication-intensive types of the task. Extract a task dependency graph from the dependency description file of each task. The task dependency graph contains the data transfer direction and sequence constraints between each subtask. Based on the task dependency graph, each task is divided into subtasks and prioritized, generating a task feature vector containing subtask identifiers, data size, and dependency depth.

4. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 3, characterized in that, The steps for real-time collection and scheduling of computing platform status data specifically include: Receive real-time monitoring data reported by each computing unit during the execution of subtasks, including task execution time, peak resource usage, and cache hit rate; Compare the task execution time with the expected execution time in the subtask execution instruction, and calculate the time deviation value; The time deviation value, the peak resource usage, and the cache hit rate are combined into an environmental feedback vector, which is then input into the value function network in the asynchronous advantage actor-critic network to calculate the time difference error.

5. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 4, characterized in that, The method further includes a step of pre-assigning tasks to the deep learning task: Based on the dependency depth and parallelism of subtasks in the task dependency graph, the critical path in the task dependency graph is identified. Starting from the initial subtask on the critical path, the initial subtask is pre-allocated to an idle computing unit that has a direct data channel with the computing unit executing the initial subtask; Before the task scheduling decision step is executed, input data for subsequent subtasks on the critical path is preloaded into the data cache of the idle computing unit.

6. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 5, characterized in that, The steps for updating model parameters specifically include: Based on the temporal difference error, calculate the gradient of the loss function of the policy network and the value function network in the asynchronous dominant actor-critic network; The gradient of the loss function of the policy network is constrained by the pruning function optimized by the near-end policy, and the truncated gradient update amount is generated. Based on the truncated gradient update amount, the Adam optimizer is used to asynchronously update the weight parameters of the policy network and the value function network, and the gradients from multiple parallel working processes are accumulated and averaged.

7. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 6, characterized in that, Both the policy network and the value function network employ a recurrent neural network structure containing long short-term memory (LSTM) layers to process temporal state information.

8. The deep reinforcement learning task scheduling method integrating heterogeneous computing power according to claim 7, characterized in that, The pruning threshold ε in the near-end policy optimization pruning function is set to 0.2, and the learning rate of the Adam optimizer is set to 0.001.

Images