Task scheduling method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a task scheduling method and device, electronic equipment and storage medium, and relate to the field of financial technology. The method comprises: in response to a received task to be scheduled, determining multi-dimensional state data of a plurality of computing nodes; for any one computing node, determining a dynamic weight value of the computing node according to the multi-dimensional state data of the computing node; determining a target node in the plurality of computing nodes according to the dynamic weight values of the plurality of computing nodes; and assigning the task to be scheduled to the target node, so that the target node executes the task to be scheduled, thereby improving the task scheduling efficiency.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more particularly to a task scheduling method, apparatus, electronic device, and storage medium. Background Technology

[0002] Against the backdrop of rapid development in technologies such as artificial intelligence, big data analytics, and high-performance computing, heterogeneous computing clusters of central processing units (CPUs) and graphics processing units (GPUs) have become the core infrastructure supporting complex computing tasks. For example, in model inference scenarios, users may submit hundreds of real-time image recognition requests simultaneously, which need to be quickly allocated to GPU clusters for parallel processing.

[0003] In related technologies, resource scheduling of CPU-GPU heterogeneous clusters mainly relies on the following methods: technicians pre-divide the computing power of each GPU node into static partitions based on experience, clarify the computing power threshold and task acceptance range of different partitions, and then rely on basic scheduling strategies such as round-robin and hashing to allocate the tasks to be executed sequentially or in a targeted manner to the corresponding GPU nodes and computing power partitions according to the preset partitioning rules. At the same time, the CPU nodes undertake auxiliary tasks such as task distribution and status monitoring, thereby completing the task scheduling process of the heterogeneous cluster.

[0004] In the above process, there is a significant contradiction between dynamic perception accuracy, scheduling real-time performance and resource utilization, resulting in low task scheduling efficiency. Summary of the Invention

[0005] This application provides a task scheduling method, apparatus, electronic device, and storage medium to solve the problem of low task scheduling efficiency.

[0006] Firstly, this application provides a task scheduling method, including:

[0007] In response to the received scheduled tasks, determine the multidimensional state data of multiple computing nodes;

[0008] For any given computing node, determine the dynamic weight value of the computing node based on its multidimensional state data.

[0009] The target node is determined among multiple computing nodes based on their dynamic weight values.

[0010] The task to be scheduled is assigned to the target node so that the target node can execute the task to be scheduled.

[0011] In one possible implementation, determining the dynamic weight value of a computing node based on its multidimensional state data includes:

[0012] The multidimensional state data of the computing nodes are normalized to obtain the multidimensional feature vectors of the computing nodes.

[0013] The initial weight values ​​are determined based on the multidimensional feature vector and the weight generator;

[0014] Based on the preset sparse penalty algorithm, the initial weight values ​​are adjusted to obtain the dynamic weight values ​​of the computing nodes.

[0015] In one possible implementation, the weight generator is configured with scaling factor parameters in multiple dimensions; initial weight values ​​are determined based on the multidimensional feature vector and the weight generator, including:

[0016] The multidimensional feature vector is input into the weight generator to obtain weight factors in multiple dimensions;

[0017] The initial weight values ​​corresponding to the multidimensional feature vectors are determined based on the weight factors of multiple dimensions.

[0018] In one possible implementation, the initial weight values ​​are adjusted according to a preset sparsity penalty algorithm to obtain the dynamic weight values ​​of the computing nodes, including:

[0019] Obtain the preset penalty coefficient for the sparse penalty algorithm;

[0020] The sparse penalty term is determined based on the preset penalty coefficient and the initial weight value;

[0021] The initial weight values ​​are adjusted based on the sparsity penalty term to obtain the dynamic weight values ​​of the computing nodes.

[0022] In one possible implementation, determining the target node among the multiple computing nodes based on their dynamic weight values ​​includes:

[0023] The node probabilities corresponding to multiple computing nodes are determined based on the dynamic weight values ​​of multiple computing nodes.

[0024] Based on the node probabilities corresponding to multiple computing nodes, the target node is determined from multiple computing nodes through a preset probability sampling and allocation algorithm.

[0025] In one possible implementation, after the target node executes the task to be scheduled, the method further includes:

[0026] Obtain the execution result of the scheduled task on the target node;

[0027] The execution results are analyzed and processed to obtain feedback data;

[0028] Based on the feedback data, the weight generator is updated to obtain the updated weight generator.

[0029] In one possible implementation, for any given computing node, determining the multidimensional state data of the computing node includes:

[0030] Acquire hardware status data, computing task status data, and network communication status data of the computing nodes;

[0031] Preprocessing and timestamp synchronization are performed on hardware status data, computing task status data, and network communication status data to obtain multidimensional status data of computing nodes.

[0032] Secondly, this application provides a task scheduling device, comprising:

[0033] The receiving module is used to determine the multi-dimensional state data of multiple computing nodes in response to the received task to be scheduled;

[0034] The calculation module is used to determine the dynamic weight value of any computing node based on its multidimensional state data.

[0035] The determination module is used to determine the target node among multiple computing nodes based on the dynamic weight values ​​of multiple computing nodes;

[0036] The scheduling module is used to assign tasks to be scheduled to target nodes so that the target nodes can execute the tasks.

[0037] In one possible implementation, the computing module is specifically used for:

[0038] The multidimensional state data of the computing nodes are normalized to obtain the multidimensional feature vectors of the computing nodes.

[0039] The initial weight values ​​are determined based on the multidimensional feature vector and the weight generator;

[0040] Based on the preset sparse penalty algorithm, the initial weight values ​​are adjusted to obtain the dynamic weight values ​​of the computing nodes.

[0041] In one possible implementation, the weight generator is configured with scaling factor parameters for multiple dimensions; the calculation module is specifically used for:

[0042] The multidimensional feature vector is input into the weight generator to obtain weight factors in multiple dimensions;

[0043] The initial weight values ​​corresponding to the multidimensional feature vectors are determined based on the weight factors of multiple dimensions.

[0044] In one possible implementation, the computing module is specifically used for:

[0045] Obtain the preset penalty coefficient for the sparse penalty algorithm;

[0046] The sparse penalty term is determined based on the preset penalty coefficient and the initial weight value;

[0047] The initial weight values ​​are adjusted based on the sparsity penalty term to obtain the dynamic weight values ​​of the computing nodes.

[0048] In one possible implementation, the determining module is specifically used for:

[0049] The node probabilities corresponding to multiple computing nodes are determined based on the dynamic weight values ​​of multiple computing nodes.

[0050] Based on the node probabilities corresponding to multiple computing nodes, the target node is determined from multiple computing nodes through a preset probability sampling and allocation algorithm.

[0051] In one possible implementation, after the target node executes the scheduled task, the device further includes an update module, which is used to:

[0052] Obtain the execution result of the scheduled task on the target node;

[0053] The execution results are analyzed and processed to obtain feedback data;

[0054] Based on the feedback data, the weight generator is updated to obtain the updated weight generator.

[0055] In one possible implementation, the receiving module is specifically used for:

[0056] Acquire hardware status data, computing task status data, and network communication status data of the computing nodes;

[0057] Preprocessing and timestamp synchronization are performed on hardware status data, computing task status data, and network communication status data to obtain multidimensional status data of computing nodes.

[0058] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0059] The memory stores the instructions that the computer executes;

[0060] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0061] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any of the first aspects.

[0062] Fifthly, this application provides a computer program product, including a computer program that, when executed by a computer, implements the method as described in any of the first aspects.

[0063] This application provides a task scheduling method, apparatus, electronic device, and storage medium. In response to a received task to be scheduled, it determines multi-dimensional state data of multiple computing nodes; for any given computing node, it determines a dynamic weight value based on the multi-dimensional state data; based on the dynamic weight values ​​of multiple computing nodes, it determines a target node among the multiple computing nodes; and it assigns the task to be scheduled to the target node so that the target node can execute the task. In this way, by calculating the dynamic weights of the nodes based on multi-dimensional data and selecting the target node to execute the task, it effectively improves the resource utilization and task execution efficiency of heterogeneous clusters, avoids resource idleness or local overload, and enhances the real-time performance and adaptability of scheduling decisions, thereby improving task scheduling efficiency. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0065] Figure 1 A schematic diagram of a cluster architecture provided for an embodiment of this application;

[0066] Figure 2 A flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0067] Figure 3 A flowchart illustrating another task scheduling method provided in an embodiment of this application;

[0068] Figure 4 This is a schematic diagram of the structure of a task scheduling device provided in an embodiment of this application;

[0069] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0073] It should be noted that the task scheduling method, apparatus, electronic device and storage medium provided in this application can be used in the field of financial technology, or in any field other than financial technology. The application field of the task scheduling method, apparatus, electronic device and storage medium in this application is not limited.

[0074] Against the backdrop of rapid development in technologies such as artificial intelligence, big data analytics, and high-performance computing, heterogeneous computing clusters of central processing units (CPUs) and graphics processing units (GPUs) have become the core infrastructure supporting complex computing tasks. For example, in model inference scenarios, users may submit hundreds of real-time image recognition requests simultaneously, which need to be quickly allocated to GPU clusters for parallel processing.

[0075] In related technologies, resource scheduling of CPU-GPU heterogeneous clusters mainly relies on the following methods: technicians pre-divide the computing power of each GPU node into static partitions based on experience, clarify the computing power threshold and task acceptance range of different partitions, and then rely on basic scheduling strategies such as round-robin and hashing to allocate the tasks to be executed sequentially or in a targeted manner to the corresponding GPU nodes and computing power partitions according to the preset partitioning rules. At the same time, the CPU nodes undertake auxiliary tasks such as task distribution and status monitoring, thereby completing the task scheduling process of the heterogeneous cluster.

[0076] In the above process, there is a significant contradiction between dynamic perception accuracy, scheduling real-time performance and resource utilization, resulting in low task scheduling efficiency.

[0077] To address the aforementioned technical problems, this application provides a task scheduling method. In response to the receipt of a task to be scheduled, the method collects multi-dimensional state data from each computing node within the cluster and calculates the dynamic weight value of each node. Then, based on the weight values, it selects a target node from multiple computing nodes and allocates the task. This achieves multi-dimensional dynamic perception and low-overhead, efficient scheduling of the computing power of heterogeneous clusters, thus improving task scheduling efficiency.

[0078] Below, in conjunction with Figure 1 The following example illustrates the architecture of the cluster.

[0079] Figure 1 This is a schematic diagram of a cluster architecture provided for an embodiment of this application. Please refer to [link / reference]. Figure 1 , Figure 1 It may include a central scheduler, a heterogeneous computing node cluster, a high-speed communication link, and a status monitoring module.

[0080] The central scheduler can be the control unit of a cluster architecture. The central scheduler can receive externally input tasks to be scheduled and execute scheduling logic such as task parsing, node selection, and task allocation.

[0081] Heterogeneous computing node clusters can be the core carrier for task execution.

[0082] A heterogeneous computing node cluster can include multiple computing nodes.

[0083] Each computing node is configured with a preset computing power threshold and task capacity. The computing nodes interact with each other and transmit instructions through a high-speed communication link.

[0084] The status monitoring module can establish a real-time connection with each computing node.

[0085] The status monitoring module can be used to collect and feed back multi-dimensional status data of each computing node to the central scheduler.

[0086] Multidimensional status data can include computing load rate, memory usage, task execution progress, network transmission rate, etc.

[0087] The central scheduler may include a data processing unit and a node selection unit.

[0088] The data processing unit can be used to clean, integrate, and extract features from the multidimensional status data collected by the monitoring module.

[0089] The node selection unit can calculate the dynamic weight value of each node based on the processed status data, filter out the target nodes that are suitable for the task to be scheduled, and distribute the task instructions and related data to the target nodes through a high-speed communication link, so that the target nodes can complete the execution of the task.

[0090] This architecture, through the coordinated operation of its various modules, provides stable hardware support and a data foundation for subsequent dynamic and precise resource scheduling strategies.

[0091] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0092] Figure 2 This is a flowchart illustrating a task scheduling method provided in an embodiment of this application. The execution entity in this embodiment can be a processor. The processor can be implemented in software or through a combination of software and hardware. Please refer to... Figure 2 The method includes:

[0093] S201. In response to the received scheduled task, determine the multidimensional state data of multiple computing nodes.

[0094] Tasks awaiting scheduling can refer to various computing tasks received by the system that have not yet been assigned to specific computing nodes for execution.

[0095] Tasks to be scheduled can include data processing tasks, model training tasks, business logic operation tasks, etc.

[0096] Multidimensional state data can refer to multidimensional indicator data that characterizes the real-time state of a computing node.

[0097] Multi-dimensional metrics data can include hardware status data, computing task status data, and network communication status data.

[0098] When the central scheduler receives a task to be scheduled from an external source, it can trigger the collection of status data of all computing nodes in the cluster, obtain multi-dimensional indicator data of multiple computing nodes, and determine the multi-dimensional status data of multiple computing nodes based on the multi-dimensional indicator data.

[0099] Optionally, for any computing node, the multidimensional state data of the computing node can be determined in the following way: obtain the hardware state data, computing task state data, and network communication state data of the computing node; perform preprocessing and timestamp synchronization processing on the hardware state data, computing task state data, and network communication state data to obtain the multidimensional state data of the computing node.

[0100] The acquisition of hardware status data, computing task status data, and network communication status data of computing nodes, as well as subsequent preprocessing and timestamp synchronization, can be achieved by deploying a lightweight monitoring agent on each computing node.

[0101] The monitoring agent can obtain core hardware indicators such as GPU computing power utilization, video memory usage, and core temperature in real time by calling dedicated hardware interfaces. At the same time, it can collect hardware-related data such as CPU utilization, core load distribution, and total memory capacity, used capacity, and remaining capacity through native operating system interfaces to obtain hardware status data.

[0102] The monitoring agent can indirectly obtain task status information such as the real-time usage ratio of CPU, GPU and memory of currently running tasks on the node, and task runtime by associating with hardware resource usage. This allows for the collection of task status data in conjunction with hardware resource consumption.

[0103] The monitoring agent can record the sending and receiving timestamps of data transmission between computing nodes and the central scheduler, as well as other computing nodes, through a built-in timestamp synchronization mechanism. It can calculate network latency by using the time difference and simultaneously collect auxiliary communication indicators such as data transmission rate to obtain network communication status data.

[0104] Abnormal fluctuations in hardware status data and computing task status data are removed and supplemented. Indicators with different dimensions are standardized to unify the data range. At the same time, the time dimension of hardware status data, computing task status data, and network communication status data is calibrated by using the timestamp synchronization mechanism of the monitoring agent to ensure that the timestamps of the three types of data are accurately aligned and to avoid deviations in subsequent weight calculations caused by misalignment of data time sequence.

[0105] All metrics that have been preprocessed and timestamped can be uniformly encapsulated into a time-series data vector containing three dimensions: hardware, task, and network. This time-series data vector is the multi-dimensional state data of the computing node and is reported to the central scheduler according to a preset period.

[0106] S202. For any computing node, determine the dynamic weight value of the computing node based on the multidimensional state data of the computing node.

[0107] Dynamic weight values ​​can be quantified values ​​calculated based on real-time multidimensional state data of computing nodes.

[0108] Dynamic weight values ​​can be used to comprehensively evaluate the adaptability and capability of computing nodes to undertake new tasks.

[0109] Optionally, for any computing node, the dynamic weight value of the computing node can be determined by using a lightweight normalized attention algorithm based on the multidimensional state data of the computing node.

[0110] Optionally, for any computing node, the dynamic weight value of the computing node can be determined based on the multidimensional state data of the computing node and through a preset weight calculation model.

[0111] S203. Determine the target node among multiple computing nodes based on the dynamic weight values ​​of multiple computing nodes.

[0112] The target node can be the computing node that is most suitable for undertaking the scheduling task, determined based on dynamic weight values.

[0113] The target node can be determined by sorting the dynamic weight values ​​of multiple computing nodes from highest to lowest.

[0114] By sorting or filtering by weight, the node with the optimal resource allocation and the most stable operating status can be accurately located, thus ensuring the efficient execution of tasks.

[0115] Optionally, the target node can be determined from multiple computing nodes based on the dynamic weight values ​​of multiple computing nodes in the following way: determine the node probability corresponding to multiple computing nodes based on the dynamic weight values ​​of multiple computing nodes; determine the target node from multiple computing nodes based on the node probability corresponding to multiple computing nodes through a preset probability sampling and allocation algorithm.

[0116] The node probabilities corresponding to multiple computing nodes can be determined as follows: the dynamic weight values ​​are input into a preset function for normalization. Through the mapping calculation of the preset function, the dynamic weight values ​​of each node are converted into probability values ​​in the interval [0,1], and the sum of the probability values ​​of all computing nodes is 1.

[0117] The central scheduler can be equipped with preset probability sampling and allocation algorithms, and can flexibly adopt weighted round-robin or probability sampling strategies to select target nodes.

[0118] If a weighted round-robin strategy is adopted, the central scheduler can convert node probability into round-robin allocation weight. The higher the probability value of a node, the higher the proportion of task allocations it receives within the round-robin period.

[0119] If a probability sampling strategy is adopted, the central scheduler can directly select nodes randomly based on the probability distribution of the nodes, and nodes with higher probability values ​​have a higher probability of being selected.

[0120] The algorithm described above accurately assigns newly arriving computing tasks to target nodes, ensuring overall load balancing across the heterogeneous cluster and avoiding suboptimal decision-making problems caused by ignoring node state differences in traditional simple polling or random allocation methods.

[0121] Optionally, for high-priority tasks or critical system processes, the central scheduler will reserve some computing resources in advance and dynamically adjust the scheduling weight of the corresponding nodes during the weight calculation and probability allocation process.

[0122] When a high-priority task arrives, the dynamic weight value of the matching node is automatically increased, thereby increasing its normalized task allocation probability. This ensures that high-priority tasks receive priority support from target nodes with sufficient resources, guaranteeing the service quality of core tasks.

[0123] S204. Assign the task to be scheduled to the target node so that the target node can execute the task to be scheduled.

[0124] The relevant information of the task to be scheduled can be transmitted to the designated target node to complete the task allocation operation. After receiving the task information, the target node starts its own task execution process and completes data processing, calculation and other operations according to the task requirements until the task ends.

[0125] Optionally, after the target node executes the task to be scheduled, the method further includes: obtaining the execution result of the target node executing the task to be scheduled; analyzing and processing the execution result to obtain feedback data; and updating the weight generator based on the feedback data to obtain the updated weight generator.

[0126] Among them, the feedback agent deployed on the computing node can perform standardized analysis and processing of the execution results, filter out the indicators required for optimization by the central scheduler, unify the data format and units, encapsulate the processed performance indicators into structured feedback data, and send it back to the central scheduler.

[0127] After receiving the feedback data, the central scheduler uses it as the basis for optimization and dynamically updates the weight generator.

[0128] The central scheduler can adaptively adjust the weight calculation logic based on feedback data and modify the weight ratio in the scaling factors of different performance indicators.

[0129] In this way, the weight generator can continuously learn the actual effect of task execution, complete iterative updates, and enable the entire scheduling system to have continuous optimization capabilities, forming a closed loop.

[0130] This embodiment provides a task scheduling method that, in response to a received task to be scheduled, determines multi-dimensional state data of multiple computing nodes; for any given computing node, a dynamic weight value is determined based on the multi-dimensional state data; based on the dynamic weight values ​​of multiple computing nodes, a target node is determined among the multiple computing nodes; and the task to be scheduled is assigned to the target node so that the target node can execute the task. In this way, by calculating the dynamic weights of nodes based on multi-dimensional data and selecting the target node to execute the task, the resource utilization and task execution efficiency of heterogeneous clusters are effectively improved, resource idleness or local overload is avoided, and the real-time performance and adaptability of scheduling decisions are enhanced, thus improving task scheduling efficiency.

[0131] Below, in conjunction with Figure 3 The process of determining the dynamic weight value of a computing node based on its multidimensional state data is explained.

[0132] Figure 3 This is a flowchart illustrating another task scheduling method provided in an embodiment of this application. Based on the above embodiments, see also... Figure 3 The method includes:

[0133] S301. Normalize the multidimensional state data of the computing node to obtain the multidimensional feature vector of the computing node.

[0134] The multidimensional state data of computing nodes can be normalized, and the normalization results of each dimension can be combined into an ordered vector in a preset order to obtain the multidimensional feature vector of the computing node.

[0135] For example, if the normalized data of a certain node is GPU utilization 0.8, CPU utilization 0.6, memory utilization 0.7, and network latency 0.21, then the multidimensional feature vector is [0.8, 0.6, 0.7, 0.21].

[0136] The normalized multidimensional feature vectors eliminate the scale interference of the original data, enabling the indicators of each dimension to participate fairly in the calculation of subsequent dynamic weight values. This ensures that the weight values ​​can truly reflect the comprehensive status of the nodes and avoid scheduling decision deviations due to differences in data dimensions.

[0137] S302. Determine the initial weight values ​​based on the multidimensional feature vector and the weight generator.

[0138] The weight generator can be used to define the contribution percentage of different dimensional features to the node weights.

[0139] The weight generator can preload a preset calculation model and configure the initial weight parameters of each dimension feature. The multidimensional feature vector of the calculation node is input into the calculation model of the weight generator, and the numerical calculation is completed according to the preset logic, and the initial weight value of the node is output.

[0140] Optionally, the weight generator is configured with scaling factor parameters in multiple dimensions; the initial weight values ​​can be determined based on the multidimensional feature vector and the weight generator in the following way: input the multidimensional feature vector into the weight generator to obtain the weight factors in multiple dimensions; determine the initial weight values ​​corresponding to the multidimensional feature vector based on the weight factors in multiple dimensions.

[0141] The weight generator has a built-in computational model based on the Batch Normalization (BN) layer. Its core is to define the weight factors of each performance dimension through the scaling factor parameter of the BN layer.

[0142] For example, the weight generator will directly call the set of preset BN layer scaling factor parameters {γ1,γ2,...,γj,...,γn} (where n is the number of dimensions of the multidimensional feature vector).

[0143] The value of the scaling factor parameter is not fixed, but is pre-calculated or adaptively obtained by analyzing the historical load data of the cluster during the initialization or update phase. Its value directly represents the intrinsic importance of the j-th feature dimension to the node weight.

[0144] The weighting factor can be determined using the following formula:

[0145] Attention_Weight = BN_γ* Feature_Value

[0146] Here, Attention_Weight can represent the weight factor corresponding to a certain dimension, BN_γ can represent the scaling factor parameter corresponding to a certain dimension, and Feature_Value can represent the vector value corresponding to a certain dimension in the multi-dimensional feature vector.

[0147] S303. According to the preset sparse penalty algorithm, the initial weight values ​​are adjusted to obtain the dynamic weight values ​​of the computing nodes.

[0148] During the initial weight calculation, all dimensions of the multidimensional feature vector participate in the operation. However, some feature dimensions have minimal impact on the tasks undertaken by the nodes and are considered redundant. The participation of these redundant dimensions not only increases computational overhead and reduces scheduling efficiency but may also introduce invalid noise, interfering with the weight values' representation of the true state of the nodes.

[0149] The sparse penalty algorithm introduces a penalty coefficient, which can impose penalty constraints on the weight factors of each feature dimension.

[0150] Optionally, the initial weight value can be adjusted according to a preset sparse penalty algorithm to obtain the dynamic weight value of the computing node: obtain the preset penalty coefficient of the sparse penalty algorithm; determine the sparse penalty term according to the preset penalty coefficient and the initial weight value; adjust the initial weight value according to the sparse penalty term to obtain the dynamic weight value of the computing node.

[0151] The dynamic weight values ​​of computing nodes can be obtained in the following way:

[0152] Score_i = Σ(γ_j * x_ij) - λ * |γ_j|

[0153] Where Score_i can represent dynamic weight value, Σ(γ_j * x_ij) can represent the initial weight value of the node, x_ij can represent the vector value corresponding to a certain dimension in the multidimensional feature vector of the computing node, γ_j can represent the scaling factor coefficient of a certain dimension, λ can represent the preset penalty coefficient, and λ * |γ_j| can represent the sparse penalty term.

[0154] The implementation details of each step in this application embodiment can be found in the description of the corresponding steps or operations in the above method embodiments; repeated content will not be repeated.

[0155] This embodiment provides a task scheduling method that normalizes the multidimensional state data of computing nodes to obtain multidimensional feature vectors for the computing nodes; determines initial weight values ​​based on the multidimensional feature vectors and a weight generator; and adjusts the initial weight values ​​according to a preset sparsity penalty algorithm to obtain dynamic weight values ​​for the computing nodes. Through the normalization of the multidimensional state data of computing nodes, the initial weight calculation based on the weight generator, and the weight adjustment using the sparsity penalty algorithm, a dynamic weight value that accurately reflects the true state of the nodes is output, effectively eliminating data dimension interference, reducing redundant dimensional noise, and improving the accuracy of weight calculation.

[0156] Figure 4 This is a schematic diagram of a task scheduling device provided in an embodiment of this application. Please refer to... Figure 4 The task scheduling device 400 includes a receiving module 401, a calculation module 402, a determination module 403, and a scheduling module 404, wherein...

[0157] The receiving module 401 is used to determine the multi-dimensional status data of multiple computing nodes in response to the received task to be scheduled;

[0158] The calculation module 402 is used to determine the dynamic weight value of any computing node based on the multidimensional state data of the computing node.

[0159] The determination module 403 is used to determine the target node among multiple computing nodes based on the dynamic weight values ​​of multiple computing nodes;

[0160] The scheduling module 404 is used to assign the task to be scheduled to the target node so that the target node can execute the task to be scheduled.

[0161] In one possible implementation, the computing module 402 is specifically used for:

[0162] The multidimensional state data of the computing nodes are normalized to obtain the multidimensional feature vectors of the computing nodes.

[0163] The initial weight values ​​are determined based on the multidimensional feature vector and the weight generator;

[0164] Based on the preset sparse penalty algorithm, the initial weight values ​​are adjusted to obtain the dynamic weight values ​​of the computing nodes.

[0165] In one possible implementation, the weight generator is configured with scaling factor parameters for multiple dimensions; the calculation module 402 is specifically used for:

[0166] The multidimensional feature vector is input into the weight generator to obtain weight factors in multiple dimensions;

[0167] The initial weight values ​​corresponding to the multidimensional feature vectors are determined based on the weight factors of multiple dimensions.

[0168] In one possible implementation, the computing module 402 is specifically used for:

[0169] Obtain the preset penalty coefficient for the sparse penalty algorithm;

[0170] The sparse penalty term is determined based on the preset penalty coefficient and the initial weight value;

[0171] The initial weight values ​​are adjusted based on the sparsity penalty term to obtain the dynamic weight values ​​of the computing nodes.

[0172] In one possible implementation, the determining module 403 is specifically used for:

[0173] The node probabilities corresponding to multiple computing nodes are determined based on the dynamic weight values ​​of multiple computing nodes.

[0174] Based on the node probabilities corresponding to multiple computing nodes, the target node is determined from multiple computing nodes through a preset probability sampling and allocation algorithm.

[0175] In one possible implementation, after the target node executes the scheduled task, the device further includes an update module 405, which is used to:

[0176] Obtain the execution result of the scheduled task on the target node;

[0177] The execution results are analyzed and processed to obtain feedback data;

[0178] Based on the feedback data, the weight generator is updated to obtain the updated weight generator.

[0179] In one possible implementation, the receiving module 401 is specifically used for:

[0180] Acquire hardware status data, computing task status data, and network communication status data of the computing nodes;

[0181] Preprocessing and timestamp synchronization are performed on hardware status data, computing task status data, and network communication status data to obtain multidimensional status data of computing nodes.

[0182] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 5 Electronic device 500 may include: memory 501, processor 502, and transceiver 503.

[0183] Memory 501 is used to store program instructions;

[0184] The processor 502 is used to execute the program instructions stored in the memory so that the electronic device 500 performs the above-described method.

[0185] Transceiver 503 may include a transmitter and / or a receiver. The transmitter may also be referred to as a transmitter, transmitter port, or transmitter interface, and the receiver may also be referred to as a receiver port, receiver interface, or similar descriptions. Exemplarily, memory 501, processor 502, and transceiver 503 are interconnected via bus 504.

[0186] This application also provides a computer program product that can be executed by a processor, and when the computer program product is executed, the above-described method can be implemented.

[0187] The task scheduling device, electronic device, computer-readable storage medium, and computer program product of the embodiments of this application can execute the technical solutions shown in the above method embodiments. Their implementation principles and beneficial effects are similar, and will not be described again here.

[0188] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0189] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0190] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0191] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0192] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0193] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0194] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0195] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0196] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A task scheduling method, characterized in that, The method includes: In response to the received scheduled tasks, determine the multidimensional state data of multiple computing nodes; For any given computing node, the dynamic weight value of the computing node is determined based on the multidimensional state data of the computing node. Based on the dynamic weight values ​​of the plurality of computing nodes, the target node is determined among the plurality of computing nodes; The task to be scheduled is assigned to the target node so that the target node executes the task to be scheduled.

2. The method according to claim 1, characterized in that, Based on the multidimensional state data of the computing node, the dynamic weight value of the computing node is determined, including: The multidimensional state data of the computing node is normalized to obtain the multidimensional feature vector of the computing node; The initial weight values ​​are determined based on the multidimensional feature vector and the weight generator; The initial weight values ​​are adjusted according to a preset sparse penalty algorithm to obtain the dynamic weight values ​​of the computing nodes.

3. The method according to claim 2, characterized in that, The weight generator is configured with scaling factor parameters in multiple dimensions; Based on the multidimensional feature vector and the weight generator, the initial weight values ​​are determined, including: The multidimensional feature vector is input into the weight generator to obtain weight factors in multiple dimensions; Based on the weight factors of the multiple dimensions, the initial weight values ​​corresponding to the multidimensional feature vector are determined.

4. The method according to claim 2, characterized in that, According to a preset sparsity penalty algorithm, the initial weight values ​​are adjusted to obtain the dynamic weight values ​​of the computing nodes, including: Obtain the preset penalty coefficient of the sparse penalty algorithm; Based on the preset penalty coefficient and the initial weight value, a sparse penalty term is determined; The initial weight value is adjusted according to the sparse penalty term to obtain the dynamic weight value of the computing node.

5. The method according to any one of claims 1-4, characterized in that, Based on the dynamic weight values ​​of the plurality of computing nodes, the target node is determined among the plurality of computing nodes, including: The node probability corresponding to the plurality of computing nodes is determined based on the dynamic weight values ​​of the plurality of computing nodes; Based on the node probabilities corresponding to the multiple computing nodes, a target node is determined from the multiple computing nodes using a preset probability sampling and allocation algorithm.

6. The method according to any one of claims 2-4, characterized in that, After the target node executes the scheduled task, the method further includes: Obtain the execution result of the scheduled task executed by the target node; The execution results are analyzed and processed to obtain feedback data; Based on the feedback data, the weight generator is updated to obtain the updated weight generator.

7. The method according to any one of claims 1-4, characterized in that, For any given computing node, determine the multidimensional state data of the computing node, including: Acquire the hardware status data, computing task status data, and network communication status data of the computing node; The hardware status data, the computing task status data, and the network communication status data are preprocessed and timestamp synchronized to obtain the multidimensional status data of the computing node.

8. A task scheduling device, characterized in that, The device includes: The receiving module is used to determine the multi-dimensional state data of multiple computing nodes in response to the received task to be scheduled; The calculation module is used to determine the dynamic weight value of any given calculation node based on its multidimensional state data. The determination module is used to determine the target node among the plurality of computing nodes based on the dynamic weight values ​​of the plurality of computing nodes; The scheduling module is used to assign the task to be scheduled to the target node, so that the target node can execute the task to be scheduled.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images