A task scheduling method and device based on a distributed large language model, a terminal device, and a storage medium

Abstract

The application discloses a task scheduling method and device based on a distributed large language model, terminal equipment and a storage medium. The method comprises the following steps: obtaining communication data of each device node in a multi-layer network of the distributed large language model, a to-be-scheduled task, a task feature of the to-be-scheduled task, and real-time monitoring data of each device node; then, according to the communication data, the to-be-scheduled task, the task feature of the to-be-scheduled task, and the real-time monitoring data of each device node, performing model segmentation and task scheduling on the multi-layer network with the minimum total delay of the multi-layer network as the target, obtaining a target device node corresponding to each to-be-scheduled task, and distributing each to-be-scheduled task to the corresponding target device node for scheduling. Through implementation of the application, resource management of the multi-layer network in the large language model can be realized, the performance of the network is improved, and the delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of model task scheduling technology, and in particular to a task scheduling method, apparatus, terminal device and storage medium based on a distributed large language model. Background Technology

[0002] In the rapidly evolving field of multi-tier cloud-edge computing, efficient task scheduling is crucial for optimizing system throughput and resource utilization. Multi-tier network architectures, by distributing computing tasks across the cloud, intermediate nodes, and edge layers, are designed to meet the needs of modern artificial intelligence applications, particularly large language models (LLMs).

[0003] In the field of edge computing, graph neural networks (GNNs) have been used to optimize task offloading strategies and improve resource utilization. However, existing GNN methods do not optimize task scheduling for multi-layer networks and LLM segmentation optimization results, resulting in low system efficiency and high latency when processing tasks. Summary of the Invention

[0004] This invention provides a task scheduling method, apparatus, terminal device, and storage medium based on a distributed large language model, which can realize resource management of the multi-layer network within the large language model, improve network performance, and reduce latency.

[0005] An embodiment of the present invention provides a task scheduling method based on a distributed large language model, comprising:

[0006] The system acquires communication data, scheduled tasks, task characteristics of scheduled tasks, and real-time monitoring data of each device node in the multi-layer network of the distributed large language model. The task characteristics include: task computing power, network conditions, task size, and computing power requirements. The real-time monitoring data includes: computing power, memory, bandwidth, and latency.

[0007] Based on the above communication data, the tasks to be scheduled, the task characteristics of the tasks to be scheduled, and the real-time monitoring data of each device node, the multi-layer network is segmented and task scheduling is performed with the goal of minimizing the total latency of the multi-layer network. The target device node corresponding to each task to be scheduled is obtained, and each task to be scheduled is assigned to the corresponding target device node for scheduling.

[0008] The model is segmented based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm to obtain the allocation scheme for each level; the allocation scheme includes: model size and computational requirements.

[0009] Based on the above allocation scheme, each task to be scheduled is assigned to each level to obtain the target task to be scheduled for each level.

[0010] Based on the aforementioned target tasks to be scheduled, the target task characteristics corresponding to the aforementioned target tasks to be scheduled, the aforementioned real-time monitoring data, and the preset task scheduling model, task scheduling is performed with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, thereby obtaining the target device nodes corresponding to each task to be scheduled; wherein, the aforementioned scheduling cost and the sum of violation of computing power constraints are calculated based on the aforementioned real-time monitoring data.

[0011] Furthermore, when performing model segmentation and task scheduling on the aforementioned multi-layer network with the objective of minimizing the total latency of the multi-layer network, the corresponding first objective function is:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] In the formula, Let L represent the total delay, L represent the total number of levels, and l represent the l-th level. This represents the transmission delay at level l. This represents the computational latency of level l. This represents the device node at level l. This represents the total number of device nodes at level l. Indicates the first Hierarchical device nodes, Indicates the first The total number of device nodes at each level Indicates whether the scheduled task is scheduled from level l to level 1. Hierarchy This represents the computational demand allocation coefficient for level l. Indicates computational requirements. This represents the computing power of the device nodes in level l. This represents the size allocation coefficient of the l-th level model. Indicates the model size. This represents the achievable transmission rate of node u. This represents the bandwidth of subchannel k. Indicates the interference-to-noise ratio of the transmitted signal. Indicates the power of interference plus noise. This indicates that other users g in the same sub-channel set are paired with the first... Hierarchical nodes Interference, This represents the signal-to-interference-to-noise ratio between level l' and level l. Indicates starting from the l-th level node To the l' level node Channel gain, Represents the node at level l To the node The transmission power.

[0019] Furthermore, based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm, the allocation schemes for each level are obtained, including:

[0020] The priority queue is initialized; wherein, the priority queue includes several nodes used to represent the above allocation scheme; the initialized priority queue includes: the root node that has not been model-segmented;

[0021] Repeat the model splitting operation until the priority queue is empty, and use the optimal allocation scheme when the priority queue is empty as the above allocation scheme;

[0022] The above-mentioned model segmentation operations include:

[0023] Get the current priority queue, and arbitrarily select a node from the current priority queue as the target node; where the initial node is the root node mentioned above;

[0024] If the current target node is a leaf node, calculate the total delay corresponding to the current target node based on the communication data corresponding to the current target node; where the leaf node represents the allocation scheme corresponding to the multi-layer network.

[0025] If the current total delay is less than the minimum total delay corresponding to the current optimal solution, update the minimum total delay based on the current total delay, take the allocation scheme corresponding to the current target node as the current optimal allocation scheme, and remove the current target node from the current priority queue; otherwise, remove the current target node from the current priority queue; where the initial optimal solution is infinite and the initial optimal allocation scheme is empty.

[0026] If the current target node is not a leaf node, select an undetermined variable in the current target node to branch and obtain several child nodes; where the variable includes: model size and computational requirements;

[0027] Based on the communication data of the current child nodes, calculate the total delay of each child node. If the current maximum total delay of the child nodes is less than the minimum total delay corresponding to the current optimal solution, or if the current minimum total delay of the child nodes is greater than the maximum total delay corresponding to the current optimal solution, then remove the current target node and its corresponding child nodes from the current priority queue. Otherwise, add the current child node to the current priority queue as a node in the priority queue, and remove the current target node to update the current priority queue.

[0028] Furthermore, based on the aforementioned target tasks to be scheduled, the target task characteristics corresponding to the aforementioned target tasks to be scheduled, the aforementioned real-time monitoring data, and the preset task scheduling model, with the objective of minimizing the sum of scheduling cost and violation of computing power constraints, the target device nodes corresponding to each task to be scheduled are obtained, including:

[0029] Initialize the list of tasks to be scheduled; the initialized list includes all the target tasks to be scheduled mentioned above.

[0030] Repeat the task scheduling operation until the above list of tasks to be scheduled is empty, and determine the target device node corresponding to each task to be scheduled.

[0031] The aforementioned task scheduling operations include:

[0032] Select a device node as the current node to be scheduled, and retrieve the current task to be scheduled from the current list of tasks to be scheduled;

[0033] Based on the current tasks to be scheduled and the corresponding characteristics of the current target task, construct the current structure graph;

[0034] Obtain the current real-time monitoring data and the above-mentioned preset task scheduling model, input the current structure graph, current task characteristics and current real-time monitoring data into the above-mentioned preset task scheduling model, so that the above-mentioned preset task scheduling model can calculate the current degree matrix and adjacency matrix based on the current structure graph.

[0035] Based on the current degree matrix, the aforementioned adjacency matrix, and the current target task features, the current node feature matrix of the last layer in the aforementioned preset task scheduling model is obtained.

[0036] Based on the current node feature matrix and activation function, with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, the scheduling probability distribution between the current node to be scheduled and each current task to be scheduled is obtained.

[0037] The current node to be scheduled is selected as the target device node for the task with the highest scheduling probability, and this node is removed from the current list of tasks to be scheduled in order to update the list of tasks to be scheduled.

[0038] Furthermore, based on the current tasks to be scheduled and the corresponding characteristics of the current target task, the above-mentioned structure graph is constructed, including:

[0039] The current task to be scheduled is used as the graph node of the current structure graph, and the graph nodes are connected according to the characteristics of the current target task to obtain the current structure graph.

[0040] Furthermore, when scheduling tasks with the objective of minimizing the sum of scheduling costs and violations of computational capacity constraints, the corresponding second objective function is:

[0041]

[0042] In the formula, H represents the second objective function. Indicates the task size. Indicates whether the task to be scheduled originates from the device node. Dispatch to device node , Indicates the penalty parameter. This represents the computational capability constraint at level l.

[0043] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments;

[0044] This invention provides a task scheduling device based on a distributed large language model, comprising:

[0045] Data acquisition module and target device node determination module;

[0046] The aforementioned data acquisition module is used to acquire communication data, scheduled tasks, task characteristics of scheduled tasks, and real-time monitoring data of each device node in the multi-layer network of the distributed large language model; wherein, the aforementioned task characteristics include: task computing power, network conditions, task size, and computing power requirements; the aforementioned real-time monitoring data includes: computing power, memory, bandwidth, and latency.

[0047] The aforementioned target device node determination module is used to perform model segmentation and task scheduling on the aforementioned multi-layer network based on the aforementioned communication data, the tasks to be scheduled, the task characteristics of the tasks to be scheduled, and the real-time monitoring data of each device node, with the goal of minimizing the total latency of the multi-layer network, to obtain the target device node corresponding to each task to be scheduled, and to assign each task to be scheduled to the corresponding target device node for scheduling.

[0048] The model is segmented based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm to obtain the allocation scheme for each level; the allocation scheme includes: model size and computational requirements.

[0049] Based on the above allocation scheme, each task to be scheduled is assigned to each level to obtain the target task to be scheduled for each level.

[0050] Based on the aforementioned target tasks to be scheduled, the target task characteristics corresponding to the aforementioned target tasks to be scheduled, the aforementioned real-time monitoring data, and the preset task scheduling model, task scheduling is performed with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, thereby obtaining the target device nodes corresponding to each task to be scheduled; wherein, the aforementioned scheduling cost and the sum of violation of computing power constraints are calculated based on the aforementioned real-time monitoring data.

[0051] Furthermore, the aforementioned target device node determination module includes:

[0052] Priority queue initialization unit and model segmentation unit;

[0053] The aforementioned priority queue initialization unit is used to initialize the priority queue; wherein, the priority queue includes several nodes for representing the aforementioned allocation scheme; the initialized priority queue includes: the root node that has not been model-segmented;

[0054] The aforementioned model segmentation unit is used to repeatedly perform the model segmentation operation until the priority queue is empty, and the optimal allocation scheme corresponding to the empty priority queue is used as the aforementioned allocation scheme.

[0055] The above-mentioned model segmentation operations include:

[0056] Get the current priority queue, and arbitrarily select a node from the current priority queue as the target node; where the initial node is the root node mentioned above;

[0057] If the current target node is a leaf node, calculate the total delay corresponding to the current target node based on the communication data corresponding to the current target node; where the leaf node represents the allocation scheme corresponding to the multi-layer network.

[0058] If the current total delay is less than the minimum total delay corresponding to the current optimal solution, update the minimum total delay based on the current total delay, take the allocation scheme corresponding to the current target node as the current optimal allocation scheme, and remove the current target node from the current priority queue; otherwise, remove the current target node from the current priority queue; where the initial optimal solution is infinite and the initial optimal allocation scheme is empty.

[0059] If the current target node is not a leaf node, select an undetermined variable in the current target node to branch and obtain several child nodes; where the variable includes: model size and computational requirements;

[0060] Based on the communication data of the current child nodes, calculate the total delay of each child node. If the current maximum total delay of the child nodes is less than the minimum total delay corresponding to the current optimal solution, or if the current minimum total delay of the child nodes is greater than the maximum total delay corresponding to the current optimal solution, then remove the current target node and its corresponding child nodes from the current priority queue. Otherwise, add the current child node to the current priority queue as a node in the priority queue, and remove the current target node to update the current priority queue.

[0061] Based on the above method embodiments, the present invention provides a corresponding terminal device embodiment;

[0062] The present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the task scheduling method based on a distributed large language model described in any embodiment of the present invention.

[0063] Based on the above method embodiments, the present invention provides a corresponding storage medium embodiment;

[0064] The present invention provides a storage medium including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the task scheduling method based on a distributed large language model described in any embodiment of the present invention.

[0065] The embodiments of the present invention have the following beneficial effects:

[0066] This invention provides a task scheduling method, apparatus, terminal device, and storage medium based on a distributed large language model. The method includes: acquiring communication data of each device node in a multi-layer network of the distributed large language model, tasks to be scheduled, task characteristics of the tasks to be scheduled, and real-time monitoring data of each device node; wherein the task characteristics include: task computing power, network conditions, task size, and computing power requirements; the real-time monitoring data includes: computing power, memory, bandwidth, and latency; subsequently, based on the communication data, tasks to be scheduled, task characteristics of the tasks to be scheduled, and real-time monitoring data of each device node, the multi-layer network is model-segmented and task-scheduled with the goal of minimizing the total latency of the multi-layer network, to obtain the target device node corresponding to each task to be scheduled, and then... The tasks to be scheduled are assigned to the corresponding target device nodes for scheduling. Specifically, the model is segmented based on the aforementioned communication data and a branch-and-bound integer linear programming algorithm to obtain allocation schemes for each level. These allocation schemes include model size and computational requirements. Each task to be scheduled is then assigned to its corresponding level according to these schemes, resulting in target tasks to be scheduled at each level. Based on the target tasks to be scheduled, their corresponding target task characteristics, the aforementioned real-time monitoring data, and a preset task scheduling model, task scheduling is performed with the objective of minimizing the sum of scheduling costs and violations of computational capability constraints, thus obtaining the target device nodes corresponding to each task to be scheduled. The sum of scheduling costs and violations of computational capability constraints is calculated based on the aforementioned real-time monitoring data. Therefore, this invention divides the entire task scheduling process into two parts: model segmentation and task scheduling. With the goal of minimizing the total latency of a multi-layered network, the model is first segmented using a branch-and-bound integer linear programming algorithm and communication data. Then, based on the segmentation results, and considering the target tasks to be scheduled, their corresponding target task characteristics, real-time monitoring data, and a pre-defined task scheduling model, task scheduling is performed with the objective of minimizing the sum of scheduling costs and violations of computational capacity constraints. This determines the target device nodes corresponding to each task to complete the entire task scheduling process. This approach enables resource management of the multi-layered network, improving network performance and reducing latency. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating a task scheduling method based on a distributed large language model, provided by an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of a two-stage, multi-level, multi-node scheduling LLM based on a collaborative AI computing method provided in an embodiment of the present invention.

[0069] Figure 3This is a schematic diagram of automatic decoupling deployment of inter-layer LLM based on the Transformer architecture provided in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram showing the performance comparison results of an inference task based on the OPT model in a three-layer framework provided by an embodiment of the present invention.

[0071] Figure 5 This is a schematic diagram showing the performance comparison results of an inference task based on the Qwen2 model in a three-layer framework provided by an embodiment of the present invention.

[0072] Figure 6 This is a schematic diagram showing the performance comparison results of an inference task based on the OPT model in a four-layer framework provided by an embodiment of the present invention.

[0073] Figure 7 This is a schematic diagram showing the performance comparison results of inference tasks based on the Qwen2 model in a four-layer framework provided by an embodiment of the present invention.

[0074] Figure 8 This is a schematic diagram of the structure of a task scheduling device based on a distributed large language model provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0076] like Figure 1 As shown, an embodiment of the present invention provides a task scheduling method based on a distributed large language model, comprising:

[0077] Step S101: Obtain the communication data of each device node in the multi-layer network of the distributed large language model, the tasks to be scheduled, the task characteristics of the tasks to be scheduled, and the real-time monitoring data of each device node; wherein, the task characteristics include: task computing power, network conditions, task size, and computing power requirements; the real-time monitoring data includes: computing power, memory, bandwidth, and latency.

[0078] Specifically, this paper considers a multi-layered cloud-edge network architecture containing L layers, denoted as l∈{1,2,...,L}. Each layer contains multiple compute nodes (mNodes), i.e., the aforementioned "device nodes." These nodes can be edge devices, intermediate servers, or cloud data centers. The inference task of large language models (LLMs) requires allocation and scheduling across these layers to achieve optimal performance and resource utilization. This represents the set of device nodes at level l. Each device node has different computing capabilities and network connectivity.

[0079] Step S102: Based on the above communication data, the tasks to be scheduled, the task characteristics of the tasks to be scheduled, and the real-time monitoring data of each device node, the multi-layer network is modeled and task scheduled with the goal of minimizing the total latency of the multi-layer network, so as to obtain the target device node corresponding to each task to be scheduled, and each task to be scheduled is assigned to the corresponding target device node for scheduling.

[0080] The model is segmented based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm to obtain the allocation scheme for each level; the allocation scheme includes: model size and computational requirements.

[0081] Based on the above allocation scheme, each task to be scheduled is assigned to each level to obtain the target task to be scheduled for each level.

[0082] Based on the aforementioned target tasks to be scheduled, the target task characteristics corresponding to the aforementioned target tasks to be scheduled, the aforementioned real-time monitoring data, and the preset task scheduling model, task scheduling is performed with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, thereby obtaining the target device nodes corresponding to each task to be scheduled; wherein, the aforementioned scheduling cost and the sum of violation of computing power constraints are calculated based on the aforementioned real-time monitoring data.

[0083] In a preferred embodiment, when performing model segmentation and task scheduling on the multi-layer network with the objective of minimizing the total latency of the multi-layer network, the corresponding first objective function is:

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] In the formula, Let L represent the total delay, L represent the total number of levels, and l represent the l-th level. This represents the transmission delay at level l. This represents the computational latency of level l. This represents the device node at level l. This represents the total number of device nodes at level l. Indicates the first Hierarchical device nodes, Indicates the first The total number of device nodes at each level Indicates whether the scheduled task is scheduled from level l to level 1. Hierarchy This represents the computational demand allocation coefficient for level l. Indicates computational requirements. This represents the computing power of the device nodes in level l. This represents the size allocation coefficient of the l-th level model. Indicates the model size. This represents the achievable transmission rate of node u. This represents the bandwidth of subchannel k. Indicates the interference-to-noise ratio of the transmitted signal. Indicates the power of interference plus noise. This indicates that other users g in the same sub-channel set are paired with the first... Hierarchical nodes Interference, This represents the signal-to-interference-to-noise ratio between level l' and level l. Indicates starting from the l-th level node To the l' level node Channel gain, Represents the node at level l To the node The transmission power.

[0091] Specifically, the entire LLM model is considered as a divisible task, with its model size and computational requirements denoted as Ms and Mc, respectively. Based on the model partitioning decision π, the model size and computational requirements are allocated to different levels. The model size and computational requirements allocated to level l are as follows: Ms and Mc, among which and These are the corresponding allocation coefficients. An LLM model can be decomposed into multiple independent parts, which can be executed in parallel on different mNodes. This decomposition can be expressed by the following formula:

[0092]

[0093] In the formula, This represents the i-th segment of the LLM model, where n represents the total number of segments.

[0094] Specifically, the transmission delay in a multi-layer network mainly depends on the size of the model segments, bandwidth, channel gain, and channel noise. First, it is necessary to calculate the transmission rate affected by noise. Assume... This represents a node at level l. Let l' represent the node at level l'. Then, the signal-to-interference-plus-noise ratio (SINR) between levels l and l' can be expressed as:

[0095]

[0096] In the formula, The power of interference plus noise is defined as follows:

[0097]

[0098] Specifically, the achievable transmission rate of node l It can be represented as:

[0099]

[0100] Specifically, the transmission delay of the l-th layer can be expressed as:

[0101]

[0102] Specifically, computational latency depends on the computational requirements of model segments, the computational capabilities of nodes, and scheduling decisions. Therefore, the computational latency of level l can be expressed as:

[0103]

[0104] Specifically, based on the above formulas, the total delay in a multi-layer network can be obtained:

[0105]

[0106] Therefore, based on the above formulas, we obtain the first objective function for model segmentation and task scheduling of the multi-layer network with the goal of minimizing the total latency of the multi-layer network. The entire optimization problem can be transformed into minimizing the total latency, with the objective of finding the optimal model segmentation decision π and scheduling decision A.

[0107] An illustrative diagram illustrating a two-stage, multi-level, multi-node scheduling method for LLM based on collaborative AI computing is shown below. Figure 2 As shown, Figure 2This paper elaborates on the multi-layer, multi-node LLM scheduling (MMSL) method, a task scheduling method based on distributed large language models as presented in this invention, aiming to optimize the inference efficiency of large language models (LLMs) in multi-layer networks. This method decomposes the complex optimization problem (P0) into two sub-problems: P1 and P2, thereby simplifying the solution process. P0 represents the overall MMSL problem, with the goal of computationally optimizing LLM inference across multi-layer networks. The core challenge lies in how to efficiently allocate computational resources across different layers to minimize total latency.

[0108] Specifically, to address the complexity of the original problem P0, the MMSL method decomposes it into two sub-problems. P1, or Inter-Layer LLM Automatic Decoupling and Slicing (CT-ADS), focuses on determining the optimal model segmentation strategy π. This involves deciding the model size and computational requirements to be handled at each layer, achieved through integer linear programming techniques, particularly based on the Branch and Bound algorithm (ILP-BB). ILP-BB searches the solution space systematically to find the optimal resource allocation scheme, including monitoring resource status at each layer, solving the optimization problem using the ILP-BB algorithm, and transforming the resource allocation problem into a mathematical model for solution. P2, or Intra-Layer LLM Adaptive Scheduling and Distributed (IT-ASD), is responsible for finding the optimal task scheduling scheme A within each layer. This requires considering node resource utilization and network conditions to achieve load balancing and minimum latency, achieved through a task scheduling algorithm based on Graph Neural Networks (GNN) (TS-GNN). GNN optimizes task scheduling decisions through information propagation and updating node features, including monitoring device resource utilization at each layer and optimizing task allocation within the layer using the GNN algorithm.

[0109] Preferably, by solving the two subproblems P1 and P2 sequentially, the MMSL method can effectively minimize the total latency of large-scale model processing in the entire multi-layer computing network. This decomposition significantly reduces computational complexity and provides a structured framework for addressing the challenges of large-scale model deployment and resource management. Ultimately, it achieves efficient scheduling of LLM inference in multi-layer networks, highlighting the key roles of problem decoupling, mathematical formulation and optimization, and graph neural networks in solving complex optimization problems.

[0110] In this preferred embodiment, by calculating the total latency within the multi-layer network, a first objective function is constructed to perform model segmentation and task scheduling on the multi-layer network with the goal of minimizing the total latency of the multi-layer network.

[0111] In another preferred embodiment, the allocation scheme for each level is obtained based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm, including:

[0112] The priority queue is initialized; wherein, the priority queue includes several nodes used to represent the above allocation scheme; the initialized priority queue includes: the root node that has not been model-segmented;

[0113] Repeat the model splitting operation until the priority queue is empty, and use the optimal allocation scheme when the priority queue is empty as the above allocation scheme;

[0114] The above-mentioned model segmentation operations include:

[0115] Get the current priority queue, and arbitrarily select a node from the current priority queue as the target node; where the initial node is the root node mentioned above;

[0116] If the current target node is a leaf node, calculate the total delay corresponding to the current target node based on the communication data corresponding to the current target node; where the leaf node represents the allocation scheme corresponding to the multi-layer network.

[0117] If the current total delay is less than the minimum total delay corresponding to the current optimal solution, update the minimum total delay based on the current total delay, take the allocation scheme corresponding to the current target node as the current optimal allocation scheme, and remove the current target node from the current priority queue; otherwise, remove the current target node from the current priority queue; where the initial optimal solution is infinite and the initial optimal allocation scheme is empty.

[0118] If the current target node is not a leaf node, select an undetermined variable in the current target node to branch and obtain several child nodes; where the variable includes: model size and computational requirements;

[0119] Based on the communication data of the current child nodes, calculate the total delay of each child node. If the current maximum total delay of the child nodes is less than the minimum total delay corresponding to the current optimal solution, or if the current minimum total delay of the child nodes is greater than the maximum total delay corresponding to the current optimal solution, then remove the current target node and its corresponding child nodes from the current priority queue. Otherwise, add the current child node to the current priority queue as a node in the priority queue, and remove the current target node to update the current priority queue.

[0120] An illustrative diagram illustrating the automatic decoupling deployment of inter-layer LLM based on the Transformer architecture is shown below. Figure 3 As shown, Figure 3 This paper describes the deployment architecture of a Transformer-based Large Language Model (LLM) in a multi-tiered computing environment. It demonstrates how to efficiently segment and process LLM models in a network consisting of cloud, middleware, and edge devices to optimize performance and reduce latency. Figure 3 A multi-layered network structure is presented, including a user layer and three layers (first, second, and third), each containing mNodes in either idle or busy states. These layers can be viewed as representing cloud, intermediate nodes, and edge nodes, each responsible for different types of computational tasks. The cloud primarily handles model training and management, intermediate nodes handle preprocessing and initial inference, and edge nodes focus on local inference and personalized fine-tuning.

[0121] Specifically, Figure 3 The presentation also detailed the encoder, decoder, multi-head attention mechanism, feed-forward layer, and addition and normalization steps within the Transformer-based LLM model. The encoder's input is word embeddings. The decoder structure is similar to the encoder but incorporates a masked multi-head attention encoder-decoder attention layer to perform specific processing tasks. The decoder's output is passed through a softmax layer to generate probabilities for the target vocabulary. Different computational tasks or data streams are distributed across different mNodes in the multi-layer network, enabling parallel processing and fully utilizing idle computing resources to reduce inference latency.

[0122] Specifically, the flexibility of the Transformer architecture allows the LLM model to be partitioned into multiple smaller segments (Mi), which can be executed independently on different device nodes in a multi-layer edge network. This flexible partitioning strategy enables the model to be flexibly deployed on different mNodes based on computing resources and task requirements.

[0123] Specifically, the branch-and-bound integer linear programming algorithm, namely the ILP-BB algorithm, aims to achieve automated decoupling and partitioning across layers in the deployment of large language models (LLMs) in multi-layer network environments. This decoupling and partitioning dynamically optimizes the allocation of model size and computational requirements based on the computational capabilities and resource conditions of different layers in the network, i.e., the corresponding coefficients. and The size of the resource allocation is determined by this refined approach, which minimizes the latency of the entire system and improves inference efficiency. Due to the complexity of cross-layer resource allocation, directly solving this problem is computationally infeasible; therefore, the algorithm employs a branch-and-bound method. This method cleverly decomposes a complex optimization problem into a series of more manageable and solvable subproblems, then progressively approximates the optimal solution.

[0124] Specifically, the algorithm's execution flow can be understood as a systematic search process. First, a priority queue (PQ) is initialized, and the root node, representing the initial state of the model size and computational allocation, is added to the queue. This root node represents the initial state of the entire optimization problem; at this point, the model has not yet been partitioned, and resources have not been allocated. Simultaneously, the objective function value z of the optimal solution is initialized to infinity, and the optimal allocation scheme π is empty. Here, z represents the total delay. Subsequently, as long as the priority queue is not empty, nodes are continuously extracted from it for processing. If the currently processed node is a leaf node, it means that all parts of the model have been allocated. At this point, the algorithm calculates the objective function value z corresponding to that node, i.e., the total delay under the current allocation scheme. If the current z value is less than the objective function value z* of the current optimal solution, it indicates that a better allocation scheme has been found, and the algorithm updates z* and the optimal allocation scheme π*. The objective function value z is calculated using the following formula:

[0125]

[0126] In the formula, This represents the computational power of the l-th layer.

[0127] If the current node is not a leaf node, it means that there are still some unassigned variables in the model. The algorithm will select one of these unassigned variables. or Proceed with a branch. Then, create two subproblems based on the chosen variable, i.e., the aforementioned "child nodes," representing the lower and upper bounds of that variable, respectively. For example, if you choose... When branching, the algorithm will try to branch out... The value of is set to the floor and floor of its current value. Then, for each subproblem, the algorithm calculates its corresponding objective function value and boundaries. Here, the algorithm evaluates the subproblem by calculating z and updates the lower and upper bounds. The lower bound is the minimum of the objective function values ​​of all subproblems, and the upper bound is the maximum of the objective function values ​​of all subproblems. These boundary values ​​are used for pruning to avoid unnecessary searches. If the lower bound is greater than the upper bound of the current optimal solution, or the upper bound is less than the lower bound of the current optimal solution, it means that the branch cannot contain the optimal solution, and the algorithm will prune and stop exploring this branch.

[0128] Preferably, by continuously selecting the highest-priority node for branching and pruning, the algorithm gradually approaches the optimal solution. This process can be viewed as searching the solution space, reducing the search range through pruning operations, and continuing until the optimal solution is found or it is confirmed that no solution meets the conditions. In summary, the ILP-BB algorithm uses the branch-and-bound method to efficiently search the solution space of integer linear programming problems, providing an effective cross-layer model partitioning and resource allocation scheme for LLM deployment in multi-layer networks. This helps to maximize the performance and efficiency of LLM in resource-constrained environments.

[0129] In this preferred embodiment, the LLM model is partitioned based on the branch-and-bound integer linear programming algorithm to obtain an allocation scheme that minimizes the total latency of the multilayer network.

[0130] In another preferred embodiment, the process of obtaining the target device node corresponding to each task to be scheduled based on the target task to be scheduled, the target task characteristics corresponding to the target task to be scheduled, the real-time monitoring data, and the preset task scheduling model, with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, includes:

[0131] Initialize the list of tasks to be scheduled; the initialized list includes all the target tasks to be scheduled mentioned above.

[0132] Repeat the task scheduling operation until the above list of tasks to be scheduled is empty, and determine the target device node corresponding to each task to be scheduled.

[0133] The aforementioned task scheduling operations include:

[0134] Select a device node as the current node to be scheduled, and retrieve the current task to be scheduled from the current list of tasks to be scheduled;

[0135] Based on the current tasks to be scheduled and the corresponding characteristics of the current target task, construct the current structure graph;

[0136] Obtain the current real-time monitoring data and the above-mentioned preset task scheduling model, input the current structure graph, current task characteristics and current real-time monitoring data into the above-mentioned preset task scheduling model, so that the above-mentioned preset task scheduling model can calculate the current degree matrix and adjacency matrix based on the current structure graph.

[0137] Based on the current degree matrix, the aforementioned adjacency matrix, and the current target task features, the current node feature matrix of the last layer in the aforementioned preset task scheduling model is obtained.

[0138] Based on the current node feature matrix and activation function, with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, the scheduling probability distribution between the current node to be scheduled and each current task to be scheduled is obtained.

[0139] The current node to be scheduled is selected as the target device node for the task with the highest scheduling probability, and this node is removed from the current list of tasks to be scheduled in order to update the list of tasks to be scheduled.

[0140] Specifically, based on the allocation scheme obtained above, the size of the segmented model deployed in each level of the LLM and the corresponding computational requirements can be determined. Then, based on the model size and computational requirements, the tasks that can be scheduled at that level can be determined.

[0141] Specifically, the core objective of graph neural network-based task scheduling algorithms is to achieve adaptive task scheduling within each layer of a multi-layered network environment for inference tasks involving large language models (LLMs). This involves more than simply allocating tasks to individual device nodes; it requires dynamically making optimal task offloading decisions based on the real-time resource status of each node (e.g., computing power, memory usage) and network conditions (e.g., bandwidth, latency). The algorithm aims to maximize the resource utilization of the entire system and reduce task execution time through this intelligent task scheduling, thereby improving overall performance. Unlike branch-and-bound integer linear programming algorithms, which focus on cross-layer model partitioning and resource allocation, graph neural network-based task scheduling algorithms concentrate on fine-grained task scheduling within the same layer.

[0142] Specifically, the core idea of ​​the algorithm lies in using Graph Neural Networks (GNNs) to process network topology data and propagate and update node features to optimize task scheduling decisions. This allows the algorithm to understand the relationships between nodes in the network and the resource status of each node, thus making more informed scheduling decisions. Specifically, the algorithm first initializes the parameters of the task scheduling model, including the weight matrix and bias terms. Then, the algorithm continuously monitors network resources, including but not limited to: task size (…). ), used to constrain task scheduling decisions (penalty parameters) ), and the computational capacity limitations of each layer of nodes ( The task size here can be understood as the computational load required for the LLM inference task; the penalty parameter is used to penalize behaviors that exceed the computational capacity of the device node during optimization; and the computational capacity limit represents the maximum processing capability of the device node. After monitoring network resources, the algorithm performs individual task scheduling for each layer of the network. For each layer, the algorithm collects the tasks that need to be scheduled within the current layer and puts these tasks into a waiting list (i.e., the list of tasks to be scheduled mentioned above). These tasks to be scheduled may be part of the model or data that needs further processing. The algorithm then schedules these tasks one by one until the waiting list is empty.

[0143] The algorithm calculates the degree matrix (D) and adjacency matrix (Λ). The degree matrix describes the number of other nodes connected to each node in the graph, while the adjacency matrix describes the connections between nodes. Both are fundamental representations of graph-structured data. Subsequently, the algorithm performs feature propagation through multiple GNN layers. In each GNN layer, nodes aggregate and update their own feature information along with features from neighboring nodes, thereby learning more comprehensive network state information. This process is performed using the following formula:

[0144]

[0145] In the formula, It is the feature matrix of the nodes in the l-th GNN layer. D is the adjacency matrix, and D is the degree matrix. It is the weight matrix of the l-th GNN layer. This is an activation function (e.g., ReLU). Feature propagation allows each node in the network to "sense" the state of other nodes, thus providing a basis for subsequent scheduling decisions. After processing through multiple GNN layers, the Softmax activation function is used on the output of the last GNN layer to determine the optimal target device node for the task. The Softmax function transforms the feature vectors of the nodes into a probability distribution, representing the likelihood of each node being the task scheduling target corresponding to the target device node. The task to be scheduled corresponding to the node with the highest probability is scheduled to that target device node.

[0146] Specifically, during the training of the task scheduling model, to optimize the model's parameters, the algorithm defines a loss function (L) based on the objective function of a quadratic unconstrained binary optimization (QUBO) model. The QUBO model is a mathematical framework for solving combinatorial optimization problems, allowing the algorithm to transform complex scheduling problems into an optimizable mathematical form. The loss function L is designed to reflect the cost of task scheduling decisions and the penalty for violating computational capacity constraints. By using backpropagation, the algorithm updates the model's parameters based on the value of the loss function L, thereby minimizing the loss function and optimizing the task scheduling strategy. In this way, the algorithm transforms a discrete optimization problem (task scheduling) into a continuous optimization problem (minimizing the loss function). This transformation enables GNNs to be effectively applied to large-scale combinatorial optimization problems in various real-world scenarios. The final output of the algorithm is the scheduling decision variable. It is a binary value used to indicate whether a task originated from a node. Scheduled to another node In summary, the TS-GNN algorithm leverages the powerful graph data processing capabilities of GNNs and the QUBO model's ability to handle combinatorial optimization problems to dynamically learn and optimize task scheduling strategies within each layer of a multi-layered network, thereby improving system resource utilization and overall performance. This adaptive scheduling approach helps to more effectively deploy and run large language models in complex network environments.

[0147] In this preferred embodiment, the target device node corresponding to each task to be scheduled is obtained by using the target task to be scheduled, the target task characteristics corresponding to the target task to be scheduled, real-time monitoring data, and a preset task scheduling model, with the goal of minimizing the sum of scheduling cost and violation of computing power constraints.

[0148] In another preferred embodiment, the above-mentioned construction of the current structure graph based on the current tasks to be scheduled and the corresponding current target task characteristics includes:

[0149] The current task to be scheduled is used as the graph node of the current structure graph, and the graph nodes are connected according to the characteristics of the current target task to obtain the current structure graph.

[0150] Specifically, to capture the spatial and temporal dependencies between tasks to be scheduled, tasks within each level are represented as nodes in a structural graph. Edges represent dependencies between tasks, which are influenced by characteristics such as task computing power, network conditions, task size, and computing power requirements. For example, when there are temporal conflicts between task executions, edges will be generated between the corresponding nodes in the graph.

[0151] In this preferred embodiment, a structure graph is constructed by the individual tasks to be scheduled and the dependencies between them.

[0152] In another preferred embodiment, when task scheduling aims to minimize the sum of scheduling cost and violation of computing power constraints, the corresponding second objective function is:

[0153]

[0154] In the formula, H represents the second objective function. Indicates the task size. Indicates whether the task to be scheduled originates from the device node. Dispatch to device node , Indicates the penalty parameter. This represents the computational capability constraint at level l.

[0155] Specifically, the above uses quadratic unconstrained binary optimization (QUBO) to model the task scheduling problem. Nodes at a specific level... and nodes of level l Scheduling decisions can use binary variables Indicates. If =1 indicates that the task originates from the node. Scheduled to node ,otherwise =0.

[0156] Specifically, Indicates scheduling cost, This indicates a punishment for violating computational capacity constraints.

[0157] In another preferred embodiment, to comprehensively evaluate the task scheduling method based on a distributed large language model proposed in this invention, three computing layers were designed, with multiple heterogeneous device nodes deployed in each layer. These nodes have varying computing and communication capabilities, aiming to simulate complex heterogeneous computing environments in the real world. The specific configuration is as follows: First layer: Six Jetson AGX Orin nodes were deployed, each with powerful computing capabilities up to 275 TFLOPS, suitable for handling computationally intensive tasks; Second layer: Six Jetson Orin NX nodes were deployed, each with a computing capability of 70 TFLOPS, serving as an intermediate layer between the cloud and the edge, responsible for handling some computationally intensive tasks; Third layer: Six Jetson AGX Xavier nodes were deployed, each with a computing capability of 32 TFLOPS, located at the network edge, directly serving end users and handling local inference tasks. To simulate real-world LLM application scenarios, the Qwen2 and OPT models were selected as test objects. These models were divided into multiple parts, and then these parts were deployed to different mNode nodes for inference tasks. The number of input tokens for the inference task is set to 4096 or 2048 to simulate large-scale and small-to-medium-scale input scenarios, respectively.

[0158] The network communication is based on the Orthogonal Frequency Division Multiple Access (OFDMA) protocol, which is widely used in wireless communication networks. A bandwidth of 5 MHz and a channel noise of -117 dBm were set to simulate a real-world communication environment. Furthermore, to simulate the diversity of user equipment, the device communication power was randomly varied between 0.3W, 0.5W, and 0.7W. To more realistically reflect the characteristics of a multi-layered network, delays for each layer were introduced, set to 10ms for the first layer, 20ms for the second, and 30ms for the third, simulating the delays in data transmission and processing at different layers. To ensure system reliability, redundant nodes were added; when a node fails, a redundant node can take over, ensuring continuous system operation. Five algorithms were compared:

[0159] The first algorithm, Uniform Model Partitioning and Random Assignment (UMPRA), uniformly partitions the LLM model into multiple tasks of equal size and then randomly assigns them to nodes in the multi-layer network. This method is simple and direct, but it may not fully utilize the computational power of different nodes.

[0160] The second algorithm, Random Model Partitioning and Random Scheduling (RMPSS), randomly partitions the LLM model into tasks of varying sizes and then randomly schedules them to nodes in the multi-layer network. This method is more flexible but may lead to load imbalance.

[0161] The third algorithm: Model Partitioning and Arbitrary Task Assignment (MPATD): This algorithm uses Integer Linear Programming (ILP) to partition the LLM model and then arbitrarily assigns tasks to mNodes. This is a baseline that considers model partitioning, but task assignment is not optimized.

[0162] The fourth algorithm: Model Partitioning and Heuristic Task Scheduling Algorithm (MPHTS): This algorithm uses Integer Linear Programming (ILP) to partition the LLM model and uses heuristic rules for task scheduling. It is a relatively optimized baseline that uses heuristic rules for task allocation.

[0163] The algorithm proposed in this application, the Multi-Level Multi-Node LLM Cooperative Computation Scheduling Algorithm (MMSL), employs dynamic model partitioning and graph-based analysis to optimize task scheduling. This algorithm achieves more efficient resource utilization and task scheduling.

[0164] The following is an illustrative comparison of the performance of inference tasks based on the OPT model within a three-layer framework: Figure 4 As shown, the performance comparison results of inference tasks based on the Qwen2 model in the three-layer framework are as follows: Figure 5 As shown. Figure 4 and Figure 5 This demonstrates the performance of different algorithms in performing LLM inference tasks under a three-layer network architecture. The horizontal axis represents the number of tasks to be processed (from 5 to 45), and the vertical axis represents the total inference time required to complete all tasks. Figure 4 The performance of the OPT model with 4096 tokens as input is demonstrated. Experimental results clearly show that UMPRA and RMPSS have significantly longer completion times than MMSL. Specifically, when there are 40 tasks, UMPRA and RMPSS require 1219 seconds and 1042 seconds, respectively, to complete all tasks, while MMSL takes only 708 seconds. This indicates that the MMSL algorithm is significantly more efficient than UMPRA and RMPSS, improving efficiency by 72.2% and 47.2%, respectively. Figure 5 The results using the Qwen2 model are presented. As the number of tasks increases, the completion time of UMPRA and RMPSS increases rapidly, while the time of MMSL increases significantly more slowly, indicating that MMSL has better scalability when handling more tasks.

[0165] The following is an illustrative comparison of the performance of inference tasks based on the OPT model within a four-layer framework: Figure 6 As shown, the performance comparison results of the inference task based on the Qwen2 model in the four-layer framework are as follows: Figure 7 As shown. Figure 6 and Figure 7The performance of a four-layer network architecture was demonstrated. This network adds an edge computing layer to the original three-layer structure and also uses Jetson AGXXavier nodes. Experimental results are similar to those of the three-layer network; the MMSL algorithm consistently maintains the lowest task execution time, further demonstrating that MMSL has efficient model segmentation performance under different network architectures. Figure 6 It uses the OPT model. Figure 7 These results are from the Qwen2 model. The results show that MMSL, through its efficient model segmentation mechanism, still achieves the shortest task execution time in a four-layer network.

[0166] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0167] like Figure 8 As shown, one embodiment of the present invention provides a task scheduling device based on a distributed large language model, comprising:

[0168] Data acquisition module and target device node determination module;

[0169] The aforementioned data acquisition module is used to acquire communication data, scheduled tasks, task characteristics of scheduled tasks, and real-time monitoring data of each device node in the multi-layer network of the distributed large language model; wherein, the aforementioned task characteristics include: task computing power, network conditions, task size, and computing power requirements; the aforementioned real-time monitoring data includes: computing power, memory, bandwidth, and latency.

[0170] The aforementioned target device node determination module is used to perform model segmentation and task scheduling on the aforementioned multi-layer network based on the aforementioned communication data, the tasks to be scheduled, the task characteristics of the tasks to be scheduled, and the real-time monitoring data of each device node, with the goal of minimizing the total latency of the multi-layer network, to obtain the target device node corresponding to each task to be scheduled, and to assign each task to be scheduled to the corresponding target device node for scheduling.

[0171] The model is segmented based on the aforementioned communication data and the branch-and-bound integer linear programming algorithm to obtain the allocation scheme for each level; the allocation scheme includes: model size and computational requirements.

[0172] Based on the above allocation scheme, each task to be scheduled is assigned to each level to obtain the target task to be scheduled for each level.

[0173] Based on the aforementioned target tasks to be scheduled, the target task characteristics corresponding to the aforementioned target tasks to be scheduled, the aforementioned real-time monitoring data, and the preset task scheduling model, task scheduling is performed with the goal of minimizing the sum of scheduling cost and violation of computing power constraints, thereby obtaining the target device nodes corresponding to each task to be scheduled; wherein, the aforementioned scheduling cost and the sum of violation of computing power constraints are calculated based on the aforementioned real-time monitoring data.

[0174] In a preferred embodiment, the target device node determination module includes:

[0175] Priority queue initialization unit and model segmentation unit;

[0176] The aforementioned priority queue initialization unit is used to initialize the priority queue; wherein, the priority queue includes several nodes for representing the aforementioned allocation scheme; the initialized priority queue includes: the root node that has not been model-segmented;

[0177] The aforementioned model segmentation unit is used to repeatedly perform the model segmentation operation until the priority queue is empty, and the optimal allocation scheme corresponding to the empty priority queue is used as the aforementioned allocation scheme.

[0178] The above-mentioned model segmentation operations include:

[0179] Get the current priority queue, and arbitrarily select a node from the current priority queue as the target node; where the initial node is the root node mentioned above;

[0180] If the current target node is a leaf node, calculate the total delay corresponding to the current target node based on the communication data corresponding to the current target node; where the leaf node represents the allocation scheme corresponding to the multi-layer network.

[0181] If the current total delay is less than the minimum total delay corresponding to the current optimal solution, update the minimum total delay based on the current total delay, take the allocation scheme corresponding to the current target node as the current optimal allocation scheme, and remove the current target node from the current priority queue; otherwise, remove the current target node from the current priority queue; where the initial optimal solution is infinite and the initial optimal allocation scheme is empty.

[0182] If the current target node is not a leaf node, select an undetermined variable in the current target node to branch and obtain several child nodes; where the variable includes: model size and computational requirements;

[0183] Based on the communication data of the current child nodes, calculate the total delay of each child node. If the current maximum total delay of the child nodes is less than the minimum total delay corresponding to the current optimal solution, or if the current minimum total delay of the child nodes is greater than the maximum total delay corresponding to the current optimal solution, then remove the current target node and its corresponding child nodes from the current priority queue. Otherwise, add the current child node to the current priority queue as a node in the priority queue, and remove the current target node to update the current priority queue.

[0184] It should be noted that the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort. The above schematic diagrams are merely examples of a task scheduling device based on a distributed large language model and do not constitute a limitation on a task scheduling device based on a distributed large language model. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0185] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments.

[0186] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the task scheduling method based on a distributed large language model described in any embodiment of the present invention.

[0187] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the device.

[0188] The aforementioned terminal devices may be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These devices may include, but are not limited to, processors and memory.

[0189] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the device, connecting various parts of the device via various interfaces and lines.

[0190] The aforementioned memory can be used to store the aforementioned computer programs and / or modules. The aforementioned processor implements various functions of the aforementioned device by running or executing the computer programs and / or modules stored in the aforementioned memory, and by calling data stored in the memory. The aforementioned memory may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0191] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments.

[0192] Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the storage medium is located to execute the task scheduling method based on a distributed large language model described in any embodiment of the present invention.

[0193] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0194] Compared with the prior art, by implementing the above embodiments of the present invention, resource management of the multi-layer network inside a large language model can be realized, thereby improving network performance and reducing latency.

[0195] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A task scheduling method based on a distributed large language model, characterized in that, Comprise: Obtain the communication data of each device node in the multi-layer network of the distributed large language model, the to-be-scheduled task, the task characteristics of the to-be-scheduled task, and the real-time monitoring data of each device node; wherein, the task characteristics include: task computing power, network condition, task size and computing power demand; the real-time monitoring data includes: computing power, memory, bandwidth and delay; According to the communication data, the to-be-scheduled task, the task characteristics of the to-be-scheduled task, and the real-time monitoring data of each device node, the model partitioning and task scheduling of the multi-layer network are performed with the minimum total delay of the multi-layer network as the target, to obtain the target device node corresponding to each to-be-scheduled task, and each to-be-scheduled task is allocated to the corresponding target device node for scheduling; wherein, when the model partitioning and task scheduling of the multi-layer network are performed with the minimum total delay of the multi-layer network as the target, the corresponding first target function is: In the formula, Let L represent the total delay, L represent the total number of levels, and l represent the l-th level. This represents the transmission delay at level l. This represents the computational latency of level l. This represents the device node at level l. This represents the total number of device nodes at level l. Indicates the first Hierarchical device nodes, Indicates the first The total number of device nodes at each level Indicates whether the scheduled task is scheduled from level l to level 1. Hierarchy This represents the computational demand allocation coefficient for level l. Indicates computational requirements. This represents the computing power of the device nodes in level l. This represents the size allocation coefficient of the l-th level model. Indicates the model size. This represents the achievable transmission rate of node u. This represents the bandwidth of subchannel k. Indicates the interference-to-noise ratio of the transmitted signal. This represents the interference plus noise power at level l. This indicates that other users g in the same sub-channel set are paired with the first... Hierarchical nodes Interference, This represents the signal-to-interference-to-noise ratio between level l' and level l. Indicates starting from the l-th level node To the l' level node Channel gain, Represents the node at level l To the node The transmission power; Wherein, according to the communication data and the branch and bound integer linear programming algorithm, the allocation scheme of each level is obtained; the allocation scheme includes: model size and computing demand; According to the allocation scheme, each to-be-scheduled task is allocated to each level to obtain the target to-be-scheduled task corresponding to each level; According to the target to-be-scheduled task, the target task characteristics corresponding to the target to-be-scheduled task, the real-time monitoring data, and a preset task scheduling model, the task scheduling is performed with the minimum sum of scheduling cost and violation of computing power constraint as the target, to obtain the target device node corresponding to each to-be-scheduled task; wherein, the sum of scheduling cost and violation of computing power constraint is calculated according to the real-time monitoring data.

2. The task scheduling method based on a distributed large language model according to claim 1, characterized in that, The allocation scheme of each level is obtained according to the communication data and the branch and bound integer linear programming algorithm, comprising: Initialize the priority queue; wherein, the priority queue includes a plurality of nodes for representing the allocation scheme; the initialized priority queue includes a root node which has not been subjected to model partitioning; Repeat the model partitioning operation until the priority queue is empty, and the optimal allocation scheme corresponding to the time when the priority queue is empty is taken as the allocation scheme; Wherein, the model partitioning operation includes: Obtain the current priority queue, and randomly select a node from the current priority queue as the target node; wherein, the initial node is the root node; If the current target node is a leaf node, calculate the total delay corresponding to the current target node according to the communication data corresponding to the current target node; wherein, the leaf node represents the allocation scheme corresponding to the multi-layer network; When the current total delay is less than the minimum total delay corresponding to the current optimal solution, update the minimum total delay according to the current total delay, take the allocation scheme corresponding to the current target node as the current optimal allocation scheme, and delete the current target node from the current priority queue; otherwise, delete the current target node from the current priority queue; wherein, the initial optimal solution is infinite, and the initial optimal allocation scheme is empty; If the current target node is not a leaf node, a variable in the current target node is selected to branch, and a plurality of child nodes are obtained; wherein the variable includes: model size and computing requirement; According to the communication data of the current child node, a child node total delay corresponding to each child node is calculated, if the current maximum child node total delay is less than the minimum total delay corresponding to the current optimal solution, or the current minimum child node total delay is greater than the maximum total delay corresponding to the current optimal solution, the current target node and the child node corresponding to the current target node are deleted from the current priority queue; otherwise, the current child node is added to the current priority queue as a node of the priority queue, and the current target node is deleted to update the current priority queue.

3. The task scheduling method based on a distributed large language model according to claim 2, characterized in that, The target device node corresponding to each to-be-scheduled task is obtained by taking the to-be-scheduled task, the target task feature corresponding to the to-be-scheduled task, the real-time monitoring data, and a preset task scheduling model as input, and taking the sum of scheduling cost and violation of computing capability constraint as minimum as an objective, and the method comprises the following steps: Initializing a to-be-scheduled task list; wherein the initialized to-be-scheduled list comprises all the target to-be-scheduled tasks; Repeating the task scheduling operation until the to-be-scheduled task list is empty, and determining the target device node corresponding to each to-be-scheduled task; The task scheduling operation comprises: Selecting a device node as a current to-be-scheduled node, and obtaining a current to-be-scheduled task from the current to-be-scheduled list; Constructing a current structure graph according to the current to-be-scheduled task and the corresponding current target task feature; Obtaining current real-time monitoring data and the preset task scheduling model, inputting the current structure graph, the current task feature, and the current real-time monitoring data into the preset task scheduling model, so that the preset task scheduling model calculates the current degree matrix and the adjacency matrix according to the current structure graph; Obtaining the last layer node feature matrix of the preset task scheduling model according to the current degree matrix, the adjacency matrix, and the current target task feature; Obtaining the scheduling probability distribution between the current to-be-scheduled node and each current to-be-scheduled task by taking the sum of scheduling cost and violation of computing capability constraint as minimum as an objective according to the current node feature matrix and the activation function; Taking the current to-be-scheduled node as the target device node of the to-be-scheduled task with the maximum scheduling probability, and deleting the to-be-scheduled node from the current to-be-scheduled list to update the to-be-scheduled list.

4. The task scheduling method based on a distributed large language model according to claim 3, characterized in that, The current structure graph is constructed according to the current to-be-scheduled task and the corresponding current target task feature, and the method comprises the following steps: Taking the current to-be-scheduled task as a graph node of the current structure graph, and connecting each graph node according to the current target task feature to obtain the current structure graph.

5. The task scheduling method based on a distributed large language model according to claim 4, characterized in that, When scheduling a task by taking the sum of scheduling cost and violation of computing capability constraint as minimum as an objective, the corresponding second objective function is: where H represents the second objective function, represents the task size, represents whether the task to be scheduled is from a device node to a device node , represents the penalty parameter, represents the computing power constraint of the lth level.

6. A task scheduling apparatus based on a distributed large language model, characterized by, It comprises: A data acquisition module and a target device node determination module; The data acquisition module is configured to acquire communication data of each device node in a multi-layer network of a distributed large language model, a to-be-scheduled task, a task feature of the to-be-scheduled task, and real-time monitoring data of each device node. The task feature includes task computing power, network conditions, task size, and computing power demand. The real-time monitoring data includes computing power, memory, bandwidth, and delay. The target device node determination module is configured to perform model partitioning and task scheduling on the multi-layer network with the minimum total delay of the multi-layer network as a target based on the communication data, the to-be-scheduled task, the task feature of the to-be-scheduled task, and the real-time monitoring data of each device node, to obtain a target device node corresponding to each to-be-scheduled task, and to allocate each to-be-scheduled task to the corresponding target device node for scheduling. When performing model partitioning and task scheduling on the multi-layer network with the minimum total delay of the multi-layer network as a target, the corresponding first target function is: In the formula, Let L represent the total delay, L represent the total number of levels, and l represent the l-th level. This represents the transmission delay at level l. This represents the computational latency of level l. This represents the device node at level l. This represents the total number of device nodes at level l. Indicates the first Hierarchical device nodes, Indicates the first The total number of device nodes at each level Indicates whether the scheduled task is scheduled from level l to level 1. Hierarchy This represents the computational demand allocation coefficient for level l. Indicates computational requirements. This represents the computing power of the device nodes in level l. This represents the size allocation coefficient of the l-th level model. Indicates the model size. This represents the achievable transmission rate of node u. This represents the bandwidth of subchannel k. Indicates the interference-to-noise ratio of the transmitted signal. This represents the interference plus noise power at level l. This indicates that other users g in the same sub-channel set are paired with the first... Hierarchical nodes Interference, This represents the signal-to-interference-to-noise ratio between level l' and level l. Indicates starting from the l-th level node To the l' level node Channel gain, Represents the node at level l To the node The transmission power; wherein the model partitioning is performed based on the communication data and an integer linear programming algorithm of branch and bound to obtain an allocation scheme of each level. The allocation scheme includes model size and computing demand. Each to-be-scheduled task is allocated to each level based on the allocation scheme to obtain a target to-be-scheduled task corresponding to each level. A target device node corresponding to each to-be-scheduled task is obtained by performing task scheduling based on the target to-be-scheduled task, a target task feature corresponding to the target to-be-scheduled task, the real-time monitoring data, and a preset task scheduling model, with the minimum sum of scheduling cost and violation of computing power constraint as a target. The sum of scheduling cost and violation of computing power constraint is calculated based on the real-time monitoring data.

7. The task scheduling apparatus based on a distributed large language model according to claim 6, characterized in that, The target device node determination module includes: a priority queue initialization unit and a model partitioning unit. The priority queue initialization unit is configured to initialize a priority queue. The priority queue includes a plurality of nodes representing the allocation scheme. The initialized priority queue includes a root node that has not been subjected to model partitioning. The model partitioning unit is configured to repeatedly perform a model partitioning operation until the priority queue is empty, and to take the optimal allocation scheme corresponding to the empty priority queue as the allocation scheme. The model partitioning operation includes: acquiring a current priority queue and randomly selecting a node from the current priority queue as a target node. The initial node is the root node. If the current target node is a leaf node, the total delay corresponding to the current target node is calculated based on the communication data corresponding to the current target node. The leaf node represents the allocation scheme corresponding to the multi-layer network. If the current total delay is less than the minimum total delay corresponding to the current optimal solution, the minimum total delay is updated based on the current total delay, the allocation scheme corresponding to the current target node is taken as the current optimal allocation scheme, and the current target node is deleted from the current priority queue. Otherwise, the current target node is deleted from the current priority queue. The initial optimal solution is infinite, and the initial optimal allocation scheme is empty. If the current target node is not a leaf node, a variable in the current target node is selected to branch, and a plurality of child nodes are obtained; wherein the variable includes a model size and a computing requirement; According to the communication data of the current child node, a total delay of each child node is calculated, if a maximum total delay of the current child node is less than a minimum total delay corresponding to the current optimal solution, or a minimum total delay of the current child node is greater than a maximum total delay corresponding to the current optimal solution, the current target node and the child nodes corresponding to the current target node are deleted from the current priority queue; otherwise, the current child node is added to the current priority queue as a node of the priority queue, and the current target node is deleted to update the current priority queue.

8. A terminal device, comprising: The computer program is configured to be executed by the processor, and the processor implements the task scheduling method based on the distributed large language model according to the computer program.

9. A storage medium, characterized by The storage medium includes a stored computer program, wherein the computer program controls a device where the storage medium is located to execute the task scheduling method based on the distributed large language model according to any one of claims 1 to 5 when the computer program is running.

Images