Large language model reasoning task scheduling method

Abstract

The invention discloses an inference task scheduling method for a large language model, and the method comprises the steps: monitoring the operation states of N computational nodes in a network in real time, extracting the multi-dimensional resource attribute of each computational node, and constructing a resource feature vector; wherein the calculation point comprises a cloud server and an edge device; calculating the resource difference degree of each computing node according to the resource feature vector of each computing node; according to the resource difference degree and the real-time resource availability of each computing node and the metadata of the inference task to be processed, based on a Markov decision process, outputting a scheduling scheme of the inference task; and according to the scheduling scheme, scheduling the reasoning task to the corresponding computing node. According to the invention, fine-grained scheduling comprehensively considering node load, network conditions and LLM reasoning characteristics can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for scheduling reasoning tasks for large language models. Background Technology

[0002] In recent years, with the widespread adoption of Internet of Things (IoT) devices and the rapid development of edge computing technology, applying Large Language Models (LLM) to IoT environments has become a trend. LLM's advantages in efficient data analysis, pattern recognition, and generation capabilities can provide unprecedented intelligent services to the IoT ecosystem.

[0003] However, deploying and running large language models on IoT devices faces significant challenges: First, terminal computing power and storage limitations. IoT devices typically have limited computing power and storage capacity, making it difficult to directly run complex deep learning models. While the task can be offloaded entirely to the cloud, this leads to network latency becoming a bottleneck, and frequent data transmission consumes significant bandwidth and increases operating costs. Second, heterogeneous edge resources and large model size. Although edge devices have better hardware performance than IoT terminals, large models like DeepSeek R1-14B require at least 24GB of memory, exceeding the capacity of most edge devices. This necessitates a trade-off between model inference accuracy and transmission latency—that is, choosing between offloading the task to the cloud or the edge. Third, the iterative nature of inference tasks. Unlike the batch processing model of traditional AI tasks, the inference process of large language models has significant iterative characteristics. The processing time of a single request depends not only on its own task complexity but also on other requests in the same batch.

[0004] Existing task offloading strategies often lack precise capture of resource correlations and tend to ignore the constraints of inference accuracy when pursuing efficiency, resulting in increased latency or a significant drop in accuracy in high-concurrency scenarios. Therefore, there is an urgent need for a fine-grained scheduling method that can comprehensively consider node load, network conditions, and LLM inference characteristics. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a method for scheduling inference tasks of large language models. This method can comprehensively consider node load, network conditions, and fine-grained scheduling of LLM inference characteristics, and schedule an appropriate number of inference tasks to be executed on edge devices. This avoids the network bandwidth bottleneck caused by uploading all data to the cloud, thereby significantly reducing the average response time and communication overhead of the system when processing high-concurrency IoT tasks.

[0006] To achieve the above objectives, this invention provides a method for scheduling inference tasks in a large language model, comprising: Real-time monitoring network NThe system monitors the running status of each computing node and extracts multi-dimensional resource attributes for each computing node to construct a resource feature vector; wherein, the computing nodes include cloud servers and edge devices; Calculate the resource dissimilarity of each computing node based on the resource feature vector of each computing node; Based on the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference tasks to be processed, a scheduling scheme for the inference tasks is output based on the Markov decision process. According to the scheduling scheme, the inference task will be scheduled to the corresponding computing node.

[0007] Preferably, the step of calculating the resource dissimilarity of each computing node based on the resource feature vector of each computing node specifically involves: Calculate the first according to the following formula 1. i Resource disparity among computing nodes : (Formula 1) in, For the first i Resource feature vectors of each computing node For the first i The resource mean vector within the neighborhood of each computing node For the first i The local covariance matrix of resources in the neighborhood of each computing node; T represents the transpose operation, and -1 represents the matrix inversion; The resource attributes include: computing power of computing units, remaining video memory capacity, network bandwidth fluctuation rate, and configuration parameters of the currently loaded large language model (LLM).

[0008] Preferably, the step of outputting a scheduling scheme for the inference task based on the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference task to be processed, using a Markov decision process, specifically includes: Based on the input Based on Markov decision processes, the output maximizes the reward. The scheduling scheme for the reasoning task is obtained; and the parameters of the Markov decision process are adjusted accordingly based on the reward function. in, The state space of the Markov decision process includes: , , ; in, Indicates time t The multidimensional resource knowledge base is calculated based on the resource differences of each computing node; This indicates the task queue status, which records the metadata of the inference tasks to be processed. This indicates the real-time resource availability of each computing node; in, The action space of the Markov decision process includes: the decision vector output for each inference task; and the action space for the inference task. k The output decision vector includes: ∈{0,1} and ;like =1 indicates that the reasoning task will be completed. k Scheduled to compute node m Execute; otherwise =0; Represents a computing node m For reasoning tasks k The specific amount of computing power allocated; The reward function is constructed based on the following penalty and reward terms; the penalty terms include: a delay penalty term. Resource efficiency penalty items The rewards include: Reasoning Quality Rewards. ; in, The sum of the task transmission delay, computation delay, and propagation delay for each inference task is negatively evaluated to penalize high-latency decisions. The sum of the resource disparities of the computing nodes scheduled for each inference task is negative; ;in, Indicates the reasoning task k Scheduled to compute node m The accuracy of predictive inference; This represents the preset positive reward weight coefficient, which is used to give rewards based on the level of accuracy when the accuracy constraint is met; This represents a preset penalty weight coefficient, used to impose severe penalties when accuracy constraints are not met; Indicates the first k Minimum reasoning accuracy requirements for each reasoning task; , , For respectively , , The set weight value.

[0009] in, Specifically, it is calculated according to the following formula 2: (Formula 2) in, The set attenuation coefficient, For a momentt -1 Multidimensional Resource Knowledge Base.

[0010] Preferably, the scheduling scheme for outputting the inference task based on the Markov decision process is specifically as follows: While ensuring the minimum inference accuracy requirement of the task, a scheduling scheme for the inference task is output based on the Markov decision process.

[0011] The present invention also provides a computer device, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for scheduling inference tasks of a large language model.

[0012] The present invention also provides a computer-readable storage medium storing a computer program that can be executed by at least one processor to perform the steps of the above-described method for scheduling inference tasks of a large language model.

[0013] In the technical solution of this invention, a large language model (LLM) is loaded into the real-time monitoring network. N The system analyzes the running status of each computing node and extracts multi-dimensional resource attributes for each node to construct a resource feature vector. The computing nodes include cloud servers and edge devices. Based on the resource feature vector of each computing node, the resource dissimilarity of each computing node is calculated. Based on the resource dissimilarity of each computing node, real-time resource availability, and metadata of the inference task to be processed, a scheduling scheme for the inference task is output using a Markov decision process. According to the scheduling scheme, the inference task is scheduled to the corresponding computing node.

[0014] Because this invention is based on a multi-dimensional resource awareness mechanism with adaptive Markov metric, it can accurately assess the real-time load status of cloud and heterogeneous edge nodes, and implement a fine-grained task offloading strategy based on Markov decision process. It can achieve fine-grained scheduling that comprehensively considers node load, network conditions and LLM inference characteristics, and schedule an appropriate amount of inference tasks to be executed on edge devices, avoiding the network bandwidth bottleneck caused by uploading all data to the cloud. Thus, when processing high-concurrency IoT tasks, it significantly reduces the average response time and communication overhead of the system. Furthermore, this invention effectively addresses the problem that traditional reinforcement learning algorithms often sacrifice inference accuracy when pursuing low latency by introducing a constrained policy optimization algorithm and a Lyapunov constrained network. By incorporating inference accuracy as a hard constraint into the optimization objective, this invention ensures that the inference accuracy of large language models always meets the requirements of the service level agreement on resource-constrained edge nodes, while maximizing the utilization efficiency of computational resources. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a reasoning task scheduling method for a large language model provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a computer device hardware structure provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] The present invention proposes a method for scheduling inference tasks in a large language model, the specific process of which is as follows: Figure 1 As shown, it includes the following steps: Step S101: Real-time monitoring of the network loaded with large language models N The running status of each computing node is recorded, and the multi-dimensional resource attributes of each computing node are extracted to construct a resource feature vector; In this step, the focus is on cloud servers and edge devices that have loaded large language models into the network. NEach computing node has a real-time monitoring system to track its operational status and extract multi-dimensional resource attributes for each node to construct a resource feature vector. For the i Each computing node, extract its K Dimensional resource attributes, construct resource feature vectors ;in, K The resource attributes include: computing power of computing units, remaining video memory capacity, network bandwidth fluctuation rate, and configuration parameters of the currently loaded large language model; the configuration parameters may include: number of model parameters, context window size, quantization bit width (such as FP16, INT8), and number of attention heads.

[0021] Step S102: Calculate the resource difference degree of each computing node based on the resource feature vector of each computing node, and construct a multi-dimensional resource knowledge base; In this step, to eliminate the dimensional differences between different resource dimensions (such as bandwidth and video memory) and to consider the distribution of computing nodes in the network topology, the calculation of the first... i Mahalanobis distance score of each computing node relative to its neighboring nodes As the first i A neighborhood-aware adaptive resource difference measure for the nth computing node; specifically, the nth node is calculated according to the following formula 1. i Resource disparity among computing nodes : (Formula 1) in, For the first i The resource mean vector within the neighborhood of each computing node For the first i The local covariance matrix of resources in the neighborhood of each computing node; T represents the transpose operation, and -1 represents the matrix inversion; Furthermore, the time-series fusion update of the resource difference measurement results is performed, that is, a sliding time window mechanism is used to fuse historical and current states to construct a multi-dimensional resource knowledge base. Specifically, using the attenuation coefficient For time t -1 Multidimensional Resource Knowledge Base Weighting is performed, and the current resource difference measurement results are concatenated to obtain the time-to-time value. t Multidimensional resource knowledge base ;Right now The update formula is shown in Formula 2: (Formula 2) in, Indicates splicing.

[0022] Through a multidimensional resource knowledge base It can dynamically maintain the resource evolution history of all network nodes, providing time-series-specific state inputs for subsequent intelligent offloading decisions.

[0023] Step S103: Based on the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference task to be processed, output the scheduling scheme of the inference task based on the Markov decision process. In this step, a scheduling scheme for inference tasks is output based on a pre-established task offloading model based on Markov decision process, taking into account the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference tasks to be processed. In fact, Markov Decision Processes (MDPs) are a core concept in reinforcement learning, primarily used to describe how a Markov decision process can perform a task given a specific state. How to choose The actions taken are aimed at maximizing long-term rewards; in the task offloading model (or agent) based on Markov decision processes pre-established in this invention, the state space... ,include: , , ; in, This indicates the task queue status, which records the metadata of the inference tasks to be processed; the... k The metadata for the first inference task includes: k Input data volume for each inference task , No. k The estimated computational complexity of each inference task , No. k Minimum inference accuracy requirement for each inference task ; This indicates the real-time resource availability of each computing node; including each computing node... m Current moment t Available network bandwidth and available computing resource margin .

[0024] In the pre-established task offloading model based on Markov decision processes in this invention, the action space This includes: the decision vector output for each inference task; and the decision vector output for each inference task. k The output decision vector includes: ∈{0,1} and ; Among them, if =1 indicates that the reasoning task will be completed. k Scheduled to compute node m Execute; if =0, which means the reasoning task will be... k Not scheduled to compute node m ; Represents a computing node m For reasoning tasks k The specific amount of computing power allocated is limited by the physical limit of the computing nodes.

[0025] In the pre-established task offloading model based on Markov decision processes in this invention, the function for calculating the reward (i.e., the reward function) is a multi-objective weighted reward function, constructed based on the following penalty and reward terms; the penalty terms include: a delay penalty term. Resource efficiency penalty items The rewards include: Reasoning Quality Rewards. Thus, the optimal balance is achieved between latency, resource efficiency, and inference quality; that is, the defined total reward. ; in, The sum of the task transmission delay, computation delay, and propagation delay for each inference task is negatively evaluated to penalize high-latency decisions. The sum of the resource disparities of the computing nodes scheduled for each inference task is negative; that is, according to This approach aims to encourage load balancing by penalizing nodes that are overloaded or have large resource deviations when assigning tasks.

[0026] If the accuracy of prediction inference When the minimum inference accuracy requirement set by the user is met It is positive if the value is positive, otherwise it is negative; that is, when the prediction inference accuracy is positive... Meets the minimum requirements set by the user. Give positive rewards when appropriate, otherwise impose severe punishment: ;in, Indicates the reasoning task k Scheduled to compute node m The accuracy of predictive inference; This represents the preset positive reward weight coefficient, which is used to give rewards based on the level of accuracy when the accuracy constraint is met; This represents a preset penalty weight coefficient, used to impose severe penalties when accuracy constraints are not met; , , For respectively , , The set weight value.

[0027] Considering the limited resources at edge nodes, the amount of allocated computing resources directly affects the quantization accuracy or branch execution depth of model inference. This invention defines the prediction inference accuracy... With resource input The exponential correlation model between them is shown in Equation 3: (Formula 3) in, Represents a computing node m The baseline accuracy of the base LLM deployed on it; The marginal revenue coefficient is set to characterize the nonlinear characteristic that the accuracy gains gradually saturate as computing power investment increases. This model is used to quantify the expected quality when performing tasks on different nodes with different resource amounts.

[0028] In a better implementation, while ensuring the minimum inference accuracy requirement of the task, the scheduling scheme of the task is output based on the Markov decision process; that is, instead of using traditional unconstrained reinforcement learning, the optimal scheduling scheme, i.e. the optimal unloading strategy, is solved based on the constrained policy optimization algorithm in the Markov decision process, so that the inference accuracy is a hard constraint that must be met (in the scheduling scheme). Therefore, when constructing a task offloading model based on Markov decision processes, an auxiliary network is introduced: in addition to the traditional main value network based on Markov decision processes used to evaluate cumulative rewards, an independent Lyapunov auxiliary network is constructed in parallel. This auxiliary network shares the state with the main value network (policy network). However, it possesses independent weight parameters, and its function is to estimate the expected risk of the current policy violating inference accuracy requirements in future steps. During the training phase of policy updates, the output value of this auxiliary network is... By directly substituting into the optimization objective function (as shown in Equation 4), the policy update direction is guided by calculating the constraint gradient, thereby forcing the updated policy parameters to be restricted to the feasible region that satisfies the accuracy constraint. In other words, this invention introduces a Lyapunov auxiliary network to estimate the expected value of the policy satisfying the accuracy constraint in the current state, as shown in Equation 4, to construct an optimization objective function that includes an adaptive penalty term. ,Will As the final reward of the task offloading model based on Markov decision process; thus, under the premise of ensuring the minimum inference accuracy requirement of the task, the scheduling scheme of the inference task can be output based on Markov decision process. (Formula 4) in, Represents a period of time T The mean, The adaptive penalty coefficient is set. This represents the minimum inference accuracy threshold set by the user (or specified in the Service Level Agreement (SLA)). This indicates that the Lyapunov auxiliary network for the task offloading model based on Markov decision processes is adjusted according to the current state. The output is the expected estimate of the accuracy constraint satisfaction. When the policy generated by the task offloading model (or agent) based on the Markov decision process violates the accuracy constraint, Automatically increasing the policy update step size forces the policy update direction to shift towards the feasible region that satisfies the constraints. The advantage function is calculated using generalized advantage estimation, and a Kullback-Leibler divergence constraint is applied when updating the policy network to limit the difference between the old and new policies, preventing excessively large policy update steps from causing system performance collapse. Finally, the optimal task scheduling scheme that satisfies the SLA constraints is output.

[0029] Step S104: According to the scheduling scheme, schedule the inference task to the corresponding computing node; In this step, based on the action space For each inference task's output decision vector, the inference task is offloaded to the corresponding computing node; for example, some inference tasks are offloaded to the cloud server, and some are offloaded to the edge node.

[0030] In the technical solution of this invention, a large language model (LLM) is loaded into the real-time monitoring network. N The system analyzes the running status of each computing node and extracts multi-dimensional resource attributes for each node to construct a resource feature vector. The computing nodes include cloud servers and edge devices. Based on the resource feature vector of each computing node, the resource dissimilarity of each computing node is calculated. Based on the resource dissimilarity of each computing node, real-time resource availability, and metadata of the inference task to be processed, a scheduling scheme for the inference task is output using a Markov decision process. According to the scheduling scheme, the inference task is scheduled to the corresponding computing node.

[0031] Because this invention is based on a multi-dimensional resource awareness mechanism with adaptive Markov metric, it can accurately assess the real-time load status of cloud and heterogeneous edge nodes, and implement a fine-grained task offloading strategy based on Markov decision process. It can achieve fine-grained scheduling that comprehensively considers node load, network conditions and LLM inference characteristics, and schedule an appropriate amount of inference tasks to be executed on edge devices, avoiding the network bandwidth bottleneck caused by uploading all data to the cloud. Thus, when processing high-concurrency IoT tasks, it significantly reduces the average response time and communication overhead of the system. Furthermore, this invention effectively addresses the problem that traditional reinforcement learning algorithms often sacrifice inference accuracy when pursuing low latency by introducing a constrained policy optimization algorithm and a Lyapunov constrained network. By incorporating inference accuracy as a hard constraint into the optimization objective, this invention ensures that the inference accuracy of large language models always meets the requirements of the service level agreement on resource-constrained edge nodes, while maximizing the utilization efficiency of computational resources.

[0032] Figure 2 This illustration schematically shows a hardware architecture diagram of a computer device 1300 for a large language model inference task scheduling method according to an embodiment of this application. In this embodiment, the computer device 1300 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. For example, it may be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 2 As shown, the computer device 1300 includes, but is not limited to, at least: a memory 1310, a processor 1320, and a network interface 1330 that can communicate with each other via a system bus. Wherein: The memory 1310 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 1310 may be an internal storage module of the computer device 1300, such as the hard disk or memory of the computer device 1300. In other embodiments, the memory 1310 may also be an external storage device of the computer device 1300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 1310 may also include both the internal storage module and the external storage device of the computer device 1300. In this embodiment, the memory 1310 is typically used to store the operating system and various application software installed on the computer device 1300, such as program code for a large language model inference task scheduling method. In addition, the memory 1310 can also be used to temporarily store various types of data that have been output or will be output.

[0033] In some embodiments, processor 1320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 1320 is typically used to control the overall operation of computer device 1300, such as performing control and processing related to data interaction or communication with computer device 1300. In this embodiment, processor 1320 is used to run program code stored in memory 1310 or process data.

[0034] Network interface 1330 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 1300 and other computer devices. For example, network interface 1330 is used to connect computer device 1300 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 1300 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0035] It should be pointed out that, Figure 2 Only computer devices with components 1310-1330 are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0036] In this embodiment, the reasoning task scheduling method for a large language model stored in memory 1310 can be further divided into one or more program modules and executed by one or more processors (processor 1320 in this embodiment) to complete the embodiment of this application.

[0037] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0038] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in the details for the sake of brevity.

[0039] Additionally, to simplify the description and discussion, and to avoid obscuring the invention, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the invention, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the invention will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) are set forth to describe exemplary embodiments of the invention, it will be apparent to those skilled in the art that the invention may be practiced without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0040] Although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0041] The embodiments of this invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for scheduling inference tasks in a large language model, characterized in that, include: Real-time monitoring network N The system monitors the running status of each computing node and extracts multi-dimensional resource attributes for each computing node to construct a resource feature vector; wherein, the computing nodes include cloud servers and edge devices; Calculate the resource dissimilarity of each computing node based on the resource feature vector of each computing node; Based on the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference tasks to be processed, a scheduling scheme for the inference tasks is output based on the Markov decision process. According to the scheduling scheme, the inference task will be scheduled to the corresponding computing node.

2. The method according to claim 1, characterized in that, The step of calculating the resource dissimilarity of each computing node based on the resource feature vector of each computing node is as follows: Calculate the first according to the following formula 1. i Resource disparity among computing nodes : (Official 1); in, For the first i Resource feature vectors of each computing node For the first i The resource mean vector within the neighborhood of each computing node For the first i The local covariance matrix of resources in the neighborhood of each computing node; T represents the transpose operation, and -1 represents the matrix inversion; The resource attributes include: computing power of computing units, remaining video memory capacity, network bandwidth fluctuation rate, and configuration parameters of the currently loaded large language model (LLM).

3. The method according to claim 1, characterized in that, The step involves outputting a scheduling scheme for the inference task based on the resource differences and real-time resource availability of each computing node, as well as the metadata of the inference task to be processed, using a Markov decision process. Specifically, this includes: Based on the input Based on Markov decision processes, the output maximizes the reward. The scheduling scheme for the reasoning task is obtained; and the parameters of the Markov decision process are adjusted accordingly based on the reward function. in, The state space of the Markov decision process includes: , , ; in, Indicates time t The multidimensional resource knowledge base is calculated based on the resource differences of each computing node; This indicates the task queue status, which records the metadata of the inference tasks to be processed. This indicates the real-time resource availability of each computing node; in, The action space of the Markov decision process includes: the decision vector output for each inference task; and the action space for the inference task. k The output decision vector includes: ∈{0,1} and ;like =1 indicates that the reasoning task will be completed. k Scheduled to compute node m Execute; otherwise =0; Represents a computing node m For reasoning tasks k The specific amount of computing power allocated; The reward function is constructed based on the following penalty and reward terms; the penalty terms include: a delay penalty term. Resource efficiency penalty items The rewards include: Reasoning Quality Rewards. ; in, The sum of the task transmission delay, computation delay, and propagation delay for each inference task is negatively evaluated to penalize high-latency decisions. The sum of the resource disparities of the computing nodes scheduled for each inference task is negative; ； in, Indicates the reasoning task k Scheduled to compute node m The accuracy of predictive inference; This represents the preset positive reward weight coefficient, which is used to give rewards based on the level of accuracy when the accuracy constraint is met; This represents a preset penalty weight coefficient, used to impose severe penalties when accuracy constraints are not met; Indicates the first k Minimum reasoning accuracy requirements for each reasoning task; , , For respectively , , The set weight value.

4. The method according to claim 3, characterized in that, Specifically, it is calculated according to the following formula 2: (Official 2); in, The set attenuation coefficient, For a moment t -1 Multidimensional Resource Knowledge Base.

5. The method according to claim 3, characterized in that, The metadata of the inference task includes: the amount of input data for the inference task, the estimated computational complexity of the inference task, and the minimum inference accuracy requirement for the inference task.

6. The method according to claim 3, characterized in that, The following formula 3 is used for calculation: (Official 3); in, Represents a computing node m The baseline accuracy of the base LLM deployed on it; The marginal revenue coefficient is set. For reasoning tasks k The estimated computational complexity.

7. The method according to claim 3, characterized in that, The scheduling scheme for outputting inference tasks based on Markov decision processes is as follows: While ensuring the minimum inference accuracy requirement of the task, a scheduling scheme for the inference task is output based on the Markov decision process.

8. The method according to claim 7, characterized in that, Under the premise of ensuring the minimum inference accuracy requirement of the task, a scheduling scheme for the inference task is output based on the Markov decision process, specifically including: The optimization objective function constructed according to the following formula 4 is calculated as follows: As the final reward of the task offloading model based on Markov decision process; (Official 4); in, Represents a period of time T The mean, The adaptive penalty coefficient is set. This represents the minimum inference accuracy threshold set. This indicates that the auxiliary network of the task offloading model based on Markov decision processes is adjusted according to the current state. The output is the expected estimate of the accuracy constraint satisfaction.

9. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the reasoning task scheduling method for a large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the inference task scheduling method for a large language model as described in any one of claims 1 to 8.

Images