Model pre-filling task dynamic scheduling method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a dynamic scheduling method and device for model pre-filling tasks, equipment and storage medium. The method comprises: obtaining the availability state and queue state of a remote pre-filling node, the feature parameters of a large language model inference request, and the load state of a local decoding node, the feature parameters including the number of prompt tokens and the request priority, and the load state including the cache idle ratio and the computing power utilization rate; analyzing the queue state and the number of prompt tokens to obtain the remote path cost, and analyzing the number of prompt tokens, the cache idle ratio and the computing power utilization rate to obtain the local path cost; determining the execution path of the pre-filling task by comparing the path costs; when the execution path is the remote path, detecting whether the remote pre-filling node is abnormal, and if it is abnormal, switching to the local path and recording the fuse count and cooling time. The method is used to improve the scheduling accuracy of pre-filling tasks, realize fast switching of remote nodes in abnormal state, and ensure the stability of large language model inference services.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to a dynamic scheduling method, apparatus, device, and storage medium for model pre-filling tasks. Background Technology

[0002] With the large-scale deployment and implementation of large language models, inference services need to balance execution efficiency, computing power utilization, and service stability. Pre-filling, as a key pre-step for large language model inference, is responsible for forward propagation of input prompts and generating key-value caches. The reasonable allocation of its execution path directly affects the response speed and resource utilization efficiency of the entire inference service.

[0003] In existing technologies, the scheduling of pre-filling tasks often adopts a single-dimensional decision-making logic. The execution process mainly involves: first, collecting local state information, including partial load data of local decoding nodes or queue length of remote pre-filling nodes; then, making a simple judgment based on preset thresholds, and distributing the pre-filling task to the local decoding node or the remote pre-filling node. Some solutions only consider task characteristics or the state of a single node, without comprehensively analyzing multi-dimensional parameters, and lack dynamic adaptation and anomaly handling mechanisms after path allocation. Once the execution path is determined, execution continues without dynamic adjustment based on real-time changes in the node's running state.

[0004] Existing pre-filled task scheduling schemes lack effective fallback mechanisms for remote execution anomalies, resulting in insufficient scheduling flexibility. This can lead to issues such as unreasonable path allocation and inability to switch to local execution in a timely manner when remote nodes malfunction, ultimately affecting the stability and overall execution efficiency of large language model inference services. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for dynamic scheduling of model pre-filling tasks, in order to improve the accuracy of pre-filling task scheduling and the stability of large language model inference services.

[0006] In a first aspect, embodiments of this application provide a dynamic scheduling method for model pre-filling tasks, including:

[0007] The availability and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node are obtained. The feature parameters include the number of prompt lexical units and the request priority. The load status includes the cache idle ratio and the computing power utilization rate.

[0008] The remote path cost is obtained by analyzing the queue status and the number of prompt words.

[0009] The local path cost is obtained by analyzing the number of prompt words, the proportion of idle cache, and the computing power utilization.

[0010] The remote path cost and the local path cost are compared based on a preset hysteresis threshold and a preset decision stickiness time window, and the execution path of the pre-filled task is determined based on the comparison results.

[0011] When the execution path is a remote path, anomaly detection is performed on the remote pre-filled node. If an anomaly is found in the remote pre-filled node, the execution path is changed to a local path, and the circuit breaker count and cooling time are recorded.

[0012] In one possible implementation, the remote path cost and the local path cost are compared based on a preset hysteresis threshold and a preset decision stickiness time window, and the execution path of the pre-filled task is determined according to the comparison result, including:

[0013] When the cost of the remote path is lower than the difference between the local path cost and the upper limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the remote path.

[0014] When the local path cost is lower than the difference between the remote path cost and the lower limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the local path.

[0015] In one possible implementation, the remote path cost is obtained by analyzing the queue state and the number of prompt lexical terms, including:

[0016] The queue waiting time is calculated by using preset sliding window statistics rules and preset queue algorithms to determine the queue status.

[0017] The network transmission time is obtained by detecting the network status of the remote pre-filled node and the local decoding node through a preset heartbeat detection mechanism and a preset exponential smoothing algorithm.

[0018] Based on a preset historical sample regression model, the pre-filling calculation time is obtained by fitting the number of prompt word elements.

[0019] The remote path cost is obtained by weighted summing of the queue waiting time, network transmission time, pre-filling calculation time, and result notification time.

[0020] In one possible implementation, anomaly detection of the remote pre-filled node includes:

[0021] Obtain the health status, queue waiting time, network round-trip time, and result notification time of the remote pre-filled node;

[0022] The health status is compared with a preset health status threshold to obtain the health status judgment result;

[0023] The queue waiting time is compared with the preset queue time limit to obtain the waiting anomaly judgment result;

[0024] The network round-trip time is compared with the preset network time limit to obtain the round-trip anomaly judgment result;

[0025] The notification time is compared with the preset notification time limit to obtain the notification anomaly judgment result;

[0026] When one or more of the health status judgment result, waiting abnormal judgment result, round-trip abnormal judgment result, and notification abnormal judgment result are abnormal, the remote pre-filled node is determined to be in an abnormal state.

[0027] In one possible implementation, after determining the execution path of the pre-filling task based on the comparison results, the method further includes:

[0028] Obtain end-to-end latency data and system status data corresponding to each pre-filled task scheduling;

[0029] The end-to-end latency data and system status data are analyzed to obtain the parameter update information corresponding to the pre-filled scheduling.

[0030] In one possible implementation, the analysis of the end-to-end delay data and system state data includes:

[0031] The end-to-end delay data is processed based on preset sliding window statistical rules to obtain delay distribution characteristics;

[0032] Based on the time delay distribution characteristics, the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal triggering condition threshold are dynamically adjusted to obtain the parameter adjustment amount;

[0033] Parameter update information is generated based on the parameter adjustment amount.

[0034] Secondly, embodiments of this application provide a dynamic scheduling apparatus for model pre-filling tasks, comprising:

[0035] The acquisition module is used to acquire the availability and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node. The feature parameters include the number of prompt lexical units and the request priority, and the load status includes the cache idle ratio and computing power utilization.

[0036] The analysis module is used to analyze the queue status and the number of prompt words to obtain the remote path cost;

[0037] The analysis module is also used to analyze the number of prompt words, the proportion of cached idle time, and the computing power utilization to obtain the local path cost;

[0038] The comparison module is used to compare the cost of the remote path with the cost of the local path based on a preset hysteresis threshold and a preset decision stickiness time window, and to determine the execution path of the pre-filled task based on the comparison result.

[0039] The detection module is used to perform anomaly detection on the remote pre-filled node when the execution path is a remote path, and to change the execution path to a local path when the remote pre-filled node has an anomaly, while recording the circuit breaker count and cooling time.

[0040] In one possible implementation, the comparison module is specifically used for:

[0041] When the cost of the remote path is lower than the difference between the local path cost and the upper limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the remote path.

[0042] When the local path cost is lower than the difference between the remote path cost and the lower limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the local path.

[0043] In one possible implementation, the analysis module is specifically used for:

[0044] The queue waiting time is calculated by using preset sliding window statistics rules and preset queue algorithms to determine the queue status.

[0045] The network transmission time is obtained by detecting the network status of the remote pre-filled node and the local decoding node through a preset heartbeat detection mechanism and a preset exponential smoothing algorithm.

[0046] Based on a preset historical sample regression model, the pre-filling calculation time is obtained by fitting the number of prompt word elements.

[0047] The remote path cost is obtained by weighted summing of the queue waiting time, network transmission time, pre-filling calculation time, and result notification time.

[0048] In one possible implementation, the detection module is specifically used for:

[0049] Obtain the health status, queue waiting time, network round-trip time, and result notification time of the remote pre-filled node;

[0050] The health status is compared with a preset health status threshold to obtain the health status judgment result;

[0051] The queue waiting time is compared with the preset queue time limit to obtain the waiting anomaly judgment result;

[0052] The network round-trip time is compared with the preset network time limit to obtain the round-trip anomaly judgment result;

[0053] The notification time is compared with the preset notification time limit to obtain the notification anomaly judgment result;

[0054] When one or more of the health status judgment result, waiting abnormal judgment result, round-trip abnormal judgment result, and notification abnormal judgment result are abnormal, the remote pre-filled node is determined to be in an abnormal state.

[0055] In one possible implementation, an update module is further included, which is specifically used for:

[0056] Obtain end-to-end latency data and system status data corresponding to each pre-filled task scheduling;

[0057] The end-to-end latency data and system status data are analyzed to obtain the parameter update information corresponding to the pre-filled scheduling.

[0058] In one possible implementation, the update module is further configured to:

[0059] The end-to-end delay data is processed based on preset sliding window statistical rules to obtain delay distribution characteristics;

[0060] Based on the time delay distribution characteristics, the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal triggering condition threshold are dynamically adjusted to obtain the parameter adjustment amount;

[0061] Parameter update information is generated based on the parameter adjustment amount.

[0062] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0063] The memory stores computer-executed instructions;

[0064] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0066] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0067] The dynamic scheduling method, apparatus, device, and storage medium for model pre-filling tasks provided in this application embodiment obtain the availability and queue status of remote pre-filling nodes, the feature parameters of large language model inference requests, and the load status of local decoding nodes. It calculates the remote path cost and the local path cost, determines the execution path of the pre-filling task based on a preset hysteresis threshold and a preset decision stickiness time window, and performs anomaly detection on the remote pre-filling node when the execution path is the remote path. When an anomaly occurs, it switches to the local path and records the circuit breaker count and cooldown time. This achieves the effects of improving the scheduling accuracy of pre-filling tasks, enabling rapid switching of remote nodes in case of anomalies, and ensuring the stable operation of large language model inference services. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0069] Figure 1 A schematic diagram illustrating an application scenario of the dynamic scheduling method for model pre-filling tasks provided in this application embodiment;

[0070] Figure 2 A flowchart illustrating the dynamic scheduling method for model pre-filling tasks provided in this application embodiment;

[0071] Figure 3 A schematic diagram of the structure of the dynamic scheduling device for model pre-filling tasks provided in the embodiments of this application;

[0072] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0074] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0075] First, let me explain the terms used in this application:

[0076] Model pre-filling task: refers to the pre-task of processing input prompts and generating key-value caches during the reasoning process of a large language model.

[0077] Remote pre-filling node: refers to a computing node deployed in a remote server cluster to perform large language model pre-filling tasks, and has independent computing power and queue processing capabilities.

[0078] Local decoding node: refers to a local computing node deployed in the same domain as the scheduling system, which can undertake the execution of pre-filled tasks and is used to handle tasks when remote nodes are abnormal or local resources are not compatible.

[0079] Availability status refers to the normal operating condition of the hardware, software, and core components of the remote pre-filled node, reflecting whether the node has the basic capability to perform pre-filled tasks.

[0080] Queue status refers to data such as the queue length and task arrival frequency of pre-filled tasks to be executed in remote pre-filled nodes, reflecting the degree of task congestion of the nodes.

[0081] Large language model inference request: refers to a request initiated by the terminal to the server for large language model calculation, which includes input prompt information.

[0082] Feature parameters: These are core metrics extracted from large language model inference requests, used to characterize the task scale and priority, including the number of prompt lexical units and request priority.

[0083] Number of prompt lexical units: refers to the total number of lexical units contained in the input prompt information in the reasoning request of a large language model, which determines the computational scale and time consumption of the pre-filling task.

[0084] Request priority: refers to the priority level assigned to inference requests for large language models. It is used to distinguish the importance of tasks and influence the weight allocation of scheduling decisions.

[0085] Load status refers to the resource usage of the local decoding node, with core components including the cache idle ratio and computing power utilization, reflecting the node's resource carrying capacity.

[0086] Cache free ratio: refers to the proportion of unoccupied cache space in the local decoding node to the total cache space, reflecting the remaining availability of node cache resources.

[0087] Computing utilization rate: refers to the proportion of computing resources occupied by the local decoding node, reflecting the busyness of the node's computing resources.

[0088] Preset hysteresis threshold: This refers to the cost difference threshold preset to avoid frequent path switching. It includes an upper limit and a lower limit and is used to define the trigger threshold for remote or local paths.

[0089] Preset decision stickiness time window: refers to the preset duration threshold for filtering instantaneous fluctuations in cost data. Path decision is triggered only when the cost comparison results meet the threshold and the duration reaches the target.

[0090] Execution path: refers to the final execution node path selected by the pre-fill task, which is divided into remote pre-fill node path and local decoding node path.

[0091] Health status refers to a quantitative indicator that characterizes the operating status of the hardware and core components of the remote pre-filled node, reflecting the node's own operational stability.

[0092] Queue waiting time: refers to the time it takes for a pre-filled task to wait in the queue of a remote pre-filled node for execution.

[0093] Network round-trip time: refers to the round-trip time for data to be transmitted between the remote pre-filled node and the local decoding node.

[0094] Result notification time: refers to the time taken for the remote pre-filling node to notify the scheduling system of the execution result after completing the pre-filling task.

[0095] Circuit breaker count: refers to the cumulative statistical value of the number of times anomalies occur in remote pre-filled nodes, used to monitor the frequency of node anomalies.

[0096] Cooldown time: refers to the set duration for pausing task allocation after a remote pre-filled node is determined to be abnormal, used for node status recovery and abnormal control.

[0097] End-to-end latency data refers to the total time taken for a pre-filled task from scheduling initiation to execution completion, reflecting the overall efficiency of scheduling and execution.

[0098] System status data refers to the collection of data such as the running status of remote pre-filled nodes and the load status of local decoding nodes during the scheduling process.

[0099] Preset sliding window statistical rules: These are processing rules that use a fixed-duration sliding window to collect, update, and statistically analyze data in segments, used to filter out instantaneous data fluctuations.

[0100] Preset queue algorithm: refers to a pre-set algorithm that calculates queue waiting time based on queue state data. It relies on queuing theory logic to achieve quantitative estimation of waiting time, for example, using Little's law.

[0101] Pre-set heartbeat detection mechanism: refers to a mechanism that periodically sends detection signals to remote pre-filled nodes and receives feedback in order to sense the network status between nodes.

[0102] Preset exponential smoothing algorithm: refers to an algorithm that weights historical and real-time network data to reduce instantaneous fluctuations and stably estimate network transmission time.

[0103] Preset historical sample regression model: This refers to a regression model built based on the number of historical prompt words and pre-filled calculation time samples, used to predict the calculation time of the current task, such as a univariate linear regression model.

[0104] In existing technologies, pre-filling task path allocation is performed by collecting node status or request characteristics from a single dimension. This lacks multi-dimensional comprehensive analysis and remote node anomaly response mechanisms, resulting in technical problems such as insufficient accuracy in pre-filling task scheduling and inability to switch execution paths in a timely manner when remote nodes malfunction.

[0105] The dynamic scheduling method for model pre-filling tasks provided in this application solves the technical problems of insufficient scheduling accuracy of pre-filling tasks and inability to switch execution paths in a timely manner when remote nodes are abnormal. This is achieved by obtaining the availability and queue status of remote pre-filling nodes, the feature parameters of large language model inference requests, and the load status of local decoding nodes. The method also includes obtaining the availability and queue status of remote pre-filling nodes, the feature parameters of large language model inference requests, and the load status of local decoding nodes.

[0106] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0107] Figure 1 This is a schematic diagram illustrating an application scenario of the dynamic scheduling method for model pre-filling tasks provided in the embodiments of this application, such as... Figure 1 As shown, it includes: terminal 101 and server 102.

[0108] Terminal 101 is used to send a large language model inference request to the server. The inference request includes the number of prompt lexical units and the request priority. Server 102 is used to receive the large language model inference request sent by the terminal, obtain the availability and queue status of the remote pre-filling node and the load status of the local decoding node, execute the dynamic scheduling method of the model pre-filling task, determine the execution path of the pre-filling task, switch the execution path when the remote pre-filling node is abnormal and record the circuit breaker count and cooldown time, generate the pre-filling task execution result and feed it back to the terminal.

[0109] Figure 2 This is a flowchart illustrating the dynamic scheduling method for model pre-filling tasks provided in this embodiment. The execution entity in this embodiment can be... Figure 1 The server 102 in the illustrated embodiment can also be other computer-related devices, and this embodiment is not particularly limited.

[0110] like Figure 2 As shown, the dynamic scheduling method for pre-filling tasks in this model includes the following steps:

[0111] Step S201: Obtain the availability and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node. The feature parameters include the number of prompt lexical units and the request priority. The load status includes the cache idle ratio and computing power utilization.

[0112] Specifically, based on the distributed system status acquisition mechanism, the synchronous acquisition of node status and request characteristics is achieved through a multi-dimensional data acquisition interface. By integrating the remote node running status, core request characteristics, and local node resource status, multi-dimensional data input is formed to ensure that the data dimensions cover node resources, request characteristics, and running status, thereby improving the accuracy of subsequent calculations and decisions and avoiding scheduling deviations caused by missing data.

[0113] Step S202: Analyze the queue status and the number of prompt words to obtain the remote path cost.

[0114] Specifically, by quantifying the time consumption during remote path execution, and achieving a numerical representation of cost through weighted summation of multiple time dimensions, the calculation of each time dimension relies on statistical methods and regression models to ensure accuracy. The resource consumption and time consumption of remote pre-filling nodes executing pre-filling tasks are transformed into quantitative cost indicators, providing a unified quantitative standard for comparing the costs of remote and local paths. This achieves accurate quantification of remote path execution costs, clarifies the time and resource consumption of remote path execution, provides a quantitative basis for comparing execution paths, and improves the rationality of path selection.

[0115] Step S203: Analyze the number of prompt words, the proportion of cached free space, and the computing power utilization to obtain the local path cost.

[0116] Specifically, the basic computing power requirement for the pre-filling task is determined based on the number of prompt words, the computing power utilization rate of the local decoding node is combined to fit the pre-filling calculation time, and the cache resource adaptability is judged by the cache idle ratio. The local path cost is obtained by comprehensively considering the calculation time and resource adaptability.

[0117] Based on the correlation between local resources and time consumption, the computing power utilization rate is directly bound to the pre-filling calculation time, quantifying the impact of computing power load on execution time. At the same time, combined with cache adaptability, the increase in calculation time caused by the computing power load of local decoding nodes is quantified, improving the constituent elements of local path cost and maintaining consistency with the multi-dimensional calculation logic of remote path cost. This accurately reflects the impact of local computing power status on pre-filling execution efficiency, ensuring that the comparison dimensions of local path cost and remote path cost are unified, and improving the rationality of path decision-making.

[0118] Step S204: Compare the cost of the remote path with the cost of the local path based on the preset hysteresis threshold and the preset decision stickiness time window, and determine the execution path of the pre-filled task based on the comparison results.

[0119] Specifically, based on hysteresis control and time window constraints, threshold limits and continuous duration monitoring are used to avoid frequent path switching caused by instantaneous fluctuations in cost data, ensuring the stability of execution path decisions. The system compares the costs of remote and local paths, and by combining hysteresis thresholds and decision stickiness time window constraints, it selects the stable and cost-optimal execution path, improving the rationality and stability of scheduling decisions.

[0120] Step S205: When the execution path is a remote path, perform anomaly detection on the remote pre-filled node. If there is an anomaly in the remote pre-filled node, change the execution path to the local path and record the circuit breaker count and cooldown time.

[0121] Specifically, based on multi-dimensional anomaly detection, the system compares node performance metrics with preset thresholds to determine abnormal states. A path switching mechanism dynamically adjusts the execution path, and circuit breaker counts and cooldown time records are used to monitor and manage abnormal states. When the execution path is a remote path, multi-dimensional anomaly detection is performed on the remote pre-filled nodes. If a node is determined to be abnormal and the circuit breaker count reaches a preset threshold, the execution path is switched to a local path. Simultaneously, the circuit breaker count and dynamically calculated cooldown time are recorded, and task allocation to that node is suspended during the cooldown period.

[0122] This system effectively enables real-time monitoring and anomaly response of remote pre-fill nodes, quickly switching execution paths when nodes malfunction to ensure continuous execution of pre-fill tasks. It also quantifies abnormal states through circuit breaker counting and cooldown time recording. Timely identification of abnormal states in remote pre-fill nodes allows for rapid switching of execution paths, preventing pre-fill task interruptions due to node anomalies. Furthermore, circuit breaker counting and cooldown time recording provide data support for subsequent node state recovery and scheduling adjustments.

[0123] The dynamic scheduling method for model pre-filling tasks provided in this invention achieves accuracy and stability in pre-filling task scheduling through multi-dimensional data collection, path cost quantification calculation, hysteresis constraint path decision-making, and dynamic switching of remote nodes in case of anomalies, thereby improving the execution efficiency and resource utilization of large language model inference services.

[0124] This embodiment details the process described in the above embodiment of comparing the cost of the remote path with the cost of the local path based on a preset hysteresis threshold and a preset decision stickiness time window, and determining the execution path of the pre-filled task based on the comparison results. The specific implementation of this process includes the following steps:

[0125] Step a1: When the cost of the remote path is lower than the difference between the cost of the local path and the upper limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the remote path.

[0126] Specifically, by calculating the difference between the local path cost and the upper limit of the preset hysteresis threshold, the remote path cost is compared with the difference in real time. The duration for which the remote path cost is lower than the difference is monitored synchronously. When the duration reaches or exceeds the preset decision stickiness time window, the execution path determination operation is triggered, and the execution path of the pre-filled task is locked as the remote path.

[0127] By setting the upper limit of the preset hysteresis threshold to define the trigger threshold for remote paths, and filtering out instantaneous fluctuations in cost data through a decision stickiness time window, the system avoids path misjudgment caused by short-term data fluctuations, ensuring decision stability, clarifying the trigger conditions for remote paths, and selecting scenarios where remote pre-filled nodes have a greater cost advantage and more stable status for task execution. This enables precise triggering and stable locking of remote paths, avoids frequent path switching caused by instantaneous cost fluctuations, fully leverages the resource advantages of remote pre-filled nodes, improves the efficiency and stability of pre-filled task execution, and meets the need for efficient utilization of remote node resources.

[0128] Step a2: When the local path cost is lower than the difference between the remote path cost and the lower limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the local path.

[0129] Specifically, by calculating the difference between the remote path cost and the lower limit of the preset hysteresis threshold, the local path cost is compared with the difference in real time. The duration for which the local path cost is lower than the difference is monitored synchronously. When the duration reaches or exceeds the preset decision stickiness time window, the execution path determination operation is triggered, and the execution path of the pre-filled task is locked as the local path.

[0130] By setting a lower limit for the preset hysteresis threshold to define the trigger threshold for local paths, and combining this with a decision stickiness time window to filter out instantaneous interference, the selection of local paths is ensured to be based on stable cost comparison results, rather than short-term fluctuations. By clearly defining the trigger conditions for local paths, scenarios where local decoding nodes offer greater cost advantages and stable status are selected. Simultaneously, local resource adaptability and utilization efficiency are ensured, achieving precise triggering and stable locking of local paths. This avoids resource consumption caused by frequent path switching, fully adapts to the resource status of local decoding nodes, and guarantees the rationality and efficiency of pre-filling tasks being executed locally.

[0131] This invention, by setting a graded hysteresis threshold and a decision stickiness time window, clarifies the triggering conditions for remote and local paths and filters out instantaneous cost fluctuations, thereby achieving stable and accurate decision-making for pre-filled task execution paths, taking into account the efficient utilization of resources at both ends of the node, and avoiding resource consumption and execution delays caused by frequent path switching.

[0132] This embodiment provides a detailed explanation of the process in the above embodiment for analyzing the queue status and the number of prompt words to obtain the remote path cost. The specific implementation of this process includes the following steps:

[0133] Step b1: Calculate the queue waiting time by using preset sliding window statistics rules and preset queue algorithm to determine the queue status.

[0134] Specifically, by setting a fixed time window as the statistical range, the system uses preset sliding window statistical rules to collect and update queue status data in segments in real time. Combined with a preset queue algorithm, it calculates data such as queue length and task arrival frequency within the window, outputting the queue waiting time. Based on the dynamic data statistical logic of the sliding window, it filters out instantaneous fluctuations in queue status and quantifies the correlation between queue congestion and task waiting time through the preset queue algorithm, achieving accurate estimation of waiting time. It also quantifies task delays caused by congestion in remote pre-filled node queues, providing queue-level time loss data for remote path cost calculation, reducing the interference of instantaneous queue fluctuations on waiting time estimation, and improving the accuracy of queue waiting time data.

[0135] Step b2: The network transmission time is obtained by detecting the network status of the remote pre-filled node and the local decoding node through a preset heartbeat detection mechanism and a preset exponential smoothing algorithm.

[0136] Specifically, a pre-set heartbeat detection mechanism periodically sends probe signals to remote pre-filled nodes and receives feedback, collecting network data such as signal round-trip time. A pre-set exponential smoothing algorithm is used to denoise the collected network data, eliminating the impact of sudden network fluctuations, and outputting the network transmission time. By quantifying the time loss of pre-filled task data transmission between remote and local nodes, network dimension data of remote path cost is supplemented, reducing the interference of sudden network fluctuations on transmission time estimation, improving the stability and accuracy of network transmission time, and ensuring the quantification accuracy of the network loss dimension in remote path cost.

[0137] Step b3: Based on the preset historical sample regression model, fit the number of prompt word elements to obtain the pre-filling calculation time.

[0138] Specifically, a preset historical sample regression model is obtained, and the number of prompt words, model architecture parameters, and hardware performance parameters are input. The model is used to fit the relationship between the number of prompt words, model architecture, hardware performance and pre-filling calculation time in the historical samples, and outputs the pre-filling calculation time that matches the features of the current task.

[0139] Based on multi-dimensional feature regression analysis, this study explores the correlation between the number of prompt words, model architecture (number of layers, number of parameters), hardware performance (GPU / TPU computing power), and pre-filling computation time. Multi-feature fitting eliminates prediction bias from a single dimension, improving the accuracy of computation time estimation. By quantifying the pre-filling computation time corresponding to multi-dimensional task and environmental features, it supplements the computational loss dimension data for remote path costs, ensuring that the cost structure closely matches the actual execution scenario. This achieves accurate matching between pre-filling computation time and multi-dimensional features, reducing estimation errors caused by differences in model architecture and hardware, and improving the realism of remote path cost quantification.

[0140] Step b4: The queue waiting time, network transmission time, pre-filling calculation time, and result notification time are weighted and summed to obtain the remote path cost.

[0141] Specifically, based on the degree of impact of each time dimension on the execution efficiency of the remote path, weight coefficients are set for queue waiting time, network transmission time, pre-filling calculation time and result notification time. According to the preset weighted summation formula, the four time data are substituted into the calculation to output the remote path cost.

[0142] By using weighted summation quantitative analysis and highlighting the contribution of each time loss dimension to the total cost of the remote path through weight allocation, the multi-dimensional time loss is integrated and quantified to form a unified cost evaluation index, providing a standardized basis for subsequent comparison with local path costs.

[0143] This invention achieves comprehensive quantification of remote path costs through accurate estimation and weighted integration of multi-dimensional time loss, improving the accuracy and standardization of cost calculation and providing reliable data support for rational decision-making on pre-filled task execution paths.

[0144] This embodiment provides a detailed description of the anomaly detection process for remote pre-filled nodes in the above embodiments. The specific implementation of this process includes the following steps:

[0145] Step c1: Obtain the health status, queue waiting time, network round trip time, and result notification time of the remote pre-filled node.

[0146] Specifically, the system collects CPU usage, memory consumption, and service response time through remote node monitoring interfaces, and calculates health indicators according to preset weighting rules. It also reuses queue waiting time calculations, synchronously collects network round-trip time and task result notification time, and integrates these to obtain four data points: health, queue waiting time, network round-trip time, and result notification time. By acquiring this foundational data system for anomaly detection, the system ensures comprehensiveness of detection dimensions, providing multi-dimensional data input for subsequent anomaly assessment. This covers the node itself, queues, network, and notification processes, avoiding the bias caused by single-dimensional data.

[0147] Step c2: Compare the health status with the preset health status threshold to obtain the health status judgment result.

[0148] Specifically, the calculated health score is compared with a threshold. A health score below the threshold is considered abnormal, while a score above the threshold is considered normal, and the health score result is output. By comparing the values, the system determines whether the health status meets the standards, reflects the working status of the node hardware and core components, identifies anomalies caused by node hardware or core component failures, improves the targeting of anomaly detection, and avoids task execution interruptions due to undetected core node failures.

[0149] Step c3: Compare the queue waiting time with the preset queue time limit to obtain the waiting exception judgment result.

[0150] Specifically, the calculated queue waiting time is compared with the upper limit. If the queue waiting time exceeds the upper limit, it is judged as abnormal; otherwise, it is judged as normal, and the abnormal waiting judgment result is output.

[0151] By comparing the duration, the degree of queue congestion can be reflected, and it can be determined whether the node has the normal task processing capacity. This avoids excessive delays in the execution of pre-filled tasks due to excessive queue congestion, and ensures task execution efficiency.

[0152] Step c4: Compare the network round-trip time with the preset network time limit to obtain the round-trip anomaly judgment result.

[0153] Specifically, the collected network round-trip time is compared with the upper limit. If the network round-trip time exceeds the upper limit, it is judged as abnormal; otherwise, it is judged as normal, and the abnormal round-trip time judgment result is output.

[0154] By comparing the duration, the stability and efficiency of network transmission between remote and local nodes are reflected. This effectively identifies the occurrence of task data interaction delays or interruptions caused by network transmission anomalies, supplements the network status basis for comprehensive anomaly judgment, promptly detects anomalies at the network transmission level, avoids failure of pre-filled task data interaction or excessive delay due to network problems, and ensures data transmission stability.

[0155] Step c5: Compare the result notification time with the preset notification time limit to obtain the notification anomaly judgment result.

[0156] Specifically, the notification time of the collected results is compared with the preset notification time limit. If the notification time exceeds the limit, it is judged as abnormal; otherwise, it is judged as normal, and the abnormal notification judgment result is output.

[0157] By comparing time intervals, the timeliness of task execution result feedback is reflected, indirectly demonstrating the completeness of node task processing. Its function is to identify anomalies in task result feedback, supplementing the comprehensive anomaly judgment with evidence for the task closed-loop process, avoiding delays in scheduling decisions due to untimely result notifications, and ensuring the stability of the task execution closed loop.

[0158] Step c6: When one or more of the following are abnormal: health status judgment result, waiting abnormal judgment result, round trip abnormal judgment result, and notification abnormal judgment result, the remote pre-filled node is determined to be in an abnormal state.

[0159] Specifically, by integrating the anomaly judgment results of each stage through multi-dimensional joint judgment, we can avoid omissions in single-dimensional judgments, ensure the comprehensiveness of anomaly identification, improve the comprehensiveness and accuracy of anomaly judgment, avoid omissions in single-dimensional anomalies, ensure that all kinds of anomalies of remote nodes can be identified in a timely manner, and provide reliable support for task execution path switching.

[0160] The system summarizes the results of various anomaly assessments. When one or more anomalies are present, the circuit breaker count is incremented. When the circuit breaker count reaches a preset threshold, the remote pre-filled node is determined to be in an abnormal state. Simultaneously, a cooldown period is calculated based on the number of anomalies and the node's historical recovery time. This circuit breaker mechanism, based on a count threshold and dynamic cooling, defines the severity of anomalies by using a preset count threshold and dynamically calculates the cooldown period based on the number of anomalies and recovery time. This achieves effective control of abnormal nodes, avoids frequent path switching due to missing thresholds or unclear cooling rules, accurately triggers the circuit breaker mechanism, and prevents repeated task allocation during node anomalies through reasonable cooldown periods, thus improving the stability of pre-filled task execution.

[0161] This invention achieves comprehensive and accurate identification of abnormal states of remote pre-filling nodes through multi-dimensional abnormal data collection, single indicator threshold comparison, and joint judgment of multiple results. This provides a reliable basis for the dynamic switching of the pre-filling task execution path and ensures the continuous and stable execution of the pre-filling task.

[0162] This embodiment provides a detailed description of the process after determining the execution path of the pre-filling task based on the comparison results in the above embodiments. The specific implementation of this process also includes the following steps:

[0163] Step d1: Obtain the end-to-end latency data and system status data corresponding to each pre-filled task scheduling.

[0164] Specifically, after the pre-filling task scheduling is completed, the entire process time from task initiation to execution completion is recorded to obtain end-to-end latency data; the running status of remote pre-filling nodes and the load status of local decoding nodes are collected synchronously during the scheduling process and integrated to form system status data.

[0165] By acquiring the actual execution latency of pre-filled task scheduling and the corresponding system operating status, a correlation between latency data and system status data is established, providing complete data input for parameter update analysis. This enables comprehensive and accurate collection of feedback data, ensuring that the data can truly reflect the scheduling effect and system status, providing reliable data basis for subsequent adaptive parameter adjustment, and supporting continuous optimization of scheduling strategies.

[0166] Step d2 involves analyzing end-to-end latency data and system status data to obtain parameter update information corresponding to the pre-filled scheduling.

[0167] Specifically, the end-to-end latency data is processed using a preset sliding window statistical method to extract latency distribution characteristics; combined with system status data, the correlation between latency fluctuations and scheduling parameters is analyzed, and the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal trigger condition threshold are dynamically adjusted to generate parameter update information for synchronously updating the calculation parameters of subsequent scheduling.

[0168] By statistically analyzing and uncovering patterns in latency and system status data, a mapping relationship between parameter adjustment and scheduling effectiveness is established. This enables dynamic adaptation of scheduling parameters, corrects deviations between initial parameters and actual system status and task characteristics, improves the accuracy of subsequent scheduling decisions, achieves adaptive updates of scheduling parameters, allows parameters to adapt to dynamic changes in system status and task characteristics, continuously optimizes the accuracy of path decision-making and anomaly detection, and enhances the overall efficiency of pre-filled task scheduling.

[0169] This invention collects scheduling feedback data, performs correlation analysis, and dynamically updates scheduling parameters to construct an adaptive optimization closed loop for pre-filled task scheduling. This continuously improves the accuracy and adaptability of scheduling decisions and ensures the stable and efficient operation of large language model inference services.

[0170] This embodiment provides a detailed description of the process for analyzing end-to-end latency data and system status data in the above embodiments. The specific implementation of this process includes the following steps:

[0171] Step e1: Process the end-to-end delay data based on the preset sliding window statistical rules to obtain the delay distribution characteristics.

[0172] Specifically, by setting a fixed-duration sliding statistical window, the end-to-end latency data is segmented and updated in real time using preset sliding window statistical rules. The latency data within the window is denoised and statistically analyzed to extract key information such as the latency mean, fluctuation range, and extreme values, and integrated to form latency distribution characteristics.

[0173] By extracting features from end-to-end latency data, the raw latency data is transformed into distribution features with reference value, providing an analytical basis for the dynamic adjustment of subsequent scheduling parameters, reducing the interference of instantaneous latency fluctuations on feature analysis, improving the accuracy and representativeness of latency distribution features, ensuring that subsequent parameter adjustments have reliable data analysis support, and avoiding blind adjustments.

[0174] Step e2: Based on the time delay distribution characteristics, dynamically adjust the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal triggering condition threshold to obtain the parameter adjustment amount.

[0175] Specifically, a correlation mapping between latency distribution characteristics and scheduling parameters is established. The adaptation relationship between features such as average latency and fluctuation range and preset hysteresis threshold, cost calculation weighting coefficient, and abnormal trigger condition threshold is analyzed. The parameter adjustment range is quantified according to the feature deviation, and targeted dynamic corrections are made to various parameters to calculate the corresponding adjustment amount for each parameter.

[0176] Through data-driven adaptive optimization, and by reflecting the scheduling effect and system adaptability based on latency distribution characteristics, deviation analysis determines the direction and magnitude of parameter adjustments. This enables dynamic matching of scheduling parameters with the actual system state and task execution effect, corrects the adaptation deviation between initial scheduling parameters and actual operating scenarios, optimizes core scheduling parameters such as hysteresis threshold and weighting coefficient, and improves the accuracy of subsequent path decision-making, cost calculation, and anomaly detection. It achieves precise adaptation adjustment of scheduling parameters, ensuring that parameters conform to the dynamic changes in system state and task latency characteristics, reducing scheduling deviations caused by unreasonable parameters, and improving the overall efficiency of pre-filled task scheduling.

[0177] Step e3: Generate parameter update information based on parameter adjustment amount.

[0178] Specifically, by summarizing the adjustment amounts corresponding to each scheduling parameter, the object of parameter update, the adjustment range, and the effective rules are clarified. Parameter adjustment information is integrated according to a standardized format to generate parameter update information that includes the parameter name, the adjusted value, and the effective time. This ensures that the update information is clear and standardized, and can be directly used for subsequent scheduling parameter synchronization. This effectively ensures the standardization and completeness of parameter update information, avoids omissions or errors during the parameter update process, and guarantees that scheduling parameters are accurately synchronized according to adjustment requirements.

[0179] This invention constructs an adaptive optimization link for scheduling parameters by extracting sliding window features from end-to-end latency data, dynamically adjusting scheduling parameters based on these features, and generating standardized update information. This continuously improves parameter adaptability and scheduling decision accuracy, ensuring stable and efficient execution of pre-filled tasks.

[0180] Figure 3 This is a schematic diagram of the structure of a dynamic scheduling device for model pre-filling tasks provided in an embodiment of this application. Figure 3 As shown, the dynamic scheduling device 30 for pre-filling tasks in this model includes:

[0181] The acquisition module 301 is used to acquire the availability status and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node. The feature parameters include the number of prompt words and the request priority, and the load status includes the cache idle ratio and computing power utilization.

[0182] Analysis module 302 is used to analyze the queue status and the number of prompt words to obtain the remote path cost;

[0183] Analysis module 302 is also used to analyze the number of prompt words, cache idle ratio and computing power utilization to obtain local path cost;

[0184] The comparison module 303 is used to compare the cost of the remote path with the cost of the local path based on a preset hysteresis threshold and a preset decision stickiness time window, and to determine the execution path of the pre-filled task based on the comparison results.

[0185] The detection module 304 is used to perform anomaly detection on the remote pre-filled node when the execution path is a remote path. When an anomaly is found in the remote pre-filled node, the execution path is changed to the local path, and the circuit breaker count and cooling time are recorded.

[0186] In one possible implementation, the comparison module 303 is specifically used for:

[0187] When the cost of the remote path is lower than the difference between the local path cost and the upper limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the remote path.

[0188] When the local path cost is lower than the difference between the remote path cost and the lower limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the local path.

[0189] In one possible implementation, the analysis module 302 is specifically used for:

[0190] The queue waiting time is calculated by using preset sliding window statistics rules and preset queue algorithms to determine the queue status.

[0191] The network transmission time is obtained by detecting the network status of the remote pre-filled node and the local decoding node through a preset heartbeat detection mechanism and a preset exponential smoothing algorithm.

[0192] Based on a preset historical sample regression model, the pre-filling calculation time is obtained by fitting the number of prompt word elements;

[0193] The remote path cost is obtained by weighting and summing the queue waiting time, network transmission time, pre-filling calculation time, and result notification time.

[0194] In one possible implementation, the detection module 304 is specifically used for:

[0195] Get the health status, queue waiting time, network round-trip time, and result notification time of the remote pre-populated node;

[0196] The health status is compared with a preset health status threshold to obtain the health status judgment result;

[0197] The queue waiting time is compared with the preset queue time limit to obtain the waiting exception judgment result;

[0198] The network round-trip time is compared with the preset network time limit to obtain the round-trip anomaly judgment result;

[0199] The notification time is compared with the preset notification time limit to obtain the notification anomaly judgment result;

[0200] When one or more of the following are abnormal: health status judgment result, waiting abnormal judgment result, round trip abnormal judgment result, and notification abnormal judgment result, the remote pre-filled node is determined to be in an abnormal state.

[0201] In one possible implementation, an update module 305 is also included, which is specifically used for:

[0202] Obtain end-to-end latency data and system status data corresponding to each pre-filled task scheduling;

[0203] By analyzing end-to-end latency data and system status data, parameter update information corresponding to pre-filled scheduling is obtained.

[0204] In one possible implementation, the update module 305 is further configured to:

[0205] The end-to-end latency data is processed based on preset sliding window statistical rules to obtain latency distribution characteristics;

[0206] Based on the time delay distribution characteristics, the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal triggering condition threshold are dynamically adjusted to obtain the parameter adjustment amount;

[0207] Parameter update information is generated based on the parameter adjustment amount.

[0208] The dynamic scheduling device for model pre-filling tasks provided in this embodiment can be used to execute the dynamic scheduling method for model pre-filling tasks described above. Its implementation principle and technical effect are similar, and will not be repeated here.

[0209] Figure 4 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device 40 includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 also includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0210] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above method.

[0211] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0212] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0213] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0214] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0215] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0216] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0217] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0218] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0219] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0221] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0222] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0224] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A dynamic scheduling method for model pre-filling tasks, characterized in that, include: The availability and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node are obtained. The feature parameters include the number of prompt lexical units and the request priority. The load status includes the cache idle ratio and the computing power utilization rate. The remote path cost is obtained by analyzing the queue status and the number of prompt words. The local path cost is obtained by analyzing the number of prompt words, the proportion of idle cache, and the computing power utilization. The remote path cost and the local path cost are compared based on a preset hysteresis threshold and a preset decision stickiness time window, and the execution path of the pre-filled task is determined based on the comparison results. When the execution path is a remote path, anomaly detection is performed on the remote pre-filled node. If an anomaly is found in the remote pre-filled node, the execution path is changed to a local path, and the circuit breaker count and cooling time are recorded.

2. The method according to claim 1, characterized in that, The remote path cost and the local path cost are compared based on a preset hysteresis threshold and a preset decision stickiness time window, and the execution path of the pre-filled task is determined according to the comparison result, including: When the cost of the remote path is lower than the difference between the local path cost and the upper limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the remote path. When the local path cost is lower than the difference between the remote path cost and the lower limit of the preset hysteresis threshold, and the duration is greater than or equal to the preset decision stickiness time window, the execution path of the pre-filled task is determined to be the local path.

3. The method according to claim 1, characterized in that, The remote path cost is obtained by analyzing the queue state and the number of prompt lexical terms, including: The queue waiting time is calculated by using preset sliding window statistics rules and preset queue algorithms to determine the queue status. The network transmission time is obtained by detecting the network status of the remote pre-filled node and the local decoding node through a preset heartbeat detection mechanism and a preset exponential smoothing algorithm. Based on a preset historical sample regression model, the pre-filling calculation time is obtained by fitting the number of prompt word elements. The remote path cost is obtained by weighted summing of the queue waiting time, network transmission time, pre-filling calculation time, and result notification time.

4. The method according to claim 1, characterized in that, Anomaly detection is performed on the remote pre-filled nodes, including: Obtain the health status, queue waiting time, network round-trip time, and result notification time of the remote pre-filled node; The health status is compared with a preset health status threshold to obtain the health status judgment result; The queue waiting time is compared with the preset queue time limit to obtain the waiting anomaly judgment result; The network round-trip time is compared with the preset network time limit to obtain the round-trip anomaly judgment result; The notification time is compared with the preset notification time limit to obtain the notification anomaly judgment result; When one or more of the health status judgment result, waiting abnormal judgment result, round-trip abnormal judgment result, and notification abnormal judgment result are abnormal, the remote pre-filled node is determined to be in an abnormal state.

5. The method according to claim 1, characterized in that, After determining the execution path of the pre-filling task based on the comparison results, the following is also included: Obtain end-to-end latency data and system status data corresponding to each pre-filled task scheduling; The end-to-end latency data and system status data are analyzed to obtain the parameter update information corresponding to the pre-filled scheduling.

6. The method according to claim 5, characterized in that, The analysis of the end-to-end latency data and system status data includes: The end-to-end delay data is processed based on preset sliding window statistical rules to obtain delay distribution characteristics; Based on the time delay distribution characteristics, the preset hysteresis threshold, cost calculation weighting coefficient, and abnormal triggering condition threshold are dynamically adjusted to obtain the parameter adjustment amount; Parameter update information is generated based on the parameter adjustment amount.

7. A dynamic scheduling device for model pre-filling tasks, characterized in that, include: The acquisition module is used to acquire the availability and queue status of the remote pre-filled node, the feature parameters of the large language model inference request, and the load status of the local decoding node. The feature parameters include the number of prompt lexical units and the request priority, and the load status includes the cache idle ratio and computing power utilization. The analysis module is used to analyze the queue status and the number of prompt words to obtain the remote path cost; The analysis module is also used to analyze the number of prompt words, the proportion of cached idle time, and the computing power utilization to obtain the local path cost; The comparison module is used to compare the cost of the remote path with the cost of the local path based on a preset hysteresis threshold and a preset decision stickiness time window, and to determine the execution path of the pre-filled task based on the comparison result. The detection module is used to perform anomaly detection on the remote pre-filled node when the execution path is a remote path, and to change the execution path to a local path when the remote pre-filled node has an anomaly, while recording the circuit breaker count and cooling time.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images