A token consumption-based computing power scheduling method

Abstract

The application discloses a computing power scheduling method based on Token consumption and belongs to the computer field. A large model inference request is received, which is parsed into an input Token sequence and the number of input Tokens is counted; a preset threshold is dynamically adjusted according to the available computing power resources of a local edge computing node and the number of queued requests, and the request is distributed to the local edge computing node or a cloud cluster computing node according to the number of input Tokens and the adjusted preset threshold; an output Token sequence is generated by calling or loading an inference model by a target node; the number of input and output Tokens and corresponding resource consumption are counted and summary records are generated. The application can also avoid repeated inference through cache key matching and forward complex tasks to a cloud large-parameter model, thereby improving the computing power utilization rate and inference service efficiency.

Description

Technical Field

[0001] This invention discloses a computing power scheduling method based on token consumption, belonging to the field of computing power scheduling and large model inference services in computer technology. Background Technology

[0002] In recent years, with the rapid development of large-scale language models, natural language inference services have been widely deployed in various terminal applications. Users submit large model inference requests through mobile devices, personal computers, or IoT terminals, and computing nodes on the cloud or edge load the inference model and generate a response. Traditional inference service scheduling methods typically perform load balancing based on the number of requests or the simple string length of the requests, lacking fine-grained measurement of the actual computational consumption of each request. Because the computing power, memory, and power consumed by inference models to process input sequences of different lengths vary greatly, traditional uniform or random allocation methods can easily cause local edge nodes to become overloaded due to processing long sequences, while cloud nodes are idle or experience reverse overflow, resulting in a decrease in overall inference throughput and an increase in user waiting time.

[0003] While existing technologies offer some scheduling schemes for inference requests, such as prioritizing requests based on the request source IP or user level, or using fixed input length thresholds for traffic distribution, these schemes fail to adequately consider the repetitiveness of request content. A large number of users repeatedly submitting large model inference requests with identical content or preprocessed input token sequences, such as common question-answering, translation, or summarizing tasks, results in the complete inference process being executed repeatedly each time, leading to a waste of computing resources and power. Furthermore, existing solutions lack a collaborative mechanism between local lightweight models and large cloud models when handling complex inference tasks, often simply uploading all requests to the cloud, increasing network transmission costs and cloud computing pressure. Simultaneously, the power consumption and heat dissipation issues of edge computing nodes are particularly prominent during peak periods, and traditional scheduling strategies lack joint optimization of grid load, equipment temperature, and queue management.

[0004] On the other hand, some aggregation models employ single pricing or coarse-grained aggregation based on the number of requests, failing to distinguish the processing cost differences between different tokens in the input and generation stages, and also failing to transparently aggregate based on the actual number of tokens consumed. For novel architectures such as hybrid expert models, some existing implementations employ a method where many experts reside on GPU memory, which may lead to high GPU memory consumption and insufficient resource utilization. When local inference is interrupted due to timeout, some existing solutions lack an effective state transition mechanism, and users may need to re-initiate a complete request, resulting in duplicate computation. In some serverless implementations, instances may still retain model GPU memory resources when idle, further increasing cloud resource consumption. Therefore, a fine-grained computing power scheduling method based on token consumption is needed to solve the above problems. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution: A method for scheduling computing power based on token consumption includes the following steps: S1, Receive a large model inference request sent by the user terminal, parse the large model inference request into an input token sequence, and count the number of input tokens in the input token sequence; S2, Generate a cache key based on the input Token sequence, inference model identifier, model version, system prompt template, decoding parameters and context identifier, query whether there is a cached output result in the local cache that matches the cache key, and if there is, directly return the cached output result and terminate; S3, if it does not exist, the preset threshold is dynamically adjusted according to the current available computing power resources and the current number of queued requests of the local edge computing node, and the large model inference request is allocated to the local edge computing node or the cloud cluster computing node according to the comparison result between the number of input tokens and the adjusted preset threshold. S4, the allocated computing node calls the loaded inference model or loads the inference model, inputs the input token sequence into the inference model, and generates the output token sequence; S5, count the number of output tokens in the output token sequence, and generate a summary record based on the number of input tokens and the number of output tokens.

[0006] Furthermore, in S4, if the allocated computing node is a local edge computing node, then before performing inference, the complex task features in the input Token sequence of the large model inference request are extracted and the complexity score is calculated. When the complexity score is greater than the preset complexity threshold, the large model inference request is forwarded to the cloud cluster computing node and processed by the large parameter inference model deployed in the cloud cluster computing node. The parameter scale of the large parameter inference model is greater than the parameter scale of the inference model in the local edge computing node. If the complexity score is less than or equal to the preset complexity threshold, the local edge computing node will continue to call the loaded inference model or load the inference model for processing. The complexity score is obtained by weighting at least one of the following features: multi-step logical reasoning features, symbolic operation features, long-range dependency features, and large-scale knowledge retrieval features.

[0007] Further, S3 includes: Obtain a preset threshold that is dynamically adjusted based on the real-time load status of the local edge computing node, and compare the number of input tokens with the adjusted preset threshold. When the number of input tokens is less than or equal to the preset threshold, the large model inference request will be allocated to a local edge computing node. When the number of input tokens exceeds the preset threshold, the large model inference request will be allocated to a cloud cluster computing node. The local edge computing nodes are equipped with lightweight inference models, while the cloud cluster computing nodes are equipped with large-scale distributed inference model clusters.

[0008] Further, in step S4, if the allocated computing node is a local edge computing node, then the local edge computing node executes a power scheduling strategy, which includes: The current power grid load rate is obtained according to a preset sampling period, and the load level corresponding to the current sampling period is determined according to the load rate range to which the power grid load rate belongs. Set an upper limit for power consumption within a sliding time window for each load level. The upper limit for power consumption of different load levels decreases as the load rate increases. Monitor the power consumption of the local edge computing node within a preset sliding time window. When the power consumption exceeds the upper limit of power consumption corresponding to the current load level, perform frequency reduction and voltage reduction operations on the processor, graphics processor or neural network accelerator of the local edge computing node, and place newly received large model inference requests that have not yet been allocated into a waiting queue, suspending the processing of requests in the waiting queue. The processor, graphics processor or neural network accelerator core temperature is monitored in real time. When the core temperature exceeds a preset temperature threshold, a start signal is sent to the liquid cooling device. The liquid cooling device adjusts the coolant flow rate according to the difference between the core temperature and the temperature threshold. For requests already placed in the waiting queue, their positions in the waiting queue are assigned according to each request's priority; When the power consumption of the local edge computing node within the sliding time window is lower than the recovery threshold, and the core temperature is lower than the temperature threshold, and the preset recovery time is continuously met, the processing of requests in the waiting queue is resumed, and requests are retrieved in sequence for inference. The recovery threshold is lower than the upper limit of power consumption corresponding to the current load level.

[0009] Furthermore, in S4, before inputting the input token sequence into the inference model, the maximum context length supported by the inference model is obtained, and the available input length is determined by subtracting the system prompts, necessary control tokens, and the length reserved for the output token from the maximum context length. Compare the length of the input token sequence with the available input length; When the length of the input token sequence is less than or equal to the available input length, the input token sequence is directly input; When the length of the input token sequence is greater than the available input length, a context adaptation process is performed on the input token sequence, the context adaptation process including any of the following: The first part of the starting position and the second part of the ending position in the input token sequence are retained. The remaining tokens in the middle position are subjected to digest compression or key information extraction. The first part, the digest compression result or key information extraction result and the second part are concatenated to form a new input token sequence, wherein the length of the new input token sequence is not greater than the available input length; or, the input token sequence is split into multiple subsequences, the length of each subsequence is not greater than the available input length minus the length reserved for connecting tokens, connecting tokens for indicating context are inserted between adjacent subsequences, and multiple subsequences are input into the inference model in sequence, one subsequence at a time and the output is generated by accumulation.

[0010] Furthermore, in S4, if the allocated computing node is a cloud cluster computing node, and the inference model deployed in the cloud cluster computing node is a hybrid expert model, then a dynamic expert loading and unloading process is executed when loading the hybrid expert model. The hybrid expert model comprises a gating network and multiple expert subnetworks, each expert subnetwork corresponding to a specific semantic pattern or reasoning type; the dynamic expert loading and unloading process includes: The semantic features of the input token sequence are analyzed, and the relevance score between each expert subnetwork and the current input token sequence is calculated through a gating network. Select the expert subnetworks with the highest relevance scores as the set of activated experts; directly call the activated expert subnetworks that are already residing in the video memory hot expert cache, and asynchronously load the activated expert subnetworks that are not residing in video memory from the CPU memory cache or persistent storage into the video memory of the graphics processor. When the available video memory space is lower than a preset threshold, at least one expert subnetwork is selected from the resident expert subnetworks that are not included in the active expert set, are not referenced by the request being processed, and are not marked as a hot expert for unloading, thereby releasing the corresponding video memory space; when the video memory pressure exceeds the emergency threshold, it is allowed to downgrade the hot expert and then unload it. When processing each token in the input token sequence, forward computation is performed first by the currently loaded set of active experts in video memory, and a downgrade or wait strategy is executed when an expert is missing. When the current input token sequence is processed and the next large model inference request is received, the semantic features of the next request are recalculated and the activated expert set is updated, and the above loading and unloading process is repeated.

[0011] Furthermore, S4 also includes an inference timeout migration mechanism and a Serverless instance management mechanism; The inference timeout migration mechanism is as follows: When the assigned computing node is a local edge computing node, a timer is started to record the cumulative processing time since the start of processing the large model inference request. When the cumulative processing time exceeds a preset timeout threshold, a query is performed to see if there is a compatible computing node in the cloud cluster computing nodes. The compatible computing node runs the same model architecture, parameter version, word segmenter, numerical precision, and parallel segmentation method as the local edge computing node, has the same key-value cache layout, and has available resources to process the large model inference request. When the compatible computing node exists, the current inference process is paused and the intermediate state of the inference model at the current moment is captured. The intermediate state includes the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state. When the compatible computing node does not exist, the inference continues to be performed by the local edge computing node, or the generated output token sequence prefix and the original input token sequence are sent to the cloud cluster computing node to re-execute the inference. After the intermediate state is serialized and encoded, it is transmitted to the designated receiving service in the compatible computing node through the communication interface; After receiving the intermediate state, the compatible computing node reconstructs the internal state of the inference model based on the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state, and continues to execute the inference process from the generated output token sequence prefix to generate the remaining output token sequence. The Serverless instance management mechanism is as follows: each inference model instance in the cloud cluster computing node runs in the form of a Serverless container. Each Serverless instance is associated with a memory allocator, a model loader, and a persistent storage volume. The persistent storage volume stores the complete parameter file of the inference model. When any Serverless instance does not receive a new inference request within a preset idle time window, and the current number of active requests is zero, the request queue is empty, and there are no unfinished model loading or migration tasks, the instance unloads the inference model and enters a hibernation state. The memory allocator releases all memory resources occupied by the instance and retains the parameter file of the inference model in the persistent storage volume. When a new inference request needs to be processed by the Serverless instance, the instance management controller wakes up the instance, the memory allocator reallocates memory for the instance, and the model loader loads the inference model parameter file in the persistent storage volume into the allocated memory, restoring the instance to normal operation.

[0012] Further, S5 includes: Obtain the pre-configured input token unit resource consumption coefficient and output token unit resource consumption coefficient. The input token unit resource consumption coefficient and the output token unit resource consumption coefficient correspond to the same resource dimension, which is one of processor execution time, energy consumption, or token-related video memory usage. The input resource consumption is calculated by multiplying the number of input tokens by the resource consumption coefficient per unit of input tokens. The output resource consumption is calculated by multiplying the number of output tokens by the resource consumption coefficient per unit of output tokens. The total resource consumption is equal to the sum of the input resource consumption and the output resource consumption. Generate a summary record, which includes the identifier of the large model inference request, the number of input tokens, the number of output tokens, the resource consumption coefficient per unit of input tokens, the resource consumption coefficient per unit of output tokens, the amount of input resources consumed, the amount of output resources consumed, the total resource consumption, and the timestamp. The summarized records are stored in the summary database and sent to the user terminal.

[0013] Furthermore, the allocation of positions in the waiting queue based on the priority of each request, wherein the priority of a request is determined according to at least one of the following factors: The number of input tokens in the request, the smaller the number of input tokens, the higher the priority; The user service level associated with the request; the higher the service level, the higher the priority of the request. The expected processing time of the request; the shorter the expected processing time, the higher the priority. The earlier the request is submitted, the higher its priority weight.

[0014] Furthermore, the preset threshold is dynamically adjusted based on the real-time load status of the local edge computing node, which includes the current available computing power resources and the current number of queued requests of the local edge computing node; the available computing power resource index is determined based on at least one preset resource index among the current available computing power resources, which includes at least one of the number of available CPU cores, available GPU memory, and memory reserves; when multiple preset resource indices are used, each preset resource index is normalized according to a preset weight and combined to obtain the available computing power resource index; When the available computing power resource index is lower than the resource lower limit threshold or the current number of queued requests exceeds the queue upper limit threshold, the value of the preset threshold is reduced. When the available computing power resource index is higher than the resource upper limit threshold and the current number of queued requests is lower than the queue lower limit threshold, the value of the preset threshold is increased; the preset threshold is adjusted between a preset minimum value and a preset maximum value according to a preset step size.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: it dynamically performs edge-to-cloud traffic splitting based on the number of input tokens and the real-time load of local edge computing nodes, reducing the overload of local edge computing nodes; it avoids repeated inference under the same configuration by matching cache keys; it forwards complex tasks to cloud-based models with large number of parameters by using complexity scores; and it improves the traceability of resource usage by separately statistically analyzing the resource consumption corresponding to input tokens and output tokens. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the operation of a token-based computing power scheduling method claimed in an embodiment of the present invention; Figure 2 This is a second flowchart of a computing power scheduling method based on token consumption, as claimed in an embodiment of the present invention. Figure 3 The third flowchart is a method for scheduling computing power based on token consumption, as claimed in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications in the embodiments of this application, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationships and movements between components in a specific orientation as shown in the accompanying drawings. If the specific orientation changes, the directional indications will change accordingly. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] According to the first embodiment of the present invention, referring to Figure 1 This invention seeks protection for a computing power scheduling method based on token consumption, comprising the following steps: S1, Receive a large model inference request sent by the user terminal, parse the large model inference request into an input token sequence, and count the number of input tokens in the input token sequence; S2, Generate a cache key based on the input Token sequence, inference model identifier, model version, system prompt template, decoding parameters and context identifier, query whether there is a cached output result in the local cache that matches the cache key, and if there is, directly return the cached output result and terminate; S3, if it does not exist, the preset threshold is dynamically adjusted according to the current available computing power resources and the current number of queued requests of the local edge computing node, and the large model inference request is allocated to the local edge computing node or the cloud cluster computing node according to the comparison result between the number of input tokens and the adjusted preset threshold. S4, the allocated computing node calls the loaded inference model or loads the inference model, inputs the input token sequence into the inference model, and generates the output token sequence; S5, count the number of output tokens in the output token sequence, and generate a summary record based on the number of input tokens and the number of output tokens.

[0021] In this embodiment, a large model inference request sent by a user terminal is received. This request is a text statement sent by the user via a mobile device, personal computer, or IoT terminal, such as "Please explain the principle of quantum entanglement." The parsing process first segments the entire text statement into tokens according to the vocabulary used by the inference model, i.e., the word segmentation rules used during model training. Each token can be a complete word, a part of a word, or a punctuation mark. For example, "Please explain the principle of quantum entanglement" is segmented into a token sequence of varying lengths. Simultaneously, the number of tokens in this sequence is counted to obtain the number of input tokens.

[0022] The system generates combined data based on the input token sequence, inference model identifier, model version, system prompt template, decoding parameters, and context identifier. A hash value is then calculated on this combined data to generate a cache key. The hash value is calculated using a general hash function, where the combined data is sequentially input into the hash function, outputting a fixed-length hash value. This cache key serves as a digital fingerprint of the request under a specific model and context configuration. Subsequently, the local cache system of the local edge computing node is queried to determine if a cache entry matching this key exists. The local cache is a key-value pair storage structure, where the key is the cache key and the value is the complete output token sequence generated when performing the same token sequence inference previously under the same model, decoding parameters, and context configuration. If a matching cache entry is found, the output token sequence in the cache is directly returned to the user terminal as a response, without executing any further steps, and the scheduling process ends.

[0023] If no matching cached item is found, the allocation phase begins. The system obtains a pre-configured initial threshold and dynamically adjusts it based on the current available computing resources of the local edge computing node and the number of currently queued requests. The number of input tokens is compared with the adjusted threshold, and the system determines whether to allocate the request to a local edge computing node or a cloud cluster computing node based on the comparison result. Local edge computing nodes are computing devices deployed at the network edge near the user, such as servers in a community data center or computing nodes deployed within an enterprise; cloud cluster computing nodes are large-scale server clusters located in remote data centers.

[0024] After allocation, if the allocated computing node already has a corresponding inference model in a loading state, the inference model is directly invoked; if the corresponding inference model has not yet been loaded, the model parameters are loaded from the storage medium into memory or video memory. Then, the input token sequence is used as the model input, and after several layers of neural network computation, the output is generated token by token until a terminator is generated or the maximum length limit is reached, finally forming the output token sequence.

[0025] The system counts the number of output tokens in the output token sequence. The number of output tokens refers to the number of tokens obtained after the model-generated response text is segmented using the same vocabulary. A summary record is generated based on the number of input tokens and the number of output tokens. This summary record records the number of input tokens, the number of output tokens, and their corresponding resource consumption for this inference, and includes a timestamp and request identifier for use by the subsequent resource statistics system.

[0026] Furthermore, in S4, if the allocated computing node is a local edge computing node, then before performing inference, the complex task features in the input Token sequence of the large model inference request are extracted and the complexity score is calculated. When the complexity score is greater than the preset complexity threshold, the large model inference request is forwarded to the cloud cluster computing node and processed by the large parameter inference model deployed in the cloud cluster computing node. The parameter scale of the large parameter inference model is greater than the parameter scale of the inference model in the local edge computing node. If the complexity score is less than or equal to the preset complexity threshold, the local edge computing node will continue to call the loaded inference model or load the inference model for processing. The complexity score is obtained by weighting at least one of the following features: multi-step logical reasoning features, symbolic operation features, long-range dependency features, and large-scale knowledge retrieval features.

[0027] In this embodiment, when the assigned computing node is a local edge computing node, an additional judgment step is added before performing inference. Specifically, this involves first extracting multi-step logical reasoning features, symbolic operation features, long-range dependency features, and large-scale knowledge retrieval features from the input token sequence of the large model inference request, and then calculating the complexity score based on the preset weights of each feature. The words or symbols used to extract these features include, for example, words representing multi-step logical reasoning such as "reasoning out," "therefore," and "if...then"; words representing symbolic operation such as "solving equations," "derivating," and "calculating integrals"; words representing long-range dependency such as "based on the content of the first three paragraphs" and "reviewing previous dialogues"; and words representing large-scale knowledge retrieval such as "listing all" and "comparing historically." These features can be extracted from the input token sequence through string matching, rule recognition, or semantic similarity matching.

[0028] When the complexity score exceeds a preset complexity threshold, the request is deemed unsuitable for processing by the lightweight model on the local edge computing node. In this case, the local edge computing node does not invoke the local lightweight inference model to perform inference. Instead, it forwards the complete content of the large model inference request, including the original input token sequence, to the cloud cluster computing node via the internal network. After receiving the request, the cloud cluster computing node assigns it to a specially deployed large-parameter inference model for processing. The parameter scale of the large-parameter inference model is significantly larger than that of the inference model on the local edge computing node. For example, the local model has billions of parameters, while the cloud model has hundreds of billions of parameters. After the cloud processing is completed, the generated output token sequence is returned to the local edge computing node along the original path, and then the local edge computing node returns it to the user terminal.

[0029] If the complexity score is less than or equal to the preset complexity threshold, the request is considered to be a regular dialogue or simple question-and-answer task that can be handled by the local edge computing node. In this case, the original process continues, with the local edge computing node calling the loaded lightweight inference model or loading a lightweight inference model and inputting the input token sequence into the model to generate the output token sequence. Throughout the process, the user terminal does not need to be aware of whether the request is forwarded to the cloud, but only receives the final output result.

[0030] The complexity score C is obtained by weighted summation of the feature scores from the four dimensions:

[0031] Where: F_logic is the feature score of multi-step logical reasoning, which is calculated by detecting the frequency and density of logical connectors in the input token sequence, such as "because...therefore...", "if...then...", "therefore it can be seen", "inferred", "first...second...finally", etc. F_symbol is the feature score for symbolic operations, obtained by detecting mathematical symbols, such as... , , , , , , The calculations are based on the occurrence of formula markers, such as "solve equations", "find derivatives", and "calculate integrals", as well as program code features, such as function definitions and loop structures. F_long is the long-range dependency feature score, which is calculated by detecting the occurrence of reference resolution requirements, such as "as mentioned above", "according to the above text", "reviewing the previous text", and cross-paragraph citation markers. F_knowledge is a feature score for large-scale knowledge retrieval, calculated by detecting the occurrence of knowledge-related interrogative words such as "list all", "compare historically", "summarize recent years", "statistics show", and multi-entity parallel structures.

[0032] The value range of each feature score is [0, 1]. It is obtained by normalizing the number of matching keywords: F = min(count / max_count, 1.0), where count is the number of matched keywords and max_count is the preset saturation threshold.

[0033] Weighting coefficient , , , It is a preset positive number, and satisfies The initial values ​​of the weights are determined in the following way: Before system deployment, a test sample set containing multi-step reasoning questions, mathematical calculation questions, long document question answering, and knowledge retrieval questions is selected. Each sample is manually labeled with a complexity level of simple, medium, or complex. Using logistic regression or grid search methods, the weight coefficients of each dimension are optimized on the validation set to maximize the correlation between the complexity score and the manually labeled level. Once determined, the weights are fixed in the system configuration file and can be adjusted based on subsequent validation results.

[0034] The initial value of the preset complexity threshold T_complexity is determined by the complexity score distribution of the above test sample set, and the quantile value that can best separate the "simple" and "complex" samples is taken, such as the 75th percentile.

[0035] The threshold supports dynamic adjustment. The adjustment strategy is as follows: The system periodically calculates the proportion of requests processed by local edge computing nodes that are identified as complex tasks and forwarded to the cloud. When the forwarding proportion exceeds a preset upper limit, such as 30%, the threshold T_complexity is appropriately increased to keep more requests processed locally. When the forwarding proportion is lower than a preset lower limit, such as 5%, the threshold is appropriately decreased to forward more complex requests to the cloud's high-parameter model for processing, thus ensuring overall service quality.

[0036] Further, S3 includes: Obtain a preset threshold that is dynamically adjusted based on the real-time load status of the local edge computing node, and compare the number of input tokens with the adjusted preset threshold. When the number of input tokens is less than or equal to the preset threshold, the large model inference request will be allocated to a local edge computing node. When the number of input tokens exceeds the preset threshold, the large model inference request will be allocated to a cloud cluster computing node. The local edge computing nodes are equipped with lightweight inference models, while the cloud cluster computing nodes are equipped with large-scale distributed inference model clusters.

[0037] In this embodiment, the system obtains an initial preset threshold in the form of a positive integer and dynamically adjusts the initial preset threshold based on the real-time load status of the local edge computing node; the number of input tokens is compared with the adjusted preset threshold. If the number of input tokens is less than or equal to the adjusted preset threshold, the large model inference request is allocated to the local edge computing node; if the number of input tokens is greater than the adjusted preset threshold, the request is allocated to the cloud cluster computing node.

[0038] A lightweight inference model is deployed on the local edge computing node. This lightweight inference model has a small number of parameters and fast inference speed, but it is prone to service quality degradation when processing long sequence inputs due to insufficient video memory or excessive computing time. This model is specifically optimized for short text inputs, such as tens to hundreds of tokens, and can respond quickly locally. The cloud cluster computing nodes are deployed with a large-scale distributed inference model cluster; the cluster consists of multiple servers, each running one or more replicas of a large-parameter inference model; when a long input token sequence, such as thousands or even tens of thousands of tokens, is received, the scheduler in the cluster will allocate the request to one of the servers, and the model on that server can make full use of distributed memory and parallel computing capabilities to process the long sequence; the models in the cluster support longer context windows and higher accuracy.

[0039] The allocation process uses the number of input tokens as the main numerical indicator on the requesting side, and dynamically adjusts the preset threshold in combination with the real-time load status of the local edge computing nodes. This allows more short requests to be retained when local resources are sufficient to reduce network latency and cloud computing costs, and the threshold to be lowered when local resources are scarce or there are many queued requests, so as to send more requests to the cloud to avoid overloading local nodes.

[0040] Furthermore, referring to Figure 2 In step S4, if the allocated computing node is a local edge computing node, then the local edge computing node executes a power scheduling strategy, which includes: The current power grid load rate is obtained according to a preset sampling period, and the load level corresponding to the current sampling period is determined according to the load rate range to which the power grid load rate belongs. Set an upper limit for power consumption within a sliding time window for each load level. The upper limit for power consumption of different load levels decreases as the load rate increases. Monitor the power consumption of the local edge computing node within a preset sliding time window. When the power consumption exceeds the upper limit of power consumption corresponding to the current load level, perform frequency reduction and voltage reduction operations on the processor, graphics processor or neural network accelerator of the local edge computing node, and place newly received large model inference requests that have not yet been allocated into a waiting queue, suspending the processing of requests in the waiting queue. The processor, graphics processor or neural network accelerator core temperature is monitored in real time. When the core temperature exceeds a preset temperature threshold, a start signal is sent to the liquid cooling device. The liquid cooling device adjusts the coolant flow rate according to the difference between the core temperature and the temperature threshold. For requests already placed in the waiting queue, their positions in the waiting queue are assigned according to each request's priority; When the power consumption of the local edge computing node within the sliding time window is lower than the recovery threshold, and the core temperature is lower than the temperature threshold, and the preset recovery time is continuously met, the processing of requests in the waiting queue is resumed, and requests are retrieved in sequence for inference. The recovery threshold is lower than the upper limit of power consumption corresponding to the current load level.

[0041] In this embodiment, when the assigned computing node is a local edge computing node, the local edge computing node must execute a complete power dispatching strategy before and during inference; this strategy includes the following detailed steps: The first step is to obtain the current grid load rate according to a preset sampling period. The local edge computing node connects to the grid monitoring system or power management platform to read the ratio of the current total grid load to the rated load in real time, expressed as a percentage. The grid load rate is updated every certain period of time, for example, once per minute.

[0042] The second step is to determine the load level for each sampling period based on the load rate range to which the grid load rate belongs. The classification rules are as follows: a load rate below the first lower limit is defined as a low-valley load level; a load rate between the first lower limit and the second upper limit is defined as a flat-peak load level; and a load rate above the second upper limit is defined as a peak load level. Different regions can set their own specific boundary values. The system sets an upper limit for power consumption within a sliding time window for each load level. The upper limit for power consumption of the low-valley load level is set higher to allow nodes to perform more calculations; the upper limit for power consumption of the peak load level is set lower to respond to the grid's peak shaving needs; and the upper limit for power consumption of the flat-peak load level is between the two. As the load rate increases, the upper limit for power consumption decreases accordingly.

[0043] The third step is to continuously monitor the cumulative power consumption of the local edge computing node within a preset sliding time window. This power consumption is obtained by integrating the power sample values ​​within the sliding time window using the power meter inside the node, and the unit is kilowatt-hours or joules. The system reads the current power at fixed intervals and removes historical sample values ​​earlier than the start time of the sliding time window to update the power consumption.

[0044] The fourth step involves triggering a power throttling operation when the power consumption exceeds the upper limit of the sliding time window corresponding to the current load level. The power throttling operation includes: performing frequency reduction and voltage reduction operations on the processors, graphics processors, or neural network accelerators of the local edge computing nodes. Frequency reduction refers to reducing the operating frequency of the processor, graphics processor, or neural network accelerator, and voltage reduction refers to correspondingly reducing its operating voltage. At the same time, newly received but unassigned large model inference requests are placed into a waiting queue. This queue has priority adjustment capabilities. The system suspends the retrieval of new requests from the waiting queue for inference.

[0045] The fifth step involves real-time monitoring of the core temperature of the processor, graphics processor, or neural network accelerator while simultaneously reducing the frequency and voltage. The CPU's integrated temperature sensor outputs the current temperature value every fixed interval, such as 100 milliseconds. A temperature threshold is set within the system. When the core temperature exceeds this threshold, a start signal is sent to the liquid cooling system. The liquid cooling system includes a water pump, radiator, coolant piping, and cooling fan. The start signal contains an intensity value calculated based on the difference between the core temperature and the temperature threshold: the larger the difference, the stronger the start signal. Upon receiving the signal, the liquid cooling system adjusts the speed of the coolant pump, thereby regulating the coolant flow rate. The temperature difference and flow rate are positively correlated.

[0046] The sixth step involves assigning a position in the waiting queue to each request based on its priority. Generally, higher-priority requests are moved to the front of the queue, while lower-priority requests are moved to the back. The moving operation is accomplished by reordering pointers or adjusting the linked list order, without changing the content of the request itself.

[0047] Step 7: Set a recovery threshold, which is lower than the upper limit of power consumption corresponding to the current load level; continuously monitor two conditions: whether the power consumption within the sliding time window is lower than the recovery threshold, and whether the core temperature is lower than the temperature threshold; when both conditions are met simultaneously and the preset recovery time is continuously maintained, the system resumes processing requests in the waiting queue; the recovery processing method is to take requests from the front of the queue sequentially and perform reasoning according to the normal process; during the recovery processing, the system continues to monitor power consumption and core temperature, and if they exceed the limits again, the above-mentioned frequency reduction and voltage reduction, pause processing and start liquid cooling operations are repeated.

[0048] Furthermore, in S4, before inputting the input token sequence into the inference model, the maximum context length supported by the inference model is obtained, and the available input length is determined by subtracting the system prompts, necessary control tokens, and the length reserved for the output token from the maximum context length. Compare the length of the input token sequence with the available input length; When the length of the input token sequence is less than or equal to the available input length, the input token sequence is directly input; When the length of the input token sequence is greater than the available input length, a context adaptation process is performed on the input token sequence, the context adaptation process including any of the following: The first part of the starting position and the second part of the ending position in the input token sequence are retained. The remaining tokens in the middle position are subjected to digest compression or key information extraction. The first part, the digest compression result or key information extraction result and the second part are concatenated to form a new input token sequence, wherein the length of the new input token sequence is not greater than the available input length; or, the input token sequence is split into multiple subsequences, the length of each subsequence is not greater than the available input length minus the length reserved for connecting tokens, connecting tokens for indicating context are inserted between adjacent subsequences, and multiple subsequences are input into the inference model in sequence, one subsequence at a time and the output is generated by accumulation.

[0049] In this embodiment, during the process of loading the inference model and generating the output token sequence by the allocated computing node, a context length adaptation step is added before the input token sequence is formally sent into the inference model; specifically as follows: The first step is to obtain the maximum context length supported by the inference model. Each inference model will expose its technical specifications when deployed, one of which is the maximum context length. This length represents the total number of tokens that the model can process at one time. Exceeding this length will cause the attention mechanism inside the model to exceed the predefined location encoding range, or cause memory overflow. The available input length is determined by subtracting the system prompts, necessary control tokens, and the length reserved for output tokens from the maximum context length.

[0050] The second step is to compare the length of the input token sequence, i.e. the number of input tokens, with the available input length.

[0051] Third, when the length of the input token sequence is less than or equal to the available input length, no truncation or splitting is required; the complete input token sequence is directly input into the inference model.

[0052] Fourth, when the length of the input token sequence exceeds the available input length, context adaptation processing is performed. The system provides the following two optional context adaptation methods: Method 1: Retain the first part of the starting position and the second part of the ending position in the input token sequence, and perform digest compression or key information extraction on the remaining tokens in the middle position; concatenate the starting part, the digest token sequence or key information text and the ending part in sequence to form a new input token sequence, so that the length of the new input token sequence is not greater than the available input length.

[0053] Method 2: Divide the input token sequence into multiple subsequences. The length of each subsequence is no greater than the available input length minus the length reserved for the connecting token or the state summary of the previous subsequence. When processing subsequent subsequences, insert a connecting token or the state summary of the previous subsequence to indicate the context before the subsequence. Then, input each subsequence into the inference model in sequence and accumulate to generate the output until all subsequences have been processed.

[0054] In this embodiment, when the length of the input token sequence exceeds the available input length, digest compression is performed on the remaining tokens in the middle position. Specifically, this includes the following steps: The first step is to extract the starting part (30% of the available input length) and the ending part (30% of the available input length) from the original input token sequence, and use the tokens in the middle part as the content to be compressed.

[0055] The second step is to construct a summary generation prompt template, which includes the following elements: 1. Compression target description - "Please compress the following text into a summary of no more than N tokens, retaining core facts, key entities and logical relationships"; 2. The middle part of the text to be compressed; 3. Output format constraints - "Only output the summary text, without adding any additional descriptions".

[0056] The third step involves calling a lightweight summarization model, such as a dedicated summarization model with fewer than 1 billion parameters, or reusing the current inference model itself, inputting the prompt words along with the content to be compressed, and generating a summarization token sequence.

[0057] The fourth step is to concatenate the start part, the digest token sequence, and the end part in sequence into a new input token sequence, ensuring that the length of the new input token sequence does not exceed the available input length.

[0058] As an alternative, an extraction-based summarization method can be used: by calculating the positional weight of each sentence in the middle part, the closer the sentence is to the middle position, the lower the weight is, the longer the sentence is penalized, the lower the weight is appropriately applied to sentences with excessive length, and the sentences containing core entity words are given bonus points. Each sentence is comprehensively scored, and sentences are extracted from high to low scores until the total number of tokens of the extracted sentences reaches the preset summary length budget.

[0059] In this embodiment, when implementing the key information extraction scheme, as an alternative to or supplement to summary compression, the key information extraction scheme includes the following steps: The first step is to define the target types of information extraction, which should include at least the following categories: entity names, place names, organization names, product names, numerical values ​​and attributes, time, quantity, specifications, descriptions of relationships between entities, and the core needs or question types of the intended user.

[0060] The second step involves extracting the aforementioned information from the intermediate token sequence using a combination of rule-based and model-based approaches: 1. Rule layer: using regular expressions to match structured information such as date, amount, and model number; 2. Entity recognition layer: calling a lightweight named entity recognition model to label entities and their types in the text; 3. Relation extraction layer: extracting predicate-argument relations between entities based on dependency parsing results.

[0061] The third step is to organize the extracted structured information into a compact text description according to a preset template, for example: "[Entity] Zhang San[ / Entity] submitted an [Intent] product after-sales consultation[ / Intent] to [Entity] Company[ / Entity] in March 2024[ / Time], involving the [Problem] displaying abnormally related to [Attribute] Model X200[ / Attribute]".

[0062] The fourth step is to concatenate the start part, the structured information text, and the end part into a new input token sequence, and ensure that the length of the new input token sequence does not exceed the available input length.

[0063] In this embodiment, when the length of the input token sequence exceeds the available input length, the sliding window method is used to process it segment by segment: The first step is to set the size of the sliding window to 50% of the available input length, and the sliding step size to 50% of the window size, that is, to slide forward half a window each time.

[0064] The second step is to start from the beginning of the input token sequence, take the token subsequence within the first window and input it into the inference model to generate the corresponding output fragment for that window.

[0065] The third step is to slide the window forward by the sliding step length, take the Token subsequence of the next window, and insert a connection prompt before the subsequence, such as "Continue to process the following content after the previous text", and input it into the inference model again to accumulate and generate subsequent outputs; the total length of the window size and the connection prompt does not exceed the available input length.

[0066] Fourth, repeat step three until the entire input token sequence is covered, and then concatenate all the output fragments in order to form a complete output token sequence.

[0067] The sliding window approach does not change the content of the original input sequence; it only adapts to the context length limit by processing in segments.

[0068] In this embodiment, when executing the retrieval enhancement generation scheme, as another context adaptation method, the retrieval enhancement generation scheme includes the following steps: The first step is to divide the input token sequence into multiple paragraphs or chapters according to semantic integrity when the length of the input token sequence exceeds the available input length.

[0069] The second step is to generate semantic vectors for each paragraph and encode them using an embedding model to build a temporary vector index.

[0070] The third step is to calculate the similarity between the core query part of the input token sequence, which is usually the final user question, and the semantic vector of each paragraph, and sort them from high to low similarity.

[0071] The fourth step is to select several paragraphs with the highest similarity from the sorting results, ensuring that the total number of tokens in the selected paragraphs and the core query part does not exceed the available input length; then, concatenate the selected paragraphs and the core query part into a new input token sequence.

[0072] The fifth step is to input the new input token sequence into the inference model to generate the output.

[0073] The above four schemes can be flexibly selected or combined according to the actual scenario: summary compression is preferred for narrative long texts; key information extraction is preferred for information-intensive technical documents; sliding window is preferred for dialogue scenarios; and search-enhanced generation is preferred for knowledge question-and-answer scenarios.

[0074] After performing context adaptation processing, the new input token sequence generated by any of the above schemes will be fed into the inference model for subsequent calculations.

[0075] Furthermore, in S4, if the allocated computing node is a cloud cluster computing node, and the inference model deployed in the cloud cluster computing node is a hybrid expert model, then a dynamic expert loading and unloading process is executed when loading the hybrid expert model. The hybrid expert model comprises a gating network and multiple expert subnetworks, each expert subnetwork corresponding to a specific semantic pattern or reasoning type; refer to Figure 3 The dynamic expert loading and unloading process includes: The semantic features of the input token sequence are analyzed, and the relevance score between each expert subnetwork and the current input token sequence is calculated through a gating network. Select the expert subnetworks with the highest relevance scores as the set of activated experts; directly call the activated expert subnetworks that are already residing in the video memory hot expert cache, and asynchronously load the activated expert subnetworks that are not residing in video memory from the CPU memory cache or persistent storage into the video memory of the graphics processor. When the available video memory space is lower than a preset threshold, at least one expert subnetwork is selected from the resident expert subnetworks that are not included in the active expert set, are not referenced by the request being processed, and are not marked as a hot expert for unloading, thereby releasing the corresponding video memory space; when the video memory pressure exceeds the emergency threshold, it is allowed to downgrade the hot expert and then unload it. When processing each token in the input token sequence, forward computation is performed first by the currently loaded set of active experts in video memory, and a downgrade or wait strategy is executed when an expert is missing. When the current input token sequence is processed and the next large model inference request is received, the semantic features of the next request are recalculated and the activated expert set is updated, and the above loading and unloading process is repeated.

[0076] In this embodiment, when the allocated computing node is a cloud cluster computing node, and the inference model deployed on the cloud cluster computing node is a Mixture of Experts (MoE) model, a dynamic expert loading and unloading process is executed when loading the model. Specific details are as follows: The hybrid expert model consists of a gating network and multiple expert subnetworks; each expert subnetwork is an independent feedforward neural network module that is specifically responsible for processing input tokens of a specific semantic pattern or reasoning type; for example, the first expert is good at processing mathematical formulas, the second expert is good at processing legal texts, and the third expert is good at processing everyday conversations; the gating network is a small neural network whose input is the implicit representation of the current input token sequence and whose output is the relevance score of each expert subnetwork.

[0077] A hot expert pool mechanism is employed, allocating a fixed-capacity hot expert cache within the GPU memory of the cloud cluster computing nodes to store frequently accessed expert sub-network parameters. Management of the hot expert pool includes: The system maintains an access frequency counter for each expert subnetwork. Whenever an expert is selected as an active expert by the gating network, its access count is incremented by 1. Every fixed time window, such as 10 minutes, the access frequencies of all experts are normalized and sorted, and the experts with the highest access frequencies, such as the top N% (top 20%), are marked as "hot experts".

[0078] Subnetwork parameters marked as thermal experts are preferentially retained in video memory during the unloading process and do not participate in the regular LRU / LFU eviction, ensuring that high-frequency experts are always available and reducing the overhead of repeated loading.

[0079] At fixed update intervals, the access frequency of each expert is recalculated, the list of hot experts is updated, newly popular experts are loaded into video memory, and experts that are no longer popular are downgraded to ordinary experts to release the video memory space they occupy.

[0080] In addition, a tiered caching architecture can be adopted: a high-priority thermal expert cache is set up in the GPU memory to store global thermal experts, a medium-priority cache is set up to store dynamically managed local thermal experts, and a low-priority cache is set up in the CPU memory to store cold experts.

[0081] An asynchronous prefetching mechanism is used to load experts into GPU memory before they are actually needed, thereby reducing waiting latency during the inference process. The first step involves the gating network calculating the semantic features of the current input token sequence. It not only selects Top-K experts as the current active expert set but also predicts the candidate expert set that might be needed for the next batch of input token sequences or the next layer of the network. The prediction method is as follows: based on the current output score distribution of the gating network, several experts whose scores rank after Top-K but whose score decay is relatively gradual are selected as pre-selection candidates.

[0082] The second step involves asynchronously loading the parameters of the prefetched candidate experts from persistent storage or CPU memory into the GPU's prefetch buffer. This loading operation is performed in parallel with the forward computation of the current batch, without blocking the main computation process.

[0083] The third step is to load the required experts directly from the buffer to the active expert set when the next batch of requests arrives, achieving zero-wait switching.

[0084] The fourth step is to trigger the on-demand loading process if the prefetch prediction is incorrect, meaning that the expert actually needed for the next batch is not in the prefetch buffer. This will load the expert from persistent storage to video memory and record the prediction error for subsequent optimization of the prefetch strategy.

[0085] Prefetching decisions can also be optimized based on historical routing patterns: the system records the semantic feature vector of each input token sequence and the corresponding expert activation sequence to build a lightweight prediction model such as a multilayer perceptron, which is used to predict the expert requirements of subsequent requests online.

[0086] An LRU / LFU unloading strategy is adopted. For expert subnetworks that are currently loaded but not activated in video memory, an unloading strategy combining LRU (Least Recently Used) and LFU (Least Frequently Used) is used. LRU dimension: Records the timestamp of the last time each expert subnetwork was activated. When it is necessary to free up memory space, the expert with the earliest recent usage time is unloaded first.

[0087] LFU dimension: Records the access frequency of each expert subnetwork within a historical time period. Experts with extremely low access frequency (below a preset frequency threshold) are prioritized for uninstallation, even if they have been recently used.

[0088] Overall Score: For each resident expert, calculate the uninstallation priority score P_evict = α·(1 - recency_score) + β·(1 - frequency_score), where recency_score is calculated based on the normalized value of the last usage time, frequency_score is calculated based on the normalized value of the access frequency, and α and β are preset weighting coefficients. Experts with higher scores are uninstalled with higher priority.

[0089] Protection mechanism: Experts in the hot expert pool are exempt from LRU / LFU offloading assessment unless the memory pressure exceeds the emergency threshold, such as when the available memory is less than 5% of the total memory.

[0090] A degradation strategy is adopted when the expert subnetwork required during inference is not cached in GPU memory and cannot be loaded on demand in a timely manner. One of the following degradation strategies is executed: Strategy 1: Cache Friend Substitution. The system pre-analyzes the functional similarity of each expert subnetwork and maintains a "friend expert" list for each expert, i.e., experts with similar functions that can be substituted for each other. When the target expert is missing, it checks whether its friend expert is already in the memory. If so, it uses the friend expert to substitute for the calculation, trading a small loss of precision for the continuity of inference.

[0091] Strategy 2: Low-precision substitution. For missing experts, if a low-precision quantized version of the expert, such as an 8-bit integer quantized version, exists in persistent storage, the low-precision version will be loaded into video memory first, compared to the original 16-bit floating-point version, thereby reducing loading latency while ensuring inference can proceed.

[0092] Strategy 3: Synchronous Waiting and Timeout Retry. If none of the above alternative strategies are feasible, the current inference request is placed in a waiting state, while an on-demand loading request for the expert is initiated. Set a waiting timeout threshold, such as 500 milliseconds. If loading is completed before the timeout, inference continues; if the timeout occurs, the request is marked as degraded, and an error message is returned or the request is transferred to a backup model.

[0093] Strategy 4: Request Forwarding. When the available expert set in the GPU memory cannot cover the inference requirements of the current input token sequence, and all fallback alternatives are not applicable, the current large model inference request is forwarded from the cloud cluster computing node to another cloud node with higher configuration, such as larger GPU memory capacity, for processing, ensuring that the inference task is eventually completed.

[0094] The dynamic expert loading and unloading process includes the following steps: The first step is to parse the semantic features of the input token sequence. The parsing process is completed by a gating network: each token in the input token sequence is converted into an embedding vector, and then passed through the forward computation of the gating network to obtain a set of score vectors. Each score corresponds to an expert subnetwork, which represents the importance of the expert to the current input token sequence.

[0095] The second step is to select the expert subnetwork with the highest relevance score as the activated expert set; the size of the activated expert set is a preset fixed value, for example, selecting the four experts with the highest scores; even if there are dozens of experts in total, only a few of them are activated.

[0096] The third step involves loading the parameters of each expert subnetwork in the active expert set from persistent storage, such as a distributed file system or solid-state drive, into the GPU's video memory. Before loading, the system checks if the expert's parameters already exist in the video memory; if not, a read operation is initiated. When the available video memory space is below a preset threshold, the system uses an LRU / LFU combined score to prioritize unloading objects from resident experts that are not included in the active expert set, are not referenced by currently processed requests, and are not marked as hot experts. Only when the video memory pressure exceeds an emergency threshold is a hot expert allowed to be downgraded and participate in unloading to free up the required video memory space.

[0097] Fourth, when processing each token in the input token sequence, the model's forward computation is only participated in by the set of activated experts currently loaded in the memory; the gating network independently calculates a routing weight for each token, assigns the weight to each activated expert, and then sums the outputs of each expert according to the weight; experts that are not activated do not participate in any computation.

[0098] Fifth, after processing the entire input token sequence and generating the output token sequence, the system receives the next large model inference request. At this time, the system repeats steps one through four: re-parses the semantic features of the next request, recalculates the relevance score, redetermines the activation expert set, and performs the corresponding loading and unloading operations. This means that each request may bring a different expert set, realizing the dynamic switching of expert resources at the request granularity.

[0099] Furthermore, S4 also includes an inference timeout migration mechanism and a Serverless instance management mechanism; The inference timeout migration mechanism is as follows: When the assigned computing node is a local edge computing node, a timer is started to record the cumulative processing time since the start of processing the large model inference request. When the cumulative processing time exceeds a preset timeout threshold, a query is performed to see if there is a compatible computing node in the cloud cluster computing nodes. The compatible computing node runs the same model architecture, parameter version, word segmenter, numerical precision, and parallel segmentation method as the local edge computing node, has the same key-value cache layout, and has available resources to process the large model inference request. When the compatible computing node exists, the current inference process is paused and the intermediate state of the inference model at the current moment is captured. The intermediate state includes the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state. When the compatible computing node does not exist, the inference continues to be performed by the local edge computing node, or the generated output token sequence prefix and the original input token sequence are sent to the cloud cluster computing node to re-execute the inference. After the intermediate state is serialized and encoded, it is transmitted to the designated receiving service in the compatible computing node through the communication interface; After receiving the intermediate state, the compatible computing node reconstructs the internal state of the inference model based on the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state, and continues to execute the inference process from the generated output token sequence prefix to generate the remaining output token sequence. The Serverless instance management mechanism is as follows: each inference model instance in the cloud cluster computing node runs in the form of a Serverless container. Each Serverless instance is associated with a memory allocator, a model loader, and a persistent storage volume. The persistent storage volume stores the complete parameter file of the inference model. When any Serverless instance does not receive a new inference request within a preset idle time window, and the current number of active requests is zero, the request queue is empty, and there are no unfinished model loading or migration tasks, the instance unloads the inference model and enters a hibernation state. The memory allocator releases all memory resources occupied by the instance and retains the parameter file of the inference model in the persistent storage volume. When a new inference request needs to be processed by the Serverless instance, the instance management controller wakes up the instance, the memory allocator reallocates memory for the instance, and the model loader loads the inference model parameter file in the persistent storage volume into the allocated memory, restoring the instance to normal operation.

[0100] In this embodiment, a specific implementation of the inference timeout migration mechanism is as follows: When the assigned computing node is a local edge computing node, a timer is started when the large model inference request is processed. The timer records the cumulative processing time from the start of processing the request to the current time. The timer continues to count as the inference model generates the output token sequence token by token. The system presets a timeout threshold. When the cumulative processing time exceeds the timeout threshold, a migration decision is triggered.

[0101] Step 1: Check if there are any computing nodes in the cloud cluster that are compatible with the local inference configuration and have available resources. Compatibility criteria include identical model architecture, parameter versions, tokenizers, numerical precision, and parallel segmentation methods, as well as consistent key-value cache layouts. If no compatible computing node exists, inference continues to be performed by the local edge computing node, or the generated output token sequence prefix and the original input token sequence are sent to the cloud cluster computing node for re-execution of inference. If a compatible computing node exists, the currently ongoing inference process is paused, and the inference process on the local edge computing node stops generating subsequent tokens.

[0102] Step 2: When the compatible computing node exists, capture the intermediate state of the inference model at the current moment. The intermediate state contains at least four parts of data: First, the prefix of the currently generated output token sequence; second, the position index corresponding to the current generation position; third, the key-value cache in the attention mechanism, which records the key vector and value vector of the processed tokens; fourth, the decoding state, which includes the termination condition, random number seed, and state information related to the decoding parameters.

[0103] Step 3: Serialize and encode the above intermediate states; the serialization process will encode the output Token sequence prefix, position index, key-value cache and decoding state into a continuous byte stream in a fixed order, and attach metadata such as model identifier, parameter version, numerical precision, parallel splitting method and key-value cache layout.

[0104] Step 4: Transmit the serialized byte stream to the designated receiving service in the compatible computing node via a communication interface such as a TCP / IP connection or message queue; the receiving service is a resident daemon process dedicated to receiving and deserializing intermediate states.

[0105] Step 5: After receiving the serialized data, the compatible computing node performs a deserialization operation to reconstruct the generated output token sequence prefix, current position index, key-value cache and decoding status, and verifies again whether the metadata is consistent with the running configuration of the cloud inference model.

[0106] Step 6: The compatible computing node loads the key-value cache and decoding state into the corresponding inference model, and continues the inference process from the next token position after the generated output token sequence prefix to generate the remaining output token sequence.

[0107] Step 7: Return the complete output token sequence prefix plus the remaining part to the user terminal.

[0108] Specific implementation of the Serverless instance management mechanism: Each inference model instance in the cloud cluster compute nodes runs as a Serverless container. A Serverless container is a lightweight virtualization unit with the ability to start and stop quickly; each Serverless instance is associated with a memory allocator, a model loader, and a persistent storage volume; the persistent storage volume is a distributed storage system that stores the complete parameter file of the inference model.

[0109] When any Serverless instance does not receive a new inference request within a preset idle time window, and the current number of active requests is zero, the request queue is empty, and there are no incomplete model loading or migration tasks, the instance automatically enters a hibernation state. The hibernation state determination is performed by the instance monitor: the monitor records the timestamp of the last request received by each instance and obtains the current number of active requests, the request queue status, and the status of model loading and migration tasks in real time; when the current time minus the timestamp of the last received request exceeds the idle time window, and the current number of active requests is zero, the request queue is empty, and there are no incomplete model loading or migration tasks, a hibernation command is sent to the instance; upon receiving the hibernation command, the model loader unloads the inference model and notifies the memory allocator to release all currently occupied GPU memory resources; the memory allocator performs a memory release operation, returning the GPU memory blocks held by the instance to the global memory pool; the container retains its metadata and network port bindings and enters a hibernation state; the parameter file of the inference model is always retained in the persistent storage volume and will not be lost during hibernation.

[0110] When a new inference request needs to be processed by this Serverless instance, the request carries the identifier of the target instance or the scheduler reselects the instance. The instance management controller receives the request and detects that the instance is in a dormant state. The controller sends a wake-up signal to the instance; the memory allocator reallocates memory for the instance; the model loader loads the parameter file of the inference model from the persistent storage volume into the allocated memory area; after loading is complete, the instance resumes normal operation and begins processing newly received inference requests.

[0111] Further, S5 includes: Obtain the pre-configured input token unit resource consumption coefficient and output token unit resource consumption coefficient. The input token unit resource consumption coefficient and the output token unit resource consumption coefficient correspond to the same resource dimension, which is one of processor execution time, energy consumption, or token-related video memory usage. The input resource consumption is calculated by multiplying the number of input tokens by the resource consumption coefficient per unit of input tokens. The output resource consumption is calculated by multiplying the number of output tokens by the resource consumption coefficient per unit of output tokens. The total resource consumption is equal to the sum of the input resource consumption and the output resource consumption. Generate a summary record, which includes the identifier of the large model inference request, the number of input tokens, the number of output tokens, the resource consumption coefficient per unit of input tokens, the resource consumption coefficient per unit of output tokens, the amount of input resources consumed, the amount of output resources consumed, the total resource consumption, and the timestamp. The summarized records are stored in the summary database and sent to the user terminal.

[0112] In this embodiment, the specific implementation method for generating a summary record based on the number of input tokens and the number of output tokens is as follows: The first step is to obtain the pre-configured input token unit resource consumption coefficient and output token unit resource consumption coefficient. These coefficients are configured or calibrated according to the inference model version and hardware configuration, and correspond to the same resource dimension. The resource dimension is one of processor execution time, energy consumption, or token-related memory usage. For example, the unit of the processor execution time coefficient can be GPU·milliseconds / Token, the unit of the energy consumption coefficient can be Joules / Token, and the unit of the token-related memory usage coefficient can be bytes / Token. The input token unit resource consumption coefficient and the output token unit resource consumption coefficient can be set to different values ​​to reflect the difference in resource consumption between the input processing stage and the autoregressive generation stage. When multiple resource dimensions need to be statistically analyzed, the corresponding input token unit resource consumption coefficient and output token unit resource consumption coefficient are configured for each resource dimension, and the input resource consumption, output resource consumption, and total resource consumption for that resource dimension are calculated respectively.

[0113] The second step is to calculate the input resource consumption. Multiply the number of input tokens by the resource consumption coefficient per unit of input tokens to obtain the resource consumption of the input stage.

[0114] The third step is to calculate the output resource consumption. Multiply the number of output tokens by the resource consumption coefficient per unit of output token to obtain the resource consumption of the output stage.

[0115] The fourth step is to calculate the total resource consumption. The input resource consumption is added to the output resource consumption to obtain the total resource consumption for this inference.

[0116] The fifth step is to generate a complete summary record. This summary record is a structured data block containing the following fields: the identifier of the large model inference request, such as UUID; the number of input tokens; the number of output tokens; the resource consumption coefficient per unit of input tokens; the resource consumption coefficient per unit of output tokens; the amount of input resources consumed; the amount of output resources consumed; the total amount of resources consumed; and the timestamp.

[0117] The sixth step is to store the generated summary record in the summary database. The summary database is a persistent storage system that supports subsequent resource statistics and operation auditing. At the same time, the summary record can be encapsulated into a message and sent to the user terminal through a secure channel between the user terminal and the system to display the resource consumption details of this inference.

[0118] This summary record is used to represent the token consumption and corresponding resource consumption of this inference request, and can be used for resource statistics, capacity planning, and operation auditing.

[0119] Furthermore, the allocation of positions in the waiting queue based on the priority of each request, wherein the priority of a request is determined according to at least one of the following factors: The number of input tokens in the request, the smaller the number of input tokens, the higher the priority; The user service level associated with the request; the higher the service level, the higher the priority of the request. The expected processing time of the request; the shorter the expected processing time, the higher the priority. The earlier the request is submitted, the higher its priority weight.

[0120] In this embodiment, each request is assigned a position in the waiting queue based on its priority, wherein the priority of a request is determined according to at least one of the following factors: The system can calculate a comprehensive priority score by combining multiple factors: Factor 1: Number of Input Tokens in the Request; The system uses the number of input tokens for each request as a basic metric. Requests with fewer input tokens have higher priority. The decision logic is as follows: All requests in the waiting queue are sorted in ascending order of the number of input tokens, and the request with the fewest tokens receives the highest priority score. This is because short requests are processed faster, and prioritizing short requests can quickly reduce the queue length.

[0121] Factor 2: The user service level associated with the request; the system maintains a service level identifier for each user account, such as regular user, paid user, enterprise user, etc. Users with higher service levels have higher request priority. The specific mapping relationship is defined by the service policy; for example, enterprise users have higher priority than paid users, and paid users have higher priority than regular users. When calculating the overall priority, the service level factor is multiplied by a weighting coefficient.

[0122] Factor 3: Expected processing time of the request; the system estimates the time required to complete the request based on the number of input tokens and the average processing speed of the inference model; requests with shorter expected processing times have higher priority. This is because short-duration requests are easier to complete within fragmented time periods, avoiding blocking subsequent requests. The estimation of expected processing time does not consider the specific computational details within the model, but is based solely on the historical average generation speed.

[0123] Factor 4: Request submission time; the system records the timestamp of each request submitted to the local edge computing node. Requests submitted earlier receive higher priority weights; this avoids starvation caused by low-priority requests remaining unprocessed for extended periods; the submission time factor typically influences the final priority in the form of a monotonically increasing function: the longer the waiting time, the greater the contribution of this factor.

[0124] In the actual waiting queue position allocation process, the system comprehensively scores the above factors. For each request, the score for each factor is calculated separately, then multiplied by its respective weight coefficient and summed to obtain a comprehensive priority score; the request with the highest comprehensive score is placed at the front of the queue, i.e., processed first; the request with the lowest comprehensive score is placed at the back of the queue; when a new request is enqueued, the priority scores of all requests in the queue are recalculated and reordered; the queue sorting operation is performed after each request is enqueued or removed for processing.

[0125] Furthermore, the preset threshold is dynamically adjusted based on the real-time load status of the local edge computing node, which includes the current available computing power resources and the current number of queued requests of the local edge computing node; the available computing power resource index is determined based on at least one preset resource index among the current available computing power resources, which includes at least one of the number of available CPU cores, available GPU memory, and memory reserves; when multiple preset resource indices are used, each preset resource index is normalized according to a preset weight and combined to obtain the available computing power resource index; When the available computing power resource index is lower than the resource lower limit threshold or the current number of queued requests exceeds the queue upper limit threshold, the value of the preset threshold is reduced. When the available computing power resource index is higher than the resource upper limit threshold and the current number of queued requests is lower than the queue lower limit threshold, the value of the preset threshold is increased; the preset threshold is adjusted between a preset minimum value and a preset maximum value according to a preset step size.

[0126] In this embodiment, the preset threshold is dynamically adjusted based on the real-time load status of the local edge computing node. The specific adjustment process is as follows: The system continuously monitors the real-time load status of local edge computing nodes; the real-time load status includes currently available computing power resources and the number of currently queued requests. The system selects at least one preset resource indicator from the number of available CPU cores, available GPU memory, and available RAM to determine the available computing power resource indicator; when multiple preset resource indicators are selected, they are normalized according to preset weights and combined to obtain the available computing power resource indicator. The number of currently queued requests refers to the number of requests that have been allocated to the local edge computing node but have not yet started processing.

[0127] The system presets four boundary thresholds: a lower limit threshold and an upper limit threshold corresponding to the available computing power resource index, and an upper limit threshold and a lower limit threshold corresponding to the current number of queued requests.

[0128] The system will lower the preset threshold value when one of the following two conditions occurs: Scenario 1: The available computing power resource index is lower than the resource lower limit threshold; this means that the computing power of the local node is close to saturation and there are few remaining resources.

[0129] Scenario 2: The current number of queued requests exceeds the queuing limit threshold. This means that the local node has a high backlog of pending tasks.

[0130] The operation of lowering the preset threshold is as follows: subtract the preset positive step size from the current threshold, and the result is not lower than the preset minimum value, to obtain a new threshold. After lowering the threshold, more large model inference requests will be judged to be greater than the threshold when comparing the number of input tokens, and thus be allocated to cloud cluster computing nodes, relieving the pressure on local nodes.

[0131] The system increases the preset threshold value when both of the following conditions are met simultaneously: Scenario 1: The available computing power is higher than the resource limit threshold. This means that the local node has a large amount of idle resources.

[0132] Scenario 2: The current number of queued requests is below the lower limit threshold; this means that the task queue of the local node is almost empty.

[0133] The operation of raising the preset threshold is as follows: add a preset positive step size to the current threshold, without exceeding the preset maximum value, to obtain a new threshold. After raising the threshold, more requests will be judged as being less than or equal to the threshold, and thus be allocated to local edge computing nodes to make full use of local idle resources.

[0134] The system re-collects the load status at fixed time intervals and determines whether the adjustment conditions are met. When the corresponding status meets the conditions for a preset number of consecutive times or for a preset duration, a threshold adjustment is performed. The adjusted threshold is used for the allocation decision of subsequent large model inference requests, thereby reducing frequent switching caused by load fluctuations near the boundary.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0137] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A computing power scheduling method based on token consumption, characterized in that, Includes the following steps: S1, Receive a large model inference request sent by the user terminal, parse the large model inference request into an input token sequence, and count the number of input tokens in the input token sequence; S2, Generate a cache key based on the input Token sequence, inference model identifier, model version, system prompt template, decoding parameters and context identifier, query whether there is a cached output result in the local cache that matches the cache key, and if there is, directly return the cached output result and terminate; S3, if it does not exist, the preset threshold is dynamically adjusted according to the current available computing power resources and the current number of queued requests of the local edge computing node, and the large model inference request is allocated to the local edge computing node or the cloud cluster computing node according to the comparison result between the number of input tokens and the adjusted preset threshold. S4, the allocated computing node calls the loaded inference model or loads the inference model, inputs the input token sequence into the inference model, and generates the output token sequence; S5, count the number of output tokens in the output token sequence, and generate a summary record based on the number of input tokens and the number of output tokens.

2. The method according to claim 1, characterized in that, In step S4, if the assigned computing node is a local edge computing node, then before performing inference, the complex task features in the input Token sequence of the large model inference request are extracted and the complexity score is calculated. When the complexity score is greater than the preset complexity threshold, the large model inference request is forwarded to the cloud cluster computing node and processed by the large parameter inference model deployed in the cloud cluster computing node. The parameter scale of the large parameter inference model is greater than the parameter scale of the inference model in the local edge computing node. If the complexity score is less than or equal to the preset complexity threshold, the local edge computing node will continue to call the loaded inference model or load the inference model for processing. The complexity score is obtained by weighting at least one of the following features: multi-step logical reasoning features, symbolic operation features, long-range dependency features, and large-scale knowledge retrieval features.

3. The method according to claim 1, characterized in that, The S3 includes: Obtain a preset threshold that is dynamically adjusted based on the real-time load status of the local edge computing node, and compare the number of input tokens with the adjusted preset threshold. When the number of input tokens is less than or equal to the preset threshold, the large model inference request will be allocated to a local edge computing node. When the number of input tokens exceeds the preset threshold, the large model inference request will be allocated to a cloud cluster computing node. The local edge computing nodes are equipped with lightweight inference models, while the cloud cluster computing nodes are equipped with large-scale distributed inference model clusters.

4. The method according to claim 1, characterized in that, In step S4, if the allocated computing node is a local edge computing node, then the local edge computing node executes a power dispatching strategy, which includes: The current power grid load rate is obtained according to a preset sampling period, and the load level corresponding to the current sampling period is determined according to the load rate range to which the power grid load rate belongs. Set an upper limit for power consumption within a sliding time window for each load level. The upper limit for power consumption of different load levels decreases as the load rate increases. Monitor the power consumption of the local edge computing node within a preset sliding time window. When the power consumption exceeds the upper limit of power consumption corresponding to the current load level, perform frequency reduction and voltage reduction operations on the processor, graphics processor or neural network accelerator of the local edge computing node, and place newly received large model inference requests that have not yet been allocated into a waiting queue, suspending the processing of requests in the waiting queue. The processor, graphics processor or neural network accelerator core temperature is monitored in real time. When the core temperature exceeds a preset temperature threshold, a start signal is sent to the liquid cooling device. The liquid cooling device adjusts the coolant flow rate according to the difference between the core temperature and the temperature threshold. For requests already placed in the waiting queue, their positions in the waiting queue are assigned according to each request's priority; When the power consumption of the local edge computing node within the sliding time window is lower than the recovery threshold, and the core temperature is lower than the temperature threshold, and the preset recovery time is continuously met, the processing of requests in the waiting queue is resumed, and requests are retrieved in sequence for inference. The recovery threshold is lower than the upper limit of power consumption corresponding to the current load level.

5. The method according to claim 1, characterized in that, In step S4, before inputting the input token sequence into the inference model, the maximum context length supported by the inference model is obtained, and the available input length is determined by subtracting the system prompts, necessary control tokens, and the length reserved for the output token from the maximum context length. Compare the length of the input token sequence with the available input length; When the length of the input token sequence is less than or equal to the available input length, the input token sequence is directly input; When the length of the input token sequence is greater than the available input length, a context adaptation process is performed on the input token sequence, the context adaptation process including any of the following: The first part of the starting position and the second part of the ending position in the input token sequence are retained. The remaining tokens in the middle position are subjected to digest compression or key information extraction. The first part, the digest compression result or key information extraction result and the second part are concatenated to form a new input token sequence, wherein the length of the new input token sequence is not greater than the available input length; or, the input token sequence is split into multiple subsequences, the length of each subsequence is not greater than the available input length minus the length reserved for connecting tokens, connecting tokens for indicating context are inserted between adjacent subsequences, and multiple subsequences are input into the inference model in sequence, one subsequence at a time and the output is generated by accumulation.

6. The method according to claim 1, characterized in that, In S4, if the allocated computing node is a cloud cluster computing node and the inference model deployed in the cloud cluster computing node is a hybrid expert model, then a dynamic expert loading and unloading process is executed when loading the hybrid expert model. The hybrid expert model comprises a gating network and multiple expert subnetworks, each expert subnetwork corresponding to a specific semantic pattern or reasoning type. The dynamic expert loading and unloading process includes: The semantic features of the input token sequence are analyzed, and the relevance score between each expert subnetwork and the current input token sequence is calculated through a gating network. Select the expert subnetworks with the highest relevance scores as the set of activated experts; directly call the activated expert subnetworks that are already residing in the video memory hot expert cache, and asynchronously load the activated expert subnetworks that are not residing in video memory from the CPU memory cache or persistent storage into the video memory of the graphics processor. When the available video memory space is lower than a preset threshold, at least one expert subnetwork is selected from the resident expert subnetworks that are not included in the active expert set, are not referenced by the request being processed, and are not marked as a hot expert for unloading, thereby releasing the corresponding video memory space; when the video memory pressure exceeds the emergency threshold, it is allowed to downgrade the hot expert and then unload it. When processing each token in the input token sequence, forward computation is performed first by the currently loaded set of active experts in video memory, and a downgrade or wait strategy is executed when an expert is missing. When the current input token sequence is processed and the next large model inference request is received, the semantic features of the next request are recalculated and the activated expert set is updated, and the above loading and unloading process is repeated.

7. The method according to claim 1, characterized in that, The S4 also includes an inference timeout migration mechanism and a Serverless instance management mechanism; The inference timeout migration mechanism is as follows: When the assigned computing node is a local edge computing node, a timer is started to record the cumulative processing time since the start of processing the large model inference request; When the cumulative processing time exceeds a preset timeout threshold, the system queries whether a compatible computing node exists in the cloud cluster computing nodes. This compatible computing node runs the same model architecture, parameter version, word segmenter, numerical precision, and parallel segmentation method as the local edge computing node, has a consistent key-value cache layout, and possesses available resources to handle the large model inference request. If the compatible computing node exists, the current inference process is paused, and the intermediate state of the inference model at the current moment is captured. This intermediate state includes the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state. If the compatible computing node does not exist, the inference continues to be performed by the local edge computing node, or the generated output token sequence prefix and the original input token sequence are sent to the cloud cluster computing node to re-execute the inference. After the intermediate state is serialized and encoded, it is transmitted to the designated receiving service in the compatible computing node through the communication interface; After receiving the intermediate state, the compatible computing node reconstructs the internal state of the inference model based on the generated output token sequence prefix, the current position index, the key-value cache, and the decoding state, and continues to execute the inference process from the generated output token sequence prefix to generate the remaining output token sequence. The Serverless instance management mechanism is as follows: each inference model instance in the cloud cluster computing node runs in the form of a Serverless container. Each Serverless instance is associated with a memory allocator, a model loader, and a persistent storage volume. The persistent storage volume stores the complete parameter file of the inference model. When any Serverless instance does not receive a new inference request within a preset idle time window, and the current number of active requests is zero, the request queue is empty, and there are no unfinished model loading or migration tasks, the instance unloads the inference model and enters a hibernation state. The memory allocator releases all memory resources occupied by the instance and retains the parameter file of the inference model in the persistent storage volume. When a new inference request needs to be processed by the Serverless instance, the instance management controller wakes up the instance, the memory allocator reallocates memory for the instance, and the model loader loads the inference model parameter file in the persistent storage volume into the allocated memory, restoring the instance to normal operation.

8. The method according to claim 1, characterized in that, S5 includes: Obtain the pre-configured input token unit resource consumption coefficient and output token unit resource consumption coefficient. The input token unit resource consumption coefficient and the output token unit resource consumption coefficient correspond to the same resource dimension, which is one of processor execution time, energy consumption, or token-related video memory usage. The input resource consumption is calculated by multiplying the number of input tokens by the resource consumption coefficient per unit of input tokens. The output resource consumption is calculated by multiplying the number of output tokens by the resource consumption coefficient per unit of output tokens. The total resource consumption is equal to the sum of the input resource consumption and the output resource consumption. Generate a summary record, which includes the identifier of the large model inference request, the number of input tokens, the number of output tokens, the resource consumption coefficient per unit of input tokens, the resource consumption coefficient per unit of output tokens, the amount of input resources consumed, the amount of output resources consumed, the total resource consumption, and the timestamp. The summarized records are stored in the summary database and sent to the user terminal.

9. The method according to claim 4, characterized in that, The position in the waiting queue is assigned based on the priority of each request, wherein the priority of the request is determined according to at least one of the following factors: The number of input tokens in the request, the smaller the number of input tokens, the higher the priority; The user service level associated with the request; the higher the service level, the higher the priority of the request. The expected processing time of the request; the shorter the expected processing time, the higher the priority. The earlier the request is submitted, the higher its priority weight.

10. The method according to claim 1, characterized in that, The preset threshold is dynamically adjusted based on the real-time load status of the local edge computing node, which includes the current available computing power resources and the current number of queued requests of the local edge computing node. Available computing power resource indicators are determined based on at least one preset resource indicator among the currently available computing power resources. The preset resource indicators include at least one of the following: the number of available CPU cores, the available GPU memory, and the remaining memory. When multiple preset resource indicators are used, each preset resource indicator is normalized according to a preset weight and combined to obtain the available computing power resource indicators. When the available computing power resource index is lower than the resource lower limit threshold or the current number of queued requests exceeds the queue upper limit threshold, the value of the preset threshold is reduced. When the available computing power resource index is higher than the resource upper limit threshold and the current number of queued requests is lower than the queue lower limit threshold, the value of the preset threshold is increased. The preset threshold is adjusted between a preset minimum value and a preset maximum value according to a preset step size.

Images