AI inference service hierarchical response scheduling method and system based on electricity price signal

Abstract

The application provides an AI inference service grading response scheduling method and system based on electricity price signals. Different delay tolerance inference requests are divided into grades, and a differentiated scheduling strategy is developed in combination with the peak and valley fluctuations of time-of-use electricity prices. For example, only real-time grade requests sensitive to delay are guaranteed during the electricity price peak period, and offline grade requests that can be delayed are transferred to the electricity price valley for execution. This approach does not indiscriminately affect the response experience of high-value requests and actively reduces the overall electricity bill of the computing center by off-peak, achieving differentiated scheduling and processing of requests with different delay tolerances. As a result, the electricity cost of the computing center is reduced while the service experience of high-value users is guaranteed, achieving a coordinated optimization between electricity cost and service quality.

Description

Technical Field

[0001] This application relates to the field of computing power scheduling technology, and in particular to a hierarchical response scheduling method and system for AI inference services based on electricity price signals. Background Technology

[0002] With the widespread application of large-scale language models and generative artificial intelligence, AI inference services have become a major source of growth in data center power consumption. Globally, the electricity consumption of AI computing centers continues to climb, and electricity costs are accounting for an increasingly larger proportion of total operating costs. At the same time, electricity market reforms have promoted the popularization of time-of-use pricing mechanisms, resulting in significant peak-valley fluctuations in electricity prices across different time periods, which has a direct cost impact on large-scale power-consuming computing centers.

[0003] In existing technologies, AI inference services primarily employ "full-scale speed reduction" or "rate limiting and service rejection" to cope with power shortages or peak electricity prices. Specifically, when electricity prices rise or grid load is too high, the system uniformly reduces computing resource allocation for all inference requests, resulting in a decrease in overall inference speed; or, in extreme cases, directly rejects some requests. Its drawbacks include: First, the full-scale speed reduction strategy indiscriminately affects all users, forcing even latency-sensitive high-value real-time requests to be slowed down, severely degrading user experience and leading to high-value customer churn and reduced service revenue; Second, existing solutions cannot proactively migrate low-priority, latency-tolerant requests to off-peak electricity periods, causing computing centers to bear unnecessary high electricity costs during peak periods, resulting in poor electricity cost control. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the technical deficiency in the prior art where AI inference services cannot differentiate and schedule requests with different latency tolerances in scenarios of fluctuating electricity prices.

[0005] This application provides a hierarchical response scheduling method for AI inference services based on electricity price signals, including:

[0006] Obtain an AI inference service request, and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request;

[0007] Obtain electricity price-related information, including real-time electricity price signals, historical electricity price data prediction results, or user selection instructions based on electricity price information;

[0008] A target scheduling instruction is generated based on the target request level and the electricity price information, and the AI ​​inference service request is executed immediately or delayed based on the target scheduling instruction.

[0009] Optionally, determining the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request includes:

[0010] If the AI ​​inference service request carries an explicit latency tolerance parameter, then the latency tolerance parameter is extracted directly.

[0011] If not carried, the preset type-tolerance mapping table is queried according to the business type field of the AI ​​inference service request to obtain the latency tolerance parameter corresponding to the business type field;

[0012] The latency tolerance parameter is compared with a preset first threshold and a second threshold. AI inference service requests below the first threshold are determined as real-time level, AI inference service requests between the first threshold and the second threshold are determined as near-real-time level, and AI inference service requests above the second threshold are determined as offline level.

[0013] Optionally, when the electricity price-related information is a real-time electricity price signal, the step of generating a target scheduling instruction based on the target request level and the electricity price-related information includes:

[0014] Collect real-time electricity price signals, compare the real-time electricity price signals with preset peak thresholds and off-peak thresholds, and determine the current electricity price status corresponding to the real-time electricity price signals based on the comparison results. The current electricity price status includes peak period, off-peak period, and flat price period.

[0015] The target request level and the current electricity price status are used as a two-dimensional index to query the preset hierarchical response strategy table and generate the target scheduling instruction;

[0016] The tiered response strategy table defines the execution method for each level of request under different electricity price conditions:

[0017] During off-peak hours, all request levels are executed immediately;

[0018] During the flat pricing period, real-time and near real-time requests are executed immediately, while offline requests are partially delayed.

[0019] During peak periods, real-time requests are executed immediately, near-real-time requests are executed to a limited extent based on resource availability, and offline requests are executed with a delay.

[0020] Optionally, the method further includes:

[0021] The current power grid load value is collected, and the current power grid load value is combined with the real-time electricity price signal as the basis for dispatching decisions;

[0022] When the current grid load value exceeds the preset high load threshold but the real-time electricity price signal has not yet reached the peak threshold, some offline level requests are transferred to the delayed execution queue in advance.

[0023] When the current power grid load value is lower than the preset low load threshold, the triggering conditions of the delayed execution queue are relaxed, and the tasks in the delayed execution queue are executed in batches ahead of schedule.

[0024] Optionally, the method further includes:

[0025] Requests for AI inference services that will be delayed will be stored in a delayed execution queue.

[0026] AI inference service requests in the delayed execution queue are executed in batches when one of the following conditions is met:

[0027] The current electricity price is at its lowest level, based on real-time electricity price signals.

[0028] The forecast results are based on historical electricity price data, and the current period is a pre-planned period of low electricity prices;

[0029] The current power grid load value is lower than the preset low load threshold.

[0030] Optionally, the method further includes:

[0031] The waiting time of each task in the delayed execution queue is continuously tracked. When the ratio of the waiting time of any task to its delay tolerance limit reaches the preset timeout protection trigger ratio, the scheduling priority of the task is raised to the highest level and executed immediately.

[0032] Optionally, the method further includes:

[0033] Collect system operation index data, which includes at least the average response time of each level of request, the computing resource utilization rate of each time period, the actual power consumption and corresponding electricity expenditure of each time period, the average waiting time of tasks in the delay queue, the task timeout rate, and the number of requests that are temporarily downgraded due to insufficient resources.

[0034] The system operation index data is compared with the preset target value, and the current scheduling strategy parameters are dynamically adjusted based on the comparison results.

[0035] Optionally, when the electricity price-related information is a user's selection instruction based on electricity price information, the step of generating a target scheduling instruction according to the target request level and the electricity price-related information includes:

[0036] Determine the submission time period corresponding to the selection instruction;

[0037] If the submission period is a low electricity price period, then generate a target scheduling instruction to be executed immediately;

[0038] If the submission period is a period of flat price or a period of high electricity price, then a target scheduling instruction is generated based on the target request level and the submission period.

[0039] Optionally, when the electricity price-related information is a prediction result based on historical electricity price data, the step of generating a target scheduling instruction based on the target request level and the electricity price-related information includes:

[0040] Before operation, scheduling strategies for each time period are pre-planned based on historical electricity price data and electricity price forecast results. The scheduling strategies define the execution methods for different levels of requests within each time period.

[0041] During operation, a target scheduling strategy is determined based on the current time period and the target request level, and a target scheduling instruction is generated based on the target scheduling strategy.

[0042] This application also provides a hierarchical response scheduling system for AI inference services based on electricity price signals, using the hierarchical response scheduling method for AI inference services based on electricity price signals as described in any of the above embodiments. The system includes:

[0043] The request level determination unit is used to obtain AI inference service requests and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request.

[0044] The electricity price information determination unit is used to acquire electricity price-related information, which includes real-time electricity price signals, historical electricity price data prediction results, or user selection instructions based on electricity price information.

[0045] The hierarchical scheduling unit is used to generate a target scheduling instruction based on the target request level and the electricity price-related information, and to execute the AI ​​inference service request immediately or with a delay based on the target scheduling instruction.

[0046] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0047] The AI ​​inference service hierarchical response scheduling method and system based on electricity price signals provided in this application classifies inference requests with different latency tolerances into different levels and formulates differentiated scheduling strategies based on the peak and valley fluctuations of time-of-use electricity prices. For example, during peak electricity price periods, only real-time level requests that are sensitive to latency are guaranteed, while offline level requests that can be delayed are transferred to low electricity price periods for execution. This approach avoids indiscriminately affecting the response experience of high-value requests and can proactively reduce the overall electricity cost of the computing center by staggering peak times. It achieves differentiated scheduling processing for requests with different latency tolerances, thereby reducing the electricity cost of the computing center while ensuring the service experience of high-value users and achieving synergistic optimization between electricity costs and service quality. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a hierarchical response scheduling method for AI inference services based on electricity price signals, provided in an embodiment of this application;

[0050] Figure 2 A flowchart illustrating the request access and request hierarchy provided for embodiments of this application;

[0051] Figure 3 This is a schematic diagram illustrating the process of generating target scheduling instructions provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the electricity price signal acquisition and status analysis workflow provided in an embodiment of this application;

[0053] Figure 5 A schematic diagram of the scheduling process of the timeout protection and dynamic priority promotion mechanism provided in the embodiments of this application;

[0054] Figure 6 A flowchart illustrating the feedback data collection and dynamic strategy adjustment process provided in this application embodiment;

[0055] Figure 7 This is a schematic diagram of the structure of a hierarchical response scheduling system for AI inference services based on electricity price signals, provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In existing technologies, AI inference services primarily employ "full-scale speed reduction" or "rate limiting and service rejection" when dealing with power shortages or peak electricity prices. Specifically, when electricity prices rise or grid load is excessive, the system uniformly reduces computing resource allocation for all inference requests, leading to a decrease in overall inference speed; or, in extreme cases, directly rejects some requests. Its drawbacks include:

[0058] First, existing solutions apply indiscriminate rate reduction or throttling to all inference requests. Specifically, during peak electricity prices or power shortages, latency-sensitive requests such as real-time dialogues and latency-tolerant requests such as batch processing are uniformly subject to the same resource restriction strategy, causing a sharp deterioration in the user experience of high-value real-time requests and directly impacting user retention and service revenue.

[0059] Second, the existing solution lacks a mechanism to use electricity price signals as a scheduling variable. Specifically, the execution time of inference tasks depends entirely on the arrival time of requests. It is impossible to perceive and utilize electricity price fluctuation information to optimize the selection of task execution time periods. This results in a large number of delay-tolerant requests occupying computing resources during peak electricity price periods, leading to high electricity costs for computing centers and low operational efficiency.

[0060] Third, the existing solution directly rejects some requests when power is scarce. Specifically, requests that exceed the current computing power capacity are dropped instead of being delayed, causing user tasks to fail and need to be resubmitted, which reduces the overall system throughput and service completion rate.

[0061] Therefore, in order to solve the problems existing in the prior art, this application proposes a scheme that can dynamically realize hierarchical scheduling of AI inference services based on electricity price signals. This scheme divides AI inference service requests into different levels according to latency tolerance, and then combines the different times of electricity price information to schedule the execution time of each level of tasks in a differentiated manner. It can prioritize the service quality of high-value latency-sensitive requests during peak electricity price periods, while transferring deferred tasks to low electricity price periods for execution, thereby effectively reducing the overall electricity cost of the computing center, while taking into account system service quality and operational efficiency, and solving the technical problem that the prior art cannot balance cost control and service experience.

[0062] Before describing this application, the relevant terms used in this application will be explained, including:

[0063] AI Inference Service: This refers to an online service that, based on a trained artificial intelligence model, receives user input data and performs forward computation to generate predictions or output content. AI inference services are typically deployed in data centers or cloud clusters, consuming significant computing and power resources; their response time and throughput directly impact user experience.

[0064] Electricity Price Signal: This refers to price information released by the electricity market or grid dispatching agency that reflects the current or future supply and demand relationship and cost level of electricity. Electricity price signals typically exhibit peak-valley characteristics as they change over time; higher prices during peak hours reflect tight power supply, while lower prices during off-peak hours reflect ample power supply. It serves as an important reference for load dispatching on the demand side.

[0065] Tiered Response: This refers to classifying inference service requests into multiple service levels based on user request latency tolerance and business priority, and formulating differentiated scheduling strategies and response time commitments for different levels. High-level requests receive priority access to computing resources, while low-level requests can be delayed or scheduled for execution within a specific time period.

[0066] Latency Tolerance: This refers to the acceptable latency level for a user's request. Users' latency tolerance varies significantly across different business scenarios. For example, real-time conversations require millisecond to second-level response times, while batch data processing or offline content generation scenarios can tolerate wait times in the minutes or even hours.

[0067] Grid load signal: refers to real-time monitoring data reflecting the current overall electricity load level of the power system, usually provided by the grid dispatch center or energy management system. Grid load signal is correlated with electricity price signal. When grid load is high, power supply tends to be tight and electricity prices rise; when load is low, power supply is abundant and electricity prices fall. The two can be used together as the basis for power dispatching decisions.

[0068] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides a flowchart illustrating a hierarchical response scheduling method for AI inference services based on electricity price signals. The method may include:

[0069] S110: Obtain the AI ​​inference service request and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request.

[0070] In this step, after the service receiving end obtains the AI ​​inference service request initiated by the user, the corresponding latency tolerance parameter can be extracted or mapped from it. Then, the request level can be classified based on the preset level classification rules. Different levels correspond to different service timeliness priorities, which provides a basis for subsequent differentiated scheduling.

[0071] For example, the pre-defined tier classification rule in this application could divide AI inference service requests into three levels: a real-time level (sensitive to latency and requiring immediate response), a near-real-time level (tolerant of moderate latency), and an offline level (acceptable for long latency). The specific durations of moderate and long latency can be pre-defined based on actual scheduling needs. For instance, this application could classify latency tolerance below 1 second as the real-time level, between 1 second and 30 minutes as the near-real-time level, and above 30 minutes as the offline level. Furthermore, in practical applications, more or fewer tiers can be set according to business needs to adapt to different scheduling scenarios.

[0072] S120: Obtain electricity price information.

[0073] In this step, after determining the target request level of the AI ​​inference service request through S110, this application can obtain the required electricity price information from the corresponding channels according to the system's deployment mode and needs, and then generate target scheduling instructions based on the electricity price information and the target request level.

[0074] For example, computing centers participating in grid demand response and using real-time pricing can directly obtain the published real-time electricity price signals. For scenarios implementing annual time-of-use pricing, they can directly use pre-prepared time-of-use price forecasts based on historical price data, or pre-plan service level configurations and resource allocation schemes for each time period based on price forecasts before daily operation, instead of querying the electricity price status in real time for each request. This effectively reduces the computational overhead of real-time decision-making and simplifies scheduling. If users are allowed to choose their own execution time periods to obtain cost discounts, there is no mandatory tiered scheduling on the system side. Instead, electricity price fluctuation information is transparently transmitted to users, allowing them to choose to submit requests during different price periods and enjoy different service prices. For example, users can proactively schedule non-urgent tasks to be submitted during off-peak hours, and the system will provide full service at a lower price during off-peak hours. The advantages of this solution are that users have complete autonomy and transparent cost information, and the system is simple to implement.

[0075] S130: Generate target scheduling instructions based on the target request level and electricity price information, and execute AI inference service requests immediately or with a delay based on the target scheduling instructions.

[0076] In this step, the priority of the request is determined by the target request level. Combined with the current or predicted electricity price status, the corresponding scheduling instructions can be generated by matching the preset differentiated scheduling rules, and scheduling arrangements can be made to execute the requests of different levels immediately or with a delay.

[0077] For example, when real-time electricity prices are at their peak and the current price level is high, only real-time requests with the lowest latency tolerance are provided with immediate execution. Near-real-time requests are queued and executed based on resource availability, while offline requests are all moved to a delayed execution queue, waiting for electricity prices to drop before being executed in batches. If the current electricity price is at its flat rate, real-time and near-real-time requests can be executed immediately, while offline requests are moved to delayed execution. If the current electricity price is at its low rate, all requests can be executed immediately without delay. This differentiated scheduling allows many deferred tasks that would normally be executed during high-price periods to be moved to low-price periods, effectively reducing overall electricity costs without affecting the response experience of high-priority requests.

[0078] Furthermore, when executing an AI inference service request according to a target scheduling instruction, this application can first determine the scheduling action definition and encapsulate it into a scheduling instruction, which is then output to the scheduling execution unit. This scheduling action definition includes three elements: execution method (immediate execution or delayed execution), resource allocation ratio (the proportion of computing resources allocated when executing immediately), and delay constraint (the maximum waiting time limit when executing delayedly). Upon receiving the scheduling instruction, the scheduling execution unit parses the instruction and executes the corresponding operation. For immediate execution instructions, the scheduling execution unit can request a specified proportion of computing resources from the inference computing resource pool, allocate the request to idle computing nodes for inference computation, and return the result to the request initiator after computation, recording the actual response time and resource consumption data of the request. For delayed execution instructions, the scheduling execution unit forwards the request data, along with its level label, delay tolerance value, timestamp of entry into the delay queue, and delay constraint parameters, to the delay queue management unit.

[0079] Furthermore, in resource contention scenarios, if the available resources in the current inference computing resource pool are insufficient to simultaneously satisfy all requests awaiting immediate execution, the scheduling execution unit can perform resource priority arbitration: first, ensuring the resource requirements of real-time requests are met; second, satisfying near-real-time requests; and finally, considering offline requests. When resources are insufficient to satisfy a request of a certain level, the request can be temporarily deferred and placed in a delay queue.

[0080] In the above embodiments, inference requests with different latency tolerances are classified into levels, and differentiated scheduling strategies are formulated in combination with the peak and valley fluctuations of time-of-use electricity prices. For example, during peak electricity price periods, only real-time level requests that are sensitive to latency are guaranteed, while offline level requests that can be delayed are transferred to low electricity price periods for execution. This not only avoids indiscriminately affecting the response experience of high-value requests, but also proactively reduces the overall electricity cost of the computing center by staggering peak hours. This achieves differentiated scheduling processing for requests with different latency tolerances, thereby reducing the electricity cost of the computing center while ensuring the service experience of high-value users, and achieving synergistic optimization between electricity costs and service quality.

[0081] In one embodiment, such as Figure 2 As shown, Figure 2 The flowchart for request access and request classification provided in this application embodiment; S110, determining the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request, may include:

[0082] S111: If the AI ​​inference service request carries an explicit latency tolerance parameter, then the latency tolerance parameter is extracted directly.

[0083] S112: If not carried, query the preset type-tolerance mapping table according to the business type field of the AI ​​inference service request to obtain the latency tolerance parameter corresponding to the business type field.

[0084] S113: Compare the latency tolerance parameter with a preset first threshold and a second threshold, and determine AI inference service requests that are lower than the first threshold as real-time level, AI inference service requests that are between the first threshold and the second threshold as near-real-time level, and AI inference service requests that are higher than the second threshold as offline level.

[0085] In this embodiment, upon receiving an AI inference service request submitted by a user, the request message is first parsed to extract the business type field and latency tolerance parameter. The business type field identifies the application scenario category to which the request belongs, such as real-time dialogue, batch processing, or offline generation; the latency tolerance parameter represents the maximum acceptable waiting time set by the user or application for this request. If the request does not carry an explicit latency tolerance parameter, a preset type-tolerance mapping table can be queried based on the business type field to automatically fill in the default latency tolerance value.

[0086] After parameter extraction, this application can compare the latency tolerance parameter with a preset grading threshold. This grading threshold may include a first threshold and a second threshold. Requests below the first threshold are classified as real-time, those between the first and second thresholds are classified as near-real-time, and those above the second threshold are classified as offline. The specific values ​​of the grading thresholds can be flexibly configured according to business needs and are not limited here. After grading, the request data will be appended with a grading label and a latency tolerance value, forming a tagged request object for subsequent grading response scheduling.

[0087] In the above embodiments, latency tolerance parameters are obtained through either explicit extraction or implicit mapping. Combined with flexibly configurable thresholds to classify request levels, this approach can adapt to the hierarchical requirements of different business scenarios. It supports both user-defined latency requirements and automatic hierarchical classification based on business type, ensuring the reliability and flexibility of the hierarchical process.

[0088] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the process of generating a target scheduling instruction provided in an embodiment of this application; when the electricity price-related information is a real-time electricity price signal, the generation of the target scheduling instruction in S130 based on the target request level and the electricity price-related information may include:

[0089] S131: Collect real-time electricity price signals, compare the real-time electricity price signals with preset peak thresholds and off-peak thresholds, and determine the current electricity price status corresponding to the real-time electricity price signals based on the comparison results. The current electricity price status includes peak period, off-peak period, and flat price period.

[0090] S132: Using the target request level and the current electricity price status as a dual-dimensional index, query the preset hierarchical response strategy table to generate a target scheduling instruction.

[0091] In this embodiment, the current electricity price level is determined by real-time collection of electricity price signals, and the corresponding scheduling instructions are quickly obtained by matching the preset hierarchical response strategy table. The decision-making process is clear and intuitive, the query efficiency is high, and it can adapt to the dynamic scheduling needs under real-time electricity price fluctuation scenarios. It can make the optimal scheduling decision based on the latest electricity price status when each request is accessed, maximizing the benefits of peak shifting and cost reduction while ensuring service quality.

[0092] In one specific implementation, the electricity price signal acquisition unit of this application acquires the latest electricity price data through a connection with the electricity market data interface at a preset acquisition cycle. The data acquired in each acquisition operation includes the real-time electricity price at the current moment, the predicted electricity price for several future time windows, and the current overall load level of the power grid. The acquired raw data undergoes data cleaning processing to remove outliers and noise interference, and is then stored in an electricity price data cache in a standardized format.

[0093] The electricity price status analysis unit can read the latest data from the electricity price data cache and compare the current real-time electricity value with preset peak and off-peak thresholds. When the real-time electricity value is higher than the peak threshold, the current electricity price status is determined to be in the peak period. The tiered response strategy table executes real-time level requests immediately during the peak period, executes near-real-time level requests to a limited extent based on resource reserves, and delays offline level requests. When the real-time electricity value is lower than the off-peak threshold, the current electricity price status is determined to be in the off-peak period. The tiered response strategy table sets all level requests for the off-peak period to execute immediately. When the real-time electricity value is between the two thresholds, the current electricity price status is determined to be in the parity period. The tiered response strategy table executes real-time and near-real-time level requests immediately during the parity period, and partially delays offline level requests.

[0094] Furthermore, the working logic of the tiered response strategy table in this application follows the following basic principles: the higher the electricity price, the narrower the range of immediately executable tiers, and the greater the possibility of low-tier requests being delayed; under the same electricity price, requests of higher tiers receive a higher proportion of resource allocation. Therefore, the partial delayed execution of offline tier requests in the flat-price tier of the tiered response strategy table can be specifically as follows: based on the remaining available computing resources, determine the number of offline requests that can be executed immediately, and transfer the remaining offline requests to the delayed queue for execution; the limited execution of near-real-time tier requests in the peak tier can be specifically as follows: only after ensuring the resource requirements of all real-time tier requests are met, allocate the remaining available resources to near-real-time tier requests, and transfer the remaining near-real-time tier requests to the delayed queue for execution after the resources are exhausted. In this implementation, by setting elastic scheduling rules for the offline tiers in the flat-price tier and the near-real-time tiers in the peak tier, the utilization of idle computing resources can be maximized, unnecessary delayed scheduling can be further reduced, and the overall service response efficiency can be improved while reducing costs.

[0095] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the electricity price signal acquisition and status analysis workflow provided in the embodiments of this application; the method may further include:

[0096] S133: Collect the current power grid load value, and combine the current power grid load value with the real-time electricity price signal as the basis for dispatching decisions.

[0097] S133: When the current grid load value exceeds the preset high load threshold but the real-time electricity price signal has not yet reached the peak threshold, some offline level requests are transferred to the delayed execution queue in advance.

[0098] S135: When the current power grid load value is lower than the preset low load threshold, relax the triggering conditions of the delayed execution queue and execute the tasks in the delayed execution queue in advance in batches.

[0099] In this embodiment, while collecting real-time electricity price signals, this application can also collect current grid load data, i.e., the current grid load value. Combining the grid load status with real-time electricity price information is used as a reference for scheduling decisions. This allows for early initiation of peak-shaving regulation when the electricity price has not yet triggered the peak tier but the grid has already entered a high-load state. This can advance the transfer of some delayed tasks to reduce overall power consumption and alleviate grid pressure. Conversely, when the grid is in a low-load state, even if the electricity price has not yet entered the off-peak tier, idle computing resources can be used in advance to process backlogged delayed tasks in batches, reducing the overall average waiting time of tasks and further optimizing the efficiency of power resource utilization, thus balancing cost reduction needs with service response efficiency.

[0100] For example, if the current grid load has exceeded the preset high load threshold, but the real-time electricity price has not yet risen to the peak threshold, the system can preemptively transfer all currently connected offline requests to the delayed execution queue, preventing immediate execution. This reduces the overall power consumption of the computing center and avoids further price increases and power supply pressure caused by a subsequent rise in grid load. It also provides a buffer for grid load regulation. If the current grid load is already below the preset low load threshold, even if the real-time electricity price is still at parity, the triggering conditions for delayed execution can be relaxed. Tasks backlogged in the delayed queue can be preemptively retrieved and executed in batches, utilizing currently idle computing resources to complete the tasks. This reduces the average waiting time of tasks and makes full use of surplus power resources during low load periods.

[0101] In one embodiment, the method may further include:

[0102] Requests for AI inference services that will be delayed will be stored in a delayed execution queue.

[0103] AI inference service requests in the delayed execution queue are executed in batches when one of the following conditions is met:

[0104] The current electricity price is at its lowest level, based on real-time electricity price signals.

[0105] The forecast results are based on historical electricity price data, and the current period is a pre-planned period of low electricity prices;

[0106] The current power grid load value is lower than the preset low load threshold.

[0107] In this embodiment, after receiving the target scheduling instruction, the system parses the instruction and executes the corresponding operation. For immediately executed instructions, the scheduling execution unit can request a specified proportion of computing resources from the inference computing resource pool, allocate the request to idle computing nodes for inference computing, and return the result to the request initiator after the computing is completed. For deferred execution instructions, the scheduling execution unit can forward the request and its delay constraint information to the delay queue management unit. When the available capacity of the resource pool is insufficient, the scheduling execution unit will prioritize the low-level tasks according to the request level from low to high, reclaiming or pausing resources for currently executing low-level tasks, and allocating the released resources to high-level requests.

[0108] Therefore, in this application, the AI ​​inference service requests that are delayed in execution will be uniformly stored in a delayed execution queue for management. After receiving the delayed requests transferred from the scheduling execution unit, the delayed queue management unit inserts them into the delayed execution queue according to their priority level and entry time. Requests with higher priority levels are placed before requests with lower priority levels in the queue; requests of the same priority level are arranged according to the order in which they entered the queue.

[0109] The delay queue management unit monitors trigger conditions in real time. When it detects a change in electricity price status from peak or flat to off-peak, or when historical electricity price data is used for prediction and the current period falls within a pre-planned low-price period, or when the current grid load is below a preset low-load threshold, tasks are retrieved from the delay execution queue. These tasks are then sorted according to their entry order and the urgency of delay constraints, and computational resources are allocated for batch execution. After all tasks are completed, the inference results are returned to the corresponding request initiator. During batch dequeueing, the delay queue management unit controls the dequeueing rate based on the available capacity of the inference computational resource pool to prevent resource overload caused by a sudden surge of requests. Simultaneously, the delay queue management unit periodically cleans up expired tasks exceeding the maximum delay constraint, returning a task timeout notification to the corresponding request initiator to avoid unnecessary resource consumption.

[0110] This trigger-based batch execution mechanism can concentrate deferred tasks for processing during periods of lower electricity costs or lower grid load, further amplifying the effect of peak shaving and cost reduction. At the same time, batch execution can also improve the utilization efficiency of computing resources and reduce resource losses caused by fragmented scheduling.

[0111] In one embodiment, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the scheduling process for the timeout protection and dynamic priority promotion mechanism provided in the embodiments of this application; the method may further include:

[0112] The waiting time of each task in the delayed execution queue is continuously tracked. When the ratio of the waiting time of any task to its delay tolerance limit reaches the preset timeout protection trigger ratio, the scheduling priority of the task is raised to the highest level and executed immediately.

[0113] In this embodiment, the delay queue management unit performs timeout protection monitoring on each task in the queue. This unit periodically scans the queue and calculates the ratio of each task's elapsed waiting time to its maximum delay tolerance. When the ratio for a task reaches a preset timeout protection trigger ratio, the task is marked as "urgently escalated," and its scheduling priority is raised to the highest level. The delay queue management unit immediately sends the task to the scheduling execution unit for priority processing, ensuring it can be prioritized even if electricity prices have not yet returned to off-peak levels. This mechanism prevents the extreme situation where low-priority requests remain unprocessed for extended periods due to persistently high electricity prices, ensuring all requests are completed within their promised timeframes, thus balancing electricity cost optimization and service completion rate assurance.

[0114] Furthermore, Figure 5 This demonstrates the logical relationship and collaboration among three key technical features. The latency tolerance grading provides the request dimension index input to the tiered response strategy table. This table, combined with electricity price status, generates differentiated scheduling decisions, determining whether to execute immediately or delay. Delayed tasks are monitored by a timeout protection mechanism to ensure execution before the tolerance limit is reached, while under normal circumstances, they are executed in batches after the arrival of off-peak periods. These three features are interconnected, jointly ensuring a balance between cost optimization and service quality in the scheduling scheme.

[0115] In one embodiment, such as Figure 6 As shown, Figure 6 This is a flowchart illustrating the feedback data collection and dynamic strategy adjustment provided in an embodiment of this application; the method may further include:

[0116] S140: Collect system operation index data, which includes at least the average response time of requests at each level, the utilization rate of computing resources in each time period, the actual power consumption and corresponding electricity expenses in each time period, the average waiting time of tasks in the delay queue, the task timeout rate, and the number of requests that are temporarily downgraded due to insufficient resources.

[0117] S150: Compare the system operation index data with the preset target value, and dynamically adjust the current scheduling strategy parameters based on the comparison results.

[0118] In this embodiment, the feedback adjustment unit continuously collects various indicator data generated during system operation, including the average response time of requests at each level, the utilization rate of computing resources in each time period, the actual power consumption and corresponding electricity expenses in each time period, the average waiting time of tasks in the delay queue, the task timeout rate, and the number of requests that are temporarily downgraded due to insufficient resources.

[0119] The feedback adjustment unit compares and analyzes the collected indicator data with preset target values: if the average response time of real-time level requests exceeds the target value, it indicates that the resource allocation ratio of real-time level requests may be insufficient during peak periods, and the feedback adjustment unit generates adjustment suggestions to increase the resource allocation ratio of real-time level requests during peak periods; if the average waiting time of tasks in the delay queue is too long or the timeout rate is too high, it indicates that the current delay scheduling strategy is too aggressive, and the feedback adjustment unit generates adjustment suggestions to reduce the delay ratio of offline level requests during the parity period; if electricity costs do not achieve the expected savings, it indicates that the delay level of low-level requests during peak periods is insufficient, and the feedback adjustment unit generates adjustment suggestions to expand the delay execution range during peak periods.

[0120] The feedback adjustment unit outputs the generated adjustment suggestions to the hierarchical response strategy unit, which then updates the corresponding parameters in the strategy table accordingly. Parameter updates employ a gradual adjustment method, with each adjustment controlled within a certain range to avoid system instability caused by sudden parameter changes. The updated strategy table takes effect in subsequent request scheduling, and the system gradually optimizes scheduling performance through continuous feedback adjustment.

[0121] Figure 6 The application demonstrates the seven steps of the complete implementation process, their execution order, and data flow: After system initialization, the application processes request classification and electricity price status determination separately. The results of the two parallel paths converge in the strategy table query stage to generate scheduling instructions. Then, according to the instructions, tasks are either immediately dispatched or transferred to a delay queue. During periods of low electricity prices, the execution of tasks in the delay queue is triggered and sent back to the scheduling execution and resource allocation stage. The feedback collection and strategy adjustment stage collects operational data and feeds it back to the strategy table for parameter optimization.

[0122] In the above embodiments, this application combines electricity price signals and grid load status to formulate differentiated hierarchical response scheduling rules for AI inference service requests with different latency tolerance levels. This not only ensures the service response quality of high-priority requests, but also transfers deferred tasks to low-electricity-price and low-load periods for execution through off-peak scheduling, effectively reducing the overall electricity cost of AI inference service operation. At the same time, through timeout protection mechanisms and dynamic feedback adjustment mechanisms, the service completion rate is guaranteed, and scheduling parameters can be continuously optimized based on actual operating results to adapt to the operating needs of different computing centers.

[0123] In one embodiment, when the electricity price-related information is a user's selection instruction based on electricity price information, generating a target scheduling instruction according to the target request level and the electricity price-related information in step S130 may include:

[0124] S231: Determine the submission time period corresponding to the selection instruction.

[0125] S232: If the submission period is a low electricity price period, then generate a target scheduling instruction to be executed immediately.

[0126] S233: If the submission period is a period of flat price or a period of high electricity price, then a target scheduling instruction is generated based on the target request level and the submission period.

[0127] In this embodiment, the application can transparently transmit electricity price fluctuation information to users, allowing them to choose to submit requests during different electricity price periods and enjoy different service prices. Users can proactively choose off-peak scheduling based on their own needs. When a user chooses to execute their own AI inference service request during a low electricity price period, even if the request itself has a low latency tolerance level, an immediate scheduling instruction will be generated to meet the user's proactive need for off-peak cost reduction. If the user chooses to submit a request during a period of flat or high electricity prices, a corresponding scheduling instruction will still be generated according to the request level and the time period, taking into account both the user's autonomous choice needs and the overall off-peak scheduling goals of the system, further improving the flexibility and adaptability of the scheduling scheme.

[0128] In one embodiment, when the electricity price-related information is a prediction result based on historical electricity price data, generating a target dispatch instruction in step S130 based on the target request level and the electricity price-related information may include:

[0129] S331: Before operation, scheduling strategies for each time period are pre-planned based on historical electricity price data and electricity price forecast results. The scheduling strategies define the execution methods for different levels of requests within each time period.

[0130] S332: During operation, a target scheduling strategy is determined based on the current time period and the target request level, and a target scheduling instruction is generated based on the target scheduling strategy.

[0131] In this embodiment, for scenarios where electricity price fluctuations exhibit stable historical patterns, predictive planning can be completed in advance based on historical electricity price data. The execution methods for different request levels at various time periods are pre-defined in the scheduling strategy. During runtime, corresponding scheduling instructions are generated directly according to the pre-planned strategy, eliminating the need for real-time electricity price signal collection. This reduces the system's dependence on real-time data interfaces, resulting in more stable operation and adaptability to scenarios lacking real-time electricity price data collection capabilities. The pre-planned strategy can be periodically updated according to a preset cycle, incorporating the latest historical electricity price data, ensuring the strategy's compatibility with actual electricity price patterns.

[0132] For example, this application can pre-determine that 9:00-12:00 and 14:00-17:00 on weekdays are peak periods for industrial and commercial electricity consumption, which are predicted to be high-price periods. During these periods, only the highest-level real-time requests are allowed to be executed immediately, while all other levels of requests are transferred to a delayed execution queue. From 0:00 to 7:00 on weekdays, the period is predicted to be a low-price period, during which all levels of requests can be executed immediately, and tasks backlogged in the delayed queue are processed in batches. The remaining periods are considered flat-price periods, where only the first and second-level requests are allowed to be executed immediately, while other levels of requests are transferred to a delayed queue according to a preset ratio. During operation, the pre-planned scheduling strategy is directly matched to the current time period to quickly generate corresponding scheduling instructions, simplifying the real-time decision-making process and reducing the complexity of system operation.

[0133] The following describes the hierarchical response scheduling system for AI inference services based on electricity price signals provided in the embodiments of this application. The hierarchical response scheduling system for AI inference services based on electricity price signals described below can be referred to in correspondence with the hierarchical response scheduling method for AI inference services based on electricity price signals described above.

[0134] In one embodiment, such as Figure 7 As shown, Figure 7 This application provides a schematic diagram of a hierarchical response scheduling system for AI inference services based on electricity price signals, as provided in an embodiment of this application. This application also provides a hierarchical response scheduling system for AI inference services based on electricity price signals, using the hierarchical response scheduling method for AI inference services based on electricity price signals as described in any of the above embodiments. The system may include a request level determination unit 210, an electricity price information determination unit 220, and a hierarchical scheduling unit 230, specifically including the following:

[0135] The request level determination unit 210 is used to obtain the AI ​​inference service request and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request.

[0136] The electricity price information determination unit 220 is used to obtain electricity price-related information, which includes real-time electricity price signals, historical electricity price data prediction results, or user selection instructions based on electricity price information.

[0137] The hierarchical scheduling unit 230 is used to generate a target scheduling instruction based on the target request level and the electricity price-related information, and to execute the AI ​​inference service request immediately or with a delay based on the target scheduling instruction.

[0138] In the above embodiments, inference requests with different latency tolerances are classified into levels, and differentiated scheduling strategies are formulated in combination with the peak and valley fluctuations of time-of-use electricity prices. For example, during peak electricity price periods, only real-time level requests that are sensitive to latency are guaranteed, while offline level requests that can be delayed are transferred to low electricity price periods for execution. This not only avoids indiscriminately affecting the response experience of high-value requests, but also proactively reduces the overall electricity cost of the computing center by staggering peak hours. This achieves differentiated scheduling processing for requests with different latency tolerances, thereby reducing the electricity cost of the computing center while ensuring the service experience of high-value users, and achieving synergistic optimization between electricity costs and service quality.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0141] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hierarchical response scheduling method for AI inference services based on electricity price signals, characterized in that, include: Obtain an AI inference service request, and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request; Obtain electricity price-related information, including real-time electricity price signals, historical electricity price data prediction results, or user selection instructions based on electricity price information; A target scheduling instruction is generated based on the target request level and the electricity price information, and the AI ​​inference service request is executed immediately or delayed based on the target scheduling instruction.

2. The method according to claim 1, characterized in that, The step of determining the target request level of the AI ​​inference service request based on the latency tolerance parameter in the AI ​​inference service request includes: If the AI ​​inference service request carries an explicit latency tolerance parameter, then the latency tolerance parameter is extracted directly. If not carried, the preset type-tolerance mapping table is queried according to the business type field of the AI ​​inference service request to obtain the latency tolerance parameter corresponding to the business type field; The latency tolerance parameter is compared with a preset first threshold and a second threshold. AI inference service requests below the first threshold are determined as real-time level, AI inference service requests between the first threshold and the second threshold are determined as near-real-time level, and AI inference service requests above the second threshold are determined as offline level.

3. The method according to claim 1, characterized in that, When the electricity price-related information is a real-time electricity price signal, the step of generating a target scheduling instruction based on the target request level and the electricity price-related information includes: Collect real-time electricity price signals, compare the real-time electricity price signals with preset peak thresholds and off-peak thresholds, and determine the current electricity price status corresponding to the real-time electricity price signals based on the comparison results. The current electricity price status includes peak period, off-peak period, and flat price period. The target request level and the current electricity price status are used as a two-dimensional index to query the preset hierarchical response strategy table and generate the target scheduling instruction; The tiered response strategy table defines the execution method for each level of request under different electricity price conditions: During off-peak hours, all request levels are executed immediately; During the flat pricing period, real-time and near real-time requests are executed immediately, while offline requests are partially delayed. During peak periods, real-time requests are executed immediately, near-real-time requests are executed to a limited extent based on resource availability, and offline requests are executed with a delay.

4. The method according to claim 3, characterized in that, The method further includes: The current power grid load value is collected, and the current power grid load value is combined with the real-time electricity price signal as the basis for dispatching decisions; When the current grid load value exceeds the preset high load threshold but the real-time electricity price signal has not yet reached the peak threshold, some offline level requests are transferred to the delayed execution queue in advance. When the current power grid load value is lower than the preset low load threshold, the triggering conditions of the delayed execution queue are relaxed, and the tasks in the delayed execution queue are executed in batches ahead of schedule.

5. The method according to claim 1 or 4, characterized in that, The method further includes: Requests for AI inference services that will be delayed will be stored in a delayed execution queue. AI inference service requests in the delayed execution queue are executed in batches when one of the following conditions is met: The current electricity price is at its lowest level, based on real-time electricity price signals. The forecast results are based on historical electricity price data, and the current period is a pre-planned period of low electricity prices; The current power grid load value is lower than the preset low load threshold.

6. The method according to claim 5, characterized in that, The method further includes: The waiting time of each task in the delayed execution queue is continuously tracked. When the ratio of the waiting time of any task to its delay tolerance limit reaches the preset timeout protection trigger ratio, the scheduling priority of the task is raised to the highest level and executed immediately.

7. The method according to claim 1, characterized in that, The method further includes: Collect system operation index data, which includes at least the average response time of each level of request, the computing resource utilization rate of each time period, the actual power consumption and corresponding electricity expenditure of each time period, the average waiting time of tasks in the delay queue, the task timeout rate, and the number of requests that are temporarily downgraded due to insufficient resources. The system operation index data is compared with the preset target value, and the current scheduling strategy parameters are dynamically adjusted based on the comparison results.

8. The method according to claim 1, characterized in that, When the electricity price-related information is a user's selection instruction based on electricity price information, the step of generating a target scheduling instruction according to the target request level and the electricity price-related information includes: Determine the submission time period corresponding to the selection instruction; If the submission period is a low electricity price period, then generate a target scheduling instruction to be executed immediately; If the submission period is a period of flat price or a period of high electricity price, then a target scheduling instruction is generated based on the target request level and the submission period.

9. The method according to claim 1, characterized in that, When the electricity price-related information is a prediction result based on historical electricity price data, the step of generating a target scheduling instruction based on the target request level and the electricity price-related information includes: Before operation, scheduling strategies for each time period are pre-planned based on historical electricity price data and electricity price forecast results. The scheduling strategies define the execution methods for different levels of requests within each time period. During operation, a target scheduling strategy is determined based on the current time period and the target request level, and a target scheduling instruction is generated based on the target scheduling strategy.

10. A hierarchical response scheduling system for AI inference services based on electricity price signals, using the hierarchical response scheduling method for AI inference services based on electricity price signals as described in any one of claims 1-9, characterized in that, The system includes: The request level determination unit is used to obtain AI inference service requests and determine the target request level to which the AI ​​inference service request belongs based on the latency tolerance parameter in the AI ​​inference service request. The electricity price information determination unit is used to acquire electricity price-related information, which includes real-time electricity price signals, historical electricity price data prediction results, or user selection instructions based on electricity price information. The hierarchical scheduling unit is used to generate a target scheduling instruction based on the target request level and the electricity price-related information, and to execute the AI ​​inference service request immediately or with a delay based on the target scheduling instruction.

Images