Inference service capacity adjustment method and device based on traffic prediction, equipment and medium

Abstract

The embodiment of the application provides a reasoning service capacity adjustment method and device based on traffic prediction, equipment and medium, belonging to the technical field of resource allocation. The method comprises collecting running indexes of the reasoning service, and obtaining hot event information by real-time monitoring and fusing dynamic events of multi-source external signals of a target field; determining a traffic prediction curve according to the running indexes and the hot event information by using a time sequence prediction model; obtaining a future request quantity confidence by prediction according to the traffic prediction curve; determining a target number of target instances required at different time points by using the future request quantity in combination with a single instance processing capacity of the reasoning service; determining a traffic arrival time according to the target number, and determining a preheating time of the reasoning service according to the traffic arrival time and a start time of the target instance; determining a scaling strategy of the reasoning service according to the future request quantity and the confidence; and performing capacity adjustment processing on a container corresponding to the reasoning service according to the scaling strategy and the preheating time to obtain a target adjustment result.

Description

Technical Field

[0001] This invention relates to the field of resource allocation technology, and in particular to a method, apparatus, equipment and medium for adjusting the capacity of inference services based on traffic prediction. Background Technology

[0002] With the widespread deployment of generative artificial intelligence (AIGC) technology in the media field (such as automatic news generation and intelligent summarization), inference services based on large language models and diffusion models consume a large amount of GPU resources. Traffic in the media industry caused by breaking news events can surge tens or even hundreds of times in a short period. In a cloud-native architecture, elastic scaling is the core mechanism for dealing with traffic fluctuations. Existing solutions include passive scaling based on thresholds, scaling based on timed rules, and simple predictive scaling based on moving averages or general time-series models. However, the long cold start time of AIGC inference services causes a significant lag in passive scaling response. Furthermore, existing methods fail to model the propagation dynamics of hot topics in the media field, resulting in insufficient prediction accuracy. They also fail to optimize for characteristics such as long loading times, high inference latency, and coarse-grained resource allocation, leading to unreasonable scaling timing and granularity. To cope with sudden traffic surges, it is often necessary to maintain a large amount of redundant GPU resources for extended periods, resulting in high costs. Therefore, a method for adjusting the capacity of inference services based on traffic prediction is urgently needed. Summary of the Invention

[0003] The main objective of this invention is to provide a method, apparatus, device, and medium for adjusting inference service capacity based on traffic prediction, aiming to solve the problems of response latency and high resource costs in inference service capacity in the prior art.

[0004] In a first aspect, embodiments of the present invention provide a method for adjusting the capacity of inference services based on traffic prediction, comprising: The system collects operational metrics corresponding to the inference service and monitors and integrates dynamic events of multi-source external signals in the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. A time-series prediction model is used to determine the traffic prediction curve corresponding to the hotspot event information based on the operational indicators and the hotspot event information. The future request volume and the corresponding confidence level are predicted based on the traffic prediction curve; and the target number of target instances required at different time points is determined by combining the future request volume with the single instance processing capacity of the inference service. The traffic arrival time is determined based on the target quantity, and the preheating timing of the inference service under the hot event information is determined based on the traffic arrival time and the startup time of the target instance. The scaling strategy for the inference service under the hot event information is determined based on the future request volume and the confidence level. The capacity of the container corresponding to the inference service is adjusted according to the scaling strategy and the warm-up timing to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result.

[0005] Secondly, embodiments of the present invention provide an inference service capacity adjustment device based on traffic prediction, comprising: The data acquisition module is used to collect the operational metrics corresponding to the inference service, and to monitor and fuse dynamic events of multi-source external signals corresponding to the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. The data prediction module is used to determine the traffic prediction curve corresponding to the hot event information based on the operating indicators and the hot event information using a time series prediction model; The quantity determination module is used to predict the future request volume and the confidence level corresponding to the future request volume based on the traffic prediction curve; and to use the future request volume in combination with the single instance processing capacity corresponding to the inference service to determine the target number of target instances required at different time points. The timing determination module is used to determine the traffic arrival time based on the target quantity, and to determine the preheating timing of the inference service under the hot event information based on the traffic arrival time and the startup time of the target instance. The strategy generation module is used to determine the scaling strategy of the inference service under the hot event information based on the future request volume and the confidence level. The capacity adjustment module is used to adjust the capacity of the container corresponding to the inference service according to the scaling strategy and the preheating time, and obtain the target adjustment result so that the inference service can perform information inference under the target adjustment result.

[0006] Thirdly, embodiments of the present invention also provide a terminal device, the terminal device including a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of any of the inference service capacity adjustment methods based on traffic prediction provided in this specification.

[0007] Fourthly, embodiments of the present invention also provide a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of any of the inference service capacity adjustment methods based on traffic prediction provided in this specification.

[0008] This invention provides a method, apparatus, device, and medium for adjusting the capacity of an inference service based on traffic prediction. The method includes: collecting operational metrics corresponding to the inference service and monitoring and fusing dynamic events of multi-source external signals corresponding to the target domain in real time to obtain hotspot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. A time-series prediction model is used to determine a traffic prediction curve corresponding to the hotspot event information based on the operational metrics and hotspot event information. The traffic prediction curve is used to predict future request volume and the confidence level corresponding to the future request volume. The future request volume is combined with the single-instance processing capacity of the inference service to determine the target number of target instances required at different time points. The traffic arrival time is determined based on the target number, and the traffic arrival time and the startup time of the target instances are used to determine the preheating timing of the inference service under the hotspot event information. The future request volume and confidence level are used to determine the scaling strategy of the inference service under the hotspot event information. The capacity of the container corresponding to the inference service is adjusted according to the scaling strategy and the preheating timing to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result. This method constructs a lightweight time-series prediction model, deeply integrating historical hotspot data with real-time external signals from the media sector to accurately predict future request volume trends. Before traffic peaks arrive, it proactively triggers pre-warming scheduling of GPU instances and rapid loading of model weights, enabling advance expansion of inference capacity. This transforms the traditional threshold-triggered passive response mode into a traffic trend-driven proactive prediction mode, significantly reducing inference service response latency and GPU resource consumption costs while ensuring service quality. It solves the problems of high response latency and resource costs associated with inference service capacity in existing technologies. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a method for adjusting the capacity of an inference service based on traffic prediction, provided in an embodiment of the present invention; Figure 2A schematic diagram of a scenario for implementing the inference service capacity adjustment method based on traffic prediction provided in this embodiment; Figure 3 A schematic diagram of the module structure of an inference service capacity adjustment device based on traffic prediction provided in an embodiment of the present invention; Figure 4 This is a schematic block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] This invention provides a method, apparatus, device, and medium for adjusting inference service capacity based on traffic prediction. The method for adjusting inference service capacity based on traffic prediction can be applied to terminal devices, such as tablets, laptops, desktop computers, personal digital assistants, and wearable devices. The terminal device can be a server or a server cluster.

[0015] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0016] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for adjusting the capacity of an inference service based on traffic prediction, provided in an embodiment of the present invention.

[0017] like Figure 1 As shown, the inference service capacity adjustment method based on traffic prediction includes steps S101 to S106.

[0018] Step S101: Collect the operational metrics corresponding to the inference service, and monitor and fuse the dynamic events of multi-source external signals corresponding to the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage.

[0019] For example, the AIGC inference service collects operational metrics such as real-time request volume (QPS), request queue depth, average response latency, GPU utilization, and video memory usage, and stores them in a time-series database at a granular level of seconds.

[0020] For example, for target fields such as the media industry, multiple external signals such as news hot search list API, social media topic trend API and search engine hot word data are accessed in parallel to monitor the dynamics of hot events in real time; when a rapid upward trend in the popularity of a certain topic is detected, the corresponding hot event information is automatically generated.

[0021] Step S102: Using a time-series prediction model, determine the traffic prediction curve corresponding to the hotspot event information based on the operational indicators and the hotspot event information.

[0022] For example, hotspot event information includes event type, popularity change curve, and occurrence time. Historical operational metrics in the time-series database are time-aligned and feature-fused with the hotspot event information to obtain a time-series prediction model. For instance, a lightweight time-series prediction model, such as one based on a Transformer variant or PatchTST, can be used. The sequence of operational metrics within a past time window and the encoded features of hotspot events are taken as input. The model's internal multi-head self-attention mechanism captures long-short-term dependencies, and an external signal fusion layer cross-integrates event features into the time-series representation. Finally, the model generates predicted traffic values ​​and confidence intervals for multiple future time points through a multi-step prediction output layer, thereby obtaining a traffic prediction curve associated with the hotspot event based on the predicted traffic values ​​at multiple time points.

[0023] In some implementations, the time-series prediction model includes a time-series segmentation embedding layer, a multi-head self-attention layer, an external signal fusion layer, and a multi-step prediction output layer. The step of using the time-series prediction model to determine the traffic prediction curve corresponding to the hot-spot event information based on the operational metrics and the hot-spot event information includes: extracting multi-dimensional time-series features from historical data, including periodic features, trend features, burst features, and correlation features; inputting the multi-dimensional time-series features into the time-series segmentation embedding layer to divide them into fixed-length segmented time-series features; and using the multi-head self-attention layer to capture long-short-term dependencies from the segmented time-series features. The relationship yields target dependency features; the burst features and target dependency features are input together into the external signal fusion layer and fused through a cross-attention mechanism to obtain target fusion features; the multi-step prediction output layer is used to train the model with the target fusion features and historical traffic information to obtain initial model parameters; after model deployment, an online learning strategy is used to continuously collect prediction error feedback and incrementally update the initial model parameters to obtain target model parameters adapted to the latest traffic distribution; when the hotspot event information is detected, traffic prediction is performed based on the target model parameters in conjunction with the operating indicators to obtain the traffic prediction curve.

[0024] For example, the time series prediction model includes a time series slice embedding layer, a multi-head self-attention layer, an external signal fusion layer, and a multi-step prediction output layer.

[0025] For example, a feature extraction model is used to extract multi-dimensional time-series features from historical data. These multi-dimensional time-series features include periodic features, trend features, burst features, and correlation features. Then, the multi-dimensional time-series features are input into the time series segmentation embedding layer of the time-series prediction model, which divides them into segments of fixed length. The long-term and short-term dependencies of the segments are then captured by a multi-head self-attention layer to obtain the target dependency features.

[0026] For example, sudden features and target dependency features are fed into an external signal fusion layer and deeply fused using a cross-attention mechanism to obtain target fusion features. Then, a multi-step prediction output layer is used to predict the target fusion features to obtain prediction results. The prediction results are then compared with historical traffic information to calculate the loss until the loss converges to a preset value, thereby obtaining the initial model parameters.

[0027] For example, the model corresponding to the initial model parameters is deployed online to continuously collect actual prediction error feedback, and an online learning strategy is used to incrementally update the initial model parameters to obtain target model parameters that can adapt to the latest traffic distribution characteristics.

[0028] For example, when hot event information is detected, traffic prediction is performed using target model parameters in conjunction with real-time operating indicators to obtain traffic prediction values ​​at different time points, and then the traffic prediction curve of the hot event is obtained by fitting the traffic prediction values ​​at different time points.

[0029] Specifically, by leveraging the unique characteristics of suddenness and multi-source dependence in the field of converged media, the accuracy of predicting traffic surges caused by breaking news events has been significantly improved, thus providing strong support for reducing operating costs while ensuring service quality.

[0030] In some implementations, after obtaining the traffic prediction curve, the method further includes: obtaining the occurrence time and traffic impact pattern of the breaking news event corresponding to the historical data through data annotation, and learning typical traffic impact curves corresponding to different types of news events by adjusting the parameters of the target model through comparative learning; when the hot event information is detected, matching the hot event information with the breaking news event to obtain matching event information corresponding to the hot event information; obtaining the relevant impact curve corresponding to the matching event information from the typical traffic impact curve; and superimposing and correcting the traffic prediction curve according to the relevant impact curve to generate the traffic prediction curve corrected by the hot event information.

[0031] For example, the occurrence time of each breaking news event can be manually or semi-automatically marked from historical traffic data, and the traffic sequence of the entire process from the outbreak to the end of the event can be extracted to obtain the corresponding traffic impact pattern. For example, disaster events often show an instantaneous peak followed by a slow decline, while entertainment events may show a step-like increase.

[0032] For example, the pre-trained target model parameters are used as the initial encoder, and a contrastive learning method is employed for training: for the traffic impact patterns of the same type of event, such as two earthquake news reports, the distance between them in the feature space is narrowed; for different types of events, such as the impact patterns between natural disasters and entertainment gossip, the impact patterns are narrowed. After several rounds of contrastive learning, the model's feature extraction layer can map the input event features to a space with class separability, and the average feature vector or typical sequence of each type of event is stored as a typical traffic impact curve.

[0033] For example, when a new trending event is detected in real time, its feature vector is first calculated. Then, it is matched with the feature vectors of all stored breaking news events using cosine similarity to find the most similar historical event category and obtain the matching event information. Based on the matching event information, a relevant impact curve is obtained from the typical traffic impact curve. Finally, the relevant impact curve and the traffic prediction curve are aligned on the time axis, and a point-by-point addition correction operation is performed to finally output the event-corrected traffic prediction curve.

[0034] Specifically, by introducing prior traffic patterns of sudden events in the media field through comparative learning, the model can quickly learn from the impact patterns of similar historical events when facing unknown hot events, significantly improving the prediction accuracy of peak and changing trends of sudden traffic. The superimposed correction mechanism avoids the lagging response to sudden inflection points by simply relying on time series prediction, providing good support for effectively mitigating resource scheduling delays caused by cold start delays in AIGC inference services.

[0035] Step S103: Predict the future request volume and the confidence level corresponding to the future request volume based on the traffic prediction curve; and use the future request volume in combination with the single instance processing capacity corresponding to the inference service to determine the target number of target instances required at different time points.

[0036] For example, based on the generated traffic prediction curve, the predicted traffic values ​​at future time points, such as every 5 minutes, are extracted as the future request volume and the confidence level corresponding to the future request volume.

[0037] For example, for each instance of the inference service, its single-instance processing capacity is pre-calibrated—this capacity is determined through stress testing, typically measured by the maximum sustainable QPS (Queries Per Second) or the maximum concurrent requests per GPU instance, taking into account the average queuing tolerance time of requests and the differences in resource consumption of different inference tasks, to obtain a safe processing capacity (e.g., a single instance is recommended to handle 100 QPS). Finally, for each future time point, the predicted future request volume is divided by the safe processing capacity per instance and rounded up to obtain the basic number of instances to meet the basic requirements; simultaneously, redundant instances are added according to the confidence level, ultimately outputting the target number of target instances required at each time point.

[0038] In some implementations, determining the target number of target instances required at different time points by combining the future request volume with the single-instance processing capacity corresponding to the inference service includes: constructing an optimization function based on the single-instance processing capacity and the future request volume, with the objective of minimizing the number of instances and constrained by service quality and resource utilization; the single-instance processing capacity includes the maximum number of concurrent inference requests, the preset queuing tolerance time for requests, the limitation of GPU memory capacity on batch size, and the resource consumption differences between different types of inference tasks; solving the optimal solution of the optimization function at different time points to obtain the target number of target instances.

[0039] For example, offline load testing is used to obtain the single-instance processing capacity of the inference service instance, such as the maximum number of concurrent inference requests, the upper limit of GPU memory capacity on batch size, and the differences in unit resource consumption for different types of inference tasks, such as text generation and image generation. Simultaneously, a queuing tolerance time for requests is set as a service quality constraint according to business requirements, for example, P99 latency not exceeding 100ms.

[0040] For example, an optimization function is constructed using the future request volume as input: the objective is to minimize the total number of instances required, and the constraints include that the total concurrency capacity of the instances must be greater than or equal to the future request volume, the total video memory must meet the batch processing requirements of all concurrent requests, and the resource consumption of different task types cannot exceed the heterogeneous resource limit of the instances (such as GPU computing power and video memory).

[0041] For example, integer linear programming or heuristic search can be used to solve the problem. For each discrete time point, such as every 5 minutes, the predicted request volume and task type distribution at that time point are substituted to find the minimum number of instances that satisfy all constraints. Finally, the target number corresponding to the target instance at each time point is output.

[0042] Step S104: Determine the traffic arrival time based on the target quantity, and determine the preheating timing of the inference service under the hot event information based on the traffic arrival time and the startup time of the target instance.

[0043] For example, the traffic arrival time is obtained based on the target quantity at each time point in the traffic prediction curve. This is the traffic arrival time when the current instance capacity cannot meet the predicted request volume and requires capacity expansion. For instance, if the predicted request volume exceeds the current capacity in the 10th minute, then that moment is the traffic arrival time.

[0044] For example, the time consumption of the entire cold start process of a GPU instance is statistically analyzed, including stages such as container image pulling, GPU driver initialization, model weight loading, and inference engine warm-up, and the cold start time of the instance is accumulated.

[0045] For example, to ensure that instances are fully ready before traffic arrives, a warm-up lead time calculation formula is set: Warm-up lead time = Traffic arrival time Instance cold start time A safety margin, typically 10-30 seconds, is used to handle network fluctuations or scheduling delays. Finally, the warm-up timing is calculated by adding a warm-up advance value to the current time.

[0046] For example, if traffic is predicted to arrive in 10 minutes, instance readiness takes 8 minutes, and a safety margin of 30 seconds is set, then the warm-up lead time is 1.5 minutes. Therefore, instance warm-up should begin 1.5 minutes later, i.e., 8.5 minutes before traffic arrives. This ensures that new instances complete all initialization before traffic peaks arrive, thus eliminating response latency caused by cold starts.

[0047] Step S105: Determine the scaling strategy of the inference service under the hot event information based on the future request volume and the confidence level.

[0048] For example, the future request volume and its confidence level are transformed into a probability distribution. Candidate scaling strategies are defined, such as rapid scaling to 120% of the target number, standard scaling to the target number, conservative scaling to 80% of the target number, and warm-up without scaling.

[0049] For example, for each candidate strategy, its expected comprehensive cost under the probability distribution is calculated. This cost consists of three parts: the resource overhead of the scaling operation itself, such as power consumption during instance startup and scheduling delays; the penalty cost of request queuing timeouts or denial of service due to insufficient capacity; and the cost of idle resource waste caused by excess capacity.

[0050] For example, the candidate strategy with the lowest expected total cost is selected as the scaling strategy corresponding to hot event information. If the costs of multiple strategies are similar, the strategy that avoids frequent scaling is preferred.

[0051] In some implementations, determining the scaling strategy of the inference service under the hotspot event information based on the future request volume and the confidence level includes: determining the corresponding traffic change amplitude through the future request volume, and determining the corresponding rising state of the inference service under the hotspot event information based on the traffic change amplitude and the confidence level; when the traffic change amplitude is less than a first preset value and the confidence level is greater than a second preset value, the rising state is determined to be a mild rise, and the scaling strategy is to gradually expand capacity according to the target number and prioritize waking up dormant warm instances for carrying; when the traffic change amplitude is greater than or equal to the first preset value and the confidence level is greater than the second preset value, the rising state is determined to be a rapid rise, and the scaling strategy is to preheat new instances in batches according to the target number, while immediately waking up all warm instances to cope with sudden traffic; when the traffic change amplitude is greater than or equal to the first preset value and the confidence level is less than the second preset value, the rising state is determined to be an uncertain rise, and the scaling strategy is to preventively preheat a preset number of instances to a warm standby state, and then quickly activate them after the actual traffic arrives.

[0052] For example, the magnitude of traffic change between the current level and the predicted peak can be calculated based on the future request volume, such as by dividing the expected maximum request volume by the current request volume, while obtaining the prediction confidence level given by the model.

[0053] For example, a first preset value and a second preset value are set, where the first preset value is a change range threshold (e.g., 50%) and the second preset value is a confidence threshold (e.g., 80%).

[0054] For example, if the traffic change is less than the first preset value and the confidence level is higher than the second preset value, it means that the traffic will increase moderately and the prediction is reliable. At this time, the inference service is determined to be in a moderate increase state under the hot event information. The scaling strategy is to gradually expand the capacity in batches according to the target number and give priority to waking up the dormant warm instances to take over the new traffic.

[0055] For example, if the traffic change reaches or exceeds the first preset value and the confidence level is still higher than the second preset value, it indicates that the traffic will surge rapidly and the prediction is reliable. The inference service is determined to be in a rapidly rising state under the hot event information. The strategy is to immediately start new instances in batches according to the target number for preheating, and at the same time wake up all warm instances to cope with the upcoming traffic surge to the maximum extent.

[0056] For example, if the traffic change reaches or exceeds the first preset value but the confidence level is lower than the second preset value, it indicates that the prediction uncertainty is high. The inference service is determined to be in an uncertain upward state under hot event information. The strategy is to only preheat a small number of instances to a warm standby state as a precaution, and then quickly activate them when the real traffic arrives.

[0057] For example, if the traffic change is less than a first preset value and the confidence level is lower than a second preset value, then the inference service will be in a stable state under the hot event information, and the scaling strategy will be to do nothing.

[0058] Specifically, this method combines the magnitude of traffic changes with confidence levels to classify emergencies in a refined manner. While ensuring service quality, it reduces the resource redundancy costs caused by inaccurate predictions, making it particularly suitable for scenarios with drastic traffic fluctuations caused by breaking news events in the media industry.

[0059] Step S106: Adjust the capacity of the container corresponding to the inference service according to the scaling strategy and the preheating timing to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result.

[0060] For example, during the warm-up period, specific capacity adjustment actions are performed on the container cluster corresponding to the inference service according to the scaling policy to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result. The target adjustment result is used to indicate the expansion result of the inference service under the scaling policy.

[0061] For example, commonly used AIGC model images can be cached locally on the node to pre-fetch container images, and then GPU devices can be allocated and drivers initialized to obtain the target adjustment results.

[0062] In some implementations, the step of adjusting the capacity of the container corresponding to the inference service according to the scaling strategy and the warm-up timing to obtain a target adjustment result, so that the inference service can perform information inference under the target adjustment result, includes: adjusting the capacity of the container corresponding to the inference service according to the scaling strategy and the warm-up timing to obtain an initial adjustment result; assigning a preset weight to the new instance after the new instance in the initial adjustment result has started and passed the health check; allocating relevant requests to the new instance according to the traffic gateway using a weighted round-robin method based on the preset weight; if the load of the new instance gradually stabilizes, gradually increasing the preset weight of the new instance by a preset step size or time interval, and simultaneously decreasing the weight information of other instances, until the new traffic is smoothly migrated to the new instance, thereby obtaining the target adjustment result.

[0063] For example, based on the scaling strategy and warm-up timing, capacity adjustment operations such as creating new instances, waking up warm instances, or reclaiming idle instances are performed on the container cluster corresponding to the inference service to obtain the initial adjustment results.

[0064] For example, once a new instance in the initial adjustment results completes startup and passes a health check, it is assigned a preset weight, such as 1%. The traffic gateway uses a weighted round-robin algorithm based on this weight to proportionally allocate relevant requests to the new instance, while the remaining traffic is handled by the existing instances. The load metrics of the new instance, such as CPU utilization and request latency, are continuously monitored. Once its operation is confirmed to be stable and without abnormal fluctuations, the weight of the new instance is gradually increased in preset steps, such as increasing by 5% each time, or at fixed time intervals, such as every 10 seconds. Simultaneously, the weights of other instances are reduced, allowing traffic to smoothly migrate from the existing instances to the new instance. This process is repeated until the weight of the new instance reaches the desired value. At this point, all new traffic is fully handled by the new instance, and both the old and new instances are in normal working order, thus achieving the final target adjustment result. This mechanism effectively avoids a sudden surge of traffic causing overload on the new instance, ensuring a smooth transition and service quality during the scaling process.

[0065] In some implementations, the method further includes: when the future request volume continues to decrease, using a tiered hibernation strategy to reclaim the capacity of the inference service; the tiered hibernation strategy includes a first-level strategy, a second-level strategy, a third-level strategy, and a fourth-level strategy; wherein, the first-level strategy is to reduce the size of the inference batch to release some GPU memory; if there are still excess resources, then the second-level strategy is entered; the second-level strategy is to put the target instance into a warm hibernation state so that it can be restored within seconds; the third-level strategy is to unload the model corresponding to the inference service, release GPU resources, put the target instance into a cold hibernation state, and increase the recovery time to minutes; the fourth-level strategy is to completely reclaim the target instance and release all expanded resources.

[0066] For example, when it is predicted that future request volume will continue to decline, a tiered hibernation strategy is initiated to reclaim the capacity of the inference service. The tiered hibernation strategy includes a first-level strategy, a second-level strategy, a third-level strategy, and a fourth-level strategy.

[0067] For example, the first-level strategy is executed, which reduces the batch size of the inference service, for example, from 8 to 4, to free up some GPU memory and observe whether resources are still excessive. If traffic continues to be low, the second-level strategy is executed, which puts some target instances into a warm sleep state, that is, suspends request processing for some target instances but retains model memory, so that they can recover quickly within seconds. If resource redundancy is still not eliminated, the third-level strategy is executed, which unloads the model weights on the target instances to free up GPU resources, putting the target instances into a cold sleep state. The recovery time of the target instance increases to minutes, and the model needs to be reloaded. If traffic remains at a low level for a long time and there is no expectation of recovery, the fourth-level strategy is executed, which completely reclaims the target instances and releases all expanded container, GPU, and memory resources. By reducing the load step by step, resource costs are saved to the greatest extent while ensuring that the service can respond quickly.

[0068] Please refer to Figure 2 , Figure 2 A schematic diagram of a scenario for implementing the inference service capacity adjustment method based on traffic prediction provided in this embodiment, as shown below. Figure 2 As shown, the monitoring system reports real-time traffic and operational metrics, along with dynamic events from multi-source external signals corresponding to the target domain, to the prediction service that deploys the time-series prediction model to obtain hotspot event information. Then, the prediction service uses the operational metrics and hotspot event information to determine the traffic prediction curve corresponding to the hotspot event information. Based on the traffic prediction curve, it predicts future request volume and the corresponding confidence level. Using the future request volume and the single-instance processing capacity of the inference service, it determines the target number of target instances required at different time points. Based on the target number, it determines the traffic arrival time, and based on the traffic arrival time and the startup time of the target instances, it determines the preheating timing of the inference service under the hotspot event information. Finally, based on the future request volume and confidence level, it determines the scaling strategy of the inference service under the hotspot event information. The inference service executes scaling strategies during the preheating period, including requesting computing resources, allocating resources, starting and preheating instances. After the instance is ready, it reports the instance status, completing the preheating. Then, when traffic reaches a certain level, it imports and processes business requests according to the traffic strategy. The monitoring system continuously reports traffic. When it detects that the future request volume is continuously decreasing, it adopts a tiered hibernation strategy to reclaim the capacity of the inference service. The first-level strategy is to reduce the size of the inference batch to release some GPU memory. If there are still excess resources, the second-level strategy is adopted. The second-level strategy is to put the target instance into a warm hibernation state, so that it can be restored within seconds. The third-level strategy is to unload the model corresponding to the inference service, release GPU resources, and put the target instance into a cold hibernation state, increasing the recovery time to minutes. The fourth-level strategy is to completely reclaim the target instance and release all expanded resources.

[0069] This application aims to achieve efficient and elastic scaling of AIGC inference services by designing an asynchronous, multi-stage preheating pipeline. This pipeline sequentially completes container image pre-pulling, GPU device allocation and driver initialization, parallel loading of model weights from distributed storage to GPU memory, and inference engine initialization and preheating inference, ensuring rapid new instance readiness. Simultaneously, it maintains three instance pool states—hot, warm, and cold—and dynamically adjusts the number of instances in each pool to achieve rapid elastic response. During scaling, the traffic gateway employs a weighted routing strategy to gradually direct new traffic to newly ready instances, avoiding instantaneous overload. A feedback loop is further constructed: real-time monitoring of prediction accuracy (e.g., MAPE, RMSE) triggers online model updates when deviations exceed thresholds; the timing, scale, and service quality indicators (P99 latency, request success rate, resource utilization) of each scaling operation are recorded to form a decision-making effect dataset for continuous strategy optimization; and cost-benefit analysis, combined with GPU resource utilization and service level agreement (SLA) achievement rate, dynamically adjusts the scaling aggressiveness to achieve an optimal balance between service quality and resource cost. This method integrates external hotspot signals into the time-series prediction model through a cross-attention mechanism, enabling early prediction of traffic surges caused by breaking news. A tiered preheating and intelligent hibernation mechanism for AIGC inference services: A three-tiered instance pool management strategy (hot / warm / cold) is designed, leveraging the cold start characteristics of GPU instances to achieve elastic responses at granular levels ranging from seconds to minutes. Sudden event detection and traffic surge pattern matching and correction technology: Matching traffic surge curves are retrieved from a historical database of similar events, and the basic prediction is overlaid and corrected to improve prediction accuracy in sudden events. A preheating timing back-calculation decision algorithm: Based on statistics of GPU instance startup time at each stage, the preheating startup time is precisely calculated to ensure instances are ready just before traffic arrives. Continuous optimization of the prediction model and decision strategy is achieved through scaling effect feedback, enabling system adaptive evolution.

[0070] This method significantly reduces the scaling response time of AIGC inference services from 5-10 minutes in traditional solutions to less than 30 seconds through traffic trend prediction and proactive preheating mechanisms, achieving several-fold elastic acceleration. In handling sudden traffic surges, it matches hot media signals with historical event patterns, achieving a prediction accuracy of over 85% for news spikes, effectively preventing service avalanche. Regarding resource utilization, intelligent hibernation and tiered instance pool management improve GPU resource utilization by over 40% while maintaining service level agreements, resulting in a 30%-50% reduction in overall resource costs. Addressing the unique challenges of long GPU initialization times and high memory usage in AIGC inference tasks, this method designs an asynchronous parallel preheating pipeline to minimize instance readiness time. Simultaneously, through online incremental learning and closed-loop feedback mechanisms, the system continuously adapts to changes in traffic patterns, resulting in continuously improving prediction accuracy and scaling efficiency over long-term operation.

[0071] Please see Figure 3 , Figure 3 This application provides an embodiment of an inference service capacity adjustment device 200 based on traffic prediction. The device 200 includes a data acquisition module 201, a data prediction module 202, a quantity determination module 203, a timing determination module 204, a strategy generation module 205, and a capacity adjustment module 206. The data acquisition module 201 is used to collect operational metrics corresponding to the inference service and to monitor and fuse dynamic events of multi-source external signals corresponding to the target domain in real time to obtain hotspot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. The data prediction module 202 is used to determine the traffic prediction curve corresponding to the hotspot event information using a time-series prediction model based on the operational metrics and the hotspot event information. The quantity determination module 203 is used to... The system predicts future request volumes and corresponding confidence levels based on the traffic prediction curve; it then uses these future request volumes in conjunction with the single-instance processing capacity of the inference service to determine the target number of target instances required at different time points; a timing determination module 204 determines the traffic arrival time based on the target number, and determines the preheating timing of the inference service under the hot event information based on the traffic arrival time and the startup time of the target instances; a strategy generation module 205 determines the scaling strategy of the inference service under the hot event information based on the future request volumes and the confidence levels; and a capacity adjustment module 206 adjusts the capacity of the container corresponding to the inference service based on the scaling strategy and the preheating timing to obtain a target adjustment result, enabling the inference service to perform information inference under the target adjustment result.

[0072] In some implementations, the time-series prediction model includes a time-series sharding embedding layer, a multi-head self-attention layer, an external signal fusion layer, and a multi-step prediction output layer. During the process of determining the traffic prediction curve corresponding to the hotspot event information based on the operational indicators and the hotspot event information using the time-series prediction model, the data prediction module 202 performs the following: Extract multi-dimensional time-series features from historical data, including periodic features, trend features, suddenness features, and correlation features; The multi-dimensional temporal features are input into the time series segmentation embedding layer and divided into segmented temporal features of fixed length; The target dependency features are obtained by capturing long-short-term dependencies from the segmented temporal features using the multi-head self-attention layer. The burst features and the target dependency features are input together into the external signal fusion layer and fused through a cross-attention mechanism to obtain the target fusion features; The multi-step prediction output layer is used to train the model with the target fusion features and historical traffic information to obtain the initial model parameters. After the model is deployed, an online learning strategy is used to continuously collect prediction error feedback and incrementally update the initial model parameters to obtain target model parameters that adapt to the latest traffic distribution. When the hotspot event information is detected, the traffic prediction curve is obtained by combining the operation indicators with the target model parameters to perform traffic prediction.

[0073] In some implementations, after obtaining the traffic prediction curve, the data prediction module 202 also performs the following: The historical data is obtained and labeled to identify the occurrence time and traffic impact pattern of breaking news events. Through comparative learning, the parameters of the target model are adjusted to learn the typical traffic impact curves corresponding to different types of news events. When the hot topic event information is detected, the hot topic event information and the breaking news event are matched to obtain the matching event information corresponding to the hot topic event information; Obtain the relevant impact curve corresponding to the matching event information from the typical flow impact curve; The traffic prediction curve is superimposed and corrected based on the relevant impact curve to generate the traffic prediction curve after the hotspot event information is corrected.

[0074] In some implementations, the quantity determination module 203 performs the following steps during the process of determining the target number of target instances required at different time points by combining the future request volume with the single-instance processing capacity corresponding to the inference service: Based on the single instance processing capacity and the future request volume, an optimization function is constructed with the goal of minimizing the number of instances and constrained by service quality and resource utilization. The single instance processing capacity includes the maximum number of concurrent inference requests, the preset queuing tolerance time for requests, the limitation of GPU memory capacity on batch size, and the resource consumption differences between different types of inference tasks. The number of target instances is obtained by solving the optimal solution of the optimization function at different time points.

[0075] In some implementations, the policy generation module 205, during the process of determining the scaling policy of the inference service corresponding to the hotspot event information based on the future request volume and the confidence level, performs the following: The corresponding traffic change magnitude is determined by the future request volume, and the rising state of the inference service under the hot event information is determined based on the traffic change magnitude and the confidence level. When the traffic change is less than a first preset value and the confidence level is greater than a second preset value, the rising state is determined to be a mild rise, and the scaling strategy is to gradually expand capacity according to the target number and prioritize waking up dormant warm instances to carry the load. When the traffic change is greater than or equal to the first preset value and the confidence level is greater than the second preset value, the rising state is determined to be a rapid rise, and the scaling strategy is to preheat new instances in batches according to the target number, while immediately waking up all warm instances to cope with sudden traffic. When the change in traffic volume is greater than or equal to the first preset value and the confidence level is less than the second preset value, the rising state is determined to be an uncertain rise, and the scaling strategy is to preheat a preset number of instances to a warm standby state, and then quickly activate them after the actual traffic arrives.

[0076] In some embodiments, the capacity adjustment module 206 performs capacity adjustment processing on the container corresponding to the inference service according to the scaling strategy and the preheating timing to obtain a target adjustment result, so that the inference service performs information inference under the target adjustment result, and executes the following: Based on the scaling strategy and the preheating timing, the capacity of the container corresponding to the inference service is adjusted to obtain the initial adjustment result; Once the new instance in the initial adjustment result has started up and passed the health check, a preset weight is assigned to the new instance. Based on the preset weight, the traffic gateway uses a weighted round-robin method to allocate relevant requests to the new instance. As the load on the new instance gradually stabilizes, the preset weight of the new instance is gradually increased by a preset step size or time interval, while the weight information of other instances is reduced simultaneously, until the new traffic is smoothly migrated to the new instance, thus obtaining the target adjustment result.

[0077] In some implementations, the inference service capacity adjustment device 200 based on traffic prediction also performs: When the future request volume continues to decrease, a tiered hibernation strategy is adopted to reclaim the capacity of the inference service. The tiered hibernation strategy includes a first-level strategy, a second-level strategy, a third-level strategy, and a fourth-level strategy. The first-level strategy is to reduce the size of the inference batch to free up some GPU memory. If there are still excess resources, the second-level strategy is adopted. The second-level strategy is to put the target instance into a warm hibernation state so that it can be restored within seconds. The third-level strategy is to unload the model corresponding to the inference service, release GPU resources, and put the target instance into a cold hibernation state, increasing the recovery time to minutes. The fourth-level strategy is to completely reclaim the target instance and release all expanded resources.

[0078] In some implementations, the inference service capacity adjustment device 200 based on traffic prediction can be applied to terminal devices.

[0079] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the inference service capacity adjustment device 200 based on traffic prediction described above can be referred to the corresponding process in the aforementioned embodiment of the inference service capacity adjustment method based on traffic prediction, and will not be repeated here.

[0080] Please see Figure 4 , Figure 4 This is a schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0081] like Figure 4 As shown, the terminal device 300 includes a processor 301 and a memory 302, which are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0082] Specifically, processor 301 provides computing and control capabilities to support the operation of the entire terminal device. Processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0083] Specifically, the memory 302 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0084] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the embodiments of the present invention, and does not constitute a limitation on the terminal device to which the embodiments of the present invention are applied. A specific server may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0085] The processor is used to run a computer program stored in a memory, and when executing the computer program, implements any of the inference service capacity adjustment methods based on traffic prediction provided in the embodiments of the present invention.

[0086] In one embodiment, the processor is configured to run a computer program stored in memory, and when executing the computer program, perform the following steps: The system collects operational metrics corresponding to the inference service and monitors and integrates dynamic events of multi-source external signals in the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. A time-series prediction model is used to determine the traffic prediction curve corresponding to the hotspot event information based on the operational indicators and the hotspot event information. The future request volume and the corresponding confidence level are predicted based on the traffic prediction curve; and the target number of target instances required at different time points is determined by combining the future request volume with the single instance processing capacity of the inference service. The traffic arrival time is determined based on the target quantity, and the preheating timing of the inference service under the hot event information is determined based on the traffic arrival time and the startup time of the target instance. The scaling strategy for the inference service under the hot event information is determined based on the future request volume and the confidence level. The capacity of the container corresponding to the inference service is adjusted according to the scaling strategy and the warm-up timing to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result.

[0087] In some implementations, the time-series prediction model includes a time-series sharding embedding layer, a multi-head self-attention layer, an external signal fusion layer, and a multi-step prediction output layer. During the process of determining the traffic prediction curve corresponding to the hotspot event information based on the operating metrics and the hotspot event information using the time-series prediction model, the processor 301 executes: Extract multi-dimensional time-series features from historical data, including periodic features, trend features, suddenness features, and correlation features; The multi-dimensional temporal features are input into the time series segmentation embedding layer and divided into segmented temporal features of fixed length; The target dependency features are obtained by capturing long-short-term dependencies from the segmented temporal features using the multi-head self-attention layer. The burst features and the target dependency features are input together into the external signal fusion layer and fused through a cross-attention mechanism to obtain the target fusion features; The multi-step prediction output layer is used to train the model with the target fusion features and historical traffic information to obtain the initial model parameters. After the model is deployed, an online learning strategy is used to continuously collect prediction error feedback and incrementally update the initial model parameters to obtain target model parameters that adapt to the latest traffic distribution. When the hotspot event information is detected, the traffic prediction curve is obtained by combining the operation indicators with the target model parameters to perform traffic prediction.

[0088] In some implementations, after obtaining the traffic prediction curve, the processor 301 also performs the following: The historical data is obtained and labeled to identify the occurrence time and traffic impact pattern of breaking news events. Through comparative learning, the parameters of the target model are adjusted to learn the typical traffic impact curves corresponding to different types of news events. When the hot topic event information is detected, the hot topic event information and the breaking news event are matched to obtain the matching event information corresponding to the hot topic event information; Obtain the relevant impact curve corresponding to the matching event information from the typical flow impact curve; The traffic prediction curve is superimposed and corrected based on the relevant impact curve to generate the traffic prediction curve after the hotspot event information is corrected.

[0089] In some implementations, during the process of determining the target number of target instances required at different time points by combining the future request volume with the single-instance processing capacity corresponding to the inference service, the processor 301 executes: Based on the single instance processing capacity and the future request volume, an optimization function is constructed with the goal of minimizing the number of instances and constrained by service quality and resource utilization. The single instance processing capacity includes the maximum number of concurrent inference requests, the preset queuing tolerance time for requests, the limitation of GPU memory capacity on batch size, and the resource consumption differences between different types of inference tasks. The number of target instances is obtained by solving the optimal solution of the optimization function at different time points.

[0090] In some implementations, during the process of determining the scaling strategy corresponding to the inference service under the hotspot event information based on the future request volume and the confidence level, the processor 301 executes: The corresponding traffic change magnitude is determined by the future request volume, and the rising state of the inference service under the hot event information is determined based on the traffic change magnitude and the confidence level. When the traffic change is less than a first preset value and the confidence level is greater than a second preset value, the rising state is determined to be a mild rise, and the scaling strategy is to gradually expand capacity according to the target number and prioritize waking up dormant warm instances to carry the load. When the traffic change is greater than or equal to the first preset value and the confidence level is greater than the second preset value, the rising state is determined to be a rapid rise, and the scaling strategy is to preheat new instances in batches according to the target number, while immediately waking up all warm instances to cope with sudden traffic. When the change in traffic volume is greater than or equal to the first preset value and the confidence level is less than the second preset value, the rising state is determined to be an uncertain rise, and the scaling strategy is to preheat a preset number of instances to a warm standby state, and then quickly activate them after the actual traffic arrives.

[0091] In some implementations, during the process of adjusting the capacity of the container corresponding to the inference service according to the scaling strategy and the warm-up timing to obtain a target adjustment result, so that the inference service performs information inference under the target adjustment result, the processor 301 executes: Based on the scaling strategy and the preheating timing, the capacity of the container corresponding to the inference service is adjusted to obtain the initial adjustment result; Once the new instance in the initial adjustment result has started up and passed the health check, a preset weight is assigned to the new instance. Based on the preset weight, the traffic gateway uses a weighted round-robin method to allocate relevant requests to the new instance. As the load on the new instance gradually stabilizes, the preset weight of the new instance is gradually increased by a preset step size or time interval, while the weight information of other instances is reduced simultaneously, until the new traffic is smoothly migrated to the new instance, thus obtaining the target adjustment result.

[0092] In some implementations, processor 301 also performs: When the future request volume continues to decrease, a tiered hibernation strategy is adopted to reclaim the capacity of the inference service. The tiered hibernation strategy includes a first-level strategy, a second-level strategy, a third-level strategy, and a fourth-level strategy. The first-level strategy is to reduce the size of the inference batch to free up some GPU memory. If there are still excess resources, the second-level strategy is adopted. The second-level strategy is to put the target instance into a warm hibernation state so that it can be restored within seconds. The third-level strategy is to unload the model corresponding to the inference service, release GPU resources, and put the target instance into a cold hibernation state, increasing the recovery time to minutes. The fourth-level strategy is to completely reclaim the target instance and release all expanded resources.

[0093] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the terminal device described above can be referred to the corresponding process in the aforementioned embodiment of the inference service capacity adjustment method based on traffic prediction, and will not be repeated here.

[0094] This invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the steps of any of the inference service capacity adjustment methods based on traffic prediction as provided in the specification of this invention.

[0095] The storage medium can be an internal storage unit of the terminal device described in the foregoing embodiments, such as the hard drive or memory of the terminal device. Alternatively, the storage medium can be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device.

[0096] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0097] It should be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0098] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The above descriptions are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for adjusting the capacity of inference services based on traffic prediction, characterized in that, include: The system collects operational metrics corresponding to the inference service and monitors and integrates dynamic events of multi-source external signals in the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. A time-series prediction model is used to determine the traffic prediction curve corresponding to the hotspot event information based on the operational indicators and the hotspot event information. The future request volume and the corresponding confidence level are predicted based on the traffic prediction curve; and the target number of target instances required at different time points is determined by combining the future request volume with the single instance processing capacity of the inference service. The traffic arrival time is determined based on the target quantity, and the preheating timing of the inference service under the hot event information is determined based on the traffic arrival time and the startup time of the target instance. The scaling strategy for the inference service under the hot event information is determined based on the future request volume and the confidence level. The capacity of the container corresponding to the inference service is adjusted according to the scaling strategy and the warm-up timing to obtain the target adjustment result, so that the inference service can perform information inference under the target adjustment result.

2. The method according to claim 1, characterized in that, The time-series prediction model includes a time-series segmentation embedding layer, a multi-head self-attention layer, an external signal fusion layer, and a multi-step prediction output layer. The step of using the time-series prediction model to determine the traffic prediction curve corresponding to the hotspot event information based on the operational metrics and the hotspot event information includes: Extract multi-dimensional time-series features from historical data, including periodic features, trend features, suddenness features, and correlation features; The multi-dimensional temporal features are input into the time series segmentation embedding layer and divided into segmented temporal features of fixed length; The target dependency features are obtained by capturing long-short-term dependencies from the segmented temporal features using the multi-head self-attention layer. The burst features and the target dependency features are input together into the external signal fusion layer and fused through a cross-attention mechanism to obtain the target fusion features; The multi-step prediction output layer is used to train the model with the target fusion features and historical traffic information to obtain the initial model parameters. After the model is deployed, an online learning strategy is used to continuously collect prediction error feedback and incrementally update the initial model parameters to obtain target model parameters that adapt to the latest traffic distribution. When the hotspot event information is detected, the traffic prediction curve is obtained by combining the operation indicators with the target model parameters to perform traffic prediction.

3. The method according to claim 2, characterized in that, After obtaining the traffic prediction curve, the method further includes: The historical data is obtained and labeled to identify the occurrence time and traffic impact pattern of breaking news events. Through comparative learning, the parameters of the target model are adjusted to learn the typical traffic impact curves corresponding to different types of news events. When the hot topic event information is detected, the hot topic event information and the breaking news event are matched to obtain the matching event information corresponding to the hot topic event information; Obtain the relevant impact curve corresponding to the matching event information from the typical flow impact curve; The traffic prediction curve is superimposed and corrected based on the relevant impact curve to generate the traffic prediction curve after the hotspot event information is corrected.

4. The method according to claim 1, characterized in that, The step of determining the target number of target instances required at different time points by combining the future request volume with the single-instance processing capacity corresponding to the inference service includes: Based on the single instance processing capacity and the future request volume, an optimization function is constructed with the goal of minimizing the number of instances and constrained by service quality and resource utilization. The single instance processing capacity includes the maximum number of concurrent inference requests, the preset queuing tolerance time for requests, the limitation of GPU memory capacity on batch size, and the resource consumption differences between different types of inference tasks. The number of target instances is obtained by solving the optimal solution of the optimization function at different time points.

5. The method according to claim 1, characterized in that, The step of determining the scaling strategy for the inference service under the hot event information based on the future request volume and the confidence level includes: The corresponding traffic change magnitude is determined by the future request volume, and the rising state of the inference service under the hot event information is determined based on the traffic change magnitude and the confidence level. When the traffic change is less than a first preset value and the confidence level is greater than a second preset value, the rising state is determined to be a mild rise, and the scaling strategy is to gradually expand capacity according to the target number and prioritize waking up dormant warm instances to carry the load. When the traffic change is greater than or equal to the first preset value and the confidence level is greater than the second preset value, the rising state is determined to be a rapid rise, and the scaling strategy is to preheat new instances in batches according to the target number, while immediately waking up all warm instances to cope with sudden traffic. When the change in traffic volume is greater than or equal to the first preset value and the confidence level is less than the second preset value, the rising state is determined to be an uncertain rise, and the scaling strategy is to preheat a preset number of instances to a warm standby state, and then quickly activate them after the actual traffic arrives.

6. The method according to claim 1, characterized in that, The step of adjusting the capacity of the container corresponding to the inference service according to the scaling strategy and the preheating timing to obtain a target adjustment result, so that the inference service can perform information inference under the target adjustment result, includes: Based on the scaling strategy and the preheating timing, the capacity of the container corresponding to the inference service is adjusted to obtain the initial adjustment result; Once the new instance in the initial adjustment result has started up and passed the health check, a preset weight is assigned to the new instance. Based on the preset weight, the traffic gateway uses a weighted round-robin method to allocate relevant requests to the new instance. As the load on the new instance gradually stabilizes, the preset weight of the new instance is gradually increased by a preset step size or time interval, while the weight information of other instances is reduced simultaneously, until the new traffic is smoothly migrated to the new instance, thus obtaining the target adjustment result.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: When the future request volume continues to decrease, a tiered hibernation strategy is adopted to reclaim the capacity of the inference service. The tiered hibernation strategy includes a first-level strategy, a second-level strategy, a third-level strategy, and a fourth-level strategy. The first-level strategy is to reduce the size of the inference batch to free up some GPU memory. If there are still excess resources, the second-level strategy is adopted. The second-level strategy is to put the target instance into a warm hibernation state so that it can be restored within seconds. The third-level strategy is to unload the model corresponding to the inference service, release GPU resources, and put the target instance into a cold hibernation state, increasing the recovery time to minutes. The fourth-level strategy is to completely reclaim the target instance and release all expanded resources.

8. A capacity adjustment device for inference services based on traffic prediction, characterized in that, The device includes: The data acquisition module is used to collect the operational metrics corresponding to the inference service, and to monitor and fuse dynamic events of multi-source external signals corresponding to the target domain in real time to obtain hot event information. The operational metrics include real-time request volume, request queue depth, average response latency, GPU utilization, and video memory usage. The data prediction module is used to determine the traffic prediction curve corresponding to the hot event information based on the operating indicators and the hot event information using a time series prediction model; The quantity determination module is used to predict the future request volume and the confidence level corresponding to the future request volume based on the traffic prediction curve; and to use the future request volume in combination with the single instance processing capacity corresponding to the inference service to determine the target number of target instances required at different time points. The timing determination module is used to determine the traffic arrival time based on the target quantity, and to determine the preheating timing of the inference service under the hot event information based on the traffic arrival time and the startup time of the target instance. The strategy generation module is used to determine the scaling strategy of the inference service under the hot event information based on the future request volume and the confidence level. The capacity adjustment module is used to adjust the capacity of the container corresponding to the inference service according to the scaling strategy and the preheating time, and obtain the target adjustment result so that the inference service can perform information inference under the target adjustment result.

9. A terminal device, characterized in that, The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and, in executing the computer program, implement the inference service capacity adjustment method based on traffic prediction as described in any one of claims 1 to 7.

10. A computer storage medium for computer storage, characterized in that, The computer storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the inference service capacity adjustment method based on traffic prediction as described in any one of claims 1 to 7.

Images