Method and device for reducing response time delay of deep learning model, and storage medium
Through the combination of sliding window sampling and polynomial regression model with LSTM model, combined with the adjustment of error compensation value, the refinement problems of workload prediction and resource scheduling of large-scale deep learning models are solved, and the effect of reducing response delay and improving service level goals is achieved.
Patent Information
- Application Number
- CN202510249305.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-06-10
- Estimated Expiration
- 2045-03-04
AI Technical Summary
The prior art cannot accurately predict the workload of large-scale deep learning models, especially the inability to handle burst fluctuations in the workload, resulting in the resource allocation strategy being not refined enough to reduce the response delay of the model's real-time inference service while ensuring cost-effectiveness.
The historical load data is obtained through sliding window sampling, combined with the pre-built polynomial regression model and the long and short-term memory network LSTM model, the initial load prediction data sequence is obtained, the prediction data is adjusted through error compensation value, the future calculation throughput and the total number of required instances is determined, the current number of instances is adjusted to match the required number, and the response delay is reduced.
Accurate prediction and resource scheduling of large-scale deep learning model workloads is realized, effectively reducing the response delay of model inference, improving the achievement rate of service level goals, and reducing service costs.
Smart Images

Figure CN120123093A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method, device and storage medium for reducing the response latency of a deep learning model. Background Art
[0002] In a cloud computing environment, large-scale deep learning models play a key role in providing advanced intelligent services. Due to their huge number of parameters and complex structures, these models have extremely high demands on computing resources. In addition, since the inference service workload changes dynamically and unpredictably with the user access volume, in a cloud computing real-time inference environment, how to reduce the inference service latency of large-scale deep learning models to meet the Service Level Objective (SLO) is a complex and challenging problem.
[0003] Existing inference systems such as Cocktail and Mark have explored multi-dimensional optimizations (such as using preemptive instances and serverless technologies) to improve inference efficiency. However, these methods cannot accurately predict the workload, especially cannot handle the sudden fluctuations in the workload, resulting in a lack of fineness in the resource allocation strategy and being unable to reduce the response latency of the real-time inference service of large-scale deep learning models while ensuring cost-effectiveness. Summary of the Invention
[0004] In view of this, the purpose of the present invention is to provide a method, device and storage medium for reducing the response latency of a deep learning model, so as to solve the problem in the prior art that the workload cannot be accurately predicted, especially the sudden fluctuations in the workload cannot be handled, resulting in a lack of fineness in the resource allocation strategy and being unable to reduce the response latency of the real-time inference service of large-scale deep learning models while ensuring cost-effectiveness.
[0005] According to the first aspect of the embodiments of the present invention, a method for reducing the response latency of a deep learning model is provided. The method includes: Obtaining a historical load data sequence of the inference work of a large-scale deep learning model through a sliding window sampling method; Based on the historical load data sequence, obtaining an initial load prediction data sequence of the inference work of a large-scale deep learning model through a pre-built polynomial regression model and a long short-term memory network (LSTM) model; Obtaining a corresponding actual load data sequence of the initial load prediction data sequence, and obtaining a workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding actual load data sequence; Obtaining an error compensation value based on the workload fluctuation data sequence; Compensate the initial load prediction data sequence with the error compensation value to obtain the final load prediction data sequence; Determine the required computing throughput within a future period based on the final load prediction data sequence, and obtain the total number of instances required within the future period according to the throughput of a single instance and the required computing throughput within the future period; Reduce the response latency of large-scale deep learning model inference work by adjusting the number of currently running instances to match the total number of instances required within a future period.
[0006] Preferably, The obtaining of the initial load prediction data sequence of the large-scale deep learning model inference work through the pre-established polynomial regression model and long short-term memory network (LSTM) model includes: Establish a polynomial regression model and initialize the polynomial function; build a long short-term memory network (LSTM) model; Fit the historical load data sequence to a polynomial function using the least squares method, and update the polynomial regression model through the fitted polynomial function; Train and update the long short-term memory network (LSTM) model with the historical load data sequence; Predict the workload of the large-scale deep learning model inference work through the updated polynomial regression model and long short-term memory network (LSTM) model respectively; Fuse the prediction results of the updated polynomial regression model and the prediction results of the long short-term memory network (LSTM) model to obtain the initial load prediction data sequence.
[0007] Preferably, The fusing of the prediction results of the updated polynomial regression model and the prediction results of the long short-term memory network (LSTM) model to obtain the initial load prediction data sequence includes: Obtain the first mean square error of the most recent historical prediction result of the updated polynomial regression model, and obtain the second mean square error of the prediction result of the long short-term memory network (LSTM) model during the same period; Obtain the output weight of the polynomial regression model and the output weight of the long short-term memory network (LSTM) model respectively according to the first mean square error and the second mean square error; Weightedly fuse the prediction results of the updated polynomial regression model and the long short-term memory network (LSTM) model according to their respective weights to obtain the initial load prediction data sequence.
[0008] Preferably, Obtaining the workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding actual load data sequence includes: Subtracting the initial load prediction data sequence from the corresponding actual load data sequence at the same time point to obtain the workload fluctuation data sequence.
[0009] Preferably, Obtaining the error compensation value based on the workload fluctuation data sequence includes: Obtaining the 95th percentile value of the workload fluctuation data sequence, and using the 95th percentile value of the workload fluctuation data sequence as the error compensation value.
[0010] Preferably, Adjusting the number of currently running instances to match the total number of instances required for a future period includes: When it is predicted that the final load prediction data is in an upward trend and the difference from the preset load peak is less than the preset threshold, increasing the number of currently running instances or adjusting the batch size; When it is predicted that the final load prediction data is in a downward trend and is always less than the preset load peak, reducing the number of currently running instances.
[0011] According to the second aspect of the embodiments of the present invention, there is provided a device for reducing the response latency of a deep learning model. The device includes: A historical load acquisition module: used to obtain a historical load data sequence of large-scale deep learning model inference work through a sliding window sampling method; A load prediction module: used to obtain an initial load prediction data sequence of large-scale deep learning model inference work based on the historical load data sequence through a pre-built polynomial regression model and a long short-term memory network (LSTM) model; A fluctuation sequence acquisition module: used to obtain the corresponding actual load data sequence of the initial load prediction data sequence, and obtain the workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding actual load data sequence; A compensation value acquisition module: used to obtain an error compensation value based on the workload fluctuation data sequence; A predicted load compensation module: used to perform error compensation on the initial load prediction data sequence through the error compensation value to obtain a final load prediction data sequence; A future instance acquisition module: used to determine the required computing throughput within a future period based on the final load prediction data sequence, and obtain the total number of instances required for a future period according to the throughput per unit instance and the required computing throughput within a future period; Adjustment module: It is used to reduce the response latency of the inference work of large-scale deep learning models by adjusting the number of currently running instances to match the total number of instances required in a future period of time.
[0012] According to the third aspect of the embodiments of the present invention, a storage medium is provided. The storage medium stores a computer program, and when the computer program is executed by a main controller, each step in the above method is implemented.
[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: In this application, historical load is obtained through sliding window sampling. Based on the historical load, and a pre-built polynomial regression model and a long short-term memory network (LSTM) model, an initial load prediction data sequence is obtained; the workload fluctuation is obtained through the corresponding actual workload data sequence, and an error compensation value is obtained based on the workload fluctuation. The initial load prediction data is error-compensated by the error compensation value to obtain the final load prediction data; the required computing throughput in a future period of time is determined based on the final load prediction data sequence, and the total number of instances required in a future period of time is obtained according to the throughput per unit instance and the required computing throughput in a future period of time; by adjusting the number of currently running instances to match the total number of instances required in a future period of time, the response latency of the inference work of large-scale deep learning models is reduced; through accurate workload prediction and resource scheduling, this application can effectively reduce the response latency of model inference, improve the achievement rate of service level objectives, and reduce service costs.
[0014] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. Brief Description of the Drawings
[0015] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.
[0016] Figure 1 is a schematic flowchart of a method for reducing the response latency of a deep learning model shown according to an exemplary embodiment; Figure 2 is a schematic system diagram of a device for reducing the response latency of a deep learning model shown according to another exemplary embodiment; In the drawings: 1 - Historical load acquisition module, 2 - Load prediction module, 3 - Fluctuation sequence acquisition module, 4 - Compensation value acquisition module, 5 - Predicted load compensation module, 6 - Future instance acquisition module, 7 - Adjustment module. Detailed Embodiments
[0017] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0018] Embodiment 1 Figure 1 is a schematic flowchart of a method for reducing the response latency of a deep learning model according to an exemplary embodiment, as Figure 1 shown. The method includes: S1. Obtain a historical load data sequence of large-scale deep learning model inference work through a sliding window sampling method; S2. Based on the historical load data sequence, obtain an initial load prediction data sequence of large-scale deep learning model inference work through a pre-built polynomial regression model and a long short-term memory network (LSTM) model; S3. Obtain the actual load data sequence corresponding to the initial load prediction data sequence, and obtain a workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding actual load data sequence; S4. Obtain an error compensation value based on the workload fluctuation data sequence; S5. Perform error compensation on the initial load prediction data sequence through the error compensation value to obtain a final load prediction data sequence; S6. Determine the required computing throughput within a future period based on the final load prediction data sequence, and obtain the total number of instances required within the future period according to the throughput per unit instance and the required computing throughput within the future period; S7. Reduce the response latency of large-scale deep learning model inference work by adjusting the number of currently running instances to match the total number of instances required within a future period; It can be understood that in this embodiment, a sliding window method is used to sample and obtain the historical data of the current large-scale deep learning model inference workload. Next, a dynamic joint prediction (DJP) mechanism is used to perform long-term trend prediction on the workload data, set the workload prediction interval, and regularly use the sliding window method to calculate the current workload and record it. Specifically, it includes: Establish a polynomial regression model and initialize the polynomial function; build a long short-term memory network (LSTM) model and use the initial data for model pre-training. Among them, the polynomial regression model can be a polynomial of any degree, and the long short-term memory network model can be an LSTM variant such as BiLSTM or Lineer LSTM; The historical workload is fitted to a polynomial function using the least squares method, and the polynomial regression model is updated; if the amount of historical workload data collected meets the requirements for one training of the LSTM model, the LSTM model is trained and updated once using this data. The polynomial regression model and the LSTM model are used to predict the workload respectively, and the sampling weights of the final workload prediction value are determined according to the recent errors of the two models, specifically including: Define the workload prediction time length as λ, and define that at time t, the predicted value of the workload at time t + λ by the polynomial regression model is and the predicted value of the workload at time t + λ by the LSTM model is ; Define the historical prediction mean squared errors MSE of the polynomial regression model and the LSTM model as and respectively, and calculate the weight of the workload prediction of the polynomial regression model:
[0019] Among them, at the initial stage of the model inference service, since there is no data for LSTM model training, the workload prediction weight is set to 1.0, that is, the prediction result of the polynomial regression model is completely adopted. Based on the sampling weights of the two models, obtain the initial workload prediction value at time t + λ:
[0020] As the inference service progresses, regularly recalculate the mean squared error MSE of the workload prediction values of the two models. The above steps can overcome the problems of large local errors and slow start-up existing in the existing system using a single polynomial regression model or LSTM model through the historical prediction errors of the two large model inference service workload prediction models; realize the dynamic weighted combination of the deep learning prediction method and the polynomial fitting prediction method, so as to achieve a basically accurate workload prediction value at any stage of the model inference service. Adopt the historical fluctuation compensation method, and provide a compensation factor to optimize the resource scheduling decision by analyzing the fluctuation intensity of the historical workload data, including: Obtain the true sequence corresponding to the above prediction sequence ; and subtract it from the initial workload prediction sequence predicted to obtain the workload fluctuation sequence , and the formula is:
[0021] Calculate the 95th percentile value of the fluctuation sequence as the error compensation for the workload prediction at time t + λ , thus avoiding insufficient resource allocation due to small fluctuations in the workload, which may lead to an increase in the model inference latency; According to the above steps, an accurate predicted value of the workload at time t + λ is obtained :
[0022] The quantile for workload error compensation can be adjusted within the range of 50 to 100 according to the requirements of the service level objective SLO; The above steps can overcome the problem that the existing workload prediction cannot handle instantaneous fluctuations in the workload, thus achieving accurate and precise predicted values of the workload, providing a basis for resource scheduling for large-scale deep learning model inference, and ensuring that the request response latency of the inference meets the service level objective SLO; To facilitate the understanding of the above solution, this embodiment briefly describes the above solution. Taking the initial current moment as "4", and the historical load sequences as "1", "2", "3" respectively, through the historical load sequences "1", "2", "3", based on the dynamic joint prediction mechanism DJP, the future load prediction sequences "5", "6", "7" are obtained. It should be noted that this process is dynamic over time. That is to say, as time progresses, the true loads "{5}", "{6}", "{7}" corresponding to the initial load prediction sequences "5", "6", "7" can be obtained. At this time, the dynamic joint prediction mechanism DJP has predicted the initial load prediction sequences "8", "9", "10". Subtracting the corresponding true loads "{5}", "{6}", "{7}" from the initial load prediction sequences "5", "6", "7" to obtain the fluctuation sequence, obtaining the error compensation value based on the fluctuation sequence, and compensating the error of the initial load prediction sequences "8", "9", "10" through this compensation value to obtain the accurate predicted values of the workload at times "8", "9", "10" ; Dynamically adjust the resource allocation strategy according to the accurate predicted value of the long-term trend, including: Determine the required computing throughput within a future period according to the accurate predicted value of the workload, and obtain the total number of required instances according to the throughput of a single instance (an instance can be understood as a server, and the more the number of instances, the more load can be digested); according to the current resource usage situation and the prediction of the total resources of the system, adjust the number of running service instances to match the predicted workload demand; when it is predicted that the workload will peak, increase the number of service instances or adjust the batch size to improve the processing capacity of the system to avoid exceeding the latency standard; when it is predicted that the workload will decrease, actively reduce the number of service instances to save resources and reduce costs; This embodiment will also adjust the configuration parameters of the model (including batch size and the number of parallel model replicas) according to the performance metrics of the service instance (including CPU utilization, GPU utilization, and memory usage) and the performance requirements of model inference, so as to optimize the inference efficiency and response latency; It is worth emphasizing that, in order to solve the model cold start problem and request response timeout problem in large-scale deep learning model inference, and thus reduce the response latency of the model real-time inference service, this embodiment also provides a method for dynamic resource and request management, which specifically includes: (1) Model cold start optimizer AoT Tuner module. This module realizes fast model loading and initialization during runtime by pre-analyzing the optimal CUDA operators corresponding to each layer of the deep learning model, minimizing the cold start time. Specifically, it includes: Perform offline analysis on each deep learning model to determine the optimal CUDA operator corresponding to each layer of the model: Use NVIDIA's TensorRT framework for offline analysis of the model. TensorRT is an optimizer for high-performance deep learning inference, which can perform operations such as accuracy calibration, layer fusion, and operator optimization on the model; Traverse each layer of the large-scale deep learning model, try different combinations of CUDA operators, and record the execution time of each combination; Compare the inference execution times of each operator combination, select the CUDA operator with the shortest execution time as the optimal operator, and record its configuration parameters; Save the optimal CUDA operator and its configuration parameters for each layer in the optimized configuration file of the model, and this file is stored in the random access memory (RAM) for quick reading; Use the Tensorflow framework to cache the optimized model for quick loading during inference: Convert the model optimized by TensorRT into the SavedModel format supported by Tensorflow; Save the optimized model and the configuration file of the optimal CUDA operator together in the model repository of Tensorflow, and this repository can be deployed on a solid-state drive with high bandwidth and low latency; Optionally, in the model repository, maintain a version control mechanism for each model to ensure that when an inference request arrives, the latest or specified version of the model and configuration file can be quickly located; To improve the efficiency of caching, model compression techniques such as weight quantization and pruning can be adopted to reduce the size of the model while maintaining the inference performance; When an inference request arrives, directly pass the loaded model and configuration file to the GPU, and use NVIDIA's CUDA technology for quick initialization, including memory allocation and operator binding, to reduce the model loading and initialization time; Through the above step (1), the AoT Tuner mechanism can ensure that when an inference request arrives, the model can be quickly loaded and initialized, reducing the latency caused by model loading and initialization, and improving the response speed and overall performance of large-scale deep learning model inference services.
[0023] (2) The Adaptive Batch Drop method for dynamic request management monitors the system resource status and request latency in real time during the inference process. According to the current resource status and the latency requirements of the requests, it dynamically decides whether to execute the batch processing of the requests, including: Define the system resource status R(t) as the total amount of available computing resources in the system at time t, including the number of GPU cores and the memory capacity; define the request latency Di as the time required for the i-th request to be processed from submission to completion; continuously track and record the values of R(t) and Di for dynamic scheduling decisions.
[0024] Dynamically decide whether to execute the batch processing of the requests according to the current resource status and the latency requirements of the requests: for each upcoming request batch, calculate its estimated completion time :
[0025] where, n is the number of requests in the batch, W j is the j -th request weight, T j is the j -th request's estimated execution time, R(t) is the current system resource status.
[0026] When the system resources are tight and cannot meet the latency requirements of all requests, according to the preset discarding rules, selectively discard some requests: define the maximum allowable latency Dmax as the latency upper limit set by the system according to the service level objective SLO requirements; for each request i , if D i > D max , then it is considered that this request has a timeout risk; according to the priority P i and latency sensitivity S i calculate the discard probability P drop ( i ):
[0027] where, g (P i ) and h ( S i ) are predefined functions used to adjust the discard probability according to the request priority and latency sensitivity.
[0028] Optimize the overall request processing flow by adjusting the batch size and execution order: Dynamically adjust the batch size according to the current resource status R(t) and the priority information in the request queue to maximize resource utilization and meet latency requirements; Define a priority queue in which each request is sorted according to its priority and latency requirements; When the system resources are insufficient to process all requests, according to the request discard probability P drop ( i ) and the priority queue, selectively discard some requests while ensuring that critical requests are processed first; For the discarded requests, you can choose to requeue or directly return an error response, and the specific strategy depends on the business requirements of the system and the SLO requirements.
[0029] Through the above Adaptive Batch Drop method, this embodiment can effectively manage resources and reduce timeout requests caused by queuing while ensuring the quality of service of critical requests, thereby improving the overall performance of large-scale deep learning model inference services and the inference service response latency.
[0030] Embodiment 2: Figure 2 It is a system schematic diagram of a device for reducing the response latency of a deep learning model shown according to another exemplary embodiment. The device includes: Historical load acquisition module 1: Used to obtain the historical load data sequence of large-scale deep learning model inference work through the sliding window sampling method; Load prediction module 2: Used to obtain the initial load prediction data sequence of large-scale deep learning model inference work based on the historical load data sequence through a pre-built polynomial regression model and a long short-term memory network LSTM model; Fluctuation sequence acquisition module 3: Used to obtain the actual load data sequence corresponding to the initial load prediction data sequence, and obtain the workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding actual load data sequence; Compensation value acquisition module 4: Used to obtain the error compensation value based on the workload fluctuation data sequence; Predicted load compensation module 5: Used to perform error compensation on the initial load prediction data sequence through the error compensation value to obtain the final load prediction data sequence; Future instance acquisition module 6: It is used to determine the required computing throughput within a future period of time based on the final load prediction data sequence, and obtain the total number of instances required within the future period of time according to the throughput of a single instance and the required computing throughput within the future period of time; Adjustment module 7: It is used to adjust the number of currently running instances to match the total number of instances required within a future period of time, thereby reducing the response latency of the inference work of large-scale deep learning models.
[0031] Embodiment 3: This embodiment provides a storage medium, which stores a computer program. When the computer program is executed by a main controller, it implements each step in the above method; It can be understood that the above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disc, etc.
[0032] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be referred to the same or similar content in other embodiments.
[0033] It should be noted that in the description of the present invention, terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise stated, the meaning of "sparsely distributed in small amounts" means at least two.
[0034] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of executable instructions including one or more sparsely distributed steps for implementing a specific logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0035] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the sparsely distributed steps or methods in small amounts can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.
[0036] Those of ordinary skill in the art can understand that all or part of the steps carried out in implementing the above-described embodiment methods can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiment.
[0037] In addition, in each of the embodiments of the present invention, the functional units can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0038] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, or the like.
[0039] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or a small number of scattered embodiments or examples.
[0040] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for reducing the response delay of a deep learning model, characterized in that: The method comprises: Obtain historical load data sequences for large-scale deep learning model inference work through a sliding window sampling method; Based on the historical load data sequence, an initial load forecast data sequence for large-scale deep learning model reasoning is obtained through a pre-built polynomial regression model and a long short-term memory network LSTM model; Obtaining a load actual data sequence corresponding to an initial load prediction data sequence, and obtaining a workload fluctuation data sequence based on the initial load prediction data sequence and the corresponding load actual data sequence; acquiring an error compensation value based on the workload fluctuation data sequence; Performing error compensation on the initial load prediction data sequence by using the error compensation value to obtain a final load prediction data sequence; Determine the required computing throughput in the future period based on the final load prediction data sequence, and obtain the total number of instances required in the future period according to the throughput of the unit instance and the required computing throughput in the future period; By adjusting the number of currently running instances to match the total number of instances required in the future, the response latency of large-scale deep learning model inference work can be reduced.
2. The method according to claim 1, characterized in that The initial load forecasting data sequence for large-scale deep learning model reasoning work obtained by using the pre-built polynomial regression model and the long short-term memory network LSTM model includes: Establish a polynomial regression model and initialize the polynomial function; build a long short-term memory network LSTM model; Fitting the historical load data sequence into a polynomial function using the least squares method, and updating the polynomial regression model using the fitted polynomial function; Training and updating the long short-term memory network LSTM model through the historical load data sequence; The workload of the large-scale deep learning model inference work is predicted respectively by using the updated polynomial regression model and the long short-term memory network LSTM model; The prediction results of the updated polynomial regression model and the prediction results of the long short-term memory network LSTM model are merged to obtain the initial load prediction data sequence.
3. The method according to claim 2, characterized in that The prediction results of the updated polynomial regression model and the prediction results of the long short-term memory network LSTM model are integrated to obtain the initial load prediction data sequence, including: Get the first mean square error of the most recent historical prediction results of the updated polynomial regression model, and get the second mean square error of the prediction results of the long short-term memory network LSTM model in the same time period; According to the first mean square error and the second mean square error, the output weight of the polynomial regression model and the output weight of the long short-term memory network LSTM model are respectively obtained; The prediction results of the updated polynomial regression model and the long short-term memory network LSTM model are weightedly fused according to their respective weights to obtain the initial load prediction data sequence.
4. The method according to claim 3, characterized in that: The step of obtaining a workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding load actual data sequence comprises: The workload fluctuation data sequence is obtained by subtracting the initial load prediction data sequence from the corresponding load actual data sequence at the same time point.
5. The method according to claim 4, characterized in that The acquiring the error compensation value based on the workload fluctuation data sequence comprises: The 95th percentile value of the workload fluctuation data sequence is obtained, and the 95th percentile value of the workload fluctuation data sequence is used as the error compensation value.
6. The method according to claim 5, characterized in that The method of adjusting the number of currently running instances to match the total number of instances required in the future includes: When it is predicted that the final load forecast data is on an upward trend and the difference with the preset load peak is less than a preset threshold, the number of currently running instances is increased or the batch size is adjusted; When it is predicted that the final load forecast data is on a downward trend and is always smaller than the preset load peak, the number of currently running instances is reduced.
7. A device for reducing the response delay of a deep learning model, characterized in that: The device comprises: Historical load collection module: used to obtain historical load data sequences for large-scale deep learning model inference work through sliding window sampling method; Load forecasting module: used to obtain the initial load forecasting data sequence for large-scale deep learning model reasoning work through a pre-built polynomial regression model and a long short-term memory network LSTM model based on the historical load data sequence; Fluctuation sequence acquisition module: used to acquire the load actual data sequence corresponding to the initial load prediction data sequence, and acquire the workload fluctuation data sequence according to the initial load prediction data sequence and the corresponding load actual data sequence; A compensation value acquisition module: used for acquiring an error compensation value based on the workload fluctuation data sequence; Prediction load compensation module: used to perform error compensation on the initial load prediction data sequence by using the error compensation value to obtain a final load prediction data sequence; A future instance acquisition module is used to determine the required computing throughput in the future period based on the final load prediction data sequence, and obtain the total number of instances required in the future period according to the throughput of the unit instance and the computing throughput required in the future period; Adjustment module: It is used to reduce the response latency of large-scale deep learning model inference work by adjusting the number of currently running instances to match the total number of instances required in the future.
8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the main controller, the various steps in the method for reducing the response delay of a deep learning model as described in any one of claims 1-6 are implemented.
Citation Information
Patent Citations
Heading machine performance prediction method and device based on deep learning
CN118428409A
Dynamic resource scheduling method and device for multi-GPU cluster reasoning service
CN119311396A
Edge prediction method based on deep autoregressive recurrent neural network
LU503318B1
Service network employing reinforcement learning based global load balancing
US20250071064A1
Cited By
Batch processing method and device based on request data, electronic equipment and storage medium
CN120994723A
Data processing method and system
CN121458523A
Wireless network performance optimization method
CN122205464A
