Large model resource optimization method and device based on flow distribution, electronic equipment and computer readable medium

Abstract

The invention relates to a large model resource optimization method and device based on flow distribution, electronic equipment and a computer readable medium. The large model resource optimization method based on flow distribution comprises the steps that flow data including key indexes are monitored in real time, and the key indexes include one or more of the request queue length, the response time and the resource utilization rate; traffic prediction is carried out based on traffic data through the multi-level time sequence model; when it is monitored that the flow reaches the flow threshold value or it is predicted that the flow needs to reach the flow threshold value, the priority of the request is determined; and allocating different requests to different resource pools according to the priorities of the requests.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence large model services, and particularly relates to a method and device for optimizing large model resources based on traffic distribution, electronic equipment and a computer readable medium. BACKGROUND

[0002] Artificial intelligence large models such as large language models (LLM) often face the problem of low resource use efficiency in the inference service process due to their large amount of computation and high resource demand. Specifically, large model inference services usually adopt a fixed resource allocation strategy, which has the following shortcomings when dealing with traffic fluctuations:

[0003] Firstly, during traffic peak periods, the system is prone to overload, resulting in prolonged service response time and decreased user experience. Secondly, during traffic trough periods, due to the limitations of the fixed resource allocation strategy, a large amount of computing resources are idle and cannot be effectively utilized, resulting in resource waste and reduced overall cost efficiency. Thirdly, the lag in resource expansion and contraction is also a major problem faced by current large model inference services. Since the loading of large models takes several minutes, when sudden traffic arrives, traditional expansion and contraction strategies often fail to respond in time, resulting in insufficient system flexibility.

[0004] In addition, the common load balancing methods in the prior art mainly distribute based on the number of requests or response time, without fully considering the particularity of large model inference services, such as the large difference in resource consumption of different complexity requests, the diversity of model service quality requirements, and other factors.

[0005] In summary, large model inference services face problems such as low resource use efficiency, and therefore a method for optimizing large model resources based on traffic distribution is needed to solve at least one of the above deficiencies. SUMMARY

[0006] Aspects of the present disclosure provide a method for optimizing large model resources based on traffic distribution.

[0007] Aspects of the present disclosure also provide a device for optimizing large model resources based on traffic distribution.

[0008] The first aspect of the present disclosure provides a method for optimizing large model resources based on traffic distribution, comprising: monitoring traffic data including key indicators in real time, wherein the key indicators include one or more of request queue length, response time and resource utilization; performing traffic prediction based on the traffic data through a multi-level time series model; determining the priority of requests when the monitored traffic reaches or is predicted to reach a traffic threshold; and allocating different requests to different resource pools according to the priority of the requests.

[0009] In an example embodiment, the traffic prediction based on the traffic data by the multi-level time series model comprises: generating time series data based on the traffic data; generating business event features based on a business event calendar; and outputting a result of the traffic prediction by the multi-level time series model based on the time series data and the business event features.

[0010] In an example embodiment, the method of large model resource optimization based on traffic splitting further comprises: performing one or more of the following based on the result of the traffic prediction: starting new instances in advance when the traffic is predicted to exceed the capacity threshold; automatically increasing or decreasing instances based on a threshold of load and a predicted trend; allocating different computing resources to different requests according to the complexity of the requests; and releasing redundant resources in stages when the load continues to be below a recovery threshold.

[0011] In an example embodiment, determining the priority of the request comprises: evaluating one or more of the complexity of the request, the context length, and the resource requirement, and determining the priority of the request based on one or more of the level of the service level agreement, the type of the request, and the current load and the result of the evaluation of the request.

[0012] In an example embodiment, the priority of the request comprises a high priority, a standard priority, and a low priority, and wherein allocating different requests to different resource pools comprises: allocating the high priority request to a dedicated resource pool; allocating the standard priority request to a shared resource pool; and allocating the low priority request to an elastic resource pool.

[0013] In an example embodiment, when the traffic is monitored to reach the traffic threshold or predicted to reach the traffic threshold, the standard priority request is routed to the shared resource pool by an intelligent routing algorithm, and / or the low priority request is delayed for processing or downgraded for processing when the load is greater than or equal to an extreme load threshold.

[0014] In an example embodiment, the method of large model resource optimization based on traffic splitting further comprises: when the load is greater than or equal to an extreme load threshold, performing one or more of the following: processing the standard priority request and / or the low priority request by a lightweight surrogate model; consolidating processing of similar multiple requests with a feature vector distance less than a predetermined threshold; turning off preset resource-intensive functions; routing the low priority request to a third-party application programming interface service when the resource utilization of the large model exceeds a resource utilization threshold; and restoring services in order according to the priority of the request when the load drops to a normal level.

[0015] The second aspect of the present disclosure provides a device for large model resource optimization based on traffic shunting, comprising: a traffic monitoring module configured to monitor traffic data including key indicators in real time, wherein the key indicators include one or more of request queue length, response time and resource utilization; a traffic prediction module configured to predict traffic based on traffic data through a multi-level time series model; a request priority determination module configured to determine the priority of a request when it is monitored that traffic reaches a traffic threshold or it is predicted that traffic will reach a traffic threshold; and a resource scheduling module configured to allocate different requests to different resource pools according to the priority of the requests.

[0016] The third aspect of the present disclosure provides an electronic device comprising a processor and a memory, the memory being configured to store a program, when the program is executed by the processor, the method as described above is executed.

[0017] The fourth aspect of the present disclosure provides a computer readable medium having a program stored thereon, when the program is executed by a processor, the method as described above is executed.

[0018] However, aspects of the present disclosure are not limited to the above aspects set forth herein. The above and other aspects of the present disclosure will become apparent to one of ordinary skill in the art by reference to the detailed description of the present disclosure given below.

[0019] Embodiments of the present disclosure aim to solve the problem of low resource use efficiency faced by large model inference services, for example, by designing a resource scheduling method based on peak traffic shunting, achieving the following effects: improving the overall resource use efficiency of large model inference services; reducing operating costs under the premise of ensuring service quality; effectively dealing with traffic fluctuations, especially peak traffic scenarios; and supporting dynamic scheduling of differentiated quality of service requirements. The above technical solutions of the present disclosure achieve one of the aforementioned effects, and do not require each technical solution to achieve all the technical effects described above.

[0020] In addition, the effects of the present disclosure not only include the effects set forth herein, but also include other effects apparent to those skilled in the art of the present disclosure by referring to the claims, the specification and the drawings accompanying the specification. BRIEF DESCRIPTION OF DRAWINGS

[0021] These and / or other aspects will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings in which:

[0022] Figure 1 is a flowchart of a method for large model resource optimization based on traffic shunting according to embodiments of the present disclosure;

[0023] Figure 2is a block diagram of an apparatus for traffic split based large model resource optimization according to an embodiment of the present disclosure;

[0024] Figure 3 is a block diagram of an electronic device according to an embodiment of the present disclosure; and

[0025] Figure 4 is a block diagram of a computer readable medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure will now be described more fully with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. The present disclosure may, however, be embodied in different forms, and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0027] It will be understood that the numerical values or numerical ranges referred to herein are merely examples, and the numerical values or numerical ranges can be any other values or other ranges, as long as the implementation of the technical solutions of the present application is not affected.

[0028] It will also be understood that, throughout the specification, the same reference numerals indicate the same components. Each of the features of the various embodiments of the present disclosure can be combined, partially or wholly, with each other, and various associations and drives are possible in the technical field. Each embodiment can be implemented independently of each other, or can be implemented together in association.

[0029] Unless otherwise defined or implied herein, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal sense, unless expressly so defined herein.

[0030] The present disclosure describes a method for improving the resource use efficiency of large model inference services through peak traffic split, but the present disclosure is not limited to the large model inference service scenario, for example, it can also be applied to other scenarios such as cloud computing, edge computing, real-time data analysis, etc. There is a dynamic imbalance between supply and demand of resources.

[0031] The disclosure proposes a dynamic resource optimization framework for large model inference services. Through the cooperation of service quality grading, multi-level service clusters, dynamic shunting mechanisms, elastic resource management, and multi-channel backup services, the resource utilization efficiency of large models is improved. Service quality grading divides inference services into high, standard, and low priority levels according to predetermined standards. Multi-level service clusters refer to deploying independent inference clusters for services (or requests) of different priorities to avoid low-priority tasks occupying resources of high-priority tasks. The dynamic shunting mechanism performs real-time traffic scheduling based on load and request characteristics, dynamically routing inference requests to appropriate service clusters. Elastic resource management intelligently scales resources based on traffic forecasts. Multi-channel backup services can supplement computing power through third-party API (Application Programming Interface) services during extreme peak traffic to ensure core business continuity.

[0032] In the following, embodiments will be described with reference to the accompanying drawings.

[0033] The first aspect of the disclosure provides a method for large model resource optimization based on traffic shunting. Figure 1 is a flowchart of a method for large model resource optimization based on traffic shunting according to an embodiment of the disclosure. As shown in Figure 1 The method for large model resource optimization based on traffic shunting according to an embodiment of the disclosure includes: monitoring traffic data including key indicators in real time, wherein the key indicators include one or more of request queue length, response time, and resource utilization (S101); performing traffic prediction based on traffic data through a multi-level time series model (S102); determining the priority of the request when the traffic reaches or is predicted to reach a traffic threshold (S103); and allocating different requests to different resource pools according to the priority of the request (S104).

[0034] In step S101, traffic data including key indicators is monitored in real time, wherein the key indicators include one or more of request queue length, response time, and resource utilization. The request queue length can represent the number of pending requests in the queue, reflecting the current load pressure of the system (e.g., when the queue length exceeds the threshold, resource expansion can be triggered). The response time can refer to the time from when the request enters the system to when the response is returned, which is used to measure service delay (e.g., when the delay exceeds the threshold, user experience may be reduced). Resource utilization can refer to the proportion of system resources such as GPU / CPU / memory usage (e.g., when GPU utilization is greater than or equal to 80%, resources can be increased). Real-time monitoring of traffic data can be stored in a time series database.

[0035] In step S102, traffic prediction is performed based on traffic data by a multi-level time series model. For example, a multi-level time series model for short-term, medium-term, and long-term (intraday, intraweek, seasonal patterns) can be constructed based on stored traffic data as training data, which can capture traffic variation patterns at different time scales. By inputting historical and / or real-time data into the multi-level time series model, the multi-level time series model can predict future traffic trends (e.g., traffic trends in a predetermined time period such as the next 10 minutes, 30 days, etc.), and respond to changes in resource demand in advance. In a large model service, traffic prediction can be triggered every predetermined time (e.g., every minute).

[0036] In step S103, when the traffic is monitored or predicted to reach a threshold, the priority of the request is determined based on various factors such as the type of request, the importance of the service, etc. For example, when the request queue length exceeds a threshold (e.g., 200 requests), the response time exceeds a threshold (e.g., 500 ms), the resource utilization exceeds a threshold (e.g., GPU utilization ≥ 85%), or the predicted traffic in the next 10 minutes exceeds the current resource capacity (e.g., request volume growth of 30%), the priority of the request is determined. In a large model service, by determining the priority of the request when the traffic reaches the traffic threshold, it can ensure that critical service requests are prioritized and improve the overall performance and user experience of the system.

[0037] In step S104, different requests are allocated to different resource pools according to their priorities. Different requests are routed to different resource pools according to their priorities, achieving differentiated service assurance. For example, high-priority requests are allocated to high-performance resource pools, and low-priority requests are allocated to ordinary resource pools. In a large model service, by allocating different requests to different resource pools, dynamic scheduling of resources can be achieved to ensure that the system can still run efficiently under high load conditions.

[0038] The above method realizes resource optimization for large model services through real-time monitoring of key indicators, traffic prediction, request priority determination, and resource allocation. It can effectively respond to traffic changes, improve system stability and performance, and ensure that critical service requests are prioritized to improve user experience.

[0039] According to an exemplary embodiment, traffic prediction based on traffic data by a multi-level time series model includes: generating time series data based on traffic data; generating business event features based on a business event calendar; and outputting the result of traffic prediction by a multi-level time series model based on time series data and business event features. The multi-level time series model is a prediction model that combines multiple time scales (such as short-term, medium-term, and long-term). By analyzing data at different time scales, it can consider both short-term fluctuations and long-term trends, improving the accuracy of predictions. The input of the multi-level time series model includes time series data and business event features. The output of the model is the result of traffic prediction, usually including predicted traffic values and predicted confidence intervals.

[0040] The original traffic data record usually includes a timestamp (accurate to minutes, hours, days, etc.) and the corresponding traffic value (such as the number of visits, requests, etc.). The original data is processed (e.g., cleaning, aggregation, and conversion, etc.) to generate structured, time-indexed sequence data, such as: [timestamp1: traffic value1], [timestamp2: traffic value2],..., [timestampN: traffic valueN]. The business event calendar can record promotional activities (such as Double 11, 618, Black Friday, etc.), product releases / updates, marketing activities, holidays, and other external events (such as the World Cup final). The events in the calendar are converted into numerical or categorical features that the model can understand, such as: [timestamp1: whether it is a holiday = 0, whether it is a promotion = 0,...], [timestamp2: whether it is a holiday = 1, whether it is a promotion = 1,...]. The multi-level time series model predicts future traffic values based on the generated time series data and business event features, such as traffic prediction values for a specific future period (such as the next 7 days, 30 days), usually including point predictions (predicted values) and interval predictions (confidence intervals / quantile predictions).

[0041] The method of traffic prediction by a multi-level time series model includes generating time series data, extracting business event features, and integrating these data for prediction. In scenarios where business events occur frequently or have a significant impact, adding business event features can significantly improve prediction accuracy, especially in the time period before and after the event. This method significantly enhances the understanding and prediction ability of time series models (especially multi-level models) for complex fluctuations in real business scenarios by using business event features as input, thereby improving the accuracy and reliability of traffic prediction and providing strong support for resource optimization and scheduling for large models.

[0042] According to an example embodiment, the method of traffic-splitting based model resource optimization further includes performing one or more of the following based on the result of the traffic prediction: preemptively launching new instances when the traffic is predicted to exceed the scaling threshold; automatically increasing or decreasing instances based on the load threshold and the predicted trend; assigning different computing resources to different requests according to the complexity of the requests; and releasing redundant resources in stages when the load continues to be below the reclaim threshold.

[0043] Active resource management is implemented based on the result of traffic prediction. For example, when the predicted value will exceed the preset scaling threshold (such as CPU utilization > 80%) at a certain time point in the future (such as 30 minutes in the future), the resource scaling process is triggered proactively, and new instances are launched in advance to avoid cold start delay. In combination with real-time load indicators (such as CPU, memory, QPS) and traffic prediction trends, the scaling strategy is dynamically adjusted. If the current load is greater than or equal to the scaling threshold (such as GPU utilization is greater than or equal to 80%) and the predicted traffic continues to grow in the next 10 minutes, the instances are increased. Conversely, if the current load is less than or equal to the reclaim threshold (such as GPU utilization is less than or equal to 30%) and the predicted traffic continues to decrease in the next 10 minutes, the instances are decreased. The requests are analyzed in real time (such as parsing request parameters, model routing) at the traffic distribution layer (such as API gateway, load balancer), the “complexity” (such as model size, input length) of the requests is identified, and the requests are routed to different instance groups of different specifications or are configured with different resource quotas, so that complex requests are preferentially guaranteed with high-quality resources, and the experience of key businesses is improved. When the system detects that the load is continuously (non-transiently) below the reclaim threshold (such as CPU < 30% for 10 minutes) and the traffic prediction also indicates that the demand will be low in the future, staged scaling down or releasing of redundant resources is started.

[0044] According to an example embodiment, determining the priority of the request includes evaluating one or more of the complexity, context length, and resource requirement of the request, and determining the priority of the request based on one or more of the level of the service level agreement, the type of the request, and the current load and the result of the evaluation of the request.

[0045] The complexity of a request refers to a measure of the amount of computational resources and time required to process the request. A request with high complexity usually requires more computational resources and longer processing time. The complexity of a request can be evaluated by analyzing factors such as the type of operations, the amount of data, the algorithm complexity, etc. contained in the request. The context length refers to the amount of context information that needs to be considered when processing the request, which can be evaluated by calculating the size of the context information (such as text length, data volume, etc.). Resource requirements refer to the amount of specific resources (such as CPU, memory, GPU, etc.) required to process the request. Service level agreement (SLA) is a contract or agreement that specifies the quality standards of services provided by service providers to users. According to the level of SLA, the priority of different requests can be determined. The priority of a request can be calculated by weighting the multi-dimensional evaluation results. For example, the multi-dimensional evaluation results are weighted and summed to generate a final priority score, and then the priority of the request is determined according to the range of the score. By evaluating the complexity, context length and resource requirements of the request, combined with the level of service level agreement, the type of request and the current load, the priority of the request can be determined comprehensively. This method balances the "intrinsic cost" (complexity / resource requirements) and "extrinsic value" (SLA / type) of the request in a dynamic system environment, and finally outputs the optimal solution of resource allocation, so as to ensure that the system can reasonably allocate resources, prioritize critical requests and improve the overall performance and user experience of the system under the condition of limited resources.

[0046] According to an example embodiment, the priority of the request includes high priority, standard priority, and low priority, wherein the allocating different requests to different resource pools includes: allocating high priority requests to a dedicated resource pool; allocating standard priority requests to a shared resource pool; and allocating low priority requests to an elastic resource pool. Based on the importance and timeliness of the business or according to the complexity of the request, the length of the context, the resource requirement, the SLA level, the request type, and the current load, etc., the inference service or request is classified into three levels: high priority service or request, standard priority service or request, and low priority service or request. High priority services are deployed in a dedicated resource pool to ensure the lowest latency and the highest availability, which is suitable for critical business scenarios. Standard priority services are deployed in a shared resource pool to provide balanced performance and cost, which is suitable for regular inference requirements. Low priority services are deployed in an elastic resource pool to allow longer response time, which is suitable for non-real-time scenarios. High priority requests usually have higher business importance, urgency, or significant impact on user experience. For example, it may involve critical business processes, real-time interaction requirements, or high SLA level services. Standard priority requests are regular business requests that do not have particularly high requirements for response time and resource requirements, but still need to be processed within a reasonable time. Low priority requests can tolerate longer response time and lower resource allocation priority. The dedicated resource pool is a dedicated resource cluster designed for high priority requests, which ensures resource exclusivity through physical or logical isolation, ensuring high availability and low latency of services. The shared resource pool is a dynamic resource sharing cluster that supports multiple standard priority requests, balancing performance and cost efficiency through scheduling algorithms. The elastic resource pool is a dynamic scaling resource pool for low priority requests, which utilizes idle resources or low-cost elastic instances to allow significant fluctuations in response time.

[0047] By classifying the priority of the request into high priority, standard priority, and low priority, and allocating them to dedicated resource pool, shared resource pool, and elastic resource pool respectively, the reasonable allocation and dynamic adjustment of resources can be achieved. This method can improve the resource utilization efficiency of the system, improve the system performance, enhance the user experience, and ensure the stable operation of the system under different load conditions.

[0048] According to an example embodiment, when the traffic reaches or is predicted to reach the traffic threshold, the standard priority requests are routed to the shared resource pool through an intelligent routing algorithm, and / or the low priority requests are delayed or downgraded when the load is greater than or equal to the extreme load threshold.

[0049] Traffic thresholds are preset traffic limits. When actual or predicted traffic reaches this value, the system triggers corresponding processing mechanisms. Extreme load thresholds are higher load limits. When the system load reaches or exceeds the extreme load threshold, the system adopts stricter resource management measures. Intelligent routing algorithms dynamically select the optimal path or resource allocation scheme based on real-time system status and preset rules. They can intelligently allocate requests to appropriate resource pools based on factors such as current traffic, resource utilization, and request type. This strategy effectively handles traffic peaks and extreme load situations, ensuring system stability and the processing efficiency of critical requests, while optimizing resource allocation and improving overall system performance. When the system load is greater than or equal to the extreme load threshold, low-priority requests can be delayed or degraded. Delayed processing means these requests will not be processed immediately but will wait until the system load decreases. Degraded processing refers to degrading the service quality of low-priority requests when the system load is extremely high. For example, reducing the processing precision of requests, reducing the amount of returned data, or slowing down processing speed. Through intelligent routing and latency / degradation processing, the system can prioritize the protection of high-priority and standard-priority requests, ensuring that critical business operations and user experience are not affected.

[0050] According to an exemplary embodiment, the method for optimizing large model resources based on traffic splitting further includes: when the load is greater than or equal to an extreme load threshold, performing one or more of the following: processing standard priority requests and / or low priority requests using a lightweight alternative model; merging multiple similar requests whose feature vector distance is less than a predetermined threshold; disabling preset resource-intensive functions; routing low priority requests to a third-party application programming interface service when the resource utilization of the large model exceeds a resource utilization threshold; and restoring services in order of request priority after the load drops to a normal level.

[0051] Lightweight alternative models are simplified versions of larger models, typically with lower computational complexity and resource requirements, yet providing similar functionality and output. Under extreme load conditions, using lightweight alternative models can significantly reduce resource consumption while still providing basic service functionality. Lightweight alternative models can be used for standard and low-priority requests that do not have high real-time or accuracy requirements. For example, in a text generation service, a lightweight model can be used to generate basic text content instead of a full, large language model. When the feature vector distance between multiple requests is less than a predetermined threshold, or when multiple requests are similar, these requests can be merged into a single request for processing. In text generation or translation services, if the input text of multiple requests is very similar, they can be merged to generate a generic result, which can then be fine-tuned as needed. Merging similar requests can improve throughput. Resource-intensive functions or modules are typically the most resource-intensive parts of the system, such as complex deep learning algorithms or high-precision computation modules. Under extreme load conditions, (temporarily) shutting down these resource-intensive functions or modules can significantly reduce resource consumption and ensure the system can continue to operate. The resource utilization threshold is a preset upper limit for resource usage. When the actual resource utilization exceeds this value, the system triggers corresponding processing mechanisms. Third-party API services refer to application programming interface services provided by other service providers. When the resource utilization of the large model exceeds the threshold, low-priority requests are routed to the third-party API service to reduce the load on the local system. When the system load drops to a normal level, it means that the system resources are sufficient to handle more requests. Requests that were previously delayed or degraded are restored sequentially according to their priority. This ensures that the system can process all requests in an orderly manner after the load returns to normal, prioritizing the restoration of high-priority requests to ensure the continuity of critical business operations.

[0052] According to the exemplary embodiment, the above method achieves efficient operation and quality of service assurance of large model inference services under peak traffic by employing multi-level resource optimization strategies combined with third-party API resource peak shaving. Furthermore, according to the exemplary embodiment, the large model resource optimization method based on traffic diversion can effectively reduce system load under extreme load conditions by using strategies such as lightweight alternative models, merging similar requests, shutting down resource-intensive modules, routing low-priority requests to third-party API services, and restoring services according to priority. This ensures that critical requests can be processed, while optimizing resource utilization and improving the overall performance and stability of the system.

[0053] In summary, according to the embodiments of this disclosure, by intelligent peak traffic splitting and collaboration with third-party API resources, one or more of the following technical effects are achieved: significantly improved resource utilization; dynamic splitting strategies and predictive scheduling mechanisms significantly improve the average utilization of system resources and reduce resource idleness; significantly reduced operating costs; through precise resource allocation and idle resource recycling, computing resource costs are effectively reduced, while energy consumption is also reduced; stable and reliable service quality; even in peak scenarios with high-multiplier baseline traffic, the response time growth of high-priority services is effectively controlled, ensuring the stability of core businesses; enhanced peak carrying capacity; combined with third-party API peak shaving solutions, the system can cope with extreme peak traffic far exceeding that of normal traffic, significantly improving elasticity; and diversified service support, meeting the service quality requirements of different scenarios, providing differentiated service capabilities, and improving the overall user experience.

[0054] A second aspect of this disclosure provides an apparatus for large-scale model resource optimization based on traffic splitting. Figure 2 A block diagram of an apparatus for large-scale resource optimization based on traffic splitting according to an embodiment of the present disclosure is shown. Figure 2 As shown, the apparatus 200 for large-scale model resource optimization based on traffic splitting according to an embodiment of the present disclosure includes: a traffic monitoring module 201 configured to monitor traffic data including key indicators in real time, wherein the key indicators include one or more of request queue length, response time, and resource utilization; a traffic prediction module 202 configured to perform traffic prediction based on traffic data using a multi-level time series model; a request priority determination module 203 configured to determine the priority of a request when traffic is detected to reach a traffic threshold or when traffic is predicted to reach a traffic threshold; and a resource scheduling module 204 configured to allocate different requests to different resource pools according to the priority of the requests.

[0055] In the description of the first aspect of this disclosure, the various method steps involved in the technical solution of this disclosure have been described in detail. Therefore, the above description can be applied to a large model resource optimization device 200 based on traffic splitting in the second aspect of this disclosure. Accordingly, the description will not be repeated here.

[0056] This disclosure also provides an electronic device including a memory and a processor. The memory stores a program, and the processor is configured to retrieve the program and, when executing the program, perform the above-described method for large-scale resource optimization based on traffic splitting.

[0057] Figure 3 This is a block diagram of an electronic device implementing a method for large-model resource optimization based on traffic splitting, according to some embodiments of this disclosure. Figure 3 As shown, the method for optimizing large model resources based on traffic splitting in the above embodiments can be achieved through...Figure 3 The electronic device shown is used to implement this, and the electronic device includes at least one processor, memory, and at least one I / O interface.

[0058] The processor can be a general-purpose central processing unit (CPU) and a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC). Memory can include at least one of volatile memory and non-volatile memory. Memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; it can be random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; it can also be electrically erasable programmable read-only memory (EEPROM), read-only optical disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing a desired program having an instruction or data structure form and accessible by a computer, but is not limited thereto. Memory can exist independently and be connected to the processor via an address bus, data bus, and control bus. Memory can also be integrated with the processor.

[0059] The memory stores programs that execute the scheme of this disclosure, and the execution is controlled by a processor. The processor executes the programs stored in the memory. The programs may include one or more software modules. The large-model resource optimization method based on traffic splitting in the above embodiments can be implemented by a processor and one or more software modules in the program in the memory. However, this disclosure is not limited thereto. The large-model resource optimization method based on traffic splitting in the above embodiments can also be implemented by circuitry.

[0060] I / O interfaces connect to input devices such as mice, microphones, keyboards, and touchscreens, as well as output devices such as speakers, printers, and monitors. I / O interfaces can also use transceivers or similar devices to communicate with other devices or communication networks such as Ethernet, Radio Access Networks (RAN), and Wireless Local Area Networks (WLAN).

[0061] As an exemplary embodiment, an electronic device may include a plurality of processors, each of which may be a single-core processor or a multi-core processor. As used herein, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0062] The aforementioned electronic device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, laptop computer, network server, PDA, mobile phone, tablet computer, wireless terminal device, communication device (e.g., access point, router, gateway, etc.), or embedded device. The embodiments of this disclosure do not limit the type of computer device, as long as it has a processor and memory.

[0063] It should be understood that Figure 3 The illustrated electronic device is merely one example of this disclosure, and the electronic devices of this disclosure may also include elements or components not shown in the examples above. For example, some electronic devices also include display units such as displays, some electronic devices also include human-computer interaction elements such as buttons and keyboards, and some electronic devices also include various sensors, such as gesture sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, grip sensors, proximity sensors, color sensors, infrared (IR) sensors, biometric sensors, temperature sensors, humidity sensors, illuminance sensors, etc. Any electronic device capable of executing a computer-readable program in its memory to implement the methods or at least some steps of the methods described in this disclosure may be considered an electronic device covered by this disclosure.

[0064] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented in software, in hardware, or in a combination of software and necessary hardware. Therefore, as... Figure 4 As shown, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-transitory computer-readable storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) or on a network, including several commands to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described methods according to the embodiments of this disclosure.

[0065] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include, but are not limited to: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0066] This disclosure also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described method for large-scale resource optimization based on traffic splitting. Figure 4 This is a block diagram illustrating a computer-readable medium according to embodiments of the present disclosure.

[0067] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying a readable program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with a command execution system, apparatus, or device. The program contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination thereof.

[0068] Programs for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. Programs can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0069] The aforementioned computer-readable medium carries one or more programs (e.g., computer-executable programs) that, when executed by one or more devices, cause the computer-readable medium to implement the methods of this disclosure.

[0070] Those skilled in the art will understand that the above modules can be distributed in one device as described in the embodiments, or they can be varied and located in one or more related devices. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0071] Exemplary embodiments of the present disclosure have been specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, the present disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

[0072] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0073] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for large-scale model resource optimization based on traffic splitting, characterized in that, include: Real-time monitoring includes traffic data with key indicators, wherein the key indicators include one or more of the following: request queue length, response time, and resource utilization. Traffic prediction is performed based on the traffic data using a multi-level time series model. When traffic is detected to have reached a traffic threshold or is predicted to reach the traffic threshold, the priority of the request is determined; and Different requests are allocated to different resource pools based on their priority.

2. The method according to claim 1, characterized in that, Traffic prediction based on the traffic data using a multi-level time series model includes: Time series data is generated based on the traffic data; Generate business event characteristics based on the business event calendar; and The multi-level time series model outputs traffic prediction results based on the time series data and the characteristics of the business events.

3. The method according to claim 2, characterized in that, Also includes: Based on the results of the traffic prediction, perform one or more of the following: When traffic is predicted to exceed the expansion threshold, start a new instance in advance; Automatically increase or decrease instances based on load thresholds and predicted trends; Allocate different computing resources to different requests based on their complexity; and When the load remains below the recovery threshold, redundant resources are released in stages.

4. The method according to claim 1, characterized in that, Determining request priority includes: Assess one or more of the following: the complexity of the request, the context length, and the resource requirements: The priority of a request is determined based on one or more of the service level agreement level, the type of the request, and the current load, as well as the result of an evaluation of the request.

5. The method according to claim 4, characterized in that, The priority of the request includes high priority, standard priority, and low priority. This includes allocating different requests to different resource pools, including: Allocate the high-priority requests to a dedicated resource pool; A request to allocate the standard priority to the shared resource pool; and The low-priority request is allocated to the elastic resource pool.

6. The method according to claim 5, characterized in that, When the monitored flow rate reaches the flow rate threshold or the flow rate is predicted to reach the flow rate threshold, The standard priority requests are routed to the shared resource pool via a smart routing algorithm, and / or The low-priority requests are delayed or downgraded when the load is greater than or equal to the extreme load threshold.

7. The method according to claim 6, characterized in that, Also includes: When the load is greater than or equal to the extreme load threshold, perform one or more of the following: The standard priority requests and / or the low priority requests are handled using a lightweight alternative model. Merge multiple similar requests whose feature vector distance is less than a predetermined threshold. Disable default resource-intensive functions; When the resource utilization of a large model exceeds the resource utilization threshold, the low-priority requests are routed to a third-party application programming interface service. as well as Once the load returns to normal levels, services are restored sequentially according to the priority of the requests.

8. A device for large-scale model resource optimization based on traffic splitting, characterized in that, include: The traffic monitoring module is configured to monitor traffic data including key indicators in real time, wherein the key indicators include one or more of the following: request queue length, response time, and resource utilization. The traffic prediction module is configured to predict traffic based on the traffic data using a multi-level time series model. The request priority determination module is configured to determine the priority of a request when traffic is detected to have reached a traffic threshold or when traffic is predicted to reach the traffic threshold; and The resource scheduling module is configured to allocate different requests to different resource pools based on the priority of the requests.

9. An electronic device, comprising: processor; And a memory for storing a program that, when executed by the processor, performs the method as described in any one of claims 1-7.

10. A computer-readable medium storing a program that, when executed by a processor, performs the method as described in any one of claims 1-7.

Images