Method and system for realizing capacity expansion / reduction in artificial intelligence scene

Abstract

The invention provides a method and a system for realizing capacity expansion / reduction in an artificial intelligence scene. The method comprises the following steps: monitoring system performance and service performance to obtain performance indexes; based on the obtained performance indexes, a real-time updating plan and an off-line elastic plan are made respectively, and the real-time updating plan and the off-line elastic plan both comprise the number of target resources; the offline elastic plan is constructed based on machine learning of historical indexes; and determining a capacity expansion / reduction scheme based on a comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity and in combination with a set strategy.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and more particularly, to a method and system for implementing expansion / contraction in an artificial intelligence scenario. Background Art

[0002] As artificial intelligence (AI) technology continues to develop and mature, more and more companies are beginning to apply AI-driven business processes to specific business scenarios, such as autonomous driving, image recognition, natural language processing, etc. This AI-driven business process is expected to completely change the business's operating model, improve efficiency, reduce costs, and bring customers a more intelligent and personalized experience.

[0003] Traditional business processes often rely on manual operations and experience, and have defects such as low efficiency and prone to errors. AI-driven business processes can use technologies such as machine learning, natural language processing, and computer vision to achieve automated decision-making and intelligent operations, greatly improving the efficiency and accuracy of the process. For example, in the field of customer service, AI-driven business processes can use intelligent dialogue systems to understand customer needs through natural language processing technology, and give accurate responses and solutions, greatly improving the response speed and quality of customer service. In the financial field, AI-driven business processes can use machine learning algorithms to analyze large amounts of historical data to achieve intelligent decisions such as risk assessment and loan approval, improving the accuracy and consistency of decisions.

[0004] In order to support these AI-driven business processes, enterprises are increasingly adopting cloud services to provide various resources required for AI workflows. Cloud services can not only provide the necessary computing resources, such as CPU resources, GPU resources, memory, disk and other storage space, but also provide various AI-related development tools, algorithm libraries and pre-trained models, greatly reducing the cost of enterprises to build and maintain AI infrastructure by themselves. Through cloud services, cloud services also provide high availability and security guarantees to ensure the stable operation of AI-driven business processes. In addition to basic resources, some leading cloud service providers have also launched various AI services and workflow management tools, such as natural language processing services, machine learning services, data processing pipelines, etc., to help enterprises build and deploy AI-driven business processes more efficiently. These tools and services seamlessly integrate AI technology with the business needs of enterprises, significantly reducing the threshold and complexity of AI applications.

[0005] For traditional cloud services, enterprises can flexibly allocate resources according to actual needs, and the main focus is to avoid idle waste of hardware resources, and there is no need to worry about performance bottlenecks caused by insufficient resources. Traditional elastic scaling strategies often only focus on CPU usage. However, AI application scenarios are often highly uncertain, and their demand for computing resources fluctuates greatly over time. In AI application scenarios, the automatic scaling functions provided by most cloud service providers mainly rely on a single threshold trigger mechanism (such as CPU usage reaching a certain ratio). Although it is easy to configure and manage, it may be too simplified in complex AI application scenarios and cannot accurately reflect the actual load status. If the usage of storage resources such as GPU resources, memory, and disk is not considered, it will lead to resource waste or performance bottlenecks. Summary of the invention

[0006] The object of the present invention is to provide a method and system for implementing expansion / contraction in an artificial intelligence scenario, including:

[0007] A method for implementing expansion / contraction in an artificial intelligence scenario, comprising:

[0008] Monitor system performance and business performance and obtain performance indicators;

[0009] Based on the obtained performance indicators, a real-time update plan and an offline elastic plan are formulated respectively, wherein both the real-time update plan and the offline elastic plan include target resource quantities; the offline elastic plan is constructed based on machine learning of historical indicators;

[0010] The expansion / contraction plan is determined based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity in combination with the set strategy.

[0011] A system for implementing expansion / contraction in an artificial intelligence scenario, comprising:

[0012] Monitoring module, the monitoring module includes a system performance monitoring module and a service performance monitoring module, wherein:

[0013] The system performance monitoring module monitors system performance and obtains performance indicators;

[0014] The service performance monitoring module monitors service performance and obtains performance indicators;

[0015] A real-time update engine, which formulates a real-time update plan based on the acquired performance indicators, wherein the real-time update plan includes a target resource quantity;

[0016] An offline elasticity engine, which formulates an offline elasticity plan based on the obtained performance indicators, wherein the offline elasticity plan includes a target resource quantity; the offline elasticity plan is constructed based on machine learning of historical indicators;

[0017] The expansion / contraction decision maker determines the expansion / contraction plan based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity and in combination with the set strategy.

[0018] The above solution integrates multiple performance indicators to build a comprehensive performance evaluation system. This multi-dimensional performance indicator integration breaks through the limitations of a single indicator and forms a more comprehensive performance evaluation standard. In particular, it integrates business priority-oriented resource scheduling, which for the first time incorporates business value into resource allocation decisions and achieves efficient resource utilization. In this way, through intelligent scheduling, idle resources can be avoided, overall utilization can be improved, unnecessary resource consumption can be reduced, and operating costs can be reduced. In this process, high-value services can be prioritized and performance degradation caused by resource competition can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 It is a schematic diagram of the principle of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0022] In the development and application of AI technology, in addition to CPU load, factors such as GPU load and I / O performance also play a vital role, especially when dealing with large-scale data sets and complex models. The following are the key reasons:

[0023] 1. GPU computing advantages

[0024] Compared with traditional CPUs, GPUs have significant advantages in parallel computing and can process large amounts of data simultaneously. Therefore, when performing deep learning training and reasoning, the GPU load directly affects the model's training speed and reasoning response time. If the system only focuses on CPU usage, GPU resources may not be fully utilized, resulting in computing bottlenecks.

[0025] 2. Data transmission and processing

[0026] In AI applications, the speed of reading and writing data is equally important. I / O performance includes the speed of reading data from storage devices and the speed of writing calculation results to storage. When I / O performance is low, it will cause data transmission delays, thus affecting the overall efficiency of computing. For example, when using large-scale data sets, if data cannot be read quickly from the hard disk for GPU use, the GPU will face the situation of "idle" waiting for data, which will seriously affect computing efficiency.

[0027] 3. Complexity of model training

[0028] Modern AI models are usually very complex, involving hundreds of millions or even billions of parameters, and the training process requires continuous large amounts of data iteration. By monitoring GPU load and I / O performance, you can better understand the bottlenecks of the system and optimize it based on actual conditions, such as adjusting data reading strategies or expanding corresponding storage reading capabilities.

[0029] 4. Multitasking

[0030] In many AI application scenarios, multiple tasks often need to be performed simultaneously. For example, a system may perform model training, reasoning, and data preprocessing at the same time. At this time, reasonable GPU load and I / O performance management can significantly improve the overall system throughput and responsiveness.

[0031] 5. Efficient use of resources

[0032] Paying attention to factors such as GPU load and I / O performance can help enterprises use computing resources more effectively, thereby reducing resource waste and lowering operating costs. In the traditional fixed resource configuration model, the dynamic changes of the entire computing task may be ignored due to the optimization of a single indicator, and intelligent resource scheduling and optimization cannot be achieved.

[0033] 6. Performance monitoring and adjustment

[0034] By comprehensively considering multiple performance indicators, enterprises can establish a more comprehensive and dynamic performance monitoring mechanism. This allows timely resource adjustments to meet current needs when task loads change, thereby improving overall system performance and reliability.

[0035] In summary, in AI-driven business processes, GPU load and I / O performance are key factors affecting overall system performance. Traditional elastic scaling strategies that focus solely on CPU usage cannot fully reflect the actual load of the system, resulting in unbalanced resource allocation. Therefore, in order to more effectively support AI application scenarios, enterprises need to adopt a more comprehensive resource management strategy that comprehensively considers computing power, data transmission speed, and other related performance indicators to adapt to highly uncertain usage needs.

[0036] Based on this, this application provides a method for implementing expansion / contraction in an artificial intelligence scenario. The overall principle is as follows: Figure 1 As shown, including:

[0037] S110: Monitor system performance and service performance to obtain performance indicators.

[0038] Google Cloud AutoML (Google Cloud's AutoML platform) provides automated machine learning model training and deployment services, but it lacks targeted support in the field of real-time elastic scaling, especially in multi-dimensional performance monitoring.

[0039] The present application introduces multi-dimensional performance monitoring, which includes not only monitoring system performance and obtaining performance indicators, but also monitoring business performance and obtaining performance indicators.

[0040] In terms of system performance, in the embodiment of the present application, in addition to collecting and analyzing CPU / GPU utilization, multiple performance indicators such as GPU utilization, memory, storage, and network I / O resource utilization can also be collected and analyzed to form a comprehensive performance view. Multiple performance indicators such as CPU / GPU utilization, memory, storage, and network I / O resource utilization can form a comprehensive system performance view.

[0041] In a cloud computing environment, elastic scaling is the key to maximizing resource utilization efficiency and meeting the changing application load requirements. Among them, CPU, GPU, memory, storage and network I / O resource pooling and utilization detection play a core role. Through resource virtualization and intelligent scheduling, dynamic supply and efficient utilization of computing, storage and network I / O resources are collaboratively achieved.

[0042] The cloud platform virtualizes the available CPU, GPU, memory, storage, and network I / O resources (Input / Output) into corresponding resource pools, and converts physical resources into uniformly managed resource units through abstraction and logical partitioning technology. Within the resource pool, these units are divided into different quotas and provided to upper-layer applications in a multi-tenant sharing mode. When the resource demand of tenants increases, the cloud management system can dynamically allocate more quotas from each resource pool to achieve on-demand capacity expansion; when the demand decreases, the idle resource quotas can also be recovered from the resource pool to avoid waste. However, resource pooling alone cannot accurately meet the changes in resource demand of different applications. Therefore, the cloud platform also needs to continuously monitor the actual utilization rate of various resources on the resource pool. By deploying monitoring agents, the CPU, GPU, memory, storage, and network I / O resource usage data of each tenant application is collected in real time, including indicators such as usage, utilization rate, and load change trend. When it is detected that resource usage exceeds the preset threshold, the scheduling system will automatically obtain more quota from the resource pool and allocate it to the application; conversely, when the usage rate is low for a long time, it will automatically reclaim excess resource quota to prevent idle waste.

[0043] The monitoring of resource usage is not only reflected at the level of a single application, but also requires comprehensive monitoring of the entire resource pool and the underlying infrastructure, so as to achieve refined management of global resources. By statistically analyzing the resource usage from all tenant applications and building a prediction model with the help of machine learning and other technologies, the management system can predict the overall load trend of each resource pool in advance, and based on the prediction results, expand or recycle the underlying computing nodes and storage and network I / O devices in advance to ensure the timeliness and elasticity of resource supply and avoid resource bottlenecks or over-configuration. At the same time, the resource scheduling system also needs to make comprehensive decisions based on multiple factors such as the energy efficiency characteristics of hardware equipment and service quality requirements. For example, for computing-intensive applications, the scheduler will schedule them to run on GPU nodes with a higher power consumption performance ratio; for applications that are sensitive to latency, it will choose to provide services nearby to reduce data transmission delays. Through intelligent scheduling algorithms, the cloud platform can allocate limited resources to the optimal location to maximize resource utilization efficiency.

[0044] In modern cloud computing environments, elastic scaling technology combined with containerization technology provides advanced, automated solutions for efficient use of CPU, GPU, memory, storage, and network I / O resources. Resource pooling, usage detection, and the Kubernetes (K8s) orchestration engine together build a dynamic, intelligent, on-demand resource management system to ensure full utilization of infrastructure resources and timely satisfaction of application needs.

[0045] First, the cloud platform abstracts all computing resources (including CPU, GPU, and memory) and storage and network I / O resources into logical resource pools, and divides and packages the resources through containerization to form portable, highly standardized computing units and data units. Containers are a lightweight application virtualization technology that packages application code and all its dependencies (such as runtime, library files, etc.) together to form a portable container image. Multiple container instances can be safely deployed on the same physical or virtual machine, sharing the underlying hardware resources, thereby significantly improving resource utilization and elasticity.

[0046] Compared with traditional hardware-level virtualization (such as virtual machines), container technology belongs to operating system-level virtualization, which runs directly on the host machine's operating system kernel, eliminating the guest OS overhead. The container only contains the application and its most basic operating environment, without the need to provide a complete operating system repeatedly. Therefore, it has the advantages of less resource usage, fast startup, and convenient migration. Mainstream container technology standards include Docker, Rocket, containerd, etc.

[0047] When the number of container instances is huge, Kubernetes, as the industry's leading container orchestration engine, is responsible for unified scheduling and management of containerized computing, storage, and IO resources. Kubernetes treats the entire resource pool as a large computing cluster and automatically orchestrates resources within the cluster through declarative APIs, greatly reducing the complexity of operation and maintenance. In terms of resource scheduling, Kubernetes combines a variety of scheduling algorithms and constraint strategies to allocate the most appropriate computing nodes, storage, and IO resources to each workload based on its resource demand characteristics, service quality requirements, etc., to avoid resource waste and provide optimal performance.

[0048] A Kubernetes cluster generally consists of a Master node and multiple worker nodes. The Master is the control plane of the cluster, responsible for the management and control decisions of the entire cluster; while the Node is the execution plane for computing loads. The Master dispatches workloads and creates, starts, and monitors the operation of containers on the Node through the kubelet agent. Each Node also runs the kube-proxy component to enable Kubernetes Service to communicate with the outside world, maintain the network, and implement load balancing strategies.

[0049] At the same time, continuous resource usage monitoring and load forecasting are key to ensuring dynamic resource supply. In the Kubernetes cluster, a dedicated monitoring agent is deployed on each node to collect the CPU, GPU, memory, and storage resource usage of all container instances on the node, and transmit these indicator data to the cluster's monitoring system. Based on a large amount of monitoring data, the autoscaler can determine in real time when a workload needs to be scaled up or down, thereby increasing or decreasing computing and storage resources in a timely manner.

[0050] In a larger resource pool, the cloud management system will also conduct real-time analysis and predictive modeling of overall resource usage trends based on artificial intelligence technologies such as machine learning, and input the prediction results into the resource scheduling system to guide the timely expansion or recovery of underlying infrastructure resources. Through AI-based intelligent resource management, it is possible to avoid delayed or over-allocation of resource supply, achieve precise allocation of resources in time and space, and maximize resource utilization.

[0051] At the same time, efficient use of computing and storage resources also requires consideration of multiple factors such as energy consumption and service quality. The resource scheduling system will combine the energy consumption characteristics of hardware devices to centrally deploy similar workloads on nodes with better energy efficiency, and regularly organize and compress resources on high-energy consumption nodes to reduce overall energy consumption. For applications that are sensitive to latency, the scheduler will give priority to nodes that provide services nearby to reduce data transmission delays.

[0052] Through this comprehensive resource management approach, modern cloud computing environments are able to achieve efficient utilization of computing and storage resources to meet the needs of various applications while optimizing energy consumption and performance.

[0053] To monitor business performance, you can get business performance indicators such as QPS (Queries Per Second), TPS (Transactions Per Second), business latency (such as P99 Latency) / business response time (RT, Response Time), etc.

[0054] QPS (Queries Per Second) indicates the number of query requests that the system can process within a period of time. It can reflect the processing capacity of the system per unit time. The higher the QPS, the faster the system processes requests.

[0055] TPS (Transactions Per Second) indicates the number of transactions that a system can process within a period of time. It emphasizes the system's ability to process a set of operations (i.e., a transaction). A transaction usually includes multiple steps, such as querying a database, updating data, sending notifications, etc.

[0056] QPS and TPS are often used interchangeably, but strictly speaking, the concept of TPS is broader and includes more complex operations. In some scenarios, QPS and TPS can be approximately equal. Relatively speaking, the application scenarios of QPS are more focused on database servers, Web servers, API interfaces, etc., while the application scenarios of QPS are more focused on systems that need to process complex business logic, such as banking systems and e-commerce platforms.

[0057] P99 Latency (P99) is a measure of system performance. It indicates the longest response time experienced by 99% of requests over a period of time. In other words, only 1% of requests will have a response time that exceeds this value. P99 latency directly affects the user experience. If the P99 latency is too high, it means that a small number of users will experience a long wait, which will seriously affect user satisfaction. P99 latency can reflect the stability and robustness of the system. If the P99 latency fluctuates greatly, it means that there may be some potential problems in the system. Especially in cloud computing and distributed systems, due to factors such as network and resource scheduling, P99 latency helps to understand the performance of the system under high load or burst traffic. By monitoring P99 latency, the bottleneck of the system can be optimized in a targeted manner to improve overall performance. P99 latency is also a commonly used indicator in service level agreements (SLAs) and can be used to measure whether the service meets SLA requirements.

[0058] In addition to P99, there are also indicators such as P50 and P90. P50 represents the median, that is, the response time of 50% of the requests; P90 represents the response time of 90% of the requests. These indicators can jointly reflect the overall performance distribution of the system.

[0059] In addition, in contrast to latency, it can also be response time (RT), such as average response time (ART). Average response time is the average response time of all requests. However, the average value is easily affected by extreme values, while percentile indicators such as P99 can better reflect performance in extreme cases.

[0060] S120: Based on the obtained performance indicators, a real-time update plan and an offline elastic plan are formulated respectively, wherein the real-time update plan and the offline elastic plan both include a target resource quantity.

[0061] To achieve refined resource management and elastic scaling in a cloud computing environment, it is necessary to continuously monitor and obtain various performance indicator data in real time, including the usage of underlying system resources and the service quality status of upper-level business loads. Only based on a comprehensive performance data view can the resource orchestration and scheduling system make accurate resource supply decisions to meet the changing needs of different businesses. For system-level resource usage, the key indicators that need to be monitored include CPU utilization, GPU utilization, memory usage, storage and network I / O. Among them, GPU utilization is a key indicator, because GPU resources are often used in computing-intensive tasks such as AI reasoning and scientific computing. They are high-value resources and their performance value needs to be maximized. At the same time, the usage of storage resources cannot be ignored. Storage capacity and IO throughput are crucial for data-intensive applications.

[0062] By obtaining the above performance indicators in real time, such as system performance indicators GPU, memory and storage resource consumption, and business performance indicators QPS, TPS, P99 Latency) / RT, a real-time update plan can be formulated.

[0063] After the above performance data is collected into the monitoring system through widely deployed monitoring agents, these comprehensive view data can be obtained and analyzed in real time. For example, if the latency indicator of a certain business is monitored to begin to deteriorate and exceed the set service quality threshold, and it is expected that this situation will continue through trend analysis, then the real-time update plan can be set to first try to allocate more resources to it within the current cluster node; if there is contention for resources within the cluster, they will be reallocated according to the priority of different businesses. If the cluster resources are still insufficient, the platform will automatically initiate an infrastructure expansion request to schedule enough new resources from the resource pool to allocate to this high-priority business load.

[0064] Similarly, if it is monitored that the resource utilization of a certain business is chronically low, the excess resources can be automatically stripped off and returned to the resource pool in the formulated real-time update plan to avoid waste; if it is further discovered that the load of the entire cluster is far below its capacity, some infrastructure resources can be taken offline in the formulated real-time update plan to reduce energy consumption.

[0065] When a new service is just launched, only the maximum number of replicas, such as the number of Pods, needs to be estimated. After the service is launched, the number of Pods required by the current service is calculated in real time based on the monitoring data of QPS and RT, but the amount of real-time evaluation data is small. Therefore, an offline elastic plan is also built here to evaluate and correct the evaluation results in the real-time update plan.

[0066] In this way, historical system performance indicators such as GPU, memory and storage, I / O resource consumption and business performance indicators such as QPS, TPS, P99 Latency) / RT can also be counted. Offline elasticity plans can be formulated based on historical indicators. Specifically, based on machine learning modeling of historical indicators, the platform can predict future resource demand trends and provide a basis for automatic expansion and contraction decisions. At the same time, the platform will also take into account various strategic goals such as business priority, maximization of resource utilization, and service quality assurance, and comprehensively formulate real-time offline elasticity plans.

[0067] In addition, the real-time update plan also includes business priority awareness. When implementing elastic scaling in a cloud computing environment, in addition to considering the maximization of resource utilization, it is also necessary to pay attention to the priority and service quality assurance of different business workloads. Business priority awareness means that the resource scheduling and scaling decision system can identify and distinguish the importance of various business workloads, and formulate differentiated strategies for resource allocation according to their priority levels to ensure the service quality and reliability of key businesses.

[0068] Specifically, business priority awareness is mainly reflected in the following aspects: First, when allocating resources, the scheduling system will refer to the priority setting of the business, and reserve sufficient resource quotas for high-priority businesses to prevent resource shortages; for low-priority businesses, resource allocation will be appropriately compressed to avoid resource waste. Secondly, in the event of resource shortages, the scheduling system will give priority to protecting the resource needs of high-priority businesses, recover resources from low-priority businesses, and ensure the continuous operation of key businesses. Thirdly, when making capacity planning and expansion decisions, the scheduling system will give priority to the growth trend of high-priority businesses, reserve sufficient resource buffers, and avoid degradation of service quality. In addition, for delay- and bandwidth-sensitive businesses, the scheduling will schedule them to run on nodes with the best network conditions to ensure low latency and high bandwidth.

[0069] The setting of business priorities is usually done by operation and maintenance personnel or automated policy engines, and can be based on a variety of factors, such as the importance of business revenue, contract service level agreements, historical load models, etc. The scheduling system will incorporate this priority information into each decision-making link, and consider multiple factors such as resource usage and service quality requirements to develop a comprehensive resource allocation and scaling plan. Business priority awareness enables limited resources to be efficiently allocated to the most critical workloads, ensuring the rational use of overall resources and the balance of service quality for different businesses. It is an important means to achieve refined management and optimization of cloud resources.

[0070] Through real-time collection, analysis and decision-making of various performance data, the resource orchestration platform can achieve refined and intelligent resource management, dynamically match changes in application requirements, allocate resources on demand, and maximize resource utilization. At the same time, it can meet the service quality requirements of key businesses, truly achieve a dynamic balance between cost and performance, and realize efficient use of cloud resources.

[0071] Both the real-time update plan and the offline elastic plan include detailed planning and configuration of the target resource quantity to ensure that the system can maintain the optimal resource supply under various operating conditions, meet service needs and optimize cost-effectiveness.

[0072] S130: Determine expansion / contraction based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity and in combination with the set strategy.

[0073] HPA (Horizontal Pod Autoscaler) Controller is a component in Kubernetes that automatically adjusts the number of Pod replicas. It dynamically adjusts the number of application replicas by monitoring specific metrics (such as CPU usage, memory usage, or custom metrics), thereby ensuring that the application can automatically expand or shrink when the load changes to meet performance requirements and optimize resource utilization. Specifically, HPA Controller can obtain Pod-related metric data such as CPU usage, memory usage, etc. regularly (for example, 15 seconds by default) through tools such as Kubernetes's MetricsServer or Prometheus Adapter. If the user defines custom metrics (Custom Metrics), HPA Controller will also obtain data for these metrics through the corresponding API. Furthermore, HPA Controller can calculate how many replicas are needed to meet the target metric value configured by the user (such as target CPU usage) and the current actual metric value. For example, if the target CPU usage is 50% and the current CPU usage is 100%, HPA Controller may increase the number of replicas to share the load. HPA Controller compares the calculated expected number of replicas with the current number of replicas and decides whether to increase or decrease the number of replicas. HPA Controller configuration is usually done by creating a HorizontalPodAutoscaler object. This object defines the target resource (such as Deployment), the monitored metrics, the target value, and the maximum and minimum number of replicas.

[0074] This also includes set policies, such as run time policy, TPS policy, and timing policy. In addition to real-time elasticity plans and offline elasticity plans based on real-time monitoring data, the cloud resource orchestration system can also support pre-setting different expansion and contraction policies to cope with predictable business peaks and valleys.

[0075] Runtime policies are often used for batch processing businesses such as big data analysis and AI training. The running time of such businesses is often long, and the demand for resources varies significantly at different stages of the job. Through the runtime policy, operation and maintenance personnel can set scheduled resource allocation rules for different job stages in advance, such as pre-allocating a certain amount of CPU / GPU resources for data loading when the job starts, dynamically expanding the capacity in the middle stage to support intensive parallel computing, and gradually recovering excess resources at the end stage. This elastic policy based on the job life cycle can maximize the hardware performance and shorten the total running time of the job.

[0076] The TPS (here refers to TPS / QPS) strategy is often used for online service businesses. For this type of business, there are usually regular daily peak and off-peak periods. For example, the browsing and purchase peak of the e-commerce system usually occurs at night, while the payment settlement peak is concentrated during office hours. Operation and maintenance personnel can set different resource demand thresholds for different time periods based on historical TPS data. When the real-time TPS exceeds the preset threshold for a certain period of time, it will automatically trigger capacity expansion; when the TPS drops below another threshold, it will automatically release excess resources. This kind of regularly fluctuating elastic adjustment can effectively cope with the predictable peak load every day, while avoiding the waste of continuous over-allocation of resources.

[0077] Similarly, for some larger-scale predictable events, such as the mid-year promotion of e-commerce platforms and the year-end statistics of operators, operation and maintenance personnel can also formulate event-level elastic strategies to reserve resources on a large scale for these important time nodes in advance. This is collectively referred to as a timing strategy. The system can automatically complete resource deployment and capacity expansion before the event occurs according to the strategy to ensure high performance and reliability of the business; after the event, these temporary resources will also be automatically recycled. In this way, for some predictable traffic peaks such as before the promotion, early morning stress testing, and daily traffic peaks, resources can be reserved in advance through a timing elastic strategy to meet the peak traffic.

[0078] The above-mentioned timed elastic strategies combine the subjective experience judgment of operation and maintenance personnel with the quantitative analysis of historical data, which can effectively support the resource requirements of various periodic and event-based businesses and realize the dynamic supply of "on-demand services". Compared with passive real-time monitoring feedback, the active prediction planning mechanism can avoid delayed configuration or over-allocation to the greatest extent, further optimize resource utilization efficiency, and improve the service capabilities and cost-effectiveness of the business.

[0079] After the above S120, a real-time update plan and an offline elastic plan are formulated, and the real-time update plan and the offline elastic plan both include the target resource quantity. Here, first, a comparison result can be obtained based on the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity.

[0080] In a cloud computing environment, dynamic adjustment of resources is the key to ensuring efficient operation of the system. By comparing the target resource quantity in the real-time update plan and the offline elastic plan with the current resource quantity, a preliminary resource adjustment direction can be obtained. However, this is only the beginning of the decision-making process. In order to ensure the accuracy and effectiveness of resource adjustment, it is also necessary to combine this comparison result with the pre-set strategy, comprehensively consider multiple factors, and finally determine whether to expand or shrink capacity.

[0081] First, you can analyze the specific circumstances of the comparison results. If the target resource quantities in both the real-time update plan and the offline elastic plan are higher than the current resource quantities, this usually means that the system is under pressure from insufficient resources and may need to consider capacity expansion. On the contrary, if the target resource quantities in both plans are lower than the current resource quantities, this may indicate that there is a waste of resources in the system and capacity reduction needs to be considered. However, more commonly, the real-time update plan and the offline elastic plan may give different recommendations. For example, the real-time update plan may recommend capacity expansion due to short-term load fluctuations, while the offline elastic plan may recommend maintaining the status quo or slightly reducing capacity based on long-term trend analysis. In this case, the decision can be further weighed in conjunction with the preset strategy.

[0082] Preset policies are a series of rules and priority settings that guide resource adjustment decisions. These policies usually take into account multiple aspects, including but not limited to: business priority, cost control, performance guarantee, resource utilization efficiency, and system stability. For example, for high-priority critical businesses, the policy may tend to maintain a high degree of resource redundancy to ensure a quick response when the load surges. For low-priority non-core businesses, the policy may focus more on cost-effectiveness and tend to more aggressive resource recovery. At the same time, the policy also needs to consider the frequency and magnitude of resource adjustments to avoid unnecessary impacts on system stability caused by frequent small adjustments.

[0083] When combining the comparison results with the preset strategy, a balanced and gradual approach is required. First, an "adjustment threshold" can be set, and adjustments are only considered when the difference between the target resource quantity and the current resource quantity exceeds this threshold. This helps to filter out short-term fluctuations in resource demand and avoid unnecessary frequent adjustments. Secondly, a concept of "adjustment step size" can be introduced, which is the amount of resources adjusted each time. For expansion operations, a larger step size may be used to quickly respond to the increase in resource demand; for reduction operations, a smaller step size may be selected to reduce the potential impact on system stability.

[0084] In addition, in the decision-making process, you may also need to consider the time window for resource adjustment. Some businesses may have obvious peak and trough periods, and we can define time periods suitable for resource adjustments in the policy. For example, for services for global users, we may choose to make large-scale resource adjustments during time periods when global traffic is relatively low to minimize the impact on user experience. At the same time, we also need to consider the response time of the resource supply chain. Some types of resources (such as dedicated hardware or specifically configured virtual machines) may require a longer preparation time, which requires foresight in resource planning and preparation in advance.

[0085] In actual operation, it is worth noting that resource adjustment decisions are not a one-time process, but a closed-loop system that is continuously optimized. After each adjustment, it may be necessary to closely monitor changes in system performance and business indicators to evaluate the effect of the adjustment. If the adjustment effect is not as expected, it is necessary to analyze the reasons, which may be due to deviations in the prediction model, unreasonable policy settings, or sudden changes in the external environment. Based on these feedbacks, the prediction model can be continuously optimized, policy parameters can be adjusted, and decision logic can be updated, so that the entire resource management system becomes more and more accurate and efficient.

[0086] In general, it is a complex and dynamic process to determine the expansion or reduction of capacity based on the comparison results of the target resource quantity and the current resource quantity based on the real-time update plan and offline elastic plan, combined with the preset strategy. It requires us to balance short-term needs and long-term trends, weigh performance and cost, and consider system stability and business continuity. Through continuous monitoring, analysis, decision-making and optimization, we can build an intelligent, efficient and flexible resource management system to provide the best resource supply for various applications in the cloud computing environment and optimize resource utilization.

[0087] Then, based on the comparison results of the target number of resources in the real-time update plan, the target number of resources in the offline elastic plan and the current number of resources and combined with the set strategy, the expansion / contraction is determined, including determining the number of replicas that need to be adjusted. If the expected number of replicas is inconsistent with the current number of replicas, the HPA Controller will send a request to the Kubernetes API Server to modify the number of replicas of the corresponding Deployment, ReplicaSet or StatefulSet. During the adjustment process, HPA can ensure that the change in the number of replicas is smooth by gradually adjusting the number of replicas to avoid drastic fluctuations that affect system stability.

[0088] Through the above solution, multiple performance indicators are integrated to build a comprehensive performance evaluation system. This multi-dimensional performance indicator integration breaks through the limitations of a single indicator and forms a more comprehensive performance evaluation standard. In particular, the integration of business priority-oriented resource scheduling incorporates business value into resource allocation decisions for the first time, achieving efficient resource utilization. In this way, through intelligent scheduling, idle resources can be avoided, overall utilization can be improved, unnecessary resource consumption can be reduced, and operating costs can be reduced. In this process, high-value services can be prioritized and performance degradation caused by resource competition can be reduced.

[0089] After determining the expansion / reduction according to the above content, you can enter the specific implementation stage and enter the following S140 step.

[0090] S140: According to the determined expansion / reduction plan, the container image is pulled based on data access acceleration technology.

[0091] In the traditional container deployment process, the service startup time usually consists of several main parts: Kubernetes (K8S) scheduling response time, container image pulling time, and service process startup time. In actual applications, container image pulling and service process startup are often the most time-consuming links, with an average time of about a few minutes, and in some cases it may even reach more than ten minutes. This delay not only affects the response speed of the system, but may also lead to a decrease in resource utilization efficiency, especially in scenarios where rapid expansion is required to cope with sudden traffic.

[0092] To solve this problem, modern cloud-native technology has introduced data access acceleration technology based on Nydus and others. The core idea of ​​this technology is to pre-position time-consuming operations such as AI application code pulling, dependency installation, and image building before service release. Specifically, through the cloud-native CI / CD (continuous integration / continuous deployment) orchestration pipeline, the system can build an optimized deployment image in advance. Nydus technology provides an innovative image format for containers, so that only the necessary files need to be loaded when the container starts, thereby reducing the image pulling speed to milliseconds. This optimization not only greatly improves the startup speed of the container, but also reduces the consumption of network bandwidth, because only the data that is really needed is transmitted.

[0093] In S140, in order to further improve the model startup speed, the algorithm service framework can also be optimized and upgraded, and the original service framework code can be decoupled from some general computing logic to make the service framework more lightweight. Among them, the general computing logic can be abstracted into optional / configurable operators, thereby reducing the dependence on middleware during the startup process, and shortening the service process and model startup time to seconds.

[0094] This optimization can also bring indirect benefits. Due to the significant reduction in startup time, the system can perform service updates and version iterations more frequently without significantly affecting the user experience. This provides strong support for continuous integration and continuous deployment (CI / CD) practices, allowing development teams to respond to market needs and fix potential problems more quickly.

[0095] In general, the S140 step effectively solves the performance bottleneck problem in traditional container deployment by combining advanced data access acceleration technology and deep framework optimization. This not only greatly improves the system's response speed and resource utilization efficiency, but also lays the foundation for a more flexible and sophisticated resource scheduling strategy. In the current rapidly changing technology environment, this optimization method will undoubtedly become an important trend in cloud-native application deployment and operation and maintenance, providing enterprises with more efficient and flexible computing resource management solutions.

[0096] The present application also provides a system embodiment for implementing expansion / contraction in an artificial intelligence scenario, including:

[0097] Monitoring module, the monitoring module includes a system performance monitoring module and a service performance monitoring module, wherein:

[0098] The system performance monitoring module monitors system performance and obtains performance indicators;

[0099] The service performance monitoring module monitors service performance and obtains performance indicators;

[0100] A real-time update engine, which formulates a real-time update plan based on the acquired performance indicators, wherein the real-time update plan includes a target resource quantity;

[0101] An offline elasticity engine, which formulates an offline elasticity plan based on the obtained performance indicators, wherein the offline elasticity plan includes a target resource quantity; the offline elasticity plan is constructed based on machine learning of historical indicators;

[0102] The expansion / contraction decision maker determines the expansion / contraction plan based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity and in combination with the set strategy.

[0103] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0104] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0105] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0106] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0107] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0113] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0114] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0116] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0117] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. A method for implementing expansion / contraction in an artificial intelligence scenario. include: Monitor system performance and business performance and obtain performance indicators; Based on the obtained performance indicators, a real-time update plan and an offline elastic plan are formulated respectively, wherein both the real-time update plan and the offline elastic plan include a target resource quantity; the offline elastic plan is constructed based on machine learning of historical indicators; The expansion / contraction plan is determined based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity in combination with the set strategy.

2. The method as claimed in claim 1, wherein the system performance includes CPU performance, and further includes at least one of GPU, memory, storage and network I / O resources.

3. The method according to claim 1, wherein the business performance includes the number of queries per second, the number of transactions per second, business delay and / or business response time.

4. The method as described in claim 1, wherein the real-time update plan also includes business priority awareness.

5. According to the method as described in claim 1, the set strategy includes a runtime strategy, a TPS strategy, and a timing strategy.

6. According to the method as described in claim 1, the expansion / contraction scheme includes adjusting the threshold and the adjustment step size.

7. The method according to claim 1, further comprising: include: According to the determined expansion / reduction plan, the container image is pulled based on data access acceleration technology.

8. The method according to claim 7, further comprising: include: The algorithm service framework is optimized and upgraded, and the original service framework code is decoupled from some general computing logic to make the service framework more lightweight.

9. The method according to claim 7, further comprising: include: Abstract general computing logic into optional / configurable operators to reduce dependence on middleware during startup.

10. A system for implementing expansion / contraction in artificial intelligence scenarios. include: Monitoring module, the monitoring module includes a system performance monitoring module and a service performance monitoring module, wherein: The system performance monitoring module monitors system performance and obtains performance indicators; The service performance monitoring module monitors service performance and obtains performance indicators; A real-time update engine, which formulates a real-time update plan based on the acquired performance indicators, wherein the real-time update plan includes a target resource quantity; An offline elasticity engine, which formulates an offline elasticity plan based on the obtained performance indicators, wherein the offline elasticity plan includes a target resource quantity; the offline elasticity plan is constructed based on machine learning of historical indicators; The expansion / contraction decision maker determines the expansion / contraction plan based on the comparison result of the target resource quantity in the real-time update plan, the target resource quantity in the offline elastic plan and the current resource quantity and in combination with the set strategy.

Images