Scaling strategy model generation and deployment execution method and device, equipment and medium

Abstract

The invention relates to the technical field of computers, can be applied to the fields of financial science and technology and medical health, and discloses a scaling strategy model generation and deployment execution method, device, equipment and medium, the method comprises the following steps: constructing a hardware-in-the-loop simulation environment, the hardware-in-the-loop simulation environment is used for simulating a running state and a performance index of a GPU inference service, and the hardware-in-the-loop simulation environment is used for simulating the running state and the performance index of the GPU inference service; the deviation between the key index in the hardware-in-the-loop simulation environment and the key index in the production hardware environment is within a preset threshold value during the GPU reasoning service; and in the hardware-in-the-loop simulation environment, generating an automatic scaling strategy model by training a reinforcement learning agent. The accuracy of scaling decision is improved, and the cost of scaling strategy model training is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology and can be applied to the fields of financial technology and healthcare. In particular, it relates to a method, apparatus, device and medium for generating and deploying scalable strategy models. Background Technology

[0002] With the rapid development of artificial intelligence technology, AI model inference services have become a core component of critical businesses such as fintech, healthcare, real-time recommendation, and autonomous driving, with stringent requirements for service response latency and availability. These services typically employ GPU-accelerated computing and are deployed on Kubernetes container orchestration platforms to achieve elasticity and maintainability. To cope with dynamically changing request loads, Kubernetes container orchestration platforms provide Horizontal Pod Autoscaler (HPA) as a standard solution. However, the existing HPA mechanism has the following problems in GPU inference service scenarios: 1. Core metric mismatch: HPA relies on general metrics such as CPU / memory utilization for decision-making, but the real bottleneck of GPU-intensive inference tasks is not directly related to CPU / memory utilization. When GPU resources reach their bottleneck, causing latency spikes, HPA may not trigger scaling because CPU metrics have not reached the threshold, resulting in a severe decline in the quality of GPU inference services. 2. Delayed response mechanism: HPA is essentially a reactive mechanism, triggering scaling only after metrics exceed thresholds. Its response chain includes metric collection latency, controller decision latency, Pod scheduling and startup latency, and application cold start time. When faced with sudden surges in traffic, delayed responses cannot prevent instantaneous performance avalanche.

[0003] Reinforcement learning (RL) is considered an ideal paradigm for solving complex dynamic decision-making problems. However, directly applying RL to train autoscaling strategies for production-grade Kubernetes clusters faces two major obstacles: 1. High training risk, which may lead to service interruptions; 2. High training cost, consuming a large amount of expensive GPU resources. Existing Kubernetes emulators are mostly pure software simulations, which cannot accurately simulate the real behavior of GPU hardware, making it difficult to directly migrate simulation-trained strategies to production environments. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for generating and deploying a scaling strategy model, in order to solve the technical problems of low accuracy in scaling decisions and high training costs of scaling strategy models in existing GPU inference services.

[0005] Firstly, a method for generating scaling strategy models for GPU inference services is provided, including:

[0006] A hardware-in-the-loop simulation environment is constructed, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold.

[0007] In the hardware-in-the-loop simulation environment, an autoscaling policy model is generated by training a reinforcement learning agent.

[0008] Secondly, a method for deploying and executing a scaling strategy model for GPU inference services is provided, including:

[0009] The scaling strategy model is loaded through the autoscaling controller, and the real-time state vector corresponding to the state space is obtained from the monitoring system of the Kubernetes production cluster.

[0010] The real-time state vector is input into the autoscaling strategy model for inference to generate a scaling decision;

[0011] The number of replicas of the GPU inference service is automatically adjusted based on the scaling decision by calling the production Kubernetes API.

[0012] Thirdly, a scaling strategy model generation device for GPU inference services is provided, comprising:

[0013] A construction unit is used to construct a hardware-in-the-loop simulation environment, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold.

[0014] The generation unit is used to generate an autoscaling policy model by training a reinforcement learning agent in the hardware-in-the-loop simulation environment.

[0015] The fourth unit provides a scaling strategy model deployment and execution device for GPU inference services, including:

[0016] The loading and retrieval unit is used to load the scaling strategy model through the autoscaling controller, from...

[0017] The monitoring system for the Kubernetes production cluster obtains the real-time state vector corresponding to the state space.

[0018] The reasoning generation unit is used to input the real-time state vector into the auto-scaling strategy model for reasoning to generate a scaling decision.

[0019] The adjustment unit is invoked to automatically adjust the number of replicas of the GPU inference service based on the scaling decision by invoking the production Kubernetes API.

[0020] Fifthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for generating a scaling strategy model for a GPU inference service or a method for deploying and executing a scaling strategy model for a GPU inference service.

[0021] In a sixth aspect, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the above-described method for generating a scaling strategy model for a GPU inference service or a method for deploying and executing a scaling strategy model for a GPU inference service.

[0022] The above-mentioned method, apparatus, device, and medium for generating and deploying scaling strategy models can construct a hardware-in-the-loop simulation environment. This environment simulates the running state and performance metrics of a GPU inference service. During GPU inference, the deviation between key metrics in the hardware-in-the-loop simulation environment and those in the production hardware environment is within a preset threshold. Within this hardware-in-the-loop simulation environment, an autoscaling strategy model is generated by training a reinforcement learning agent. In this invention, the deviation between key metrics in the hardware-in-the-loop simulation environment and those in the production hardware environment is within a preset threshold during GPU inference, thus improving the accuracy of scaling decisions. Furthermore, generating an autoscaling strategy model within the constructed hardware-in-the-loop simulation environment by training a reinforcement learning agent avoids directly applying reinforcement learning to train an autoscaling strategy model on a production-grade Kubernetes cluster, reducing the cost of training the scaling strategy model. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a method for generating a scaling strategy model for GPU inference services according to an embodiment of the present invention.

[0025] Figure 2 yes Figure 1 A schematic diagram of a specific implementation of step S120;

[0026] Figure 3 yes Figure 1 A flowchart illustrating a specific implementation of step S123;

[0027] Figure 4 yes Figure 1 A schematic diagram of a specific implementation of step S124;

[0028] Figure 5 This is a flowchart illustrating a method for deploying and executing a scaling strategy model for GPU inference services according to an embodiment of the present invention.

[0029] Figure 6 This is a schematic block diagram of a scaling strategy model generation device for GPU inference services according to an embodiment of the present invention;

[0030] Figure 7 This is a schematic block diagram of a GPU inference service scaling strategy model deployment and execution device according to an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The scaling strategy model generation method for GPU inference services provided in this invention can be applied to either the client or server. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. Currently, AI model inference services have become a core component of critical businesses such as fintech, healthcare, real-time recommendation, and autonomous driving. Existing GPU inference services suffer from low accuracy in scaling decisions and high training costs for scaling strategy models. To address these issues, this invention proposes a scaling strategy model generation method for GPU inference services. This method first ensures that the deviation between key metrics in the hardware-in-the-loop simulation environment and key metrics in the production hardware environment is within a preset threshold during GPU inference, thus improving the accuracy of scaling decisions. In the constructed hardware-in-the-loop simulation environment, an automatic scaling strategy model is generated by training a reinforcement learning agent, avoiding the direct application of reinforcement learning to train the automatic scaling strategy model on a production-grade Kubernetes cluster, thereby reducing the training cost of the scaling strategy model. The invention will be described in detail below through specific embodiments.

[0035] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a method for generating a scaling strategy model for GPU inference services provided in an embodiment of the present invention includes the following steps: S110-S120.

[0036] S110. Construct a hardware-in-the-loop simulation environment, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold.

[0037] Specifically, the scaling strategy model generation method for GPU inference services in this application is applied to an automatic scaling system. The core of this scaling system is a hardware-in-the-loop simulation architecture, in which a reinforcement learning agent interacts with a controlled object containing real hardware, forming a closed-loop training system. It should be noted that a hardware-in-the-loop simulation environment is built within the hardware-in-the-loop simulation architecture. It should also be noted that the hardware-in-the-loop simulation environment is a high-fidelity simulation environment, and its construction process is as follows: 1. Building the basic environment: Install the Linux operating system on physical nodes of a platform equipped with GPU hardware and artificial intelligence computing capabilities (e.g., real-time risk control and quantitative trading platforms in the fintech field, medical image analysis and new drug development platforms in the healthcare field), deploy a Kubernetes cluster, and integrate drivers with GPU hardware and artificial intelligence computing capabilities, toolkits with parallel computing platform and programming model capabilities, and containers with GPU hardware and artificial intelligence computing capabilities; 2. Monitoring system: Deploy tools with GPU metric acquisition capabilities that can collect metrics through pull mode, support multi-dimensional data storage and querying, automatically identify resources such as Pods and Nodes in the cluster, and analyze metric trends (such as GPU utilization fluctuations) in real time, or collect fine-grained performance data from the GPU hardware layer; 3. Heterogeneous and load simulation: Simulate a heterogeneous node environment through Kubernetes node tagging and taint mechanisms. Deploy one or more Pod replicas of GPU inference services and set resource requests and limits to accurately simulate resource allocation. In essence, a Kubernetes cluster refers to a distributed computing platform based on container orchestration technology, consisting of a set of physical or virtual nodes, used to deploy, manage, and elastically scale cloud-native applications such as GPU inference services. A Pod replica refers to a single Kubernetes Pod, containing a complete inference service container, dependency libraries, and GPU resource requests. By implementing step S110, the risks and costs of directly training reinforcement learning in production systems are resolved, significantly accelerating the R&D iteration cycle of intelligent policies.

[0038] Furthermore, constructing a high-fidelity hardware-in-the-loop simulation environment refers to combining actual hardware devices with a computer-simulated virtual environment. The deviations of key performance indicators (KPIs) in the hardware-in-the-loop simulation environment from those in the production hardware environment are within a preset threshold; that is, the error of the key performance indicators is less than or equal to the preset threshold. By integrating real GPU hardware, accurately simulating Kubernetes scheduling behavior, and deploying a monitoring system consistent with the production environment, the simulation environment is ensured to be highly consistent with the real production hardware environment in the following aspects: 1. Hardware behavior: Accurately reflects the computing performance, memory bandwidth, driver behavior, and device plugin interactions of the real GPU; 2. Resource scheduling: Accurately simulates the Kubernetes scheduler's allocation logic for GPU resources and Pod startup / termination latency; 3. Monitoring indicators: Ensures that the errors between key indicators such as GPU utilization, memory usage, and task execution latency collected by the monitoring system and those collected in the real production hardware environment are within a preset threshold; 4. Workload performance: Under the same input requests, ensures that the performance indicators of the GPU inference service in the hardware-in-the-loop simulation environment deviate from those in the real production hardware environment within a preset threshold.

[0039] S120. In the hardware-in-the-loop simulation environment, an autoscaling strategy model is generated by training a reinforcement learning agent.

[0040] Specifically, the hardware-in-the-loop simulation environment includes a Kubernetes cluster and a monitoring system. Before training the reinforcement learning agent, the scaling system needs to be initialized, for example, the reinforcement learning method, such as the Actor-Critic neural network of the PPO algorithm. In this embodiment, both the Actor network and the Critic network adopt a multilayer perceptron structure and set hyperparameters such as learning rate and discount factor.

[0041] Among them, such as Figure 2 As shown, step S120 includes the following steps: S121-S124:

[0042] S121. At each decision time step, obtain a state vector representing its current state from the monitoring system;

[0043] S122, The reinforcement learning agent outputs a scaling action to adjust the GPU inference service based on the current state and the internal policy of the monitoring system;

[0044] S123. Execute the scaling action in the hardware-in-the-loop simulation environment through the Kubernetes API corresponding to the Kubernetes cluster, and calculate the reward value according to the predefined composite reward function after the scaling action is executed;

[0045] S124. Update the policy of the reinforcement learning agent according to the state vector, the scaling action and the reward value until the policy converges to obtain the auto-scaling policy model.

[0046] Specifically, the state vector includes GPU utilization, service quality metrics, the current number of replicas of the GPU inference service, and trend features for predictive decision-making. The trend features include request throughput difference and service response latency difference, where the request throughput difference is a first-order difference, the service response latency difference is a first-order difference, and the service quality metric is service response latency (e.g., P95 latency). The scaling action represents the adjustment amount to the number of replicas of the GPU inference service, with the adjustment value set being {-2, -1, 0, +1, +2}. Through steps S121-S124, by introducing trend features, predictive scaling is achieved, proactively preparing resources before peak loads. Compared to baseline methods such as HPA, this can reduce service P95 latency. Furthermore, by using a clear composite reward function to guide the strategy to maximize GPU utilization while meeting latency targets, GPU utilization can be improved and resource costs reduced. The autoscaling strategy model based on reinforcement learning agents does not rely on fixed rules or traffic assumptions. It can learn autonomously and adapt to complex and ever-changing traffic patterns (such as periodic, random, and bursty traffic) through continuous interaction with a high-fidelity environment, demonstrating excellent robustness and generalization ability.

[0047] Among them, such as Figure 3 As shown, step S123 includes the following steps: S1231-S1232:

[0048] S1231. After the scaling action is executed, collect performance metrics, including GPU utilization, observed service latency, and current number of replicas.

[0049] S1232. Obtain the target utilization rate, and calculate the reward value according to the predefined composite reward function based on the target utilization rate and the performance index.

[0050] Specifically, at each decision time step, for example every 15 seconds, after the scaling action is executed, Prometheus is queried to collect performance metrics. The performance metrics include at least the current number of replicas, GPU utilization, and observed service latency (e.g., P95 latency). The performance metrics may also include request throughput and latency difference and throughput difference as trend features. The latency difference is calculated using the difference between the P95 latency of the current time step and the previous time step, and the throughput difference is calculated using the difference between the request throughput of the current time step and the previous time step. Action Decision (A_t) The Actor network receives the state S_t and outputs a scaling action A_t, whose value set is {-2,-1,0,+1,+2}, representing the adjustment amount of the number of GPU replicas. The scaling operation is executed through the Kubernetes API. Calculate Reward (R_t) After the action is executed, the reward value is calculated according to the following composite reward function, as shown in formula (1):

[0051] R t =-α·Latency t -β·|GPUUtil t -TargetUtil|-γ·Replicas t (1)

[0052] Latency t For the observed service latency, GPUUtil t For GPU utilization, TargetUtil represents the target utilization, and Replicas represents the target utilization. t The current number of replicas is α, β, and γ, which are positive weighting coefficients used to balance the importance of various indicators. In this embodiment, they can be set to α = 1.0, β = 0.5, and γ = 0.1.

[0053] Among them, such as Figure 4 As shown, step S124 includes the following steps: S1241-S1242:

[0054] S1241. Update the policy of the reinforcement learning agent using a reinforcement learning method based on the state vector, the scaling action, and the reward value, and output a temporary optimized policy;

[0055] S1242. Evaluate the temporary optimization strategy. If the evaluation result shows that the temporary optimization strategy has converged, output the automatic scaling strategy model.

[0056] Specifically, in a hardware-in-the-loop simulation environment, when the reinforcement learning agent performs a scaling action (such as adjusting the number of replicas by ±2) based on the state vector obtained from the monitoring system (including GPU utilization, P95 latency, current number of replicas, and trend features such as request throughput differential and service latency differential used for predictive decision-making), the scaling system calculates an immediate reward value based on a composite reward function (this function comprehensively penalizes service latency bias, the difference between GPU utilization and the target value, and resource overhead to balance multi-objective optimization). The state vector, scaling action, and reward value are then input into the reinforcement learning method (e.g., proximal policy optimization). In Optimization (PPO), the policy network parameters of the agent are updated by calculating the policy gradient (specifically involving the decision weights of the Actor network and the value evaluation of the Critic network), thus outputting a temporary optimized policy. This update process aims to guide the policy to favor higher-reward actions (e.g., scaling up in advance under increasing latency), while the constraint mechanism of the PPO algorithm avoids training instability caused by policy mutations. Subsequently, this temporary optimized policy needs to be rigorously evaluated in various simulated load scenarios (such as stepped and pulsed traffic). The convergence criteria include: a small fluctuation range in the average reward value across consecutive training rounds. Within a preset threshold (e.g., ±2%), the distribution of action outputs under the same input state tends to be consistent (the standard deviation of action probability approaches zero), and key performance indicators reach saturation (e.g., service latency is consistently lower than the preset target, and GPU utilization deviation is controlled within 5%). If the evaluation confirms that the policy has converged, the final policy network parameters are saved, generating an auto-scaling policy model (lightweight inference model) that can be directly deployed to the production environment for real-time adjustment of the number of replicas of the GPU inference service; if it has not converged, the training loop is returned to continue iterating. This closed-loop process runs safely and efficiently in a hardware-in-the-loop simulation environment, ultimately generating a highly robust model.

[0057] Figure 5 This is a flowchart illustrating a method for deploying and executing a scaling strategy model for a GPU inference service according to an embodiment of the present invention. The deployment and execution of the scaling strategy model for the GPU inference service is applied to devices in the production environment, such as... Figure 5 As shown, in this embodiment, the method includes steps S120-S230:

[0058] S210. Load the scaling strategy model through the autoscaling controller and obtain the real-time state vector corresponding to the state space from the monitoring system in the Kubernetes production cluster.

[0059] S220. Input the real-time state vector into the automatic scaling strategy model for inference to generate a scaling decision;

[0060] S230. Based on the scaling decision, call the production Kubernetes API to automatically adjust the number of replicas of the GPU inference service.

[0061] Specifically, after model training is complete, the autoscaling strategy model is saved and deployed to a Kubernetes production cluster. This production cluster can be a pre-production or production Kubernetes cluster. An autoscaling controller, a lightweight controller, is deployed within the production cluster. This controller loads the autoscaling strategy model and continuously retrieves real-time state vectors corresponding to the state space from the Kubernetes production cluster's monitoring system. These vectors are then input into the autoscaling model for inference, generating scaling decisions. The production Kubernetes API is then invoked to automatically adjust the number of replicas of the GPU inference service. It's important to note that the number of replicas of the GPU inference service refers to the number of Pod instances running concurrently for the same inference service within the Kubernetes cluster. Each Pod instance exclusively utilizes GPU resources and processes requests independently.

[0062] To verify the effectiveness of the autoscaling strategy model, tests were conducted in the following environment: 1. Hardware environment: Server nodes equipped with NVIDIA A800 GPUs; 2. Software environment: Kubernetes v1.23.3, NVIDIA driver version 535.104.05; 3. Inference service: A large language model inference service deployed based on an engine with ultra-large-scale language model service capabilities; 4. Load generation: Simulating various typical loads such as stepped, periodic, and pulsed loads using tools with performance testing tools; 5. Baseline method: Standard Kubernetes HPA, where HPA is a horizontal Pod autoscaler. The test results are shown in Table 1.

[0063] Table 1

[0064] Evaluation indicators Method of the present invention HPA baseline Improved proportions P95 latency (ms) 15.2 102.3 Reduced by up to 6.7 times GPU utilization (%) 78.5 63.6 An increase of 23.4% Resource costs (relative value) 0.82 1.00 18% lower

[0065] The test results show that the present invention is significantly better than the HPA baseline method in key indicators. In particular, when faced with pulsed loads, the predictive scaling mechanism of the present invention can effectively avoid performance avalanche.

[0066] The method for generating a scaling strategy model for GPU inference services in this invention first ensures that the deviation between key indicators in the hardware-in-the-loop simulation environment and key indicators in the production hardware environment is within a preset threshold during GPU inference services, thus improving the accuracy of scaling decisions. In the constructed hardware-in-the-loop simulation environment, an automatic scaling strategy model is generated by training a reinforcement learning agent, avoiding the direct application of reinforcement learning to train an automatic scaling strategy model for production-grade Kubernetes clusters, thereby reducing the cost of training the scaling strategy model.

[0067] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0068] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0069] In one embodiment, a scaling strategy model generation apparatus 200 for GPU inference services is provided, which corresponds one-to-one with the scaling strategy model generation method for GPU inference services in the above embodiments. For example... Figure 6 As shown, the scaling strategy model generation device 200 for the GPU inference service includes a construction unit 201 and a generation unit 202. Detailed descriptions of each functional module are as follows:

[0070] The construction unit 201 is used to construct a hardware-in-the-loop simulation environment, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold.

[0071] The generation unit 202 is used to generate an autoscaling policy model by training a reinforcement learning agent in the hardware-in-the-loop simulation environment.

[0072] In one embodiment, the generation unit 202 is specifically used for:

[0073] At each decision time step, a state vector representing its current state is obtained from the monitoring system;

[0074] The reinforcement learning agent outputs a scaling action to adjust the GPU inference service based on the current state and the internal policy of the monitoring system.

[0075] The scaling action is executed in the hardware-in-the-loop simulation environment through the Kubernetes API corresponding to the Kubernetes cluster, and the reward value is calculated according to the predefined composite reward function after the scaling action is executed.

[0076] The policy of the reinforcement learning agent is updated based on the state vector, the scaling action, and the reward value until the policy converges, thus obtaining the autoscaling policy model.

[0077] In one embodiment, the generation unit 202 is further configured to:

[0078] The state vector includes GPU utilization, service quality metrics, the number of replicas of the current GPU inference service, and trend features for making predictive decisions, wherein the trend features include request throughput differential and service response latency differential.

[0079] In one embodiment, the generation unit 202 is further configured to:

[0080] After the scaling action is executed, performance metrics are collected, including GPU utilization, observed service latency, and current number of replicas.

[0081] Obtain the target utilization rate, and calculate the reward value based on the target utilization rate and the performance index according to the predefined composite reward function.

[0082] In one embodiment, the generation unit 202 is further configured to:

[0083] The policy of the reinforcement learning agent is updated using a reinforcement learning method based on the state vector, the scaling action, and the reward value, and a temporary optimized policy is output.

[0084] The temporary optimization strategy is evaluated. If the evaluation result shows that the temporary optimization strategy has converged, the automatic scaling strategy model is output.

[0085] In one embodiment, a scaling strategy model deployment and execution apparatus 300 for a GPU inference service is provided, which corresponds one-to-one with the scaling strategy model deployment and execution method for the GPU inference service described in the above embodiments. For example... Figure 7 As shown, the GPU inference service scaling strategy model deployment execution device 300 includes a loading and acquisition unit 301, an inference generation unit 302, and a call and adjustment unit 303. Detailed descriptions of each functional module are as follows:

[0086] Loading and acquisition unit 301 is used to load the scaling strategy model through the automatic scaling controller, from

[0087] The monitoring system for the Kubernetes production cluster obtains the real-time state vector corresponding to the state space.

[0088] The reasoning generation unit 302 is used to input the real-time state vector into the automatic scaling strategy model for reasoning to generate scaling decisions;

[0089] The adjustment unit 303 is invoked to automatically adjust the number of replicas of the GPU inference service by invoking the production Kubernetes API based on the scaling decision.

[0090] The GPU inference service scaling strategy model generation device of this invention first ensures that the deviation between key indicators in the hardware-in-the-loop simulation environment and key indicators in the production hardware environment is within a preset threshold during GPU inference service, thus improving the accuracy of scaling decisions; in the constructed hardware-in-the-loop simulation environment, an automatic scaling strategy model is generated by training a reinforcement learning agent, avoiding the direct application of reinforcement learning to train an automatic scaling strategy model in a production-grade Kubernetes cluster, thereby reducing the cost of training the scaling strategy model.

[0091] Specific limitations regarding the scaling strategy model generation device for GPU inference services can be found in the limitations regarding the scaling strategy model generation method for GPU inference services described above, and will not be repeated here. Specific limitations regarding the scaling strategy model deployment and execution device for GPU inference services can also be found in the limitations regarding the scaling strategy model deployment and execution method for GPU inference services described above, and will not be repeated here. Each unit in the aforementioned scaling strategy model generation device and scaling strategy model deployment and execution device for GPU inference services can be implemented entirely or partially through software, hardware, or a combination thereof. Each of these units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0092] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a GPU inference service scaling strategy model generation method on the server side, or implements the functions or steps of a GPU inference service scaling strategy model deployment and execution method.

[0093] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a GPU inference service scaling strategy model generation method on the client side, or implements the functions or steps of a GPU inference service scaling strategy model deployment and execution method.

[0094] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described GPU inference service scaling strategy model generation method or the steps of the above-described GPU inference service scaling strategy model deployment and execution method.

[0095] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of the above-described method for generating a scaling strategy model for a GPU inference service, or the steps of the above-described method for deploying and executing a scaling strategy model for a GPU inference service.

[0096] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0098] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating a scaling strategy model for GPU inference services, characterized in that, include: A hardware-in-the-loop simulation environment is constructed, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold. In the hardware-in-the-loop simulation environment, an autoscaling policy model is generated by training a reinforcement learning agent.

2. The method for generating a scaling strategy model for GPU inference services as described in claim 1, characterized in that, The hardware-in-the-loop simulation environment includes a Kubernetes cluster and a monitoring system. The step of generating an autoscaling strategy model by training a reinforcement learning agent includes: At each decision time step, a state vector representing its current state is obtained from the monitoring system; The reinforcement learning agent outputs a scaling action to adjust the GPU inference service based on the current state and the internal policy of the monitoring system. The scaling action is executed in the hardware-in-the-loop simulation environment through the Kubernetes API corresponding to the Kubernetes cluster, and the reward value is calculated according to the predefined composite reward function after the scaling action is executed. The policy of the reinforcement learning agent is updated based on the state vector, the scaling action, and the reward value until the policy converges, thus obtaining the autoscaling policy model.

3. The method for generating a scaling strategy model for GPU inference services as described in claim 2, characterized in that, The state vector includes GPU utilization, service quality metrics, the number of replicas of the current GPU inference service, and trend features for making predictive decisions, wherein the trend features include request throughput differential and service response latency differential.

4. The method for generating a scaling strategy model for GPU inference services as described in claim 2, characterized in that, The step of calculating the reward value according to a predefined composite reward function after the scaling action is executed includes: After the scaling action is executed, performance metrics are collected, including GPU utilization, observed service latency, and current number of replicas. Obtain the target utilization rate, and calculate the reward value based on the target utilization rate and the performance index according to the predefined composite reward function.

5. The method for generating a scaling strategy model for GPU inference services as described in claim 2, characterized in that, The step of updating the policy of the reinforcement learning agent based on the state vector, the scaling action, and the reward value until the policy converges to obtain the autoscaling policy model includes: The policy of the reinforcement learning agent is updated using a reinforcement learning method based on the state vector, the scaling action, and the reward value, and a temporary optimized policy is output. The temporary optimization strategy is evaluated. If the evaluation result shows that the temporary optimization strategy has converged, the automatic scaling strategy model is output.

6. A method for deploying and executing a scaling strategy model for GPU inference services, characterized in that, include: The scaling strategy model is loaded through the autoscaling controller, and the real-time state vector corresponding to the state space is obtained from the monitoring system of the Kubernetes production cluster. The real-time state vector is input into the autoscaling strategy model for inference to generate a scaling decision; The number of replicas of the GPU inference service is automatically adjusted based on the scaling decision by calling the production Kubernetes API.

7. A scaling strategy model generation device for GPU inference services, characterized in that, include: A construction unit is used to construct a hardware-in-the-loop simulation environment, wherein the hardware-in-the-loop simulation environment is used to simulate the running state and performance indicators of the GPU inference service, and during the GPU inference service, the deviation between the key indicators in the hardware-in-the-loop simulation environment and the key indicators in the production hardware environment is within a preset threshold. The generation unit is used to generate an autoscaling policy model by training a reinforcement learning agent in the hardware-in-the-loop simulation environment.

8. A scaling strategy model deployment and execution device for GPU inference services, characterized in that, include: The loading and acquisition unit is used to load the scaling strategy model through the autoscaling controller and obtain the real-time state vector corresponding to the state space from the monitoring system of the Kubernetes production cluster. The reasoning generation unit is used to input the real-time state vector into the auto-scaling strategy model for reasoning to generate a scaling decision. The adjustment unit is invoked to automatically adjust the number of replicas of the GPU inference service based on the scaling decision by invoking the production Kubernetes API.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the GPU inference service scaling strategy model generation method as described in any one of claims 1 to 5 or the GPU inference service scaling strategy model deployment execution method as described in claim 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the scaling strategy model generation method for the GPU inference service as described in any one of claims 1 to 7, or the scaling strategy model deployment and execution method for the GPU inference service as described in claim 6.

Images