AI model reasoning cache configuration method oriented to cloud native container environment

Abstract

The invention discloses a cloud native container environment-oriented AI model reasoning cache configuration method. The method comprises the following steps of constructing a model loading time estimation model in an AI reasoning task; collecting model loading time under different AI models and different data access throughout capacities, and fitting parameters of a loading time model; constructing a minimum cache cost optimization problem under the constraint of model loading time; simplifying the constraint optimization problem, and converting the constraint optimization problem into continuous differentiable sub-optimization problems under a plurality of interval sections; the system solves a constraint optimization problem according to a service level target submitted by a user and an AI reasoning service parameter, and recommends a cache configuration with the lowest cost under a constraint condition; the system carries out affinity scheduling on the cache container and the inference service container according to the cache configuration, and determines a short-circuit read optimization strategy; and the system creates cache resources according to cache configuration and a short-circuit read optimization strategy, and provides acceleration capability for the model loading process of the AI inference service.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to an AI model inference cache configuration method for a cloud-native container environment. Background Art

[0002] Today, artificial intelligence technologies, exemplified by deep learning and large models, are rapidly developing. Artificial intelligence models (AI models) are widely used in numerous fields, such as data analysis, recommendation systems, image recognition, content generation, and chatbots. Deep learning applications involve two phases: model training and model inference. The latter phase can directly generate value for various industries. The powerful reasoning capabilities demonstrated by large language models in recent years have prompted a growing number of industries to accelerate the deployment and construction of AI model inference services. However, AI model inference workloads rely on powerful computing power, requiring low resource consumption, high throughput, and low latency in terms of both inference performance and cost.

[0003] In recent years, thanks to the advantages of the cloud-native application development paradigm, such as convenient operations and maintenance, resource elasticity, and pay-as-you-go, deploying AI inference services on cloud platforms has become a trend to meet the resource overhead and performance requirements of AI model inference workloads. However, the traditional cloud-native development paradigm still requires developers to maintain infrastructure and Kubernetes clusters. To help developers focus more on business development and avoid infrastructure operations and maintenance, major cloud vendors have launched elastic container services. Users do not need to manage the underlying servers; they only need to provide packaged container images and customize container specifications to run the container, and pay according to the runtime and container specifications. However, cloud-native container environments typically adopt a storage-compute separation architecture, which leads to cross-network data transmission. The high latency of AI model loading exacerbates the cold start problem of AI inference services, thereby affecting end-to-end inference latency.

[0004] To accelerate the startup of inference services, a common approach is to use distributed cache to pre-fetch AI models distributed across different storage data sources, fully utilizing the bandwidth within the cluster to accelerate model loading. In a cloud-native container environment, each component of the distributed cache is deployed in a containerized manner. To further accelerate model loading, high-performance, pay-as-you-go cloud disks provided by the cloud platform are typically used as cache storage media. However, cache containers and cloud disks have fine billing granularity and a wide variety of configuration combinations. The performance of the inference service is related to resource configuration, making it difficult for developers to choose the appropriate configuration to achieve a balance between monetary cost and service quality. Summary of the Invention

[0005] Purpose of the invention: To address the problem of how to efficiently select appropriate cache resource configuration and achieve a trade-off between monetary cost and service quality, the purpose of the present invention is to provide an AI model inference cache configuration method for cloud-native container environments, so that developers can efficiently select low-cost cache configurations while meeting the service level goals of AI model loading time.

[0006] Technical Solution: To achieve the above-mentioned purpose, the present invention proposes an AI model inference cache configuration method for a cloud-native container environment, comprising the following steps:

[0007] (1) Based on the cloud disk configuration, cache container configuration, and inference service container configuration, a mathematical expression for the model loading time estimation model is constructed;

[0008] (2) Collect the model loading time during the AI model inference process under different data access throughputs, and use the least squares method to fit the parameters to be fitted in the functional relationship between model loading time and data access throughput;

[0009] (3) Model the monetary cost of model loading during AI model inference in a cloud-native container environment, construct a constrained optimization problem, and find the cloud disk configuration and cache container configuration that minimizes the monetary cost under the constraints;

[0010] (4) Simplify and split the objective function and constraints of the constrained optimization problem in step (3) and transform them into continuous differentiable sub-optimization problems in multiple intervals;

[0011] (5) Based on the service level goals submitted by the user and the inferred service container configuration, the segmented sequential quadratic programming algorithm is used to solve the constrained optimization problem to obtain the cloud disk configuration and cache container configuration that minimizes the monetary cost and satisfies the constraints;

[0012] (6) Based on the quantitative relationship between the cloud disk configuration and the cache container configuration, the affinity scheduling strategy between the cache container and the inference service container is determined to achieve the effect of short-circuit read optimization;

[0013] (7) Create cache resources in the cloud platform based on the cache container configuration and short-circuit read optimization strategy to provide acceleration capabilities for the model loading process of AI reasoning.

[0014] Furthermore, in step (1), the cloud disk configuration includes the cloud disk type diskType, cloud disk capacity capacity (GB), and cloud disk throughput tp (MB / s); the cache container configuration includes the number of containers n for caching AI models. cache , the data access bandwidth of the container used to cache the AI model B cache (converted to MB / s), the number of CPUs n in the container used to cache the AI model cpu, the memory size of the container used to cache the AI model n mem (GB); the inference service container configuration includes the number of inference service containers N infer ; Model loading time estimation model t load The mathematical expression of t load =t(min(tp,B)), where B is the actual data access bandwidth during model loading, and t(·) represents the functional relationship between model loading time and data access throughput. The cloud disk throughput tp is calculated as tp=min(α diskType *capacity+β diskType ,maxThroughput diekTypm ), α diskType is the correlation coefficient, β diskType The minimum throughput of the cloud disk (MB / s), maxThroughput diskType is the maximum throughput of the cloud disk (MB / s). The actual data access bandwidth B during model loading is calculated as B=n cache *B cache / N infer .

[0015] Furthermore, in step (2), the functional relationship between the model loading time t and the data access throughput b is log(t)=θ1*log(b)+θ2, where log(t) is the logarithm of t with the natural constant e as the base, log(b) is the logarithm of b with the natural constant e as the base, and θ1 and θ2 are parameters to be fitted; by collecting the model loading time in the AI model inference process under different data access throughputs, the least squares method is used to fit θ1 and θ2 in the functional relationship.

[0016] Furthermore, in step (3), the monetary cost of model loading in the AI model inference process is calculated as cost=t load *n cache *(cost diskType *capacity+cost cpu *n cpu +cost mem *n mem ), where cost diskType Indicates the unit cost of diskType cloud disk (yuan / GB / second), cost cpu Indicates the unit CPU cost (yuan / core / second) of the container used to cache the AI model. mem represents the unit memory cost of the container used to cache the AI model (yuan / GB / second). The constrained optimization problem is formally described as:

[0017] min cost

[0018] stt load ≤T SLO

[0019] diskType∈Φ

[0020] M / n cache ≤capacity≤maxCapacity diskType

[0021] B cache ,n cache ,capacity,n cpu ,n mem ∈Z

[0022] Where T SLO is the model loading time service level target (seconds), Φ is the set of cloud disk types provided by the cloud vendor, M is the data size of the AI model (GB), and maxCapacity diskType is the maximum capacity (GB) of a diskType cloud disk, and Z is an integer set.

[0023] Furthermore, in step (4), since the memory size of the container used to cache the AI model does not affect the model loading time, the cost of the money cost is eliminated. mem Related items; due to the data access bandwidth B of the cache container cache The number of CPU cores n of the cache container cpu In a linear relationship, we use B cache To express n cpu , so the simplified objective function cost′ is expressed as cost′=t load *n cache *(cost diskType *capacity+cost cpu *(γ*B cache +δ)), where γ and δ are correlation coefficients; in addition, diskType is not used as a decision variable, and each diskType value can be traversed later when solving the optimization problem; since the cost′ function and the constraints contain the min(tp,B) term, and the calculation expression of tp also contains the maximum value function, resulting in the function being discontinuously differentiable, the objective function cost′ is split into piecewise functions on four interval segments through classification discussion, and the function of each interval segment is continuously differentiable; therefore, the original constrained optimization problem can be transformed into a sub-optimization problem on four continuously differentiable interval segments, solving the configuration value B that minimizes the objective function cost′ and satisfies the constraints. cache ,n cache ,capacity.

[0024] Furthermore, in step (5), the segmented sequential quadratic programming algorithm traverses each cloud disk type and each segment on the objective function according to the service level target and the inference service container configuration input by the user, and solves each sub-optimization problem using the sequential quadratic programming method, and finally records the configuration B that minimizes the objective function. cache ,n cache ,capacity, and uses the rounding method to map the configuration value on the continuous domain to the discrete domain.

[0025] Furthermore, in step (6), the quantitative relationship between the cloud disk configuration and the cache container configuration refers to the size relationship between the cloud disk throughput tp and the actual data access bandwidth B during the model loading process. If the cloud disk throughput tp is greater than the actual data access bandwidth B during the model loading process, then any cache container and the inference service container are scheduled on the same node, and the inference service container can directly load the model from the local file system to achieve short-circuit read optimization effect.

[0026] Beneficial effects: The present invention can recommend the cache configuration with the lowest monetary cost to users based on the model loading service level target of the AI model inference service given by the user in a cloud-native container environment. First, the present invention describes the AI model loading time through mathematical modeling and parameter fitting; secondly, it constructs a constrained optimization problem, and by exploring the correlation between decision variables, reduces the number of decision variables and simplifies the objective function. In view of the non-smooth characteristics of the objective function and the constraints, the maximum and minimum functions are eliminated through classification discussion, thereby decomposing the original problem into continuous differentiable sub-optimization problems under multiple interval segments; then, the constrained optimization problem is solved using a piecewise sequential quadratic programming algorithm to obtain a cache configuration that minimizes the monetary cost; finally, according to the cache configuration, affinity scheduling is performed through the cache container and the inference service container, a short-circuit read optimization strategy is determined, and cache resources are created to provide acceleration capabilities for the model loading process of the AI inference service. The present invention can help users achieve fast AI model loading at a low monetary cost, so that the startup time of the model inference service is within the service level target, and the resource configuration algorithm proposed by the present invention can recommend cache configurations to users in a relatively short time, with high parameter search efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0028] Figures 2(a) and 2(b) are comparison diagrams of the model loading monetary cost and service level objective violation rate of the present invention and the prior art on the stable-diffusion-v2-1 dataset, respectively. Figure 2(c) is a comparison diagram of the parameter search time experiment. DETAILED DESCRIPTION

[0029] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0030] This invention proposes an AI model inference cache configuration method for cloud native container environments, which solves the problem that existing methods are difficult to balance the performance and monetary cost of the AI inference service startup phase. Figure 1 As shown, the complete process of the present invention includes seven steps. The following takes a stable-diffusion-v2-1 model loading process as an example to illustrate the specific implementation of the present invention.

[0031] The specific implementation of step (1) is as follows: different cloud disk configurations of a cloud vendor are collected. Table 1 lists different cloud disk types and their performance parameters. These parameters are stored in the database. The corresponding cloud disk parameters can be selected based on the cloud disk type diskType to construct the corresponding mathematical expression of the model load time estimation model.

[0032] Table 1 Various cloud disk configuration parameters of a cloud vendor

[0033]

[0034] The specific implementation of step (2) is as follows: use Linux's cgroup tool and the network speed limit tool wondershaper to control data access throughput, measure the model loading time of multiple groups of stable-diffusion-v2-1 under different data access throughputs, and use the least squares method to perform linear regression analysis on the relationship between the two to obtain the θ1 and θ2 parameters in the function relationship log(t) = θ1*log(b) + θ2. At this time, the relationship between model loading time and data access throughput can be expressed as

[0035] The specific implementation method of step (3) is: collect the unit CPU cost (0.1764 yuan / core / hour) and the unit memory cost (0.0221 yuan / GB / hour) of a cloud vendor's cache container, store them in the database, and use them to construct the monetary cost function cost and the constrained optimization problem.

[0036] The specific implementation of step (4) is as follows: first collect multiple groups of B on the cloud platform cache With n cpu Data, using the least squares method to establish B cache With n cpuAfter obtaining the specific values of γ and δ, the original monetary cost function is converted into the objective function cost′; then, cost′ is split into four continuously differentiable functions on four smooth interval segments. The piecewise function can be expressed as:

[0037]

[0038]

[0039] In order to simplify the expression of the formula, define tp1=α diskType *acpacity+β diskType , tp2 = maxThroughput diskType . Non-smooth constraint on model loading time t load ≤T SLO By analyzing the relative size relationship of tp1, tp2, and B in the four intervals, t load The maximum value function in the expression is expanded to decompose the original model loading time constraint into four continuously differentiable sub-constraints, namely: and Corresponding to the four interval segments of the objective function.

[0040] The specific implementation of step (5) is: collecting the service level target T submitted by the user SLO Configure N with the inference service container infer , substitute it into the constrained optimization problem in step (4) to solve. The specific solution is as follows: First, traverse the cloud disk types in turn. For each cloud disk type, use the sequential quadratic programming algorithm to solve the optimal cache configuration for each smooth interval of the objective function under the current cloud disk configuration, and dynamically update the global optimal cache resource configuration; then, after the traversal is completed, according to the data access bandwidth B of the container used to cache the AI model in the optimal configuration cache , calculate the corresponding number of CPU cores n cpu Finally, the optimization results in the continuous domain are discretized, and the cache resource configuration value in the discrete domain is obtained by rounding the cache resource value. The final recommended configuration value includes the number of containers n used to cache the AI model. cache , the number of container CPU cores n used to cache AI models cpu , cloud disk type diskType, cloud disk capacity capacity.

[0041] The specific implementation method of step (6) is as follows: based on the final cache resource configuration in step (5) and the inference service configuration input by the user, the cloud disk throughput tp and the actual data access bandwidth B during the model loading process are calculated. The scheduler will compare the size relationship between the two. If the cloud disk throughput is greater than the actual data access bandwidth during the model loading process, any cache container and the inference service container will be scheduled on the same node, and the short-circuit read function of the distributed cache will be enabled.

[0042] The specific implementation method of step (7) is as follows: according to the configuration recommended by the method, a cache container of specified specifications and a pay-as-you-go cloud disk are created on the cloud platform, and the cloud disk is mounted to the cache container in the form of a persistent volume declaration (PVC). The distributed cache system first pre-fetches the remotely stored model into the cache container. Thereafter, when the upper-level inference service container loads the model during startup, the model is directly loaded from the cache container of the distributed cache system, thereby achieving the effect of accelerating model loading.

[0043] In order to test the performance of this method, the performance of the piecewise sequential quadratic programming algorithm (PWSQP) proposed in this patent is compared with the BAYES (Bayesian optimization), GA (genetic algorithm), and GS (grid search) methods. Figure 2 (a) compares the monetary cost of different methods, Figure (2) b compares the model loading time and service level target violation rate of different methods, and Figure 2 (c) compares the parameter search time of different methods. The test results show that compared with the most advanced methods, the method proposed in this invention reduces the monetary cost by an average of 10.58% and is second only to the global optimal solution, improves the parameter search efficiency by 25 times, and ensures a 0% service level target violation rate.

Claims

1. A method for configuring AI model inference cache in a cloud-native container environment, comprising the following steps: (1) Based on the cloud disk configuration, cache container configuration, and inference service container configuration, a mathematical expression for the model loading time estimation model is constructed; (2) Collect the model loading time during the AI model inference process under different data access throughputs, and use the least squares method to fit the parameters to be fitted in the functional relationship between model loading time and data access throughput; (3) Model the monetary cost of model loading during AI model inference in a cloud-native container environment, construct a constrained optimization problem, and find the cloud disk configuration and cache container configuration that minimizes the monetary cost under the constraints; (4) Simplify and split the objective function and constraints of the constrained optimization problem in step (3) and transform them into continuous differentiable sub-optimization problems in multiple intervals; (5) Based on the service level goals submitted by the user and the inferred service container configuration, the segmented sequential quadratic programming algorithm is used to solve the constrained optimization problem to obtain the cloud disk configuration and cache container configuration that minimizes the monetary cost and satisfies the constraints; (6) Based on the quantitative relationship between the cloud disk configuration and the cache container configuration, the affinity scheduling strategy between the cache container and the inference service container is determined to achieve the effect of short-circuit read optimization; (7) Create cache resources in the cloud platform based on the cache container configuration and short-circuit read optimization strategy to provide acceleration capabilities for the model loading process of AI reasoning.

2. The AI model inference cache configuration method for a cloud-native container environment according to claim 1 is characterized by: In step (1), the cloud disk configuration includes the cloud disk type diskType, cloud disk capacity capacity, and cloud disk throughput tp; the cache container configuration includes the number of containers n used to cache the AI model. cache , the data access bandwidth of the container used to cache the AI model B cache , the number of CPUs n of the container used to cache the AI model cpu , the memory size of the container used to cache the AI model n mem ; The inference service container configuration includes the number of inference service containers N infer ; Model loading time estimation model t load The mathematical expression of t load =t(min(tp,B)), where B is the actual data access bandwidth during the model loading process, and t(·) represents the functional relationship between the model loading time and the data access throughput.

3. The AI model inference cache configuration method for a cloud-native container environment according to claim 2 is characterized by: In step (1), the cloud disk throughput tp is calculated as tp=min(α diskType *capacity+β diskType ,maxThroughput diskType ), α diskType is the correlation coefficient, β diskType The minimum throughput of the cloud disk, maxThroughput diskType The maximum throughput of the cloud disk; The actual data access bandwidth B during model loading is calculated as B = n cache *B cache / N infer .

4. The AI model inference cache configuration method for a cloud-native container environment according to claim 1, characterized in that: In step (2), the functional relationship between the model loading time t and the data access throughput b is log(t)=θ1*log(b)+θ2, where log(t) is the logarithm of t with the natural constant e as the base, log(b) is the logarithm of b with the natural constant e as the base, and θ1 and θ2 are parameters to be fitted; by collecting the model loading time in the AI model inference process under different data access throughputs, the least squares method is used to fit θ1 and θ2 in the functional relationship.

5. The AI model inference cache configuration method for a cloud-native container environment according to claim 2 is characterized by: In step (3), the monetary cost of model loading during the AI model inference process is calculated as cost=t load *n cache n(cost diskType *capacity+cost cpu *n cpu +cost mem *n mem ), where cost diskType Indicates the unit cost of diskType type cloud disk, cost cpu Indicates the unit CPU cost of the container used to cache the AI model, cost mem Indicates the unit memory cost of the container used to cache AI models.

6. The AI model inference cache configuration method for a cloud-native container environment according to claim 2, characterized in that: In step (3), the constrained optimization problem is to find the cloud disk configuration and cache container configuration that minimizes the monetary cost. cache ,B cache ,n cpu ,n mem , and satisfy the following constraints: t load ≤T SLO , diskType∈Φ, M / n cache ≤capacity≤maxCapacity diskType , B cache ,n cache ,capacity,n cpu ,n mem ∈Z, where T SLO is the model loading time service level target, Φ is the cloud disk type set provided by the cloud vendor, M is the data size of the AI model, and maxCapacity diskType The maximum capacity of a diskType cloud disk, and Z is an integer set.

7. The AI model inference cache configuration method for a cloud-native container environment according to claim 6, characterized in that: In step (4), since the memory size of the container used to cache the AI model does not affect the model loading time, the n in the monetary cost cost is eliminated. mem Related items; due to the data access bandwidth B of the cache container cache The number of CPU cores n of the cache container cpu In a linear relationship, we use B cache To express n cpu , so the simplified objective function cost′ is expressed as cost′=t load *n cache *(cost diskType *capacity+cost cpu *(γ*B cache +δ)), where γ and δ are correlation coefficients; in addition, diskType is not used as a decision variable, and each diskType value can be traversed later when solving the optimization problem; since the cost′ function and the constraints contain the min(tp,B) term, and the calculation expression of tp also contains the maximum value function, resulting in the function being discontinuously differentiable, the objective function cost′ is split into piecewise functions on four interval segments through classification discussion, and the function of each interval segment is continuously differentiable; therefore, the original constrained optimization problem can be transformed into a sub-optimization problem on four continuously differentiable interval segments, solving the configuration value B that minimizes the objective function cost′ and satisfies the constraints. cache ,n cache ,capacity.

8. The AI model inference cache configuration method for a cloud-native container environment according to claim 1, characterized in that: In step (5), the segmented sequence quadratic programming algorithm refers to traversing each cloud disk type and each segment on the objective function based on the service level target and inference service container configuration input by the user, using the quadratic sequence programming method to solve each sub-optimization problem, and finally recording the configuration B that minimizes the objective function. cache ,n cache ,capacity, and uses the rounding method to map the cache configuration value on the continuous domain to the discrete domain.

9. The AI model inference cache configuration method for a cloud-native container environment according to claim 1, characterized in that: In step (6), the quantitative relationship between the cloud disk configuration and the cache container configuration refers to the relationship between the cloud disk throughput tp and the actual data access bandwidth B during the model loading process. If the cloud disk throughput tp is greater than the actual data access bandwidth B during the model loading process, then any cache container and the inference service container are scheduled on the same node. The inference service container can load the model directly from the local file system, achieving a short-circuit read optimization effect.

Images