Model inference container deployment method and apparatus

Abstract

The application discloses a model inference container deployment method and device. The method comprises the following steps: obtaining a test data set of a first deep learning model in a cloud computing platform; analyzing the test data set of the first deep learning model by using a pre-trained resource configuration strategy decision model to obtain a target resource configuration strategy of the first deep learning model, wherein the resource configuration strategy decision model is trained based on a double deep Q learning algorithm, and the target resource configuration strategy is a resource configuration strategy that makes the average value of model scores of all deep learning models with the same model type as the first deep learning model in the cloud computing platform highest; and deploying an inference container of the first deep learning model according to the target resource configuration strategy. The application solves the technical problem that the related art adopts a static resource configuration method to deploy a model inference container, which is difficult to adapt to the real-time performance requirement of the model, and results in waste of computing resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for deploying a model inference container. Background Technology

[0002] In traditional cloud computing platforms, resource allocation for model inference containers often adopts a static resource allocation model. This means that administrators allocate a fixed amount of computing resources, such as GPUs (Graphics Processing Units), CPUs (Central Processing Units), and memory, to each model instance based on the model's initial requirements and past experience. While this static resource allocation model was initially designed to simplify the resource allocation process and reduce human intervention, it still suffers from the following technical drawbacks:

[0003] (1) Resource waste. The computational requirements of the same type of deep learning model (such as different versions of image recognition models) vary depending on factors such as model complexity and dataset size. Therefore, the fixed number of GPUs or other computing resources allocated using the static resource allocation mode may far exceed the actual needs of some models, resulting in low utilization of these resources during model inference and thus resource waste.

[0004] (2) It cannot dynamically adjust to adapt to the optimal operating state of each model. In scenarios with concurrent inference of multiple models, the static resource configuration mode cannot optimize the resource allocation strategy based on model performance feedback. This means that when the actual load of a model is low, its allocated computing resources cannot be borrowed by other high-load models, and vice versa. This problem greatly limits the elastic scaling capability and overall utilization of cloud platform resources.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a model inference container deployment method and apparatus to at least solve the technical problem that the static resource configuration method used in related technologies makes it difficult to adapt to the real-time performance requirements of models, resulting in wasted computing resources.

[0007] According to one aspect of the embodiments of this application, a method for deploying a model inference container is provided, comprising: acquiring a test dataset of a first deep learning model within a cloud computing platform; analyzing the test dataset of the first deep learning model using a pre-trained resource allocation strategy decision model to obtain a target resource allocation strategy for the first deep learning model, wherein the resource allocation strategy decision model is trained based on a dual-deep Q-learning algorithm, and the target resource allocation strategy is a resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model within the cloud computing platform; and deploying the inference container of the first deep learning model according to the target resource allocation strategy.

[0008] Optionally, obtaining the test dataset of the first deep learning model within the cloud computing platform includes: determining the first deep learning model within the cloud computing platform and determining the model information of the first deep learning model, wherein the model information includes at least: the input and output parameter format, the number of neurons in the first deep learning model and the quantization parameters, and the quantization parameters are obtained by quantizing the parameters of the neurons using a preset scaling factor and zero offset; and determining the test dataset of the first deep learning model based on the model information.

[0009] Optionally, the training process of the resource allocation strategy decision model includes: constructing a Q-network and a target network for solving the optimization strategy, and initializing the network weight parameters of the Q-network and the target network; setting an experience pool and determining a first number of iteration cycles, and iterating the network weight parameters of the Q-network through the following steps: in each iteration cycle, initializing the state space and determining a second number of computation cycles, wherein the state space includes at least: test datasets of multiple second deep learning models of the same type; in each computation cycle, inputting the current state space into the Q-network, calculating the Q-values ​​corresponding to different resource allocation strategies, and selecting the optimal resource allocation strategy with the largest Q-value based on a greedy strategy. The source allocation strategy executes the optimal resource allocation strategy and calculates the reward, obtaining a new state space. The current state space, the optimal resource allocation strategy, the reward, and the new state space are treated as a sample and stored in the experience pool. Multiple samples randomly sampled from the experience pool are input into the neural network, and the mean squared error loss function and loss function weights for each sample are calculated. Based on the calculation results, the fitness value is determined and the network weight parameters of the Q network are updated. After a third number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the Q network, where the third number is less than the second number. After the iteration is completed, the obtained target network is used as the resource allocation strategy decision model.

[0010] Optionally, the current state space is input into the Q-network to calculate the Q-values ​​corresponding to different resource configuration strategies, including: determining multiple resource configuration strategies, wherein the resource configuration strategies include at least one of the following: the number of allocated CPU cores, the number and type of allocated GPUs, and the amount of allocated available GPU memory; for each resource configuration strategy, deploying inference containers for multiple second deep learning models of the same type according to the resource configuration strategy; within each inference container, obtaining the index values ​​under multiple evaluation metrics obtained by the second deep learning model from analyzing various test data in the test dataset, wherein the evaluation metrics include at least: accuracy, F1 score, and latency; determining the model score of each second deep learning model under the resource configuration strategy based on the index values ​​of each second deep learning model under the multiple evaluation metrics; determining the Q-value corresponding to each resource configuration strategy based on the model scores of multiple second deep learning models of the same type under various resource configuration strategies, wherein the higher the average model score of multiple second deep learning models of the same type, the larger the Q-value corresponding to the resource configuration strategy.

[0011] Optionally, based on the index values ​​of each second deep learning model under multiple evaluation metrics, the model score of each second deep learning model under the resource allocation strategy is determined, including: constructing a model evaluation matrix with multiple second deep learning models as rows and multiple evaluation metrics as columns; determining the information entropy of each of the multiple evaluation metrics based on the multiple columns in the model evaluation matrix, and determining the weight value of each evaluation metric based on the information entropy of each of the multiple evaluation metrics; for each second deep learning model, performing a weighted sum of the index values ​​and weight values ​​of the second deep learning model under each evaluation metric to obtain the model score of the second deep learning model under the resource allocation strategy.

[0012] Optionally, the information entropy of each of the multiple evaluation indicators is determined based on the multiple columns within the model evaluation matrix, including: performing max-min normalization on the model evaluation matrix; traversing each column of the processed model evaluation matrix, calculating the ratio of each element in the current column to the sum of all elements in the current column to obtain the proportion of each element in the current column; and determining the information entropy of the evaluation indicator corresponding to the current column according to the following formula based on the proportion of each element in the current column:

[0013]

[0014] In the formula, E j Let p represent the information entropy of the evaluation index corresponding to the j-th column, m represent the number of rows in the model evaluation matrix, and p represent the information entropy of the evaluation index corresponding to the j-th column. ij This represents the percentage of the i-th element in the j-th column.

[0015] Optionally, the weight value of each evaluation indicator is determined based on the information entropy of each of the multiple evaluation indicators, including: calculating the sum of the differences between the information entropy of 1 and the evaluation indicators corresponding to all columns in the model evaluation matrix; traversing each column in the model evaluation matrix, calculating the quotient of the difference between the information entropy of 1 and the evaluation indicator corresponding to the current column and the sum of the differences, and using the quotient as the weight value of the evaluation indicator corresponding to the current column.

[0016] According to another aspect of the embodiments of this application, a model inference container deployment apparatus is also provided, comprising: a determining module, configured to acquire a test dataset of a first deep learning model within a cloud computing platform; a decision module, configured to analyze the test dataset of the first deep learning model using a pre-trained resource allocation strategy decision model to obtain a target resource allocation strategy for the first deep learning model, wherein the resource allocation strategy decision model is trained based on a dual-deep Q-learning algorithm, and the target resource allocation strategy is a resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model within the cloud computing platform; and a deployment module, configured to deploy the inference container of the first deep learning model according to the target resource allocation strategy.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described model inference container deployment method.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described model inference container deployment method through the computer program.

[0019] In this embodiment, test datasets of multiple first deep learning models of the same type within a cloud computing platform are collected as input to a pre-trained resource allocation strategy decision model. This model outputs a target resource allocation strategy that maximizes the average model score of the multiple first deep learning models. Finally, the inference containers for each of the multiple first deep learning models are deployed according to this target resource allocation strategy. This automated deployment scheme achieves the technical effect of automated and intelligent resource optimization allocation for deep learning models. It achieves the goal of improving resource utilization and reducing costs while ensuring model performance, thereby solving the technical problem of related technologies that use static resource allocation to deploy model inference containers, which is difficult to adapt to the real-time performance requirements of models and leads to wasted computing resources. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart illustrating an optional model inference container deployment method according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of an optional model inference container deployment device according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0027] Container technology divides the resources of a single operating system into independent groups in order to better balance conflicting resource usage demands among these groups.

[0028] DDQN (Double Deep Q-learning Network): An improved deep Q-learning algorithm that solves the problem of overestimation of Q-values ​​that may occur in traditional Q-learning by separating the action selection and evaluation processes.

[0029] Information entropy is a fundamental concept in information theory, used to describe the uncertainty of various possible events occurring in an information source. Later, Shannon borrowed concepts from thermodynamics, referring to the average amount of information remaining after removing redundancy as "information entropy."

[0030] Example 1

[0031] According to an embodiment of this application, a method for deploying a model inference container is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Figure 1 This is a flowchart illustrating a model inference container deployment method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0033] Step S102: Obtain the test dataset of the first deep learning model within the cloud computing platform.

[0034] Cloud computing platforms are the infrastructure that provides distributed computing resources, capable of supporting large-scale data processing and model inference tasks. They encompass a wide range of hardware resources, such as CPUs, GPUs, video memory, and software stacks. The target type refers to the specific problem domain or type that the deep learning model addresses. The primary deep learning model within a cloud computing platform is a deep learning model deployed in a specific cloud computing environment. It can cover various AI application areas, including but not limited to computer vision, natural language processing, speech recognition, and recommender systems. These models typically learn complex representations and patterns by training on large datasets to perform specific artificial intelligence tasks.

[0035] Step S104: Analyze the test dataset of the first deep learning model using the pre-trained resource allocation strategy decision model to obtain the target resource allocation strategy of the first deep learning model.

[0036] The resource allocation strategy decision model is trained using a dual-deep Q-learning algorithm. It learns the relationship between states (the model's test dataset) and actions (resource allocation strategies) to maximize rewards (model scores). Therefore, the core objective of the resource allocation strategy decision model is to obtain a target resource allocation strategy that maximizes resource utilization and cost-effectiveness while meeting the performance requirements of the first deep learning model. Thus, the target resource allocation strategy is the one that maximizes the average model score of all deep learning models of the same type as the first deep learning model within the cloud computing platform.

[0037] Step S106: Deploy the inference container of the first deep learning model according to the target resource configuration strategy.

[0038] Based on the scheme defined in steps S102 to S106 above, it can be understood that in this embodiment, by collecting the test dataset of the first deep learning model within the cloud computing platform as input to the pre-trained resource allocation strategy decision model, the model outputs a target resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model. Finally, the inference containers of the first deep learning model are deployed according to this target resource allocation strategy. This automated deployment scheme achieves the technical effect of automated and intelligent optimization and allocation of deep learning model resources, achieving the goal of improving resource utilization and reducing costs while ensuring model performance.

[0039] The following section describes each step of the model inference container deployment method in conjunction with the specific implementation process.

[0040] As an optional implementation, in the technical solution provided in step S102 above, the method may include:

[0041] Step 1: Determine the first deep learning model within the cloud computing platform and determine the model information of the first deep learning model.

[0042] The first deep learning model can be of one of the following types: image classification model, natural language processing model, or speech recognition model. Model information includes, but is not limited to, the input and output features, complexity, and model capacity of the first deep learning model. The model capacity includes at least the number of neurons and quantization parameters within the first deep learning model. The quantization parameters are obtained by quantizing the neuron parameters (including the weight matrix / weight vector) using a preset scaling factor and zero-point offset. The expression for the quantization parameters can be written as:

[0043]

[0044] In the formula, S represents the scaling factor, Z represents the zero-point offset, and W... float W represents the weight matrix / weight vector in floating-point format. quant This represents the weight matrix / weight vector in quantized integer format. Round indicates the rounding operation, used to convert floating-point numbers to integers.

[0045] Step 2: Determine the test dataset for the first deep learning model based on the model information. The test dataset includes multiple test data sets and the corresponding real sample labels for each test data set. The data characteristics of the test dataset include, but are not limited to: data type (e.g., image, text, audio), data format (e.g., JPEG, PNG, JSON, CSV), data volume, data preprocessing requirements (e.g., normalization, standardization, feature engineering), and data batch processing strategies.

[0046] Further, the test dataset determined in step S102 above is input into the pre-trained resource allocation strategy decision model. The resource allocation strategy decision model uses two neural networks, namely the Q-network and the target network, to evaluate the Q-value (i.e., model score) of the first deep learning model under different resource configurations, and makes a resource allocation strategy decision based on the Q-value, outputting the target resource allocation strategy corresponding to the first deep learning model. This target resource allocation strategy includes recommended combinations of computing resources (such as CPU, GPU type and quantity), memory resources, and storage resources to achieve the best model score.

[0047] It should be noted that the input to the resource allocation strategy decision model described above, in addition to the test dataset, can also include the service level agreement (SLA) requirements of the first deep learning model. The SLA requirements define the performance metrics of the model's inference service, such as response time, throughput, and availability. Including these as elements in the resource allocation strategy decision ensures that the final deployed model can operate under conditions that meet the SLA requirements.

[0048] Specifically, the training process for the aforementioned resource allocation strategy decision model can be achieved through the following steps:

[0049] First, construct the Q-network and the target network for solving the optimization strategy, and initialize the network weight parameters of the Q-network and the target network.

[0050] Set up an experience pool and determine the first number of iteration periods. Iterate the network weight parameters of the Q-network using the following steps:

[0051] Within each iteration cycle, the state space is initialized, and a second number of computation cycles are determined. The state space includes at least: test datasets for multiple second deep learning models of the same type. Multiple second deep learning models of the same type refer to multiple deep learning models of the same type or function selected in the cloud computing platform to perform the same type of task (their model architectures may be different), or the same deep learning model in different training stages in the cloud computing platform.

[0052] Within each computation cycle, the current state space is input into the Q-network to calculate the Q-values ​​corresponding to different resource allocation strategies. Based on a greedy strategy, the optimal resource allocation strategy with the largest Q-value is selected, the optimal resource allocation strategy is executed, and the reward is calculated. A new state space is obtained. The current state space, the optimal resource allocation strategy, the reward, and the new state space are treated as a sample and stored in the experience pool. Multiple samples randomly sampled from the experience pool are input into the neural network to calculate the mean squared error loss function and loss function weights for each sample. Based on the calculation results, the fitness value is determined and the network weight parameters of the Q-network are updated.

[0053] After a third number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the Q network, where the third number is less than the second number.

[0054] After the iteration is completed, the resulting target network is used as a resource allocation strategy decision model.

[0055] The aforementioned experience pool can be a SumTree structure, and the samples in the experience pool are the leaf nodes in the SumTree. When storing a sample into the experience pool, it can be determined first whether the capacity of the experience pool is full. If the capacity of the experience pool is not full, the sample is directly stored into the experience pool. If the capacity of the experience pool is full, based on the first-in-first-out principle, the oldest sample stored in the experience pool is replaced by a new sample.

[0056] Optionally, the calculation of the Q-value corresponding to the resource allocation strategy can be achieved through the following steps:

[0057] Step 1: Determine multiple resource allocation strategies, which include, but are not limited to, combinations of key resource parameters such as the number of CPU cores allocated, the number and type of GPUs allocated, and the amount of available video memory allocated.

[0058] Step 2: For each resource configuration strategy, use container orchestration tools to automatically deploy inference containers for multiple second-level deep learning models of the same type according to the resource configuration strategy. Within each inference container, obtain the metric values ​​of multiple evaluation metrics obtained by the second-level deep learning model from analyzing various test data in the test dataset. The evaluation metrics include at least: accuracy, F1 score, latency, etc. These metric values ​​provide a multi-dimensional perspective on model performance and help to fully understand the actual performance of the model under a specific resource configuration. Based on the metric values ​​of each second-level deep learning model under multiple evaluation metrics, determine the model score of each second-level deep learning model under the resource configuration strategy.

[0059] Step 3: Determine the Q-value for each resource allocation strategy based on the model scores of multiple second-level deep learning models of the same type under various resource allocation strategies. The higher the average model score of multiple second-level deep learning models of the same type, the larger the Q-value corresponding to the resource allocation strategy.

[0060] Specifically, in the technical solution provided in the second step above, the model score of each second deep learning model under the resource allocation strategy can be determined according to the following method:

[0061] First, construct a model evaluation matrix using multiple second-level deep learning models as rows and multiple evaluation metrics as columns. Then, each element in the model evaluation matrix is ​​the metric value of each second-level deep learning model under the corresponding evaluation metric.

[0062] Next, the information entropy of each of the multiple evaluation indicators is determined based on the multiple columns in the model evaluation matrix, and the weight value of each evaluation indicator is determined based on the information entropy of each of the multiple evaluation indicators.

[0063] The information entropy of the aforementioned evaluation indicators can be determined as follows: The model evaluation matrix is ​​subjected to max-min normalization; each column of the processed model evaluation matrix is ​​traversed, and the ratio of each element in the current column to the sum of all elements in the current column is calculated to obtain the proportion of each element in the current column, denoted as [the percentage of each element in the current column]. Where m represents the row number of the model evaluation matrix, and i and l represent the i-th and l-th indicator values ​​in the current column, respectively; based on the proportion of each element in the current column, the information entropy of the evaluation indicator corresponding to the current column is determined according to the following formula:

[0064]

[0065] In the formula, E j Let p represent the information entropy of the evaluation index corresponding to the j-th column, m represent the number of rows in the model evaluation matrix, and p represent the information entropy of the evaluation index corresponding to the j-th column. ij This represents the percentage of the i-th element in the j-th column.

[0066] The weight values ​​of the evaluation indicators are determined as follows: Calculate the sum of the differences in information entropy between 1 and the evaluation indicators corresponding to all columns in the model evaluation matrix; iterate through each column in the model evaluation matrix, calculate the quotient of the difference in information entropy between 1 and the evaluation indicator corresponding to the current column and the sum of these differences, and use this quotient as the weight value of the evaluation indicator corresponding to the current column. The expression can be written as: Where n represents the number of columns in the model evaluation matrix.

[0067] Finally, for each second deep learning model, the weighted sum of the indicator values ​​and weight values ​​of the second deep learning model under each evaluation metric is performed to obtain the model score of the second deep learning model under the resource allocation strategy. Its expression can be written as:

[0068] Therefore, by analyzing the test dataset of the first deep learning model using the resource allocation strategy decision model obtained through the above training, the resulting target resource allocation strategy can not only ensure the performance of the first deep learning model itself, but more importantly, ensure that the performance of different deep learning models of the same type on the cloud computing platform will be more consistent and stable, thereby ensuring the reliability of the inference service and the consistency of the user experience.

[0069] Furthermore, the system can utilize container coding tools to deploy the inference container of the first deep learning model according to the target resource configuration strategy. The first deep learning model is encapsulated within a corresponding inference container image, which contains the model code, dependency libraries, and runtime environment. Before deployment, the system adjusts the environment variables within the container image according to the target resource configuration strategy to load the corresponding model logic and optimization algorithms.

[0070] After the inference container is deployed, the system will continuously monitor the performance metrics of the first deep learning model under various performance indicators during the actual inference stage, and visualize these metrics. At the same time, the system can comprehensively determine the model score of the first deep learning model based on the various metrics.

[0071] If the model score of the first deep learning model falls below the preset expected score during the actual inference phase, the system can adjust the model's resource allocation strategy independently based on performance metrics. For example, if the latency of the first deep learning model is high, it is usually due to insufficient GPU resources. In this case, the system can automatically increase the allocation of GPU resources through Kubernetes' HPA (Horizontal PodAutoscaler), or dynamically adjust other resources such as CPU, memory, and network bandwidth to overcome the performance bottleneck.

[0072] In the aforementioned model inference container deployment method, the resource allocation strategy decision model based on the dual deep Q-learning algorithm possesses self-learning and self-optimization capabilities. As the model continuously runs during actual inference, the decision model accumulates more performance data, thereby continuously adjusting its strategy to more accurately predict which resource configurations are most beneficial to the model score. Therefore, this embodiment can recommend the optimal resource configuration combination to the deep learning model within the cloud computing platform based on the model's characteristics and real-time load through the resource allocation strategy decision model. This ensures that the deployed inference container receives the most suitable resource support, significantly improving operational efficiency and platform maintainability. Simultaneously, the entire configuration strategy decision process reduces the need for manual intervention, lowers the possibility of human error, and improves management efficiency and the speed of model deployment.

[0073] Example 2

[0074] According to an embodiment of this application, a model inference container deployment apparatus for implementing the model inference container deployment method in Embodiment 1 is also provided, such as... Figure 2 As shown, the model inference container deployment device includes at least: an acquisition module 22, a decision module 24, and a deployment module 26, wherein:

[0075] Module 22 is used to obtain the test dataset of the first deep learning model within the cloud computing platform;

[0076] Decision module 24 is used to analyze the test dataset of the first deep learning model using a pre-trained resource allocation strategy decision model to obtain the target resource allocation strategy of the first deep learning model. The resource allocation strategy decision model is trained based on the dual deep Q learning algorithm, and the target resource allocation strategy is the resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model in the cloud computing platform.

[0077] Deployment module 26 is used to deploy the inference container of the first deep learning model according to the target resource configuration strategy.

[0078] The following section describes the functions of each module of the model inference container deployment device in conjunction with the specific implementation process.

[0079] As an optional implementation, the acquisition module 22 can acquire the test dataset of the first deep learning model in the following manner:

[0080] Step 1: Determine the first deep learning model within the cloud computing platform and determine the model information of the first deep learning model.

[0081] The first deep learning model can be one of the following types: image classification model, natural language processing model, or speech recognition model. Model information includes, but is not limited to, the input and output features, complexity, and model capacity of the first deep learning model. The model capacity includes at least the number of neurons and quantization parameters within the first deep learning model. The quantization parameters are obtained by quantizing the neuron parameters (including the weight matrix / weight vector) using a preset scaling factor and zero-point offset.

[0082] Step 2: Determine the test dataset for the first deep learning model based on the model information. The test dataset includes multiple test data sets and the corresponding real sample labels for each test data set. The data characteristics of the test dataset include, but are not limited to: data type (e.g., image, text, audio), data format (e.g., JPEG, PNG, JSON, CSV), data volume, data preprocessing requirements (e.g., normalization, standardization, feature engineering), and data batch processing strategies.

[0083] Furthermore, the decision module 24 inputs the acquired test dataset into the pre-trained resource allocation strategy decision model. This model uses two neural networks, a Q-network and a target network, to evaluate the Q-value (i.e., model score) of the first deep learning model under different resource configurations. Based on this Q-value, it makes a resource allocation strategy decision and outputs the target resource allocation strategy corresponding to the first deep learning model. This target resource allocation strategy includes recommended combinations of computing resources (such as CPU and GPU types and quantities), memory resources, and storage resources to achieve the best model score.

[0084] Furthermore, the model inference container deployment device provided in this application embodiment also includes a model training module, and the model training module can train the resource allocation strategy decision model through the following steps:

[0085] First, construct the Q-network and the target network for solving the optimization strategy, and initialize the network weight parameters of the Q-network and the target network.

[0086] Set up an experience pool and determine the first number of iteration periods. Iterate the network weight parameters of the Q-network using the following steps:

[0087] Within each iteration cycle, the state space is initialized, and a second number of computation cycles are determined. The state space includes at least: test datasets for multiple second deep learning models of the same type. Multiple second deep learning models of the same type refer to multiple deep learning models of the same type or function selected in the cloud computing platform to perform the same type of task (their model architectures may be different), or the same deep learning model in different training stages in the cloud computing platform.

[0088] Within each computation cycle, the current state space is input into the Q-network to calculate the Q-values ​​corresponding to different resource allocation strategies. Based on a greedy strategy, the optimal resource allocation strategy with the largest Q-value is selected, the optimal resource allocation strategy is executed, and the reward is calculated. A new state space is obtained. The current state space, the optimal resource allocation strategy, the reward, and the new state space are treated as a sample and stored in the experience pool. Multiple samples randomly sampled from the experience pool are input into the neural network to calculate the mean squared error loss function and loss function weights for each sample. Based on the calculation results, the fitness value is determined and the network weight parameters of the Q-network are updated.

[0089] After a third number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the Q network, where the third number is less than the second number.

[0090] After the iteration is completed, the resulting target network is used as a resource allocation strategy decision model.

[0091] Optionally, when calculating the Q-value corresponding to the resource allocation strategy, the model training module can achieve the following steps:

[0092] Step 1: Determine multiple resource allocation strategies, which include, but are not limited to, combinations of key resource parameters such as the number of CPU cores allocated, the number and type of GPUs allocated, and the amount of available video memory allocated.

[0093] Step 2: For each resource configuration strategy, use container orchestration tools to automatically deploy inference containers for multiple second-level deep learning models of the same type according to the resource configuration strategy. Within each inference container, obtain the metric values ​​of multiple evaluation metrics obtained by the second-level deep learning model from analyzing various test data in the test dataset. The evaluation metrics include at least: accuracy, F1 score, latency, etc. These metric values ​​provide a multi-dimensional perspective on model performance and help to fully understand the actual performance of the model under a specific resource configuration. Based on the metric values ​​of each second-level deep learning model under multiple evaluation metrics, determine the model score of each second-level deep learning model under the resource configuration strategy.

[0094] Step 3: Determine the Q-value for each resource allocation strategy based on the model scores of multiple second-level deep learning models of the same type under various resource allocation strategies. The higher the average model score of multiple second-level deep learning models of the same type, the larger the Q-value corresponding to the resource allocation strategy.

[0095] Specifically, the model training module can determine the model score of each second deep learning model under the resource allocation strategy as follows:

[0096] First, construct a model evaluation matrix using multiple second-level deep learning models as rows and multiple evaluation metrics as columns. Then, each element in the model evaluation matrix is ​​the metric value of each second-level deep learning model under the corresponding evaluation metric.

[0097] Next, the information entropy of each of the multiple evaluation indicators is determined based on the multiple columns in the model evaluation matrix, and the weight value of each evaluation indicator is determined based on the information entropy of each of the multiple evaluation indicators.

[0098] The information entropy of the aforementioned evaluation indicators can be determined as follows: The model evaluation matrix is ​​subjected to max-min normalization; each column of the processed model evaluation matrix is ​​traversed, and the ratio of each element in the current column to the sum of all elements in the current column is calculated to obtain the proportion of each element in the current column; based on the proportion of each element in the current column, the information entropy of the evaluation indicator corresponding to the current column is determined according to the following formula:

[0099]

[0100] In the formula, E j Let p represent the information entropy of the evaluation index corresponding to the j-th column, m represent the number of rows in the model evaluation matrix, and p represent the information entropy of the evaluation index corresponding to the j-th column. ij This represents the percentage of the i-th element in the j-th column.

[0101] The weight values ​​of the evaluation indicators are determined as follows: Calculate the sum of the differences between the information entropy of 1 and the evaluation indicators corresponding to all columns in the model evaluation matrix; traverse each column in the model evaluation matrix, calculate the quotient of the difference between the information entropy of 1 and the evaluation indicator corresponding to the current column and the sum of the differences, and use the quotient as the weight value of the evaluation indicator corresponding to the current column.

[0102] Finally, for each second deep learning model, the index values ​​and weight values ​​of the second deep learning model under each evaluation metric are weighted and summed to obtain the model score of the second deep learning model under the resource allocation strategy.

[0103] Furthermore, deployment module 26 can utilize container coding tools to deploy the inference container of the first deep learning model according to the target resource configuration strategy. The first deep learning model is encapsulated into a corresponding inference container image, which contains the model code, dependency libraries, and runtime environment. Before deployment, the system adjusts the environment variables within the container image according to the target resource configuration strategy to load the corresponding model logic and optimization algorithms.

[0104] It should be noted that each module in the model inference container deployment device in this application corresponds one-to-one with each implementation step of the model inference container deployment method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0105] Example 3

[0106] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein the computer program, when executed by a processor, implements the model inference container deployment method in embodiment 1.

[0107] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the model inference container deployment method in Embodiment 1 by running the computer program.

[0108] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the model inference container deployment method of embodiment 1 during runtime.

[0109] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the model inference container deployment method of embodiment 1 through the computer program.

[0110] Specifically, the computer program executes the following steps during runtime: obtaining the test dataset of the first deep learning model within the cloud computing platform; analyzing the test dataset of the first deep learning model using a pre-trained resource allocation strategy decision model to obtain the target resource allocation strategy of the first deep learning model, wherein the resource allocation strategy decision model is trained based on the dual-deep Q-learning algorithm, and the target resource allocation strategy is the resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model within the cloud computing platform; and deploying the inference container of the first deep learning model according to the target resource allocation strategy.

[0111] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 3 A hardware block diagram of an electronic device for implementing a model inference container deployment method is shown. Figure 3As shown, the electronic device 30 may include one or more processors 302 (shown as 302a, 302b, ..., 302n in the figure) (processor 302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 30 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.

[0112] It should be noted that the aforementioned one or more processors 302 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 30. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0113] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model inference container deployment method in this embodiment. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the aforementioned application vulnerability detection method. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to the electronic device 30 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The transmission device 306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 30. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0115] The display may be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the electronic device 30.

[0116] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0117] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0122] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for deploying a model inference container, characterized in that, include: Obtain the test dataset for the first deep learning model within the cloud computing platform; The test dataset of the first deep learning model is analyzed using a pre-trained resource allocation strategy decision model to obtain the target resource allocation strategy of the first deep learning model. The resource allocation strategy decision model is trained based on a dual deep Q-learning algorithm, and the target resource allocation strategy is the resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model within the cloud computing platform. The inference container of the first deep learning model is deployed according to the target resource allocation strategy.

2. The method according to claim 1, characterized in that, Obtain the test dataset for the first deep learning model within the cloud computing platform, including: A first deep learning model within the cloud computing platform is determined, and the model information of the first deep learning model is determined, wherein the model information includes at least: input and output parameter formats, the number of neurons in the first deep learning model, and quantization parameters, and the quantization parameters are obtained by quantizing the parameters of the neurons using a preset scaling factor and zero-point offset; The test dataset for the first deep learning model is determined based on the model information.

3. The method according to claim 1, characterized in that, The training process of the resource allocation strategy decision model includes: Construct a Q-network and a target network for solving the optimization strategy, and initialize the network weight parameters of the Q-network and the target network; Set up an experience pool and determine the first number of iteration periods. Iterate the network weight parameters of the Q-network through the following steps: Within each iteration cycle, the state space is initialized, and a second number of computation cycles are determined, wherein the state space includes at least: test datasets of multiple second deep learning models of the same type; Within each computation cycle, the current state space is input into the Q-network, the Q-values ​​corresponding to different resource allocation strategies are calculated, and the optimal resource allocation strategy with the largest Q-value is selected based on a greedy strategy. The optimal resource allocation strategy is executed and the reward is calculated to obtain a new state space. The current state space, the optimal resource allocation strategy, the reward, and the new state space are used as a sample and stored in the experience pool. Multiple samples randomly sampled from the experience pool are input into the Q-network, the mean squared error loss function of each sample is calculated, the gradient value is determined based on the calculation results, and the network weight parameters of the Q-network are updated. After a third number of calculation cycles, the network weight parameters of the target network are updated based on the network weight parameters of the Q network, wherein the third number is less than the second number; After the iteration is completed, the obtained target network is used as the resource allocation strategy decision model.

4. The method according to claim 3, characterized in that, Input the current state space into the Q-network and calculate the Q-values ​​corresponding to different resource allocation strategies, including: Multiple resource allocation strategies are determined, wherein the resource allocation strategies include at least one of the following: the number of CPU cores allocated, the number and type of GPUs allocated, and the amount of available video memory allocated. For each resource allocation strategy, the inference containers of the multiple second deep learning models of the same type are deployed according to the resource allocation strategy; within each inference container, the index values ​​of the second deep learning model under multiple evaluation metrics obtained by analyzing each test data in the test dataset are obtained, wherein the evaluation metrics include at least: accuracy, F1 score, and latency; based on the index values ​​of each second deep learning model under multiple evaluation metrics, the model score of each second deep learning model under the resource allocation strategy is determined. The Q-value corresponding to each resource allocation strategy is determined based on the model scores of the multiple second deep learning models of the same type under various resource allocation strategies. The higher the average model score of the multiple second deep learning models of the same type, the larger the Q-value corresponding to the resource allocation strategy.

5. The method according to claim 4, characterized in that, Based on the index values ​​of each of the second deep learning models under multiple evaluation metrics, the model score of each of the second deep learning models under the resource allocation strategy is determined, including: A model evaluation matrix is ​​constructed using the multiple second deep learning models as rows and the multiple evaluation metrics as columns. The information entropy of each of the multiple evaluation indicators is determined based on the multiple columns in the model evaluation matrix, and the weight value of each evaluation indicator is determined based on the information entropy of each of the multiple evaluation indicators. For each of the second deep learning models, the index values ​​and weight values ​​of the second deep learning model under each of the evaluation indicators are weighted and summed to obtain the model score of the second deep learning model under the resource allocation strategy.

6. The method according to claim 5, characterized in that, The information entropy of each of the multiple evaluation indicators is determined based on multiple columns within the model evaluation matrix, including: The model evaluation matrix is ​​subjected to max-min normalization. Traverse each column of the processed model evaluation matrix, calculate the ratio of each element in the current column to the sum of all elements in the current column, and obtain the proportion of each element in the current column; based on the proportion of each element in the current column, determine the information entropy of the evaluation index corresponding to the current column according to the following formula: In the formula, Let represent the information entropy of the evaluation index corresponding to the j-th column, and m represent the number of rows in the model evaluation matrix. This represents the percentage of the i-th element in the j-th column.

7. The method according to claim 6, characterized in that, Determining the weight value of each of the multiple evaluation indicators based on their respective information entropy includes: Calculate the sum of the differences in information entropy between 1 and the evaluation indicators corresponding to all columns in the model evaluation matrix; Traverse each column of the model evaluation matrix, calculate the quotient of the difference between the information entropy of 1 and the evaluation index corresponding to the current column and the sum of the differences, and use the quotient as the weight value of the evaluation index corresponding to the current column, denoted as: In the formula, n represents the number of columns in the model evaluation matrix. This represents the information entropy of the evaluation index corresponding to the j-th column. This represents the information entropy of the evaluation index corresponding to the k-th column.

8. A model inference container deployment device, characterized in that, include: The acquisition module is used to acquire the test dataset of the first deep learning model within the cloud computing platform; The decision module is used to analyze the test dataset of the first deep learning model using a pre-trained resource allocation strategy decision model to obtain the target resource allocation strategy of the first deep learning model. The resource allocation strategy decision model is trained based on the dual deep Q-learning algorithm, and the target resource allocation strategy is the resource allocation strategy that maximizes the average model score of all deep learning models of the same model type as the first deep learning model in the cloud computing platform. The deployment module is used to deploy the inference container of the first deep learning model according to the target resource configuration strategy.

9. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the model inference container deployment method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the model inference container deployment method of any one of claims 1 to 7 through the computer program.

Images