Model deployment method, system and device and computer readable storage medium
By performing similarity measurement and clustering on the parameter distribution information of multiple sub-models to be deployed, shared quantization parameters are generated, solving the problem of wasted storage and computing resources in multi-model systems and achieving efficient utilization of storage and computing resources.
Patent Information
- Application Number
- CN202511808211.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-03
- Publication Date
- 2026-03-06
AI Technical Summary
In existing technologies, multi-model collaborative systems suffer from severe waste of storage and computing resources due to the independent storage of parameters. How to reduce the waste of storage and computing resources caused by model deployment has become an urgent problem to be solved.
By acquiring the parameter distribution information of multiple sub-models to be deployed, a distribution-based similarity measurement is performed to generate similarity values. Then, clustering is performed according to the set clustering granularity to generate shared quantization parameters, enabling multiple sub-models to be deployed using shared quantization parameters.
It reduces the waste of storage and computing resources in model deployment, and is expected to achieve a 50%-70% reduction in storage, a 30%-50% reduction in memory access overhead, and a significant improvement in inference speed.
Smart Images
Figure CN121614149A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence model technology, and more specifically, to a model deployment method, system, device, and computer-readable storage medium. Background Technology
[0002] As the scale of artificial intelligence models expands, especially in deep integration applications in fields such as natural language processing and computer vision, the need for multi-model collaborative work is growing. These systems typically contain multiple sub-models or expert modules, leading to an exponential increase in the overall parameter size and resulting in huge memory consumption and computational overhead. To reduce memory consumption and computational overhead, model quantization is necessary.
[0003] For example, different quantization bit widths can be dynamically allocated within a single MoE (Mixture of Experts) model based on the activation value distribution of the expert layer. However, this method quantizes according to the logic that "the parameters of each model are independent and must be stored separately," which leads to a waste of storage and computing resources.
[0004] In summary, how to reduce the waste of storage and computing resources in model deployment is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide a model deployment method that can, to some extent, address the technical problem of reducing the waste of storage and computing resources during model deployment. This application also provides a model deployment system, an electronic device, and a computer-readable storage medium.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] A model deployment method, comprising:
[0008] Obtain the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2;
[0009] Perform distribution-based similarity measurement on parameter distribution information to generate similarity values between parameter distribution information;
[0010] Based on the similarity values, the parameter distribution information is clustered according to the set clustering granularity to obtain the parameter clustering results;
[0011] The clustering results of each parameter are quantified to generate shared quantization parameters for the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model to be deployed based on the shared quantization parameters.
[0012] Preferably, the step of clustering the parameter distribution information based on the similarity value according to the set clustering granularity to obtain the parameter clustering result includes:
[0013] Determine the model characteristics of the sub-model to be deployed, including model type and task domain;
[0014] The parameter distribution information is grouped according to the model characteristics to obtain the parameter grouping results;
[0015] Based on the set clustering granularity, the parameter grouping results are clustered according to the similarity value to obtain the parameter clustering results.
[0016] Preferably, the step of quantizing the clustering results of each parameter to generate shared quantization parameters for the sub-model to be deployed at the clustering granularity includes:
[0017] Obtain a preset cutting standard, which includes a target compression ratio and / or a minimum precision loss threshold;
[0018] The clustering results of each parameter are cut according to the cutting criteria to obtain the parameter cutting results;
[0019] The results of each parameter segmentation are quantized to generate shared quantized parameters for the sub-model to be deployed at the clustering granularity.
[0020] Preferably, after quantizing the clustering results of each parameter to generate shared quantized parameters for the sub-model to be deployed at the clustering granularity, the method further includes:
[0021] Determine the task loss, memory usage, and inference latency of the sub-model to be deployed under different shared quantization parameters;
[0022] Determine the weight coefficients of the sub-model to be deployed based on its deployment environment.
[0023] The task loss value, memory usage, and inference latency value are processed according to the weighting coefficients to generate the decision loss value of the sub-model to be deployed under different shared quantization parameters;
[0024] The shared quantization parameter corresponding to the decision loss value with the largest value is used as the target quantization parameter for the sub-model to be deployed for inference application.
[0025] Preferably, after using the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application, the method further includes:
[0026] Discretely evaluate the observed output of the sub-model to be deployed to obtain the observed evaluation value;
[0027] Discretely evaluate the expected output of the sub-model to be deployed to obtain the expected evaluation value;
[0028] The shared quantization parameters are adjusted based on the difference between the observed evaluation value and the expected evaluation value.
[0029] Preferably, after using the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application, the method further includes:
[0030] Detect whether joint knowledge distillation has been triggered;
[0031] In response to triggering joint knowledge distillation, the shared quantization parameters are jointly adjusted.
[0032] Preferably, after quantizing the clustering results of each parameter to generate shared quantized parameters for the sub-model to be deployed at the clustering granularity, the method further includes:
[0033] Generate the mapping relationship between the sub-models to be deployed and the corresponding shared quantization parameters;
[0034] The mapping relationship is stored in the metadata mapping table.
[0035] A model deployment system, comprising:
[0036] The parameter distribution information acquisition module is used to acquire the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2;
[0037] The similarity value generation module is used to perform distribution-based similarity measurement on parameter distribution information and generate similarity values between parameter distribution information.
[0038] The clustering module is used to cluster the parameter distribution information based on the similarity value according to the set clustering granularity, and obtain the parameter clustering result;
[0039] The quantization module is used to quantize the clustering results of each parameter and generate shared quantization parameters for the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model based on the shared quantization parameters.
[0040] An electronic device, comprising:
[0041] Memory, used to store computer programs;
[0042] A processor for implementing the steps of any of the above-described model deployment methods when executing the computer program.
[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the model deployment methods described above.
[0044] This application provides a model deployment method that obtains parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2; performs a distribution-based similarity measurement on the parameter distribution information to generate similarity values between the parameter distribution information; clusters the parameter distribution information based on the similarity values according to a set clustering granularity to obtain parameter clustering results; quantifies each parameter clustering result to generate shared quantized parameters for the sub-models to be deployed at the clustering granularity, and deploys the sub-models to be deployed based on the shared quantized parameters. In this application, after obtaining the parameter distribution information of multiple sub-models to be deployed, a similarity value between the parameter distribution information is generated based on the similarity measure of the distribution. Since the similarity measure based on the distribution can better capture the overall similarity of parameter distributions between different models, clustering the parameter distribution information based on the similarity value can accurately classify similar parameter distribution information into the same parameter clustering result. Quantizing each parameter clustering result can then generate shared quantized parameters for the sub-models to be deployed with similar parameter distribution information, breaking through the boundary of single-model optimization and allowing multiple sub-models to be deployed to share shared quantized parameters, reducing the waste of storage and computing resources in model deployment. The model deployment system, electronic device, and computer-readable storage medium provided in this application also solve the corresponding technical problems. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0046] Figure 1 A flowchart illustrating a model deployment method provided in an embodiment of this application;
[0047] Figure 2 This is a flowchart of model deployment based on multi-granularity hierarchical clustering;
[0048] Figure 3 This is a flowchart for model deployment based on cutting.
[0049] Figure 4 A flowchart for model deployment and parameter scheduling;
[0050] Figure 5 This is a schematic diagram of the overall deployment method of the model in this application;
[0051] Figure 6 This is a schematic diagram of the structure of a model deployment system provided in an embodiment of this application;
[0052] Figure 7 Interaction diagrams for users to invoke the solution proposed in this application;
[0053] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0054] Figure 9 This is another structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0056] Please see Figure 1 , Figure 1 This is a flowchart of a model deployment method provided in an embodiment of this application.
[0057] This application provides a model deployment method that may include the following steps:
[0058] Step S101: Obtain the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2.
[0059] In practical applications, during the model deployment process, the parameter distribution information of all sub-models to be deployed can be obtained first. The parameter distribution information can be the weights and / or activation values of fully connected layers, attention layers, etc.
[0060] It should be noted that the type of sub-model to be deployed can be flexibly determined according to the application scenario. For example, the sub-model to be deployed may include sub-models in hybrid expert models (MoE), multilingual large models, multimodal fusion models, etc.; and the sub-model to be deployed may be a model for image processing, text processing, or language processing, etc.
[0061] Step S102: Perform a distribution-based similarity measurement on the parameter distribution information to generate similarity values between parameter distribution information.
[0062] In practical applications, if the parameter distribution information of multiple sub-models to be deployed is similar, these sub-models can share the parameter distribution information, so only one set of parameter distribution information needs to be deployed. Based on this, considering that the parameter distribution information is not a simple point-to-point relationship, it is necessary to perform distribution-based similarity measurement on the parameter distribution information, such as performing Wasserstein distance, JS divergence calculation, etc., to generate similarity values between parameter distribution information.
[0063] It should be noted that performing distribution-based similarity measurement on parameter distribution information essentially involves measuring similarity based on the probability distribution corresponding to the parameters. Taking JS divergence as an example, the process of generating similarity values can be as follows: ; ;P i and P j Let i and j represent the probability distributions corresponding to the parameters of model i and model j, respectively.
[0064] Step S103: Cluster the parameter distribution information based on similarity values according to the set clustering granularity to obtain the parameter clustering results.
[0065] In practical applications, since similarity values can reflect the similarity between parameter distribution information of two sub-models to be deployed, parameter distribution information can be clustered based on similarity values according to the set clustering granularity to obtain parameter clustering results, such as tree-like parameter clustering results, so as to aggregate similar parameter distribution information into the same parameter clustering result.
[0066] It's important to note that the clustering granularity determines the compression ratio and accuracy of the deployed sub-models. For example, a smaller clustering granularity (k) forces multiple significantly different model layers to share the same set of quantization parameters, resulting in a high compression ratio but potentially significant accuracy loss. Conversely, a larger clustering granularity (k) means only very similar model layers will share parameters, leading to a lower compression ratio but better accuracy. In specific application scenarios, the clustering granularity can be flexibly adjusted as needed.
[0067] Step S104: Quantize the clustering results of each parameter to generate shared quantization parameters of the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model to be deployed based on the shared quantization parameters.
[0068] In practical applications, after aggregating similar parameter distribution information into the same parameter clustering result, the clustering results of each parameter can be quantified to generate shared quantization parameters for the sub-model to be deployed at this clustering granularity, so that the sub-model to be deployed can be deployed based on the shared quantization parameters.
[0069] It should be noted that model quantization refers to reducing model storage requirements and computational complexity by lowering parameter precision (e.g., converting 32-bit floating-point numbers to 8-bit integers). It is one of the model compression methods. During the quantization process, multiple different sets of parameters need to be tried. scale For each parameter group, the clustering results are converted to low precision, the total error for that group is calculated, and the error results of all groups are compared. The group with the smallest error is selected as the final shared quantization parameter. The shared quantization parameter obtained after quantization may include a quantization scaling factor. scale And the quantization zero point z, the quantization formula is q=clip(round((xz) / scale ),q min ,q max ), where clip(·) is the truncation operation, used to ensure that the quantization result is within the valid range of low-precision integers to avoid overflow; round(·) is the rounding operation; q min and q max These are the minimum and maximum values after quantization, respectively; x is the input value. scale The range of z is determined by the maximum and minimum values of the data within the cluster, and must satisfy the requirement that "a finite number of low-precision integers (such as 8-bit integers that can represent 256 values) can cover the entire range of values in the cluster"; the range of z is selected around the central region of the data within the cluster (such as near the mean or median) to ensure that the benchmark value corresponding to "the integer 0" falls in a region with a relatively dense data distribution, thereby reducing the overall error.
[0070] In an exemplary embodiment, shared quantization parameters can be deployed to the device using a hash table, a cache-aware memory layout, or shared memory, so that multiple sub-models or multiple inference threads can efficiently and concurrently access the same set of shared quantization parameters, reducing memory bandwidth pressure.
[0071] In an exemplary embodiment, considering that the sub-model to be deployed needs to call shared quantization parameters for data processing, and given that there are multiple sets of shared quantization parameters, errors may occur during the call. To avoid this, the clustering results of each parameter are quantized. After generating the shared quantization parameters of the sub-model to be deployed at the clustering granularity, a mapping relationship between the sub-model to be deployed and the corresponding shared quantization parameters can also be generated. This mapping relationship can specifically be the mapping relationship between layers in the sub-model and the shared quantization parameters, etc. The mapping relationship is stored in a metadata mapping table. In this way, the sub-model to be deployed can accurately call the shared quantization parameters according to the mapping relationship, ensuring the accurate invocation of the shared quantization parameters.
[0072] This application provides a model deployment method that obtains parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2; performs a distribution-based similarity measurement on the parameter distribution information to generate similarity values between the parameter distribution information; clusters the parameter distribution information based on the similarity values according to a set clustering granularity to obtain parameter clustering results; and quantizes each parameter clustering result to generate shared quantized parameters for the sub-models to be deployed at the clustering granularity. In this application, after obtaining the parameter distribution information of multiple sub-models to be deployed, a similarity value between the parameter distribution information is generated based on the similarity measure of the distribution. Since the similarity measure based on the distribution can better capture the overall similarity of the parameter distribution between different models, if the parameter distribution information is clustered according to the similarity value, similar parameter distribution information can be accurately classified into the same parameter clustering result. Then, if each parameter clustering result is quantized, shared quantized parameters shared by the sub-models to be deployed with similar parameter distribution information can be generated, breaking through the boundary of single model optimization. This allows multiple sub-models to be deployed to share shared quantized parameters for deployment, reducing the waste of storage and computing resources in model deployment. It is expected to achieve a 50%-70% reduction in storage and a 30%-50% reduction in memory access overhead, significantly improving inference speed.
[0073] Based on the above embodiments, parameter distribution information can be directly clustered based on similarity values. However, when there are many types of parameter distribution information, this clustering method is time-consuming and not conducive to rapid model deployment. To avoid this situation, please refer to [link / reference needed]. Figure 2 The model deployment method provided in this application embodiment may include the following steps:
[0074] Step S201: Obtain the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2.
[0075] Step S202: Perform a distribution-based similarity measurement on the parameter distribution information to generate similarity values between parameter distribution information.
[0076] Step S203: Determine the model characteristics of the sub-model to be deployed. The model characteristics include model type and task domain.
[0077] Step S204: Group the parameter distribution information according to the model characteristics to obtain the parameter grouping results.
[0078] Step S205: Cluster the parameter grouping results based on similarity values according to the set clustering granularity to obtain the parameter clustering results.
[0079] In an exemplary embodiment, considering that the parameter distribution information of the sub-models to be deployed with the same model features is relatively similar, hierarchical clustering of the parameter distribution information can be performed accordingly. That is, according to the set clustering granularity, the parameter distribution information is clustered based on the similarity value. In the process of obtaining the parameter clustering result, the model features of the sub-models to be deployed can be determined first. The model features include model type, task domain, etc. The parameter distribution information is grouped according to the model features, that is, the parameter distribution information corresponding to the same model features is grouped together to obtain the parameter grouping result. Then, clustering is performed on the parameter grouping result as a unit, that is, according to the set clustering granularity, the parameter grouping result is clustered based on the similarity value to obtain the parameter clustering result.
[0080] Step S206: Quantize the clustering results of each parameter to generate shared quantization parameters of the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model to be deployed based on the shared quantization parameters.
[0081] As can be seen from the implementation process, this application, in the process of clustering parameter distribution information based on similarity values according to a set clustering granularity to obtain parameter clustering results, determines the model features of the sub-model to be deployed. These model features include model type and task domain. The parameter distribution information is then grouped according to these model features to obtain parameter grouping results. This involves first grouping similar parameter distribution information together based on model features; then, according to the set clustering granularity, the parameter grouping results are clustered based on similarity values to obtain the final parameter clustering results. This achieves multi-granularity hierarchical clustering of parameter distribution information based on model features and similarity values, improving clustering efficiency and quality, thereby accelerating model deployment efficiency.
[0082] Based on the above embodiments, and considering the different deployment requirements of the sub-models to be deployed—such as the different performance levels that devices with different capabilities can support for the sub-models to be deployed, and the different deployment requirements of users for different sub-models to be deployed—for flexible deployment of the sub-models to be deployed, please refer to [link / reference needed]. Figure 3 The model deployment method provided in this application embodiment may include the following steps:
[0083] Step S301: Obtain the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2.
[0084] Step S302: Perform a distribution-based similarity measurement on the parameter distribution information to generate similarity values between parameter distribution information.
[0085] Step S303: Cluster the parameter distribution information based on similarity values according to the set clustering granularity to obtain the parameter clustering results.
[0086] Step S304: Obtain the preset cutting criteria, which include the target compression ratio and / or minimum precision loss threshold.
[0087] In an exemplary embodiment, considering that the compression ratio can determine the proportion of reduction in the total model volume, for example, if the target compression ratio is 70%, it means that the model storage usage should be reduced to 30% of the original; the accuracy loss threshold can determine the proportion of model performance degradation and thus affect the model quantization degree, for example, when the accuracy loss threshold is 90%, the coarsest codebook will be selected to achieve maximum compression while meeting the accuracy requirements; therefore, in the process of quantizing the clustering results of each parameter and generating the shared quantization parameters of the sub-model to be deployed at the clustering granularity, a preset cutting standard can be obtained for model quantization, and the cutting standard includes the target compression ratio and / or the minimum accuracy loss threshold.
[0088] Step S305: Cut the clustering results of each parameter according to the cutting criteria to obtain the parameter cutting results.
[0089] In an exemplary embodiment, after obtaining the cutting criteria, the clustering results of each parameter can be cut according to the cutting criteria to obtain parameter cutting results that meet the compression and accuracy requirements, so that subsequent quantization can be performed in units of parameter cutting results.
[0090] Step S306: Quantize the segmentation results of each parameter to generate shared quantization parameters of the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model to be deployed based on the shared quantization parameters.
[0091] As can be seen from the implementation process, in the process of quantizing the clustering results of each parameter and generating shared quantization parameters of the sub-model to be deployed at the clustering granularity, this application obtains a preset cutting standard, which includes the target compression ratio and / or minimum precision loss threshold; the clustering results of each parameter are cut according to the cutting standard to obtain parameter cutting results that are adapted to the target compression ratio and / or minimum precision loss threshold; and then the cutting results of each parameter are quantized to generate shared quantization parameters of the sub-model to be deployed at the clustering granularity. This is equivalent to controlling the quantization process according to the target compression ratio and / or minimum precision loss threshold, so that the generated shared quantization parameters can meet the requirements and achieve flexible deployment.
[0092] Based on the above embodiments, the clustering granularity value can be a single value or multiple values. When the clustering granularity has multiple values, it is equivalent to obtaining shared quantization parameters under different clustering granularities. Since the shared quantization parameters under different clustering granularities have different requirements for storage and computational performance, the shared quantization parameters need to be flexibly scheduled according to requirements. Please refer to... Figure 4 The model deployment method provided in this application embodiment may include the following steps:
[0093] Step S401: Obtain the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2.
[0094] Step S402: Perform a distribution-based similarity measurement on the parameter distribution information to generate similarity values between parameter distribution information.
[0095] Step S403: Cluster the parameter distribution information based on similarity values according to the set clustering granularity to obtain the parameter clustering results.
[0096] Step S404: Quantize the clustering results of each parameter to generate shared quantization parameters for the sub-model to be deployed at the clustering granularity.
[0097] Step S405: Determine the task loss value, memory usage, and inference latency value of the sub-model to be deployed under different shared quantization parameters.
[0098] In an exemplary embodiment, the impact of different shared quantization parameters on the sub-model to be deployed can be expressed through the task loss value (Task). loss ), memory usage footprint ) and inference delay value (Inference) latency Therefore, it is necessary to determine the task loss, memory usage, and inference latency of the sub-model to be deployed under different shared quantization parameters, in order to obtain feedback. Among them, the task loss directly reflects the impact of quantization on accuracy; memory usage is directly related to the granularity of the selected shared quantization parameters, for example, coarse-grained shared quantization parameters consume less memory; inference latency is affected by computational load and memory access efficiency, and shared quantization parameters can reduce latency by reducing memory bandwidth pressure.
[0099] Step S406: Determine the weight coefficients of the sub-model to be deployed based on the deployment environment of the sub-model to be deployed.
[0100] In an exemplary embodiment, the performance of the device hosting the sub-model to be deployed affects the operation of the sub-model. For example, when memory is extremely tight, coarser-grained shared quantization parameters can be selected, while finer-grained shared quantization parameters can be selected when handling tasks requiring high precision, in order to improve task performance. That is, when resources are limited, memory saving is prioritized, and when high precision is required, performance is prioritized. Therefore, the weight coefficients of the sub-model to be deployed need to be determined according to the deployment environment. For example, on edge devices with limited memory and computing power, the weights of memory usage and inference latency can be increased to save resources and improve speed. On high-performance servers, the weight of task loss values can be increased to achieve higher task accuracy.
[0101] Step S407: Process the task loss value, memory usage, and inference latency value according to the weight coefficients to generate the decision loss value of the sub-model to be deployed under different shared quantization parameters.
[0102] Step S408: Use the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application.
[0103] In an exemplary embodiment, after determining the weight coefficients of the sub-model to be deployed, the task loss value, memory usage, and inference latency value can be processed based on the weight coefficients to generate the decision loss value of the sub-model to be deployed under different shared quantization parameters. For example, the process of generating the decision loss value L can be as follows: α, β, and γ represent the corresponding weight values; then, the shared quantization parameter corresponding to the decision loss value with the largest value is used as the target quantization parameter for the sub-model to be deployed for inference application.
[0104] As can be seen from the implementation process, this application quantifies the clustering results of each parameter, generates shared quantization parameters for the sub-model to be deployed at the clustering granularity, and then determines the task loss value, memory usage, and inference latency value of the sub-model to be deployed under different shared quantization parameters. This allows the impact of the shared quantization parameters on the sub-model to be characterized by the task loss value, memory usage, and inference latency value. Next, based on the deployment environment of the sub-model to be deployed, the weight coefficients of the sub-model to be deployed are determined to ensure that the weight coefficients are compatible with the deployment environment. Finally, the task loss value, memory usage, and inference latency value are processed according to the weight coefficients to generate the decision loss value of the sub-model to be deployed under different shared quantization parameters. If the shared quantization parameter corresponding to the decision loss value with the largest value is used as the target quantization parameter for the sub-model to be deployed for inference application, the most suitable shared quantization parameter can be selected for the sub-model to be deployed based on the deployment environment, ensuring the normal operation of the sub-model in the deployment environment.
[0105] In practical applications, if there is a deviation in the generation process of the shared quantization parameters, the performance of the sub-model will be affected. Therefore, after taking the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference, the shared quantization parameter can be adjusted according to the performance of the sub-model to keep the performance loss within <1%.
[0106] In specific application scenarios, the adjustment of shared quantization parameters can be achieved through online learning of adaptive scaling factors. For example, the observed output of the sub-model to be deployed can be discretely evaluated to obtain observed evaluation values; the expected output of the sub-model to be deployed can also be discretely evaluated to obtain expected evaluation values; and the shared quantization parameters can be adjusted based on the difference between the observed and expected evaluation values to achieve local, fast, and unsupervised parameter compensation. This can be achieved, for example, through formulas. Adjustments are made to the shared quantization parameters, where Var represents the variance calculation function, Var(y observed Var(y) represents the variance of the observed evaluation values. expected ) represents the variance of the expected evaluation value, and σ is used to adjust the magnitude of the control; Var(x) = (1 / N) * Σ(x i -μ) 2 N represents the total number of data points, x i Let represent the i-th data point, and μ represent the mean of the data.
[0107] In specific application scenarios, during the adjustment of shared quantization parameters, a shared, lightweight "teacher" model (or utilizing the original high-precision model) can be introduced to perform joint distillation on all "student" sub-models using shared quantization parameters. This allows the sub-models to collaboratively approximate the output distribution of the high-precision model while sharing low-precision parameters. In other words, it can detect whether joint knowledge distillation is triggered. For example, if the drift of the quantization scaling factor of multiple sub-models is greater than the set value over a continuous period of time, and the overall task accuracy shows a downward trend, then it can be determined that joint knowledge distillation needs to be triggered. In response to triggering joint knowledge distillation, the shared quantization parameters are jointly adjusted. For example, using a small amount of unlabeled data or user feedback, the parameters of all sub-models are jointly fine-tuned to fundamentally repair the accuracy loss of the shared quantization parameters.
[0108] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a model deployment system provided in an embodiment of this application.
[0109] This application provides a model deployment system that may include:
[0110] The parameter distribution information acquisition module 101 is used to acquire the parameter distribution information of N sub-models to be deployed, where N is an integer greater than or equal to 2;
[0111] The similarity value generation module 102 is used to perform distribution-based similarity measurement on parameter distribution information and generate similarity values between parameter distribution information.
[0112] Clustering module 103 is used to cluster parameter distribution information based on similarity values according to a set clustering granularity to obtain parameter clustering results;
[0113] The quantization module 104 is used to quantize the clustering results of each parameter and generate shared quantization parameters of the sub-model to be deployed at the clustering granularity, so as to deploy the sub-model to be deployed based on the shared quantization parameters.
[0114] This application provides a model deployment system in which the clustering module may include:
[0115] The model feature determination unit is used to determine the model features of the sub-model to be deployed. The model features include model type and task domain.
[0116] Grouping units are used to group parameter distribution information according to model characteristics to obtain parameter grouping results;
[0117] Clustering units are used to cluster the parameter grouping results based on similarity values according to a set clustering granularity, and obtain parameter clustering results.
[0118] This application provides a model deployment system, in which the quantization module may include:
[0119] The cutting standard acquisition unit is used to acquire preset cutting standards, which include target compression ratio and / or minimum precision loss threshold.
[0120] The cutting unit is used to cut the clustering results of each parameter according to the cutting criteria to obtain the parameter cutting results;
[0121] The quantization unit is used to quantize the results of each parameter segmentation and generate shared quantization parameters for the sub-model to be deployed at the clustering granularity.
[0122] The model deployment system provided in this application embodiment may further include:
[0123] The feedback parameter acquisition module is used by the quantization module to quantify the clustering results of each parameter, generate the shared quantization parameters of the sub-model to be deployed at the clustering granularity, and then determine the task loss value, memory usage and inference latency value of the sub-model to be deployed under different shared quantization parameters.
[0124] The weight coefficient determination module is used to determine the weight coefficients of the sub-model to be deployed based on the deployment environment of the sub-model to be deployed.
[0125] The decision loss value generation module is used to process the task loss value, memory usage and inference latency value according to the weight coefficients to generate the decision loss value of the sub-model to be deployed under different shared quantization parameters;
[0126] The parameter filtering module is used to select the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application.
[0127] The model deployment system provided in this application embodiment may further include:
[0128] The observation evaluation value acquisition module is used by the parameter screening module to select the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application, and then to perform discrete evaluation on the observation output of the sub-model to be deployed to obtain the observation evaluation value.
[0129] The expected evaluation value acquisition module is used to perform discrete evaluation of the expected output of the sub-model to be deployed, and obtain the expected evaluation value.
[0130] The difference adjustment module is used to adjust the shared quantization parameters based on the difference between the observed evaluation value and the expected evaluation value.
[0131] The model deployment system provided in this application embodiment may further include:
[0132] The knowledge distillation module is used by the parameter selection module to select the shared quantization parameter corresponding to the decision loss value with the largest value as the target quantization parameter for the sub-model to be deployed for inference application, and then detects whether joint knowledge distillation is triggered; in response to triggering joint knowledge distillation, the shared quantization parameter is jointly adjusted.
[0133] The model deployment system provided in this application embodiment may further include:
[0134] The mapping relationship generation module is used by the quantization module to quantize the clustering results of each parameter, generate the shared quantization parameters of the sub-model to be deployed at the clustering granularity, and then generate the mapping relationship between the sub-model to be deployed and the corresponding shared quantization parameters.
[0135] The storage module is used to store the mapping relationships to the metadata mapping table.
[0136] It should be noted that the scheme in this application can be applied flexibly according to needs, such as... Figure 6 and Figure 7 As shown, the process of providing services to users according to this application may include the following steps:
[0137] The user loads all sub-models into the system;
[0138] During the initialization phase, the global analysis and clustering module extracts the global parameter distribution of the sub-model, calculates the distribution similarity, performs hierarchical clustering, generates and stores the shared quantization codebook (shared quantization parameters) to the shared parameter pool, and generates a metadata mapping table to the shared parameter pool.
[0139] During the inference phase, the user submits input data and task context (such as the current language type and modality) to the inference engine;
[0140] The inference engine requests parameter scheduling strategy from the dynamic scheduling module;
[0141] The dynamic scheduling model evaluates the status of tasks / resources, queries the metadata mapping table in the shared parameter pool, and returns the best shared codebook ID to the scheduling engine.
[0142] The scheduling engine uses the codebook ID to read shared parameters from the shared parameter pool and performs low-precision quantization calculations. If adjustments to the shared codebook are needed, it requests the error compensation module to perform error compensation.
[0143] The error compensation module returns a correction signal to the inference engine;
[0144] The inference engine returns high-precision inference results to the user.
[0145] This application also provides an electronic device and a computer-readable storage medium, both of which have the corresponding effects of the model deployment method provided in the embodiments of this application. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0146] This application provides a fan speed processing device, including a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to implement the steps of the model deployment method described in any of the above embodiments.
[0147] Please see Figure 9Another electronic device provided in this application embodiment may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the electronic device and the outside. The display unit 204 may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module 205 includes, but is not limited to, Mobile High-Definition Link (MHL), Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), wireless connection: Wireless Fidelity (WiFi), Bluetooth communication technology, Bluetooth Low Energy communication technology, and communication technology based on IEEE 802.11s.
[0148] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the model deployment method described in any of the above embodiments.
[0149] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs (compact disc read-only memory), or any other form of storage media known in the art.
[0150] This application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the model deployment method described in any of the above embodiments.
[0151] For descriptions of relevant parts in the model deployment system, electronic device, and computer-readable storage medium provided in this application's embodiments, please refer to the detailed description of the corresponding parts in the model deployment method provided in this application's embodiments; they will not be repeated here. Furthermore, parts of the technical solutions provided in this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0152] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0153] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A model deployment method, characterized by, The method comprises the following steps: obtaining parameter distribution information of N to-be-deployed sub-models, N being an integer greater than or equal to 2; performing distribution-based similarity measurement on the parameter distribution information to generate a similarity value between the parameter distribution information; performing clustering on the parameter distribution information based on the similarity value according to a set clustering granularity to obtain a parameter clustering result; quantizing each parameter clustering result to generate shared quantized parameters of the to-be-deployed sub-models under the clustering granularity, so as to deploy the to-be-deployed sub-models based on the shared quantized parameters.
2. The method of claim 1, wherein, The method further comprises the following steps: determining model characteristics of the to-be-deployed sub-models, the model characteristics comprising a model type and a task field; grouping the parameter distribution information according to the model characteristics to obtain a parameter grouping result; performing clustering on the parameter grouping result based on the similarity value according to the set clustering granularity to obtain the parameter clustering result.
3. The method of claim 1, wherein, The method further comprises the following steps: obtaining a preset cutting standard, the cutting standard comprising a target compression rate and / or a minimum accuracy loss threshold; cutting each parameter clustering result according to the cutting standard to obtain a parameter cutting result; quantizing each parameter cutting result to generate the shared quantized parameters of the to-be-deployed sub-models under the clustering granularity.
4. The method of claim 1, wherein, The method further comprises the following steps after the quantizing of each parameter clustering result to generate the shared quantized parameters of the to-be-deployed sub-models under the clustering granularity: determining task loss values, memory occupation amounts and inference delay values of the to-be-deployed sub-models under different shared quantized parameters; determining a weight coefficient of the to-be-deployed sub-models according to a deployment environment of the to-be-deployed sub-models; processing the task loss values, the memory occupation amounts and the inference delay values according to the weight coefficient to generate decision loss values of the to-be-deployed sub-models under different shared quantized parameters; taking a shared quantized parameter corresponding to a decision loss value with the maximum value as a target quantized parameter for inference application of the to-be-deployed sub-model.
5. The method of claim 4, wherein, The method further comprises the following steps after the taking of the shared quantized parameter corresponding to the decision loss value with the maximum value as the target quantized parameter for inference application of the to-be-deployed sub-model: performing discrete evaluation on an observation output of the to-be-deployed sub-model to obtain an observation evaluation value; performing discrete evaluation on an expected output of the to-be-deployed sub-model to obtain an expected evaluation value; adjusting the shared quantized parameter according to a difference between the observation evaluation value and the expected evaluation value.
6. The method of claim 4, wherein, The method further comprises the following steps after the taking of the shared quantized parameter corresponding to the decision loss value with the maximum value as the target quantized parameter for inference application of the to-be-deployed sub-model: detecting whether joint knowledge distillation is triggered; in response to the joint knowledge distillation being triggered, adjusting the shared quantized parameter jointly.
7. The method of claim 1, wherein, The method further comprises the following steps after the quantizing of each parameter clustering result to generate the shared quantized parameters of the to-be-deployed sub-models under the clustering granularity: generating a mapping relationship between the to-be-deployed sub-models and the corresponding shared quantized parameters; storing the mapping relationship to a metadata mapping table.
8. A model deployment system, characterized by, The method comprises the following steps: The parameter distribution information acquisition module is configured to acquire parameter distribution information of N to-be-deployed sub-models, where N is an integer greater than or equal to 2. The similarity value generation module is configured to perform distribution-based similarity measurement on the parameter distribution information, and generate a similarity value between the parameter distribution information. The clustering module is configured to perform clustering on the parameter distribution information based on the similarity value according to a set clustering granularity, and obtain a parameter clustering result. The quantization module is configured to quantize each parameter clustering result, generate shared quantized parameters of the to-be-deployed sub-models at the clustering granularity, and deploy the to-be-deployed sub-models based on the shared quantized parameters.
9. An electronic device, comprising: The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps of the model deployment method according to any one of claims 1 to 7. The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps of the model deployment method according to any one of claims 1 to 7. 10. A computer-readable storage medium, characterized in that,