Method for deploying machine learning model, electronic device, and storage medium

Abstract

The application discloses a machine learning model deployment method, an electronic device and a storage medium, and relates to the technical field of large models and model deployment. The method comprises the following steps: in response to a quantization configuration instruction acting on an operation interface, displaying a target quantization mode corresponding to an initial machine learning model on the operation interface, wherein the initial machine learning model is a neural network model obtained through pre-training, and the target quantization mode is used for representing a quantization scale for quantizing the initial machine learning model; and in response to a model deployment instruction acting on the operation interface, displaying a deployment result of the initial machine learning model on the operation interface, wherein the deployment result is obtained after the target machine learning model is deployed, and the target machine learning model is obtained by quantizing the initial machine learning model based on the target quantization mode. The application solves the technical problem of high deployment cost of a model in the related art.

Description

Technical Field

[0001] This application relates to large model technology and model deployment, and more specifically, to a method for deploying a machine learning model, an electronic device, and a storage medium. Background Technology

[0002] Currently, large models have shown amazing performance on various tasks. However, due to their huge scale, large models consume a lot of computation, GPU memory, and RAM. General hardware devices are difficult to support inference, and they can only be deployed using expensive and scarce equipment such as high-performance computing accelerators (e.g., A100), resulting in high deployment costs for large models.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method for deploying a machine learning model, an electronic device, and a storage medium to at least address the technical problem of high deployment costs for models in related technologies.

[0005] According to one aspect of the embodiments of this application, a method for deploying a machine learning model is provided, comprising: responding to a quantization configuration command applied to an operation interface, displaying a target quantization method corresponding to an initial machine learning model on the operation interface, wherein the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; and responding to a model deployment command applied to the operation interface, displaying a deployment result of the initial machine learning model on the operation interface, wherein the deployment result is the result obtained after deploying a target machine learning model, and the target machine learning model is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0006] According to another aspect of the embodiments of this application, a method for deploying a large model is also provided, comprising: responding to a quantization configuration command applied to an operation interface, displaying a target quantization method corresponding to an initial large model on the operation interface, wherein the initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model; and responding to a model deployment command applied to the operation interface, displaying a deployment result of the initial large model on the operation interface, wherein the deployment result is the result obtained after deploying a target large model, and the target large model is a model obtained by quantizing the initial large model based on the target quantization method.

[0007] According to another aspect of the embodiments of this application, a method for processing a machine learning model is also provided, comprising: responding to an input instruction applied to an operation interface, displaying data to be processed on the operation interface; responding to a processing instruction applied to the operation interface, displaying a processing result of the data to be processed on the operation interface, wherein the processing result is a result obtained by accelerating the quantized data based on a target machine learning model, the target machine learning model is obtained by quantizing an initial machine learning model based on a target quantization method, the quantized data is obtained by quantizing the data to be processed based on a target quantization strategy corresponding to the target quantization method, the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0008] According to another aspect of the embodiments of this application, a method for processing large models is also provided, including: responding to an input command applied to an operation interface and displaying data to be processed on the operation interface; responding to a processing command applied to the operation interface and displaying the processing result of the data to be processed on the operation interface, wherein the processing result is the result obtained by accelerating the quantized data based on a target large model, the target large model is obtained by quantizing an initial large model based on a target quantization method, the quantized data is obtained by quantizing the data to be processed based on a target quantization strategy corresponding to the target quantization method, the initial large model is a pre-trained large model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model.

[0009] According to another aspect of the embodiments of this application, a method for deploying a machine learning model is also provided, comprising: obtaining an initial machine learning model, wherein the initial machine learning model is a pre-trained neural network model; determining a target quantization method, wherein the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model; and deploying the target machine learning model.

[0010] According to another aspect of the embodiments of this application, a method for deploying a machine learning model is also provided, comprising: obtaining a target quantization method by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the target quantization method, the target quantization method being used to characterize the quantization scale for quantizing an initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model, wherein the initial machine learning model is a pre-trained neural network model; and outputting the target machine learning model by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the target machine learning model.

[0011] In this embodiment, in response to a quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method characterizes the quantization scale used to quantize the initial machine learning model. In response to a model deployment command applied to the operation interface, the deployment result of the initial machine learning model is displayed on the operation interface. The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method. This achieves the purpose of accelerating the deployment of the machine learning model. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0012] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to a method for deploying a machine learning model according to an embodiment of this application;

[0015] Figure 2 This is a flowchart of a method for deploying a machine learning model according to Embodiment 1 of this application;

[0016] Figure 3 This is a flowchart of another method for deploying a machine learning model according to an embodiment of this application;

[0017] Figure 4 This is a flowchart of a large model deployment method according to Embodiment 2 of this application;

[0018] Figure 5 This is a flowchart of a large model deployment method according to Embodiment 3 of this application;

[0019] Figure 6 This is a flowchart of a large model deployment method according to Embodiment 4 of this application;

[0020] Figure 7 This is a flowchart of a large model deployment method according to Embodiment 5 of this application;

[0021] Figure 8 This is a flowchart of a method for deploying a machine learning model according to Embodiment 6 of this application;

[0022] Figure 9 This is a schematic diagram of a machine learning model deployment device according to Embodiment 7 of this application;

[0023] Figure 10 This is a schematic diagram of a large-scale model deployment device according to Embodiment 8 of this application;

[0024] Figure 11 This is a schematic diagram of a processing device for a machine learning model according to Embodiment 9 of this application;

[0025] Figure 12 This is a schematic diagram of a large-scale model deployment device according to Embodiment 10 of this application;

[0026] Figure 13 This is a schematic diagram of a machine learning model deployment apparatus according to Embodiment 11 of this application;

[0027] Figure 14 This is a schematic diagram of a machine learning model deployment device according to Embodiment 12 of this application;

[0028] Figure 15 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a machine learning model with a large number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability, such as Large Language Models (LLMs) and multi-modal pre-training models.

[0032] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them for various tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and other fields. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the deployment of a large model in a model deployment scenario is used as an example for explanation, as follows:

[0033] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0034] Accelerating large model inference: This refers to improving the inference speed of large models with limited computer hardware resources by optimizing and accelerating algorithms.

[0035] Quantization acceleration refers to accelerating the model through quantization to improve its inference speed.

[0036] This application provides a method for deploying machine learning models, which can significantly optimize the inference performance of machine learning models in terms of both computing power and memory, and repair the loss of quantization accuracy through various means. It eliminates the need for training quantization and fine-tuning, reducing the cost of quantization for users, thereby accelerating the deployment of machine learning models.

[0037] Example 1

[0038] According to an embodiment of this application, a method for deploying a machine learning model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the deployment method of the machine learning model provided in this application can be applied to, for example, Figure 1 The application scenarios shown are as follows: Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to a method for deploying a machine learning model according to an embodiment of this application, but it is not limited thereto. Figure 1 In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0040] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device receives a quantization configuration instruction applied to the operation interface; the server, based on the quantization configuration instruction, displays the target quantization method corresponding to the initial machine learning model on the operation interface; the client receives a model deployment instruction applied to the operation interface; and the server, based on the model deployment instruction, displays the deployment result of the initial machine learning model on the operation interface. It should be noted that this embodiment can be performed on the client device if the client device's operating resources can meet the deployment and operation conditions of a large model.

[0041] Under the aforementioned operating environment, this application provides the following: Figure 2 The deployment method of the machine learning model is shown. Figure 2 This is a flowchart of a method for deploying a machine learning model according to Embodiment 1 of this application. Figure 2 As shown, the method may include the following steps:

[0042] Step S202: In response to the quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface.

[0043] The initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale used to quantize the initial machine learning model.

[0044] The aforementioned user interface refers to the interface through which users interact with computer programs. It can be provided in a graphical or text-based manner to facilitate user operation of the computer programs on the interface. For example... Figure 1 As shown, the operation interface can include a model display area, a target quantization method display area, and a model deployment result feedback area. The initial machine learning model mentioned above can be displayed through the model display area.

[0045] The above-mentioned user interface may include a model input control and a model display area. The model input control can be used to generate quantization configuration instructions based on the user's touch operation, and the model display area can be used to display the initial machine learning model. This is only an example for illustration.

[0046] The initial machine learning model mentioned above can be a model with a large memory footprint or a large resource consumption, such as a large model. However, the initial machine learning model here can also be any model, without any restrictions.

[0047] The target quantization method described above can be determined by the user. Alternatively, the target quantization method can be determined without user instruction, based on the number of model weights, the model's purpose, and the performance and accuracy requirements of the model task.

[0048] In one optional embodiment, the user can first click the model input control on the operation interface, at which point the initial machine learning model to be accelerated deployment can be displayed. After the user confirms that the initial machine learning model is correct, the user can generate a quantization configuration instruction, and display the initial machine learning model in a specified area or any area on the operation interface according to the quantization configuration instruction, so that the user can view it.

[0049] In another optional embodiment, if a user needs to accelerate the deployment of an initial machine learning model, they can first click the model input control on the operation interface. At this time, a folder containing multiple initial machine learning models can be displayed. The user can select the initial machine learning model that needs to be accelerated and upload it, thereby generating the aforementioned quantization configuration instruction. According to the quantization configuration instruction, the initial machine learning model can be displayed in a specified area or any area on the operation interface for the user to view.

[0050] In another alternative embodiment, after the neural network model has been trained, the trained neural network model, i.e. the initial machine learning model mentioned above, can be directly displayed on the operation interface according to the quantization configuration command in response to the operation interface.

[0051] The aforementioned target quantization methods include at least one of the following: quantization accuracy, quantization granularity, and quantization accuracy enhancement.

[0052] like Figure 1 As shown, the target quantization method described above can be displayed in the target quantization area.

[0053] The quantization precision, quantization granularity, and quantization precision enhancement mentioned above can be selected by the user.

[0054] The quantization precision mentioned above can include full quantization and half quantization. Full quantization has lower precision than half quantization, but its performance is higher. By adding half quantization precision to full quantization, quantization errors can be further reduced.

[0055] The quantization granularity mentioned above can have different levels, and different levels of quantization granularity correspond to different precision and performance to meet the user's needs for models with different precision and performance. The quantization granularity can be divided into three levels, such as Tensor, Channel, and Sub Channel. This is only an example for illustration. The quantization granularity can also be divided into other levels, and the criteria for the level can be set according to actual needs.

[0056] Since quantization introduces a loss of parameter precision, model accuracy is often affected. To overcome this issue, precision enhancement techniques can be used to improve model accuracy. This can be achieved by removing outliers from the parameters to compensate for the precision loss caused by quantization, thereby improving model accuracy.

[0057] In one optional embodiment, the user can select a target quantization method according to the actual quantization requirements. The target quantization method can be one or more of quantization accuracy, quantization granularity, and quantization accuracy enhancement. The user can select the desired target quantization method on the operation interface, and the selected target quantization method will be displayed on the operation interface.

[0058] Step S204: In response to the model deployment command applied to the operation interface, display the deployment results of the initial machine learning model on the operation interface.

[0059] The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0060] like Figure 1 As shown, the deployment results of the initial machine learning model described above can be displayed in the model deployment result feedback area.

[0061] The aforementioned user interface may include model deployment controls.

[0062] In one optional embodiment, the user can click the model deployment control on the operation interface to generate a model deployment instruction. The user can first quantize the initial machine learning model according to the model deployment instruction to obtain the target machine learning model, and then deploy the target machine learning model to obtain the deployment result.

[0063] The deployment results described above can be used to indicate whether the target machine learning model has been successfully deployed. If the target machine learning model has been successfully deployed, the application interface of the target machine learning model can be displayed on the screen. If the target machine learning model has not been successfully deployed, relevant prompts about the deployment failure can be displayed on the screen.

[0064] The above deployment results can also be the model application interface displayed on the operation interface after the target machine learning model has been deployed.

[0065] Through the above steps, in response to the quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale used to quantize the initial machine learning model. In response to the model deployment command applied to the operation interface, the deployment result of the initial machine learning model is displayed on the operation interface. The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method. This achieves the goal of accelerating the deployment of the machine learning model. It is noteworthy that different quantization methods can be used to quantize the initial machine learning model to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0066] In the above embodiments of this application, the initial machine learning model includes: a multi-layer network layer, and the method further includes: quantizing the weights of the network layer based on the target quantization method to obtain the target machine learning model.

[0067] In one optional embodiment, the weights of the network layer can be quantized according to different quantization scales given by the target quantization method to obtain the target machine learning model. Optionally, the weights of the network layer can be quantized from 16-bit / 32-bit stored weights to 4-bit or 8-bit stored weights to reduce the model's memory space and resource consumption, thereby obtaining a target machine learning model with less memory footprint and less resource consumption.

[0068] In the above embodiments of this application, the quantization method includes: quantization type. The method further includes: when the quantization type is a first quantization type, quantizing the activation layer and weights contained in the multi-layer network layer according to the first quantization parameter corresponding to the first quantization type to obtain the target machine learning model; when the quantization type is a second quantization type, quantizing the weights according to the second quantization parameter corresponding to the second quantization type to obtain the target machine learning model.

[0069] The activation layers in the aforementioned multi-layer network are mainly used to enhance the representational power of the deep learning model. The weights mentioned above refer to the values ​​of various parameters in the deep learning model. By adjusting the weights, the accuracy of the output results of the deep learning model can be improved.

[0070] The first quantization type mentioned above can be used to represent full quantization, where full quantization means that the weights and activation layers are calculated with low precision, such as A8W8 and A4W4. Here, A8W8 represents a floating-point number with a precision of 8 bits, and A4W4 represents a floating-point number with a precision of 4 ...

[0071] The first quantization parameter mentioned above can be set according to the precision requirements corresponding to the quantization type. For the first quantization type, the exponent and the number of decimal places of the first quantization parameter can be the same.

[0072] The second quantization type mentioned above can be semi-quantization, where semi-quantization means that the weights use low precision and the activation layer uses floating-point numbers, such as A16W8 and A16W4. Here, A16W8 means that the precision of the input parameters is 16 bits and the precision of the output parameters is 8 bits, and A16W4 means that the precision of the input parameters is 16 bits and the precision of the output parameters is 4 bits.

[0073] The second quantization parameter mentioned above can be set according to the precision requirements corresponding to the quantization type. For the second quantization type, the exponent and the number of decimal places of the second quantization parameter can be different.

[0074] In an optional embodiment, when the quantization type is the first quantization type, the multi-layer activation layer and model weights can be quantized according to the first quantization parameter, so that the multi-layer activation layer and model weights are calculated with low precision, thereby obtaining a target machine learning model with relatively high performance and general accuracy.

[0075] In another alternative embodiment, when the quantization type is the second quantization type, the model weights can be quantized according to the second quantization parameters, so that the model weights are calculated with low precision, thereby obtaining a target machine learning model with relatively high precision and average performance.

[0076] In the above embodiments of this application, the target quantization method includes: quantization granularity. Based on the target quantization method, the weights of the network layer are quantized to obtain a target machine learning model. This includes: when the quantization granularity is tensor granularity, the weights of the network layer are quantized according to a set of quantization parameters to obtain a target machine learning model; when the quantization granularity is channel granularity, multiple channels contained in the weights are quantized according to multiple sets of first quantization parameters to obtain a target machine learning model, wherein the multiple sets of first quantization parameters correspond one-to-one with multiple channels; when the quantization granularity is sub-channel granularity, multiple sub-channels in different channels contained in the weights are quantized according to multiple sets of second quantization parameters to obtain a target machine learning model, wherein the multiple sets of second quantization parameters correspond one-to-one with multiple sub-channels.

[0077] The tensor granularity (Per Tensor) mentioned above is used to indicate that a tensor has a set of quantization parameters. Tensor granularity has relatively high performance but generally low accuracy. The performance of tensor granularity can be greater than a first preset threshold, and the accuracy of tensor granularity can be less than or equal to a second preset threshold.

[0078] The aforementioned channel granularity (Per Channel) is used to indicate that a tensor has multiple sets of quantization parameters. The performance and accuracy of channel granularity are generally average. Specifically, the performance of channel granularity can be less than or equal to a first preset threshold, and the accuracy of channel granularity can be less than or equal to a second preset threshold.

[0079] The aforementioned sub-channel granularity is used to represent further refinement of the quantization parameters based on the channel granularity. The sub-channel granularity has relatively high precision but generally low performance. Specifically, the performance of the sub-channel granularity is less than or equal to the first preset threshold, and the performance of the sub-channel granularity is greater than the second preset threshold.

[0080] In one alternative embodiment, users can reduce the cost of quantization and select appropriate quantization methods to quantize the weights of multiple network layers in a machine learning model based on the accuracy and performance requirements of different tasks, thereby obtaining a target machine learning model with a relatively small memory footprint.

[0081] If the requirements for model performance are relatively high but the requirements for accuracy are moderate, while the memory usage of the machine learning model is reduced, tensor granularity can be selected. The weights of the network layer are quantized according to a set of quantization parameters to obtain the target machine learning model.

[0082] If the requirements for model performance and accuracy are moderate, and the memory usage of the machine learning model is reduced, the target machine learning model can be obtained by quantizing multiple channels contained in the weights according to multiple sets of first quantization parameters based on the quantization granularity of the channel granularity. Here, multiple sets of first quantization parameters correspond one-to-one with multiple channels.

[0083] If the model requires high accuracy but low performance, while reducing the memory footprint of the machine learning model, the target machine learning model can be obtained by quantizing multiple sub-channels in different channels according to multiple sets of second quantization parameters based on the quantization granularity of the sub-channels. In this case, multiple sets of second quantization parameters correspond one-to-one with multiple sub-channels.

[0084] The aforementioned quantization parameters, first quantization parameter, and second quantization parameter can be the same or different.

[0085] In the above embodiments of this application, the quantization method includes: enhancing quantization granularity and quantization accuracy. The weights of the network layer are quantized based on the target quantization method to obtain a target machine learning model. This includes: determining a first weight corresponding to the quantization granularity; determining a second weight in the first weight based on the mean and variance of the first weight, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model.

[0086] The aforementioned quantization accuracy enhancement is mainly used to indicate that the accuracy of the model is enhanced based on the quantization granularity.

[0087] The quantization granularity mentioned above can be selected by the user from different quantization granularity levels according to their needs.

[0088] The first weight mentioned above can be a floating-point number.

[0089] In an alternative embodiment, since some domains have high requirements for the accuracy of large models, an outlier sparse matrix can be introduced to repair the loss of quantization accuracy. An outlier sparse matrix is ​​a matrix in which most elements are zero and only a few non-zero elements are contained. These non-zero elements are generally called "outliers" or "outliers".

[0090] The deviation of the second weight from the mean is used to represent the square of the difference between the second weight and the mean. That is, the square of the difference between the second weight and the mean is greater than the variance of the first weight.

[0091] The first weight, corresponding to the quantization granularity, can be determined. Based on the mean and variance of the first weight, the second weight can be determined. The quantization parameter corresponding to the second weight is the aforementioned non-zero element. The remaining quantization parameters can be obtained by removing the quantization parameter corresponding to the second weight from the initial quantization parameters and filling the empty positions with zero elements. The weights can be quantized based on the remaining quantization parameters to obtain the sparse matrix calculation result. The quantization matrix calculation result can be obtained by quantizing the weights based on the initial quantization parameters. The sparse matrix calculation result can be used to repair the accuracy loss caused by quantization in the quantization matrix calculation result to obtain the target machine learning model. Optionally, the target machine learning model can be obtained by summing the quantization matrix calculation result and the sparse matrix calculation result.

[0092] In the above embodiments of this application, the weights are quantized based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model, including: quantizing the weights based on the initial quantization parameters to obtain a first quantization result; quantizing the weights based on the remaining quantization parameters to obtain a second quantization result; and summing the first quantization result and the second quantization result to obtain the target machine learning model.

[0093] The first quantization result mentioned above can be the result of quantization matrix calculation.

[0094] The second quantization result mentioned above can be the result of sparse matrix calculation.

[0095] In one alternative embodiment, the weights can be quantized based on the initial quantization parameters using a quantization formula to obtain the quantization matrix calculation result.

[0096] In another alternative embodiment, the weights can be quantized based on the remaining quantization parameters using a quantization formula to obtain the sparse matrix calculation result. It should be noted that this quantization process can be performed using a low-precision method.

[0097] The results of the quantization matrix calculation and the sparse matrix calculation can be summed to obtain the target machine learning model mentioned above.

[0098] In the above embodiments of this application, the method further includes: displaying multiple accelerators on the operation interface in response to a model deployment instruction applied to the operation interface; and displaying a target accelerator on the operation interface in response to a processor selection instruction applied to the multiple accelerators, wherein the target accelerator is used to accelerate the processing of a target machine learning model.

[0099] The aforementioned accelerators may include, but are not limited to: hardware accelerators (Tensor Cores), Advanced Single Instruction Multiple Data (NEON) accelerators, and Advanced Vector Extensions (AVX) accelerators.

[0100] In one optional embodiment, after the user performs the relevant operations of generating model deployment instructions on the operation interface, multiple accelerators can be displayed on the operation interface so that the user can select a target accelerator suitable for the model deployment scenario, thereby generating a processor selection instruction. The target accelerator can be displayed on the operation interface according to the processor selection instruction, and the target accelerator can be used to accelerate the target machine learning model.

[0101] In another alternative embodiment, the target machine learning model can be accelerated using General Matrix Multiplication (GEMM) and attention operators.

[0102] In another alternative embodiment, the target acceleration processor can be determined based on the type of platform on which the target machine learning model is to be deployed.

[0103] If the platform to be deployed is a graphics processing unit (GPU) platform, the target acceleration processor can be determined as a Tensor Core, and the Tensor Core can be used to accelerate the deployment of the target machine learning model.

[0104] If the platform to be deployed is an Advanced RISC Machine (ARM) processor architecture, the target accelerator processor can be identified as NEON, and NEON can be used to accelerate the deployment of the target machine learning model.

[0105] If the platform to be deployed is an x86 architecture processor (X86-CPU) platform, the target acceleration processor can be determined as AVX, and AVX can be used to accelerate the deployment of the target machine learning model.

[0106] In the above embodiments of this application, the method further includes: obtaining a target quantization strategy corresponding to the target quantization method, wherein the target quantization strategy includes: quantization step size and quantization zero point; determining the data to be quantized in the initial machine learning model; obtaining the ratio of the data to be quantized to the quantization step size to obtain the quantization ratio; and obtaining the sum of the quantization ratio and the quantization zero point to obtain the quantized data in the target machine learning model.

[0107] The quantization data mentioned above can be used as a quantization formula. Since most hardware can directly provide floating-point multiply-add instructions (FMA), fast quantization can be achieved through multiply-add operations in the quantization formula.

[0108] In one alternative embodiment, a quantization zero point can be determined based on the quantization step size, which is primarily used to maintain the accuracy of the model.

[0109] The weights stored in the initial machine learning model (16-bit / 32-bit) can be quantized and stored in the target machine learning model (4-bit or 8-bit).

[0110] For example, the weights stored in the initial machine learning model (16 bits) can be quantized into the target machine learning model (8 bits) for storage. The quantization formula can be INT8 = FP16 / Scale + Zero Point.

[0111] The quantization step size mentioned above is Scale, where Scale = (FP16_Max – FP16_Min) / (INT8_Max – INT8_Min), where FP16_Max represents the maximum value of a 16-bit floating-point data type, FP16_Min represents the minimum value of a 16-bit floating-point data type, INT8_Max represents the maximum value of an 8-bit integer data type, and INT8_Min represents the minimum value of an 8-bit integer data type.

[0112] The data to be quantified above is FP16.

[0113] The quantization ratio mentioned above is FP16 / Scale.

[0114] The aforementioned quantization zero point is called Zero Point, where Zero Point = INT8_Min - FP16_Min / Scale. By adding a zero point during the quantization process, the accuracy of the model can be better maintained.

[0115] In another alternative embodiment, the quantization and dequantization processes can be integrated into other operators of the machine learning model, such as the Layer Norm operator or the Element Wise operator, in order to reduce the overhead of operator computation.

[0116] Figure 3 This is a flowchart illustrating another method for deploying a machine learning model according to an embodiment of this application. Figure 3 As shown, this step includes:

[0117] Step S301: Input the initial machine learning model;

[0118] The initial machine learning model described above can be a floating-point model, but is not limited to this.

[0119] Step S302: Determine whether to quantize the floating-point model. If yes, proceed to step S303; otherwise, proceed to step S305.

[0120] Step S303: Select the target quantification method;

[0121] Step S304: Quantize the initial machine learning model according to the selected target quantization method to obtain the target machine learning model, and select the platform to use the target machine learning model according to the target quantization method.

[0122] Optionally, different quantization methods can be deployed using different hardware devices.

[0123] Step S305: Accelerate the deployment of the target machine learning model according to the target acceleration processor corresponding to the platform.

[0124] If the platform to be deployed is a GPU platform, the target acceleration processor can be identified as Tensor Core, and Tensor Core can be used to accelerate the deployment of the target machine learning model.

[0125] If the platform to be deployed is an ARM platform, the target acceleration processor can be determined as NEON, and NEON can be used to accelerate the deployment of the target machine learning model.

[0126] If the platform to be deployed is an x86-CPU platform, the target acceleration processor can be identified as AVX, and AVX can be used to accelerate the deployment of the target machine learning model.

[0127] This application provides a quantization acceleration solution for large model inference. It ensures lossless quantization accuracy through various means such as refining quantization granularity and outlier correction, while significantly improving performance. Compared with other quantization solutions, this application is more convenient and seamless to use, and supports multiple heterogeneous computing platforms, providing a more efficient and economical acceleration solution for large model inference scenarios.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0129] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0131] Example 2

[0132] According to an embodiment of this application, a method for deploying a large model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0133] Figure 4 This is a flowchart of a large model deployment method according to Embodiment 2 of this application, as shown below. Figure 4 As shown, the method includes the following steps:

[0134] Step S402: In response to the quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial large model is displayed on the operation interface.

[0135] The initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model.

[0136] The large models mentioned above can refer to machine learning models with a large number of parameters and complexity. These models typically require a large amount of computing resources and data for training and inference.

[0137] The large model described above can be applied in multiple fields, such as natural language processing, computer vision, and speech recognition. There are no limitations here, and the application scenario of the large model can be determined according to the actual task requirements.

[0138] The aforementioned target quantization methods include at least one of the following: quantization type, quantization granularity, and quantization precision enhancement.

[0139] Step S404: In response to the model deployment command applied to the operation interface, display the deployment result of the initial large model on the operation interface.

[0140] The deployment result is the result obtained after deploying the target large model, which is a model obtained by quantizing the initial large model based on the target quantization method.

[0141] Through the above steps, in response to the quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial large model is displayed on the operation interface. The initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model. In response to the model deployment command applied to the operation interface, the deployment result of the initial large model is displayed on the operation interface. The deployment result is the result obtained after deploying the target large model, which is a model obtained by quantizing the initial large model based on the target quantization method. This achieves the goal of accelerating the deployment of machine learning models. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0142] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0143] Example 3

[0144] According to an embodiment of this application, a method for deploying a large model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0145] Figure 5 This is a flowchart of a large-scale model deployment method according to Embodiment 3 of this application, as shown below. Figure 5 As shown, the method includes the following steps:

[0146] Step S502: In response to the input command applied to the operation interface, display the data to be processed on the operation interface.

[0147] The aforementioned data to be processed can be the data that the target machine learning model needs to perform inference on.

[0148] Step S504: In response to the processing instructions applied to the operation interface, display the processing result of the data to be processed on the operation interface.

[0149] The processing result is obtained by accelerating the quantized data based on the target machine learning model. The target machine learning model is obtained by quantizing the initial machine learning model based on the target quantization method. The quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method. The initial machine learning model is a pre-trained neural network model. The target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0150] In one optional embodiment, the user can input the data to be processed into the target machine learning model through the operation interface. First, the target quantization strategy corresponding to the target quantization method can be used to quantize the data to be processed in order to reduce the memory resources occupied by the data to be processed and obtain quantized data. The target machine learning model can then be used to infer the quantized data to obtain the processing result.

[0151] Through the above steps, in response to the input commands applied to the operation interface, the data to be processed is displayed on the operation interface.

[0152] Responding to processing commands applied to the user interface, the system displays the processing results of the data to be processed. These results are obtained by accelerating the quantized data based on the target machine learning model. The target machine learning model is obtained by quantizing the initial machine learning model using a target quantization method. The quantized data is obtained by quantizing the data to be processed using the target quantization strategy corresponding to the target quantization method. The initial machine learning model is a pre-trained neural network model. The target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model, thus achieving the goal of accelerating the deployment of the machine learning model. It is noteworthy that different quantization methods can be used to quantize the initial machine learning model to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost and enabling large-scale deployment of the target machine learning model. This solves the technical problem of high model deployment costs in related technologies.

[0153] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0154] Example 4

[0155] According to an embodiment of this application, a method for processing large models is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0156] Figure 6 This is a flowchart of a large model processing method according to Embodiment 4 of this application, as shown below. Figure 6 As shown, the method includes the following steps:

[0157] Step S602: In response to the input command applied to the operation interface, display the data to be processed on the operation interface.

[0158] Step S604: In response to the processing instructions applied to the operation interface, display the processing result of the data to be processed on the operation interface.

[0159] The processing result is obtained by accelerating the quantized data based on the target large model. The target large model is obtained by quantizing the initial large model based on the target quantization method. The quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method. The initial large model is a pre-trained large model. The target quantization method is used to characterize the quantization scale of quantizing the initial large model.

[0160] In one optional embodiment, the user can input the data to be processed into the target large model through the operation interface. First, the target quantization strategy corresponding to the target quantization method can be used to quantize the data to be processed in order to reduce the memory resources occupied by the data to be processed and obtain quantized data. The target large model can then be used to infer the quantized data to obtain the processing result.

[0161] In the above embodiments of this application, the initial large model includes: a multi-layer network layer, and the target large model is a model obtained by quantizing the weights of the network layers based on the target quantization method.

[0162] Through the above steps, in response to input commands applied to the operation interface, the data to be processed is displayed on the operation interface; in response to processing commands applied to the operation interface, the processing result of the data to be processed is displayed on the operation interface. The processing result is obtained by accelerating the quantized data based on the target large model. The target large model is obtained by quantizing the initial large model based on the target quantization method. The quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method. The initial large model is a pre-trained large model. The target quantization method is used to characterize the quantization scale for quantizing the initial large model, thus achieving the goal of accelerating the deployment of the machine learning model. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0163] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0164] Example 5

[0165] According to an embodiment of this application, a method for deploying a machine learning model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0166] Figure 7 This is a flowchart of a large model deployment method according to Embodiment 5 of this application, as shown below. Figure 7 As shown, the method includes the following steps:

[0167] Step S702: Obtain the initial machine learning model.

[0168] The initial machine learning model is a pre-trained neural network model.

[0169] Step S704: Determine the target quantification method.

[0170] Among them, the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0171] The target quantization method described above can be determined by the user. Alternatively, the target quantization method can be determined without user instruction, based on the number of model weights, the model's purpose, and the performance and accuracy requirements of the model task.

[0172] Step S706: Quantize the initial machine learning model based on the target quantization method to obtain the target machine learning model.

[0173] Step S708: Deploy the target machine learning model.

[0174] In one alternative embodiment, the initial machine learning model can be quantized according to the target quantization method to obtain a target machine learning model with a smaller memory footprint, thereby improving the efficiency of deploying the target machine learning model.

[0175] Through the above steps, an initial machine learning model is obtained, which is a pre-trained neural network model; a target quantization method is determined, which is used to characterize the quantization scale for quantizing the initial machine learning model; the initial machine learning model is quantized based on the target quantization method to obtain the target machine learning model; the target machine learning model is deployed, thus achieving the goal of accelerating the deployment of the machine learning model; it is worth noting that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model, enabling the target machine learning model to be deployed on a large scale, and thus solving the technical problem of high model deployment cost in related technologies.

[0176] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0177] Example 6

[0178] According to an embodiment of this application, a method for deploying a machine learning model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0179] Figure 8 This is a flowchart of a method for deploying a machine learning model according to Embodiment 6 of this application, as shown below. Figure 8 As shown, the method includes the following steps:

[0180] Step S802: Obtain the target quantization method by calling the first interface.

[0181] The first interface includes a first parameter, the value of which is the target quantization method. The target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0182] The first interface in the above steps can be an interface for data interaction between the cloud server and the client. The client can pass the target quantization method into the interface function as the first parameter of the interface function, so as to achieve the purpose of uploading the first parameter to the cloud server.

[0183] Step S804: Quantize the initial machine learning model based on the target quantization method to obtain the target machine learning model.

[0184] The initial machine learning model is a pre-trained neural network model.

[0185] Step S806: Output the target machine learning model by calling the second interface.

[0186] The second interface includes a second parameter, the value of which is the target machine learning model.

[0187] The aforementioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the target machine learning model into the interface function as the second parameter of the interface function, thereby achieving the purpose of sending the target machine learning model to the client.

[0188] Through the above steps, the target quantization method is obtained by calling the first interface, where the first interface includes a first parameter whose value is the target quantization method. The target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model. The initial machine learning model is then quantized based on the target quantization method to obtain the target machine learning model, where the initial machine learning model is a pre-trained neural network model. The target machine learning model is then output by calling the second interface, where the second interface includes a second parameter whose value is the target machine learning model. This achieves the goal of accelerating the deployment of the machine learning model. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows the target machine learning model to be deployed on a large scale, thus solving the technical problem of high model deployment cost in related technologies.

[0189] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0190] Example 7

[0191] According to an embodiment of this application, an accelerated deployment apparatus for implementing the deployment method of the above-described machine learning model is also provided. Figure 9 This is a schematic diagram of a machine learning model deployment device according to Embodiment 7 of this application, as shown below. Figure 9 As shown, the device 900 includes: a first display module 902 and a second display module 904.

[0192] The first display module is used to respond to the quantization configuration command applied to the operation interface and display the target quantization method corresponding to the initial machine learning model on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model. The second display module is used to respond to the model deployment command applied to the operation interface and display the deployment result of the initial machine learning model on the operation interface. The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0193] It should be noted that the first display module 902 and the second display module 904 mentioned above correspond to steps S202 to S204 in Embodiment 1. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0194] In the above embodiments of this application, the initial machine learning model includes: a multi-layer network layer, and the device further includes: a quantization module.

[0195] The quantization module is used to quantize the weights of the network layer based on the target quantization method to obtain the target machine learning model.

[0196] The target quantization method includes: quantization type. The quantization module is also used to quantize the activation layers and weights contained in the multi-layer network layer according to the first quantization parameter corresponding to the first quantization type when the quantization type is the first quantization type, so as to obtain the target machine learning model; and to quantize the weights according to the second quantization parameter corresponding to the second quantization type when the quantization type is the second quantization type, so as to obtain the target machine learning model.

[0197] In the above embodiments of this application, the target quantization method includes: quantization granularity. The quantization module is further configured to: when the quantization granularity is tensor granularity, quantize the weights according to a set of quantization parameters to obtain a target machine learning model; when the quantization granularity is channel granularity, quantize the multiple channels contained in the weights according to multiple sets of first quantization parameters to obtain a target machine learning model, wherein the multiple sets of first quantization parameters correspond one-to-one with the multiple channels; when the quantization granularity is sub-channel granularity, quantize the multiple sub-channels in different channels contained in the weights according to multiple sets of second quantization parameters to obtain a target machine learning model, wherein the multiple sets of second quantization parameters correspond one-to-one with the multiple sub-channels.

[0198] In the above embodiments of this application, the quantization method includes: enhancing quantization granularity and quantization precision. The quantization module is further used to determine a first weight corresponding to the quantization granularity in the weights; determine a second weight in the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; remove the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantize the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model.

[0199] In the above embodiments of this application, the quantization module is further configured to quantize the weights based on the initial quantization parameters to obtain a first quantization result; quantize the weights based on the remaining quantization parameters to obtain a second quantization result; and sum the first quantization result and the second quantization result to obtain the target machine learning model.

[0200] In the above embodiments of this application, the device further includes a third display module.

[0201] The third display module is also used to respond to model deployment instructions applied to the operation interface, display multiple accelerators on the operation interface, and respond to processor selection instructions applied to the multiple accelerators, display the target accelerator on the operation interface. The target accelerator is used to accelerate the processing of the target machine learning model.

[0202] In the above embodiments of this application, the device further includes: an acquisition module and a determination module.

[0203] The acquisition module is also used to acquire the target quantization strategy corresponding to the target quantization method, wherein the target quantization strategy includes: quantization step size and quantization zero point; the determination module is also used to determine the data to be quantized in the initial machine learning model; the acquisition module is also used to acquire the ratio of the data to be quantized to the quantization step size to obtain the quantization ratio; the acquisition module is also used to acquire the sum of the quantization ratio and the quantization zero point to obtain the quantized data in the target machine learning model.

[0204] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0205] Example 8

[0206] According to embodiments of this application, an accelerated deployment apparatus for implementing the above-described large-scale model deployment method is also provided. Figure 10 This is a schematic diagram of a large-scale model deployment device according to Embodiment 8 of this application, as shown below. Figure 10 As shown, the device 1000 includes: a first display module 1002 and a second display module 1004.

[0207] The first display module is used to respond to the quantization configuration command applied to the operation interface and display the target quantization method corresponding to the initial large model on the operation interface. The initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model. The second display module is used to respond to the model deployment command applied to the operation interface and display the deployment result of the initial large model on the operation interface. The deployment result is the result obtained after deploying the target large model, which is a model obtained by quantizing the initial large model based on the target quantization method.

[0208] It should be noted that the first display module 1002 and the second display module 1004 mentioned above correspond to steps S402 to S404 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0209] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0210] Example 9

[0211] According to embodiments of this application, an accelerated deployment apparatus for implementing the processing method of the above-described machine learning model is also provided. Figure 11 This is a schematic diagram of a processing device for a machine learning model according to Embodiment 9 of this application, as shown below. Figure 11 As shown, the device 1100 includes: a first display module 1102 and a second display module 1104.

[0212] The first display module responds to input commands applied to the operation interface and displays the data to be processed on the operation interface; the second display module responds to processing commands applied to the operation interface and displays the processing result of the data to be processed on the operation interface. The processing result is obtained by accelerating the quantized data based on the target machine learning model. The target machine learning model is obtained by quantizing the initial machine learning model based on the target quantization method. The quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method. The initial machine learning model is a pre-trained neural network model. The target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0213] It should be noted that the first display module 1102 and the second display module 1104 mentioned above correspond to steps S502 to S504 in Embodiment 3. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0214] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0215] Example 10

[0216] According to embodiments of this application, an accelerated deployment apparatus for implementing the above-described large-scale model deployment method is also provided. Figure 12 This is a schematic diagram of a large-scale model deployment device according to Embodiment 10 of this application, as shown below. Figure 12 As shown, the device 1200 includes: a first display module 1202 and a second display module 1204.

[0217] The first display module responds to input commands on the operation interface and displays the data to be processed on the operation interface; the second display module responds to processing commands on the operation interface and displays the processing result of the data to be processed on the operation interface. The processing result is obtained by accelerating the quantized data based on the target large model. The target large model is obtained by quantizing the initial large model based on the target quantization method. The quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method. The initial large model is a pre-trained large model. The target quantization method is used to characterize the quantization scale for quantizing the initial large model.

[0218] It should be noted that the first display module 1202 and the second display module 1204 mentioned above correspond to steps S602 to S604 in Embodiment 4. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0219] In the above embodiments of this application, the initial large model includes: a multi-layer network layer, and the target large model is a model obtained by quantizing the weights of the network layers based on the target quantization method.

[0220] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0221] Example 11

[0222] According to an embodiment of this application, an accelerated deployment apparatus for implementing the deployment method of the above-described machine learning model is also provided. Figure 13 This is a schematic diagram of a machine learning model deployment apparatus according to Embodiment 11 of this application, as shown below. Figure 13 As shown, the device 1300 includes: an acquisition module 1302, a determination module 1304, a quantization module 1306, and a deployment module 1308.

[0223] The acquisition module is used to acquire an initial machine learning model, which is a pre-trained neural network model; the determination module is used to determine the target quantization method, which is used to characterize the quantization scale for quantizing the initial machine learning model; the quantization module is used to quantize the initial machine learning model based on the target quantization method to obtain the target machine learning model; and the deployment module is used to deploy the target machine learning model.

[0224] It should be noted that the acquisition module 1302, determination module 1304, quantization module 1306, and deployment module 1308 mentioned above correspond to steps S702 to S708 in Embodiment 5. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0225] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0226] Example 12

[0227] According to an embodiment of this application, an accelerated deployment apparatus for implementing the deployment method of the above-described machine learning model is also provided. Figure 14 This is a schematic diagram of a machine learning model deployment device according to Embodiment 12 of this application, as shown below. Figure 14 As shown, the device 1400 includes: an acquisition module 1402, a quantization module 1404, and a calling module 1406.

[0228] The acquisition module is used to acquire the target quantization method by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which is the target quantization method, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; the quantization module is used to quantize the initial machine learning model based on the target quantization method to obtain the target machine learning model, wherein the initial machine learning model is a pre-trained neural network model; the calling module is used to output the target machine learning model by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which is the target machine learning model.

[0229] It should be noted that the acquisition module 1402, quantization module 1404, and invocation module 1406 mentioned above correspond to steps S802 to S806 in Embodiment 5. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0230] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0231] Example 13

[0232] Embodiments of this application may provide an electronic device, which may be a computer terminal, and the computer terminal may be any one of a group of computer terminal devices. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0233] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0234] In this embodiment, the computer terminal described above can execute program code for the following steps in the method for deploying a machine learning model: responding to a quantization configuration command applied to the operation interface, displaying the target quantization method corresponding to the initial machine learning model on the operation interface, wherein the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; responding to a model deployment command applied to the operation interface, displaying the deployment result of the initial machine learning model on the operation interface, wherein the deployment result is the result obtained after deploying the target machine learning model, and the target machine learning model is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0235] Optionally, Figure 15 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 15 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, memory 104, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0236] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the machine learning model deployment method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned machine learning model deployment method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to computer terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0237] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: In response to a quantization configuration command applied to the operating interface, display the target quantization method corresponding to the initial machine learning model on the operating interface, wherein the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; In response to a model deployment command applied to the operating interface, display the deployment result of the initial machine learning model on the operating interface, wherein the deployment result is the result obtained after deploying the target machine learning model, and the target machine learning model is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0238] Optionally, the processor may also execute program code that performs the following steps: quantizes the weights of the network layer based on the target quantization method to obtain the target machine learning model.

[0239] Optionally, the processor may also execute program code with the following steps: when the quantization type is the first quantization type, quantize the activation layers and weights contained in the multi-layer network according to the first quantization parameter corresponding to the first quantization type to obtain the target machine learning model; when the quantization type is the second quantization type, quantize the weights according to the second quantization parameter corresponding to the second quantization type to obtain the target machine learning model.

[0240] Optionally, the processor may also execute program code with the following steps: when the quantization granularity is tensor granularity, quantize the weights according to a set of quantization parameters to obtain the target machine learning model; when the quantization granularity is channel granularity, quantize the multiple channels contained in the weights according to multiple sets of first quantization parameters to obtain the target machine learning model, wherein the multiple sets of first quantization parameters correspond one-to-one with the multiple channels; when the quantization granularity is sub-channel granularity, quantize the multiple sub-channels in different channels contained in the weights according to multiple sets of second quantization parameters to obtain the target machine learning model, wherein the multiple sets of second quantization parameters correspond one-to-one with the multiple sub-channels.

[0241] Optionally, the processor may also execute program code that performs the following steps: determining a first weight corresponding to the quantization granularity in the weights; determining a second weight in the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model.

[0242] Optionally, the processor may also execute program code that performs the following steps: quantizes the weights based on the initial quantization parameters to obtain a first quantization result; quantizes the weights based on the remaining quantization parameters to obtain a second quantization result; and sums the first quantization result and the second quantization result to obtain the target machine learning model.

[0243] Optionally, the processor may also execute program code that performs the following steps: in response to a model deployment instruction applied to the operating interface, displays multiple accelerators on the operating interface; in response to a processor selection instruction applied to the multiple accelerators, displays a target accelerator on the operating interface, wherein the target accelerator is used to accelerate the processing of a target machine learning model.

[0244] Optionally, the processor may also execute program code for the following steps: obtaining the target quantization strategy corresponding to the target quantization method, wherein the target quantization strategy includes: quantization step size and quantization zero; determining the data to be quantized in the initial machine learning model; obtaining the ratio of the data to be quantized to the quantization step size to obtain the quantization ratio; obtaining the sum of the quantization ratio and the quantization zero to obtain the quantized data in the target machine learning model.

[0245] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: responding to a quantization configuration command applied to the operation interface, displaying the target quantization method corresponding to the initial large model on the operation interface, wherein the initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model; responding to a model deployment command applied to the operation interface, displaying the deployment result of the initial large model on the operation interface, wherein the deployment result is the result obtained after deploying the target large model, and the target large model is a model obtained by quantizing the initial large model based on the target quantization method.

[0246] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: responding to input instructions applied to the operation interface and displaying the data to be processed on the operation interface; responding to processing instructions applied to the operation interface and displaying the processing result of the data to be processed on the operation interface, wherein the processing result is the result obtained by accelerating the quantized data based on the target machine learning model, the target machine learning model is obtained by quantizing the initial machine learning model based on the target quantization method, the quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method, the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0247] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: responding to input instructions applied to the operation interface and displaying the data to be processed on the operation interface; responding to processing instructions applied to the operation interface and displaying the processing result of the data to be processed on the operation interface, wherein the processing result is the result obtained by accelerating the quantized data based on the target large model, the target large model is obtained by quantizing the initial large model based on the target quantization method, the quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method, the initial large model is a pre-trained large model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model.

[0248] Optionally, the processor may also execute program code for the following steps: the initial large model includes: multiple network layers, and the target large model is a model obtained by quantizing the weights of the network layers based on the target quantization method.

[0249] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtaining an initial machine learning model, wherein the initial machine learning model is a pre-trained neural network model; determining a target quantization method, wherein the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model; and deploying the target machine learning model.

[0250] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: obtaining a target quantization method by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the target quantization method, the target quantization method being used to characterize the quantization scale for quantizing the initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model, wherein the initial machine learning model is a pre-trained neural network model; and outputting the target machine learning model by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target machine learning model.

[0251] In this embodiment, in response to a quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method characterizes the quantization scale used to quantize the initial machine learning model. In response to a model deployment command applied to the operation interface, the deployment result of the initial machine learning model is displayed on the operation interface. The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method. This achieves the goal of accelerating the deployment of the machine learning model. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0252] Those skilled in the art will understand that Figure 15 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 15 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 15 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 15 The different configurations shown.

[0253] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0254] Example 14

[0255] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the deployment method of the machine learning model provided in Embodiment 1.

[0256] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0257] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: information and application programs stored in the memory, to perform the following steps: in response to a quantization configuration command applied to the operation interface, displaying the target quantization method corresponding to the initial machine learning model on the operation interface, wherein the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; in response to a model deployment command applied to the operation interface, displaying the deployment result of the initial machine learning model on the operation interface, wherein the deployment result is the result obtained after deploying the target machine learning model, and the target machine learning model is a model obtained by quantizing the initial machine learning model based on the target quantization method.

[0258] Optionally, the aforementioned storage medium is also configured to store program code for performing the following steps: quantizing the weights of the network layer based on the target quantization method to obtain the target machine learning model.

[0259] Optionally, the storage medium is further configured to store program code for performing the following steps: when the quantization type is a first quantization type, quantizing the activation layers and weights contained in the multi-layer network according to the first quantization parameters corresponding to the first quantization type to obtain the target machine learning model; when the quantization type is a second quantization type, quantizing the weights according to the second quantization parameters corresponding to the second quantization type to obtain the target machine learning model.

[0260] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: when the quantization granularity is tensor granularity, quantizing the weights according to a set of quantization parameters to obtain a target machine learning model; when the quantization granularity is channel granularity, quantizing the multiple channels contained in the weights according to multiple sets of first quantization parameters to obtain a target machine learning model, wherein the multiple sets of first quantization parameters correspond one-to-one with the multiple channels; when the quantization granularity is sub-channel granularity, quantizing the multiple sub-channels in different channels contained in the weights according to multiple sets of second quantization parameters to obtain a target machine learning model, wherein the multiple sets of second quantization parameters correspond one-to-one with the multiple sub-channels.

[0261] Optionally, the storage medium is further configured to store program code for performing the following steps: determining a first weight corresponding to the quantization granularity in the weights; determining a second weight in the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model.

[0262] Optionally, the storage medium is further configured to store program code for performing the following steps: quantizing the weights based on initial quantization parameters to obtain a first quantization result; quantizing the weights based on the remaining quantization parameters to obtain a second quantization result; and summing the first quantization result and the second quantization result to obtain the target machine learning model.

[0263] Optionally, the storage medium is further configured to store program code for performing the following steps: displaying multiple accelerators on the operation interface in response to a model deployment instruction applied to the operation interface; and displaying a target accelerator on the operation interface in response to a processor selection instruction applied to the multiple accelerators, wherein the target accelerator is used to accelerate the processing of a target machine learning model.

[0264] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: obtaining the target quantization strategy corresponding to the target quantization method, wherein the target quantization strategy includes: quantization step size and quantization zero point; determining the data to be quantized in the initial machine learning model; obtaining the ratio of the data to be quantized to the quantization step size to obtain the quantization ratio; obtaining the sum of the quantization ratio and the quantization zero point to obtain the quantized data in the target machine learning model.

[0265] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: in response to a quantization configuration command applied to the operation interface, displaying the target quantization method corresponding to the initial large model on the operation interface, wherein the initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model; in response to a model deployment command applied to the operation interface, displaying the deployment result of the initial large model on the operation interface, wherein the deployment result is the result obtained after deploying the target large model, and the target large model is a model obtained by quantizing the initial large model based on the target quantization method.

[0266] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying data to be processed on the operation interface in response to an input command applied to the operation interface; displaying the processing result of the data to be processed on the operation interface in response to a processing command applied to the operation interface, wherein the processing result is the result obtained by accelerating the quantized data based on a target machine learning model, the target machine learning model is obtained by quantizing an initial machine learning model based on a target quantization method, the quantized data is obtained by quantizing the data to be processed based on a target quantization strategy corresponding to the target quantization method, the initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model.

[0267] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying the data to be processed on the operation interface; in response to a processing command applied to the operation interface, displaying the processing result of the data to be processed on the operation interface, wherein the processing result is the result obtained by accelerating the quantized data based on the target large model, the target large model is obtained by quantizing the initial large model based on the target quantization method, the quantized data is obtained by quantizing the data to be processed based on the target quantization strategy corresponding to the target quantization method, the initial large model is a pre-trained large model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model.

[0268] Optionally, the storage medium is also configured to store program code for performing the following steps: the initial large model includes: multiple network layers, and the target large model is a model obtained by quantizing the weights of the network layers based on the target quantization method.

[0269] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining an initial machine learning model, wherein the initial machine learning model is a pre-trained neural network model; determining a target quantization method, wherein the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model; and deploying the target machine learning model.

[0270] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a target quantization method by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the target quantization method, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model; quantizing the initial machine learning model based on the target quantization method to obtain a target machine learning model, wherein the initial machine learning model is a pre-trained neural network model; and outputting the target machine learning model by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the target machine learning model.

[0271] In this embodiment, in response to a quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method characterizes the quantization scale used to quantize the initial machine learning model. In response to a model deployment command applied to the operation interface, the deployment result of the initial machine learning model is displayed on the operation interface. The deployment result is the result obtained after deploying the target machine learning model, which is a model obtained by quantizing the initial machine learning model based on the target quantization method. This achieves the goal of accelerating the deployment of the machine learning model. It is noteworthy that the initial machine learning model can be quantized according to different quantization methods to improve the inference performance of the target machine learning model and reduce the memory occupied by the target machine learning model, thereby reducing the deployment cost of the target machine learning model. This allows for large-scale deployment of the target machine learning model, thus solving the technical problem of high model deployment costs in related technologies.

[0272] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0273] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0274] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0275] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0276] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0277] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0278] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for deploying a machine learning model, characterized in that, include: In response to a quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial machine learning model is displayed on the operation interface. The initial machine learning model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model. The target quantization method includes: quantization granularity and quantization accuracy enhancement. In response to a model deployment command applied to the operation interface, the deployment result of the initial machine learning model is displayed on the operation interface, wherein the deployment result is the result obtained after deploying the target machine learning model, and the target machine learning model is a model obtained by quantizing the initial machine learning model based on the target quantization method; The method further includes: The process involves: determining a first weight corresponding to the quantization granularity in the weights of the network layer corresponding to the initial machine learning model; determining a second weight in the first weight based on the mean and variance of the first weight, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model, wherein the input of the target machine learning model includes at least one of the following: image and text.

2. The method according to claim 1, characterized in that, The target quantization method further includes: quantization type, and the method further includes: When the quantization type is the first quantization type, the activation layers and weights contained in the multi-layer network in the initial machine learning model are quantized according to the first quantization parameter corresponding to the first quantization type to obtain the target machine learning model. When the quantization type is the second quantization type, the weights are quantized according to the second quantization parameters corresponding to the second quantization type to obtain the target machine learning model.

3. The method according to claim 1, characterized in that, The target quantization method further includes: quantization granularity, which involves quantizing the weights of the network layer based on the target quantization method to obtain the target machine learning model, including: When the quantization granularity is tensor granularity, the weights are quantized according to a set of quantization parameters to obtain the target machine learning model; When the quantization granularity is channel granularity, the multiple channels contained in the weight are quantized according to multiple sets of first quantization parameters to obtain the target machine learning model, wherein the multiple sets of first quantization parameters correspond one-to-one with the multiple channels; When the quantization granularity is sub-channel granularity, the multiple sub-channels in different channels included in the weight are quantized according to multiple sets of second quantization parameters to obtain the target machine learning model, wherein the multiple sets of second quantization parameters correspond one-to-one with the multiple sub-channels.

4. The method according to claim 1, characterized in that, The weights are quantized based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model, including: The weights are quantized based on the initial quantization parameters to obtain a first quantization result. The weights are quantized based on the remaining quantization parameters to obtain a second quantization result; The target machine learning model is obtained by summing the first quantization result and the second quantization result.

5. The method according to claim 1, characterized in that, The method further includes: In response to a model deployment command applied to the user interface, multiple accelerator processors are displayed on the user interface. In response to a processor selection instruction applied to the plurality of accelerators, a target accelerator is displayed on the user interface, wherein the target accelerator is used to accelerate the target machine learning model.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the target quantization strategy corresponding to the target quantization method, wherein the target quantization strategy includes: quantization step size and quantization zero point; Identify the data to be quantized in the initial machine learning model; The ratio of the data to be quantized to the quantization step size is obtained to get the quantization ratio. The sum of the quantization ratio and the quantization zero point is obtained to obtain the quantized data in the target machine learning model.

7. A method for deploying a large model, characterized in that, include: In response to the quantization configuration command applied to the operation interface, the target quantization method corresponding to the initial large model is displayed on the operation interface. The initial large model is a pre-trained neural network model, and the target quantization method is used to characterize the quantization scale for quantizing the initial large model. The target quantization method includes: quantization granularity and quantization accuracy enhancement. In response to a model deployment command applied to the operation interface, the deployment result of the initial large model is displayed on the operation interface, wherein the deployment result is the result obtained after deploying the target large model, and the target large model is a model obtained by quantizing the initial large model based on the target quantization method; The method further includes: Determine the first weight corresponding to the quantization granularity in the weights of the network layer corresponding to the initial large model; determine the second weight in the first weight based on the mean and variance of the first weight, wherein the deviation of the second weight from the mean is greater than the variance; remove the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; quantize the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target large model, wherein the input of the target large model includes at least one of the following: image and text.

8. A method for processing large models, characterized in that, include: In response to input commands applied to the user interface, the user interface displays the data to be processed. In response to processing instructions applied to the operation interface, the processing result of the data to be processed is displayed on the operation interface. The processing result is obtained by accelerating the quantized data based on a target large model. The target large model is obtained by quantizing an initial large model based on a target quantization method. The quantized data is obtained by quantizing the data to be processed based on a target quantization strategy corresponding to the target quantization method. The initial large model is a pre-trained large model. The target quantization method is used to characterize the quantization scale for quantizing the initial large model. The target quantization method includes: quantization granularity and quantization accuracy enhancement. The method further includes: determining a first weight corresponding to the quantization granularity among the weights of the network layer corresponding to the initial large model; determining a second weight among the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target large model, wherein the input of the target large model includes at least one of the following: image and text.

9. The method according to claim 8, characterized in that, The initial large model includes: a multi-layer network layer, and the target large model is a model obtained by quantizing the weights of the network layers based on the target quantization method.

10. A method for deploying a machine learning model, characterized in that, include: Obtain an initial machine learning model, wherein the initial machine learning model is a pre-trained neural network model; A target quantization method is determined, wherein the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model, and the target quantization method includes: quantization granularity and quantization accuracy enhancement; The initial machine learning model is quantized based on the target quantization method to obtain the target machine learning model; The target machine learning model is deployed, and the target machine learning model is obtained by quantizing the weights of the network layers in the initial machine learning model based on the initial quantization parameters and the remaining quantization parameters corresponding to the quantization granularity. The remaining quantization parameters are obtained by removing the quantization parameters corresponding to the second weight from the initial quantization parameters. The second weight is obtained based on the first weight corresponding to the quantization granularity. The method further includes: determining a first weight corresponding to the quantization granularity among the weights of the network layer corresponding to the initial machine learning model; determining a second weight among the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model, wherein the input of the target machine learning model includes at least one of the following: image and text.

11. A method for deploying a machine learning model, characterized in that, include: The target quantization method is obtained by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the target quantization method, the target quantization method is used to characterize the quantization scale for quantizing the initial machine learning model, and the target quantization method includes: quantization granularity and quantization accuracy enhancement. The initial machine learning model is quantized based on the target quantization method to obtain the target machine learning model, wherein the initial machine learning model is a pre-trained neural network model; The target machine learning model is output by calling the second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter is the target machine learning model, the target machine learning model is obtained by quantizing the weights of the network layers in the initial machine learning model based on the initial quantization parameters and the remaining quantization parameters corresponding to the quantization granularity, the remaining quantization parameters are obtained by removing the quantization parameters corresponding to the second weight from the initial quantization parameters, and the second weight is obtained based on the first weight corresponding to the quantization granularity; The method further includes: determining a first weight corresponding to the quantization granularity among the weights of the network layer corresponding to the initial machine learning model; determining a second weight among the first weights based on the mean and variance of the first weights, wherein the deviation of the second weight from the mean is greater than the variance; removing the quantization parameter corresponding to the second weight from the initial quantization parameters corresponding to the quantization granularity to obtain the remaining quantization parameters; and quantizing the weights based on the initial quantization parameters and the remaining quantization parameters to obtain the target machine learning model, wherein the input of the target machine learning model includes at least one of the following: image and text.

12. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 11.

Images