A model deployment method and device, electronic equipment and storage medium

Abstract

The application discloses a model deployment method and device, electronic equipment and storage medium. First, each first quantization bit of each structure layer of a neural network model to be deployed is randomly generated, then each first quantization bit combination of the neural network model is obtained, each first candidate quantization model is obtained after quantization compression according to each first quantization bit combination, then each first candidate quantization model is tested on a target device to be deployed, a target quantization model is determined according to a first candidate quantization model meeting requirements in terms of prediction time and accuracy, and finally the target quantization model is deployed to the target device. Therefore, the application can realize a scheme of adaptively determining a target quantization model according to the performance of a target device, and can ensure that speed and accuracy are considered when the target device runs the target quantization model.

Description

Technical Field

[0001] This application relates to the field of model deployment technology, and in particular to a model deployment method, apparatus, electronic device and storage medium. Background Technology

[0002] After the neural network model is trained on a high-performance device, it needs to be quantized and compressed before being deployed to the target device.

[0003] Quantization compression of neural network models can generally improve the speed of model output, but it inevitably leads to a loss of model accuracy. Existing model deployment solutions typically quantize neural network models to a fixed number of bits, such as 16 bits or 8 bits.

[0004] The problem with existing solutions is that the quantized model cannot be adapted to the performance of the target device, making it impossible to balance speed and accuracy when the target device runs the model. Summary of the Invention

[0005] This application provides a model deployment method, apparatus, electronic device, and storage medium to solve the problem that the quantized model in the prior art cannot be adapted to the performance of the target device, making it impossible to balance speed and accuracy when the target device runs the model.

[0006] This application provides a model deployment method, the method comprising:

[0007] For each structural layer of the neural network model to be deployed, each first quantization bit of the structural layer is randomly generated; based on each first quantization bit of each structural layer, the combination of each first quantization bit of the neural network model is determined.

[0008] The neural network model is quantized and compressed according to each of the first quantization bit combinations to obtain each first candidate quantization model;

[0009] The prediction time and accuracy of each first candidate quantization model are obtained by testing the target device to be deployed.

[0010] Based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, a target quantization model is determined, and the target quantization model is deployed to the target device.

[0011] Furthermore, if no first candidate quantization model meets the requirements for both prediction time and accuracy, the method further includes:

[0012] Based on the prediction time and accuracy of each first candidate quantization model, a preset first number of first dominant quantization bit combinations are selected; based on each first dominant quantization bit combination, a first probability distribution of each quantization bit is determined.

[0013] Based on the first probability distribution of each quantized bit, each second quantized bit of each structural layer is randomly generated.

[0014] Based on each of the second quantization bits of each of the aforementioned structural layers, a quantization model is deployed.

[0015] Furthermore, the deployment of the quantization model based on each of the second quantization bits of each of the respective structural layers includes:

[0016] Based on each of the second quantization bits of each of the structural layers, the combination of each of the second quantization bits of the neural network model is determined;

[0017] The neural network model is quantized and compressed according to each of the second quantization bit combinations to obtain each second candidate quantization model;

[0018] The prediction time and accuracy of each second candidate quantization model are obtained by testing the target device to be deployed.

[0019] Based on the prediction time and accuracy of each second candidate quantization model, a preset second number of second dominant quantization bit combinations are selected; based on each second dominant quantization bit combination, the second probability distribution of each quantization bit is determined; until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

[0020] Further, determining the first probability distribution of each quantized bit based on each first dominant quantized bit combination includes:

[0021] For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination;

[0022] The first probability distribution of each quantized bit is determined based on the first sub-probability distribution of each quantized bit corresponding to each first dominant quantized bit combination.

[0023] Furthermore, the first sub-probability distribution includes at least one of a histogram sub-distribution and a Gaussian sub-distribution.

[0024] Further, if the first sub-probability distribution includes a histogram sub-distribution and a Gaussian sub-distribution, the method includes:

[0025] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each quantization bit corresponding to the first dominant quantization bit combination;

[0026] Based on the histogram sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the histogram distribution of each quantized bit is determined; based on the Gaussian sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the Gaussian distribution of each quantized bit is determined.

[0027] The first probability distribution of each quantized bit is determined by taking a weighted average of the histogram distribution and Gaussian distribution of each quantized bit.

[0028] Furthermore, for each structural layer of the neural network model to be deployed, each first quantization bit of the structural layer is randomly generated; based on each first quantization bit of each structural layer, the combination of each first quantization bit of the neural network model is determined, including:

[0029] For each structural layer of the neural network model to be deployed, randomly generate each first weight quantization bit and each first activation value quantization bit of the structural layer;

[0030] Based on the first weight quantization bits and the first activation value quantization bits of each of the structural layers, the combinations of the first quantization bits of the neural network model are determined.

[0031] Further, determining the first probability distribution of each quantized bit based on each combination of first dominant quantized bits, and randomly generating each second quantized bit of each structural layer based on the first probability distribution of each bit, includes:

[0032] Based on the combination of each first dominant quantized bit, determine the first probability distribution of each weighted quantized bit and the first probability distribution of each activated value quantized bit.

[0033] Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated.

[0034] Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated.

[0035] Further, determining the first probability distribution of each weighted quantized bit and the first probability distribution of each activated value quantized bit based on each first dominant quantized bit combination includes:

[0036] For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; and determine the first probability distribution of each weighted quantization bit based on the first sub-probability distribution of each of the first dominant quantization bit combinations.

[0037] For each of the first dominant quantization bit combinations, a first sub-probability distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination is determined; based on the first sub-probability distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, a first probability distribution of each activation value quantization bit is determined.

[0038] Furthermore, the method includes:

[0039] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination;

[0040] Based on the histogram sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each weighted quantization bit is determined; based on the Gaussian sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each weighted quantization bit is determined.

[0041] The first probability distribution of each weighted quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each weighted quantized bit.

[0042] Furthermore, the method includes:

[0043] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination;

[0044] Based on the histogram sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each activation value quantization bit is determined; based on the Gaussian sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each activation value quantization bit is determined.

[0045] The first probability distribution of each activation value quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each quantized bit.

[0046] Further, the step of randomly generating each second quantization bit of each structural layer according to the first probability distribution of each quantization bit includes:

[0047] Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated.

[0048] Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated.

[0049] On the other hand, this application provides a model deployment apparatus, the apparatus comprising:

[0050] The determination module is used to randomly generate each first quantization bit of each structural layer of the neural network model to be deployed; and determine the combination of each first quantization bit of the neural network model based on each first quantization bit of each structural layer.

[0051] The quantization module is used to quantize and compress the neural network model according to the respective combinations of first quantization bits to obtain each first candidate quantization model.

[0052] The testing module is used to test each of the first candidate quantization models through the target device to be deployed, and to obtain the prediction time and accuracy of each of the first candidate quantization models.

[0053] The deployment module is used to determine the target quantization model based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, and to deploy the target quantization model to the target device.

[0054] Furthermore, the determining module is also configured to select a preset first number of first dominant quantization bit combinations based on the prediction time and accuracy of each first candidate quantization model; determine a first probability distribution of each quantization bit based on each first dominant quantization bit combination; and randomly generate each second quantization bit of each structural layer based on the first probability distribution of each quantization bit.

[0055] The deployment module is also used to deploy a quantization model based on each of the second quantization bits of each of the structural layers.

[0056] Further, the deployment module is specifically configured to: determine the combinations of second quantization bits of the neural network model based on the respective second quantization bits of each structural layer; quantize and compress the neural network model according to the combinations of second quantization bits to obtain the second candidate quantization models; test the second candidate quantization models on the target device to be deployed to obtain the prediction time and accuracy of the second candidate quantization models; select a preset second number of second dominant quantization bit combinations based on the prediction time and accuracy of the second candidate quantization models; determine the second probability distribution of each quantization bit based on the combinations of second dominant quantization bits; until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

[0057] Furthermore, the determining module is also used to determine, for each first dominant quantization bit combination, a first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination; and to determine a first probability distribution of each quantization bit based on the first sub-probability distribution of each quantization bit corresponding to each first dominant quantization bit combination.

[0058] Further, the determining module is specifically configured to, for each first dominant quantization bit combination, determine the histogram sub-distribution and Gaussian sub-distribution of each quantization bit corresponding to the first dominant quantization bit combination; determine the histogram distribution of each quantization bit based on the histogram sub-distribution of each quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each quantization bit based on the Gaussian sub-distribution of each quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each quantization bit to determine the first probability distribution of each quantization bit.

[0059] Furthermore, the determining module is specifically used to randomly generate each first weight quantization bit and each first activation value quantization bit of each structural layer of the neural network model to be deployed; and to determine the combination of each first quantization bit of the neural network model based on each first weight quantization bit and each first activation value quantization bit of each structural layer.

[0060] Further, the determining module is specifically used to determine the first probability distribution of each weighted quantized bit and the first probability distribution of each activation value quantized bit based on each first dominant quantized bit combination; to randomly generate each second weighted quantized bit of each structural layer based on the first probability distribution of each weighted quantized bit; and to randomly generate each second activation value quantized bit of each structural layer based on the first probability distribution of each activation value quantized bit.

[0061] Further, the determining module is specifically configured to: determine, for each first dominant quantization bit combination, a first sub-probability distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; determine a first probability distribution of each weighted quantization bit based on the first sub-probability distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; determine, for each first dominant quantization bit combination, a first sub-probability distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination; and determine a first probability distribution of each activation value quantization bit based on the first sub-probability distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination.

[0062] Further, the determining module is specifically configured to, for each first dominant quantization bit combination, determine the histogram sub-distribution and Gaussian sub-distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; determine the histogram distribution of each weighted quantization bit based on the histogram sub-distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each weighted quantization bit based on the Gaussian sub-distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each weighted quantization bit to determine the first probability distribution of each weighted quantization bit.

[0063] Further, the determining module is specifically configured to, for each first dominant quantization bit combination, determine the histogram sub-distribution and Gaussian sub-distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination; determine the histogram distribution of each activation value quantization bit based on the histogram sub-distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each activation value quantization bit based on the Gaussian sub-distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each activation value quantization bit to determine the first probability distribution of each activation value quantization bit.

[0064] Furthermore, the determining module is specifically used to randomly generate each second weight quantization bit of each structural layer according to the first probability distribution of each weight quantization bit; and to randomly generate each second activation value quantization bit of each structural layer according to the first probability distribution of each activation value quantization bit.

[0065] In another aspect, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0066] Memory, used to store computer programs;

[0067] When a processor executes a program stored in memory, it implements the steps of the method described in any of the preceding statements.

[0068] In another aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in any of the above claims.

[0069] This application provides a model deployment method, apparatus, electronic device, and storage medium. The method includes: randomly generating each first quantization bit of each structural layer of a neural network model to be deployed; determining each combination of first quantization bits of the neural network model based on each first quantization bit of each structural layer; quantizing and compressing the neural network model according to each combination of first quantization bits to obtain each first candidate quantization model; testing each first candidate quantization model on a target device to be deployed to obtain the prediction time and accuracy of each first candidate quantization model; determining a target quantization model based on the first candidate quantization models whose prediction time and accuracy both meet the requirements; and deploying the target quantization model to the target device.

[0070] The above technical solution has the following advantages or beneficial effects:

[0071] This application first randomly generates each first quantization bit of each structural layer of the neural network model to be deployed. Then, it obtains the combination of each first quantization bit of the neural network model. After quantization and compression according to each combination of first quantization bits, it obtains each first candidate quantization model. Then, it tests each first candidate quantization model on the target device to be deployed. Based on the first candidate quantization models that meet the requirements for both prediction time and accuracy, the target quantization model is determined. Finally, the target quantization model is deployed to the target device. This application provides a scheme for generating each first candidate quantization model based on the combination of each first quantization bit, and testing each first candidate quantization model through the target device to be deployed. That is, it combines the perception of the target device to finally determine the target quantization model for deployment. Therefore, this application can realize a scheme for adaptively determining the target quantization model according to the performance of the target device, which can ensure that the target device balances speed and accuracy when running the target quantization model. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 A schematic diagram of the model deployment process provided in this application;

[0074] Figure 2 A schematic diagram of the target quantification model provided for this application;

[0075] Figure 3 A schematic diagram of the model deployment device provided in this application;

[0076] Figure 4 A schematic diagram of the electronic device structure provided in this application. Detailed Implementation

[0077] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0078] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0079] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0080] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0081] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0083] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0084] Figure 1 The model deployment process provided in this application is illustrated in the following diagram, which includes the following steps:

[0085] S101: For each structural layer of the neural network model to be deployed, randomly generate each first quantization bit of the structural layer; determine the combination of each first quantization bit of the neural network model based on each first quantization bit of each structural layer.

[0086] S102: The neural network model is quantized and compressed according to each of the first quantization bit combinations to obtain each first candidate quantization model.

[0087] S103: Test each of the first candidate quantization models using the target device to be deployed, and obtain the prediction time and accuracy of each of the first candidate quantization models.

[0088] S104: Based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, determine the target quantization model and deploy the target quantization model to the target device.

[0089] The model deployment method provided in this application is applied to electronic devices, such as tablets, PCs, and servers.

[0090] After the neural network model to be deployed has been trained, the electronic device randomly generates the first quantization bits for each structural layer of the neural network model. The range of quantization bits can be set, for example, greater than 0 bits and less than or equal to 16 bits. The first quantization bits of each structural layer are randomly generated within this range. For example, the first quantization bits of each structural layer may include 5 bits, 6 bits, 7 bits, 8 bits, 9 bits, 10 bits, etc.

[0091] Based on the individual first quantization bits of each structural layer, the combinations of first quantization bits for the neural network model are determined. This can be achieved by selecting one first quantization bit from each structural layer in a specific order to obtain a combination. For example, if the neural network model contains four structural layers, each containing five first quantization bits, the first first quantization bit can be selected from each of the four structural layers to form one combination; the second first quantization bit from the first structural layer and the first first quantization bit from each of the remaining structural layers can form another combination; the third first quantization bit from the first structural layer and the first first quantization bit from each of the remaining structural layers can form yet another combination. This process continues until all possible combinations of first quantization bits are obtained.

[0092] The neural network model is quantized and compressed according to each combination of first quantization bits to obtain each first candidate quantization model. Then, each first candidate quantization model is tested on the target device to be deployed to obtain the prediction time and accuracy of each first candidate quantization model.

[0093] Electronic devices are pre-set to meet required prediction time and accuracy, such as a prediction time of less than 3 seconds and an accuracy of more than 95%. Based on the first-choice quantization models that meet both the prediction time and accuracy requirements, a target quantization model is determined and deployed to the target device. If multiple first-choice quantization models meet both the prediction time and accuracy requirements, the target quantization model can be selected according to business needs. For example, if the business prioritizes model prediction efficiency, the first-choice quantization model with the shortest prediction time among those meeting the accuracy requirement is selected as the target quantization model. Conversely, if the business prioritizes model accuracy, the first-choice quantization model with the highest accuracy among those meeting the prediction time requirement is selected as the target quantization model.

[0094] This application provides a scheme for generating first candidate quantization models based on combinations of first quantization bits, and testing each first candidate quantization model on a target device to be deployed. In other words, the target quantization model is ultimately determined and deployed to the target device based on the target device's perception. Therefore, this application can achieve a scheme for adaptively determining the target quantization model based on the performance of the target device, ensuring that the target device balances speed and accuracy when running the target quantization model.

[0095] In this application, if there is no first candidate quantization model that meets both the prediction time and accuracy requirements, the method further includes:

[0096] Based on the prediction time and accuracy of each first candidate quantization model, a preset first number of first dominant quantization bit combinations are selected; based on each first dominant quantization bit combination, a first probability distribution of each quantization bit is determined.

[0097] Based on the first probability distribution of each quantized bit, each second quantized bit of each structural layer is randomly generated.

[0098] Based on each of the second quantization bits of each of the aforementioned structural layers, a quantization model is deployed.

[0099] In this application, if none of the first candidate quantization models meets the requirements for both prediction time and accuracy, then a preset first number of first dominant quantization bit combinations are selected based on the prediction time and accuracy of each first candidate quantization model. That is, a preset first number of first dominant quantization bit combinations are selected from each quantization bit combination according to the principle of short prediction time and high accuracy. The preset first number is, for example, 300, 500, etc., and this application does not limit this.

[0100] Based on each first-dominant quantization bit combination, determine the first probability distribution of each quantization bit. Optionally, determine the first probability distribution of each quantization bit based on the frequency of occurrence of each quantization bit in each first-dominant quantization bit combination.

[0101] Based on the first probability distribution of each quantization bit, the second quantization bits of each structural layer are randomly selected and generated again. Then, based on the second quantization bits of each structural layer, the quantization model is deployed.

[0102] Specifically, based on each of the second quantization bits of each of the structural layers, the combination of each of the second quantization bits of the neural network model is determined;

[0103] The neural network model is quantized and compressed according to each of the second quantization bit combinations to obtain each second candidate quantization model;

[0104] The prediction time and accuracy of each second candidate quantization model are obtained by testing the target device to be deployed.

[0105] Based on the prediction time and accuracy of each second candidate quantization model, a preset second number of second dominant quantization bit combinations are selected; based on each second dominant quantization bit combination, the second probability distribution of each quantization bit is determined; until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

[0106] In this application, the combinations of second quantization bits of the neural network model are first determined based on the respective second quantization bits of each structural layer. This process is similar to that of determining the combinations of first quantization bits and will not be described in detail here. Then, the neural network model is quantized and compressed according to each combination of second quantization bits to obtain each second candidate quantization model. Finally, each second candidate quantization model is tested using the target device to be deployed to obtain the prediction time and accuracy of each second candidate quantization model.

[0107] If a second candidate quantization model exists among the various second candidate quantization models that meets both the prediction time and accuracy requirements, then the target quantization model is determined based on this model. Otherwise, a preset second number of second dominant quantization bit combinations are selected based on the prediction time and accuracy of each second candidate quantization model; the preset second number and the preset first number can be the same or different. Similarly, the selection of second dominant quantization bit combinations is based on the principle of short prediction time and high accuracy. A second probability distribution for each quantization bit is determined based on each second dominant quantization bit combination; then, based on the second probability distribution of each quantization bit, each third quantization bit for each structural layer is randomly generated. Based on each third quantization bit for each structural layer, the above process is repeated until a candidate quantization model that meets both the prediction time and accuracy requirements is determined.

[0108] In this application, determining the first probability distribution of each quantized bit based on each first dominant quantized bit combination includes:

[0109] For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination;

[0110] The first probability distribution of each quantized bit is determined based on the first sub-probability distribution of each quantized bit corresponding to each first dominant quantized bit combination.

[0111] In this application, for each first dominant quantization bit combination, a first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination is determined based on the frequency of occurrence of each quantization bit in the first dominant quantization bit combination. Then, based on each first sub-probability distribution, a first probability distribution of each quantization bit is determined. Optionally, the first probability distribution is obtained by summing the first sub-probability distributions.

[0112] In this application, the first sub-probability distribution includes at least one of a histogram sub-distribution and a Gaussian sub-distribution.

[0113] If the first sub-probability distribution includes a histogram sub-distribution, then for each first dominant quantization bit combination, the histogram sub-distribution of each quantization bit corresponding to that first dominant quantization bit combination is determined as the first sub-probability distribution based on the frequency of occurrence of each quantization bit in that first dominant quantization bit combination. The process of determining the histogram sub-distribution can be that the ratio of the frequency of occurrence of a quantization bit to the total frequency is used as the probability value of that quantization bit in the histogram sub-distribution.

[0114] If the first sub-probability distribution includes a Gaussian sub-distribution, then for each first dominant quantization bit combination, the Gaussian sub-distribution of each quantization bit corresponding to that first dominant quantization bit combination is determined as the first sub-probability distribution based on the frequency of occurrence of each quantization bit in that first dominant quantization bit combination. The process of determining the Gaussian sub-distribution can be achieved by substituting the frequency of occurrence of each quantization bit into a Gaussian function, and then using the Gaussian function to obtain the Gaussian sub-distribution as the first sub-probability distribution.

[0115] If the first sub-probability distribution includes a histogram sub-distribution and a Gaussian sub-distribution, the method includes:

[0116] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each quantization bit corresponding to the first dominant quantization bit combination;

[0117] Based on the histogram sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the histogram distribution of each quantized bit is determined; based on the Gaussian sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the Gaussian distribution of each quantized bit is determined.

[0118] The first probability distribution of each quantized bit is determined by taking a weighted average of the histogram distribution and Gaussian distribution of each quantized bit.

[0119] Specifically, the histogram sub-distributions of each quantized bit can be summed to obtain the histogram distribution of each quantized bit; the Gaussian sub-distributions of each quantized bit can be summed to obtain the Gaussian distribution of each quantized bit. Weight values ​​for the histogram distribution and the Gaussian distribution are set respectively; optionally, the weight value of the Gaussian distribution is greater than the weight value of the histogram distribution. Then, based on the weight values ​​of the Gaussian distribution and the histogram distribution, a weighted average is performed on the histogram distribution and the Gaussian distribution of each quantized bit to determine the first probability distribution of each quantized bit.

[0120] In this application, the first quantization bit of each structural layer generally includes a weight quantization bit and an activation value quantization bit. The sum of the weight quantization bit and the activation value quantization bit can be used as the first quantization bit of the structural layer. The first quantization bit can be determined by the above scheme and then randomly allocated to the weight quantization bit and the activation value quantization bit, or allocated to the weight quantization bit and the activation value quantization bit according to a certain ratio.

[0121] Ideally, the precise weight quantization bits and activation value quantization bits can be determined separately. Details are as follows.

[0122] For each structural layer of the neural network model to be deployed, each first quantization bit of the structural layer is randomly generated; based on each first quantization bit of each structural layer, the combination of each first quantization bit of the neural network model is determined as follows:

[0123] For each structural layer of the neural network model to be deployed, randomly generate each first weight quantization bit and each first activation value quantization bit of the structural layer;

[0124] Based on the first weight quantization bits and the first activation value quantization bits of each of the structural layers, the combinations of the first quantization bits of the neural network model are determined.

[0125] For example, each first quantization bit of each structural layer includes (3 bits, 5 bits), (4 bits, 6 bits), (5 bits, 6 bits), (6 bits, 7 bits), etc., where the first item in parentheses represents the weight quantization bit and the second item represents the activation value quantization bit.

[0126] Based on the first weight quantization bits and the first activation value quantization bits of each structural layer, the combination of first quantization bits of the neural network model is determined. This application refines the first quantization bits into first weight quantization bits and first activation value quantization bits.

[0127] The step of determining a first probability distribution for each quantized bit based on each combination of first dominant quantized bits, and randomly generating each second quantized bit for each structural layer based on the first probability distribution of each bit, includes:

[0128] Based on the combination of each first dominant quantized bit, determine the first probability distribution of each weighted quantized bit and the first probability distribution of each activated value quantized bit.

[0129] Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated.

[0130] Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated.

[0131] In this application, the first probability distribution of each weighted quantized bit and the first probability distribution of each activation value quantized bit are determined based on each combination of first dominant quantized bits. That is, the first probability distribution of each weighted quantized bit is determined based on the frequency of occurrence of each weighted quantized bit. Similarly, the first probability distribution of each activation value quantized bit is determined based on the frequency of occurrence of each activation value quantized bit.

[0132] Based on the first probability distribution of each weight quantization bit, each second weight quantization bit for each structural layer is randomly generated. This process is similar to the process of generating quantization bits, except that it generates weight quantization bits based on the first probability distribution of each weight quantization bit, so this process will not be described in detail.

[0133] Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated. This process is similar to the process of generating quantized bits, except that it generates activation value quantized bits based on the first probability distribution of the quantized bits of each activation value, so this process will not be described in detail.

[0134] In this application, determining the first probability distribution of each weighted quantized bit and the first probability distribution of each activated value quantized bit based on each first dominant quantized bit combination includes:

[0135] For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; and determine the first probability distribution of each weighted quantization bit based on the first sub-probability distribution of each of the first dominant quantization bit combinations.

[0136] For each of the first dominant quantization bit combinations, a first sub-probability distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination is determined; based on the first sub-probability distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, a first probability distribution of each activation value quantization bit is determined.

[0137] In this application, after determining the first sub-probability distribution of each weighted quantized bit, the first probability distribution of each weighted quantized bit is obtained by summing the first sub-probability distributions. Similarly, after determining the first sub-probability distribution of each activation value quantized bit, the first probability distribution of each activation value quantized bit is obtained by summing the first sub-probability distributions.

[0138] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination;

[0139] Based on the histogram sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each weighted quantization bit is determined; based on the Gaussian sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each weighted quantization bit is determined.

[0140] The first probability distribution of each weighted quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each weighted quantized bit.

[0141] For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination;

[0142] Based on the histogram sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each activation value quantization bit is determined; based on the Gaussian sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each activation value quantization bit is determined.

[0143] The first probability distribution of each activation value quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each quantized bit.

[0144] The process of determining the first probability distribution of each weight quantization bit and the first probability distribution of the activation value quantization bit is similar to the process of determining the first probability distribution of each first quantization bit.

[0145] In this application, the step of randomly generating each second quantization bit of each structural layer according to the first probability distribution of each quantization bit includes:

[0146] Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated.

[0147] Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated.

[0148] Figure 2 A schematic diagram of the target quantification model provided in this application, such as Figure 2 As shown, the third convolutional layer (Layer 3) of the target quantization model has 3 bits for weight quantization and 5 bits for activation quantization; the fourth convolutional layer (Layer 4) has 6 bits for weight quantization and 7 bits for activation quantization; the fifth convolutional layer (Layer 5) has 4 bits for weight quantization and 6 bits for activation quantization; and the sixth convolutional layer (Layer 6) has 5 bits for weight quantization and 6 bits for activation quantization.

[0149] This application first clarifies the deployment equipment, which can be the Hisilicon 3559a chip. It also clarifies the equipment performance testing methods, detailing performance testing for each network layer, including metrics such as bandwidth and speed. Next, it clarifies the algorithm model, which can be the YOLO detection model. Training data, test data, and quantization data are specified, along with model evaluation criteria. In this application, the evaluation metrics are recall, precision, and inference time, representing the ability to detect targets, the accuracy of target detection, and the model's performance, respectively. Training data is used to train the model, while test data is used to measure the model's performance. Finally, it identifies the model to be quantized. This model is trained and tested on the training set. Based on this trained model, device-aware mixed-precision quantization is performed. Mixed-precision quantization differs from mixed-precision training, which refers to using FP16 instead of traditional FP32 during model training to achieve lower memory usage and faster computation. In contrast, the mixed-precision quantization in this application refers to designing a strategy to rationally allocate the quantization bit width of the weights and activation values ​​of each layer of the model.

[0150] The Estimation of Distribution Algorithm (EDA) is essentially a novel genetic algorithm based on a probabilistic model. The combination of genetic algorithms and statistical learning represents another typical implementation mode of natural computation. It guides the algorithm's next search step by establishing a probabilistic model on the currently found set of better individuals, and samples new individuals from the probability distribution function of the obtained better solutions.

[0151] A probability model is used to describe the spatial distribution of candidate solutions. Statistical learning is used to establish a probability model describing the solution distribution from a macroscopic perspective of the population. Then, random sampling is performed on the probability model to generate a new population. This process is repeated to achieve population evolution until the termination condition is met.

[0152] In this application, the quantization bit distribution of each layer of the model is the value that EDA needs to learn. The EDA algorithm flow is as follows:

[0153] Step 1: Randomly generate M individuals as the initial population, which in this application is a combination of M quantized bits;

[0154] Step 2: Calculate the individual fitness of the Lth generation population, i.e. test the prediction time and accuracy, and determine whether the termination condition is met. If it is met, terminate the loop; if not, continue. In this application, the quantized model is tested on the target device to see if it can meet the standards. Therefore, the target device can directly perceive the quality of the quantized model and participate in it.

[0155] Step 3: Select the top N (N≤M) dominant individuals from the population based on their fitness values ​​to form the dominant subpopulation of the L+1 generation, thus forming a new bit combination in this application;

[0156] Step 4: Update the probability model based on the dominant subpopulation.

[0157] The probabilistic model is expressed as follows: When P(x) i When P(x) is the histogram subdistribution of population i, P(x) is the histogram distribution of each quantized bit; when P(x) is the histogram subdistribution of population i, P(x) is the histogram distribution of each quantized bit. i When is the Gaussian subdistribution of population i, P(x) is the Gaussian distribution of each quantized bit. n is the population size.

[0158] The core of the EDA algorithm lies in the probability model. In this application, most of the neural network models have been trained with large-scale data, and the data distribution mostly shows a normal distribution trend. Therefore, this application has improved EDA. In the probability search, a rough global search is first performed using a histogram distribution, so that the quantization bit information of each layer will refer to the content of the global variable. Then, a precise search is performed using a Gaussian distribution, which is a normal distribution. This can greatly improve the search efficiency and shorten the optimization time of EDA.

[0159] Step 5: Randomly sample the probability model to generate a new population; in this application, this is a combination of quantized bits. Determine if the updated population contains any that meet the requirements; if not, return to step 3.

[0160] This application uses EDA to construct a quantization model based on the model. Through continuous iteration and optimization of EDA, each layer is adapted to its optimal quantization bits, and finally a high-performance quantization model is obtained.

[0161] This application proposes a system for deploying a model using adaptive sensing hardware with mixed-precision quantization. By employing mixed-precision quantization of a trained model and incorporating hardware devices into the system through a distribution estimation genetic algorithm, the entire quantization process is made aware of on-device and participates in the model's quantization process online. Simultaneously, based on the characteristics of the task, the distribution estimation genetic algorithm is improved to enable faster convergence and output a quantization model with optimal overall performance.

[0162] Figure 3 The schematic diagram of the model deployment device provided in this application includes:

[0163] The determining module 31 is used to randomly generate each first quantization bit of each structural layer of the neural network model to be deployed; and to determine the combination of each first quantization bit of the neural network model based on each first quantization bit of each structural layer.

[0164] Quantization module 32 is used to quantize and compress the neural network model according to the respective combinations of first quantization bits to obtain respective first candidate quantization models;

[0165] The testing module 33 is used to test each of the first candidate quantization models through the target device to be deployed, and to obtain the prediction time and accuracy of each of the first candidate quantization models.

[0166] The deployment module 34 is used to determine the target quantization model based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, and to deploy the target quantization model to the target device.

[0167] The determining module 31 is further configured to select a preset first number of first dominant quantization bit combinations based on the prediction time and accuracy of each first candidate quantization model; determine a first probability distribution of each quantization bit based on each first dominant quantization bit combination; and randomly generate each second quantization bit of each structural layer based on the first probability distribution of each quantization bit.

[0168] The deployment module 34 is also used to deploy a quantization model based on each of the second quantization bits of each of the structural layers.

[0169] The deployment module 34 is specifically used to determine the combinations of second quantization bits of the neural network model based on the respective second quantization bits of each structural layer; to quantize and compress the neural network model according to the combinations of second quantization bits to obtain the second candidate quantization models; to test the second candidate quantization models through the target device to be deployed to obtain the prediction time and accuracy of the second candidate quantization models; to select a preset second number of second dominant quantization bit combinations based on the prediction time and accuracy of the second candidate quantization models; to determine the second probability distribution of each quantization bit based on the combinations of second dominant quantization bits; until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

[0170] The determining module 31 is further configured to, for each first dominant quantization bit combination, determine the first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination; and determine the first probability distribution of each quantization bit according to the first sub-probability distribution of each quantization bit corresponding to each first dominant quantization bit combination.

[0171] The determining module 31 is specifically configured to, for each first dominant quantization bit combination, determine the histogram sub-distribution and Gaussian sub-distribution of each quantization bit corresponding to the first dominant quantization bit combination; determine the histogram distribution of each quantization bit based on the histogram sub-distribution of each quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each quantization bit based on the Gaussian sub-distribution of each quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each quantization bit to determine the first probability distribution of each quantization bit.

[0172] The determining module 31 is specifically used to randomly generate each first weight quantization bit and each first activation value quantization bit of each structural layer of the neural network model to be deployed; and to determine the combination of each first quantization bit of the neural network model based on each first weight quantization bit and each first activation value quantization bit of each structural layer.

[0173] The determining module 31 is specifically used to determine the first probability distribution of each weighted quantized bit and the first probability distribution of each activation value quantized bit according to each first dominant quantized bit combination; to randomly generate each second weighted quantized bit of each structural layer according to the first probability distribution of each weighted quantized bit; and to randomly generate each second activation value quantized bit of each structural layer according to the first probability distribution of each activation value quantized bit.

[0174] The determining module 31 is specifically configured to: determine, for each first dominant quantization bit combination, a first sub-probability distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; determine a first probability distribution of each weighted quantization bit based on the first sub-probability distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; determine a first sub-probability distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; and determine a first probability distribution of each activation value quantization bit based on the first sub-probability distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination.

[0175] The determining module 31 is specifically configured to, for each first dominant quantization bit combination, determine the histogram sub-distribution and Gaussian sub-distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; determine the histogram distribution of each weighted quantization bit based on the histogram sub-distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each weighted quantization bit based on the Gaussian sub-distribution of each weighted quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each weighted quantization bit to determine the first probability distribution of each weighted quantization bit.

[0176] The determining module 31 is specifically used to: determine the histogram sub-distribution and Gaussian sub-distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; determine the histogram distribution of each activation value quantization bit based on the histogram sub-distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; determine the Gaussian distribution of each activation value quantization bit based on the Gaussian sub-distribution of each activation value quantization bit corresponding to each first dominant quantization bit combination; and perform a weighted average of the histogram distribution and Gaussian distribution of each activation value quantization bit to determine the first probability distribution of each activation value quantization bit.

[0177] The determining module 31 is specifically used to randomly generate each second weight quantization bit of each structural layer according to the first probability distribution of each weight quantization bit; and to randomly generate each second activation value quantization bit of each structural layer according to the first probability distribution of each activation value quantization bit.

[0178] This application also provides an electronic device, such as Figure 4 As shown, it includes: processor 301, communication interface 302, memory 303 and communication bus 304, wherein processor 301, communication interface 302 and memory 303 communicate with each other through communication bus 304;

[0179] The memory 303 stores a computer program, which, when executed by the processor 301, causes the processor 301 to perform any of the above method steps.

[0180] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0181] Communication interface 302 is used for communication between the above-mentioned electronic device and other devices.

[0182] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0183] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0184] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform any of the above method steps.

[0185] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0186] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A model deployment method, characterized in that, The method includes: For each structural layer of the neural network model to be deployed, each first quantization bit of the structural layer is randomly generated; based on each first quantization bit of each structural layer, the combination of each first quantization bit of the neural network model is determined. The neural network model is quantized and compressed according to each of the first quantization bit combinations to obtain each first candidate quantization model; The prediction time and accuracy of each first candidate quantization model are obtained by testing the target device to be deployed. Based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, a target quantization model is determined, and the target quantization model is deployed to the target device. If no first candidate quantization model meets the requirements for both prediction time and accuracy, the method further includes: Based on the prediction time and accuracy of each first candidate quantization model, a preset first number of first dominant quantization bit combinations are selected; based on each first dominant quantization bit combination, a first probability distribution of each quantization bit is determined. Based on the first probability distribution of each quantized bit, each second quantized bit of each structural layer is randomly generated. Based on each of the second quantization bits of each of the aforementioned structural layers, a quantization model is deployed; The deployment of the quantization model based on each of the second quantization bits of each of the respective structural layers includes: Based on each of the second quantization bits of each of the structural layers, the combination of each of the second quantization bits of the neural network model is determined; The neural network model is quantized and compressed according to each of the second quantization bit combinations to obtain each second candidate quantization model; The prediction time and accuracy of each second candidate quantization model are obtained by testing the target device to be deployed. Based on the prediction time and accuracy of each second candidate quantization model, a preset second number of second dominant quantization bit combinations are selected; based on each second dominant quantization bit combination, the second probability distribution of each quantization bit is determined; until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

2. The method as described in claim 1, characterized in that, The step of determining the first probability distribution of each quantized bit based on each combination of first dominant quantized bits includes: For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each quantization bit corresponding to the first dominant quantization bit combination; The first probability distribution of each quantized bit is determined based on the first sub-probability distribution of each quantized bit corresponding to each first dominant quantized bit combination.

3. The method as described in claim 2, characterized in that, The first sub-probability distribution includes at least one of the histogram sub-distribution and the Gaussian sub-distribution.

4. The method as described in claim 3, characterized in that, If the first sub-probability distribution includes a histogram sub-distribution and a Gaussian sub-distribution, the method includes: For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each quantization bit corresponding to the first dominant quantization bit combination; Based on the histogram sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the histogram distribution of each quantized bit is determined; based on the Gaussian sub-distribution of each quantized bit corresponding to each of the first dominant quantized bit combinations, the Gaussian distribution of each quantized bit is determined. The first probability distribution of each quantized bit is determined by weighted averaging of the histogram distribution and Gaussian distribution of each quantized bit.

5. The method as described in claim 3, characterized in that, For each structural layer of the neural network model to be deployed, each first quantization bit of the structural layer is randomly generated; based on each first quantization bit of each structural layer, the combination of each first quantization bit of the neural network model is determined as follows: For each structural layer of the neural network model to be deployed, randomly generate each first weight quantization bit and each first activation value quantization bit of the structural layer; Based on the first weight quantization bits and the first activation value quantization bits of each of the structural layers, the combinations of the first quantization bits of the neural network model are determined.

6. The method as described in claim 3, characterized in that, The step of determining a first probability distribution for each quantized bit based on each combination of first dominant quantized bits, and randomly generating each second quantized bit for each structural layer based on the first probability distribution of each quantized bit, includes: Based on the combination of each first dominant quantized bit, determine the first probability distribution of each weighted quantized bit and the first probability distribution of each activated value quantized bit. Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated. Based on the first probability distribution of the quantized bits of each activation value, the second activation value quantized bits of each structural layer are randomly generated.

7. The method as described in claim 6, characterized in that, The step of determining the first probability distribution of each weighted quantized bit and the first probability distribution of each activation value quantized bit based on each combination of first dominant quantized bits includes: For each of the first dominant quantization bit combinations, determine the first sub-probability distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; and determine the first probability distribution of each weighted quantization bit based on the first sub-probability distribution of each of the first dominant quantization bit combinations. For each of the first dominant quantization bit combinations, a first sub-probability distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination is determined; based on the first sub-probability distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, a first probability distribution of each activation value quantization bit is determined.

8. The method as described in claim 3, characterized in that, The method includes: For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each weighted quantization bit corresponding to the first dominant quantization bit combination; Based on the histogram sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each weighted quantization bit is determined; based on the Gaussian sub-distribution of each weighted quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each weighted quantization bit is determined. The first probability distribution of each weighted quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each weighted quantized bit.

9. The method as described in claim 3, characterized in that, The method includes: For each of the first dominant quantization bit combinations, determine the histogram sub-distribution and Gaussian sub-distribution of each activation value quantization bit corresponding to the first dominant quantization bit combination; Based on the histogram sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the histogram distribution of each activation value quantization bit is determined; based on the Gaussian sub-distribution of each activation value quantization bit corresponding to each of the first dominant quantization bit combinations, the Gaussian distribution of each activation value quantization bit is determined. The first probability distribution of each activation value quantized bit is determined by weighting the histogram distribution and Gaussian distribution of each quantized bit.

10. The method as described in claim 3, characterized in that, The step of randomly generating each second quantization bit of each structural layer according to the first probability distribution of each quantization bit includes: Based on the first probability distribution of each weight quantization bit, each second weight quantization bit of each structural layer is randomly generated. Based on the first probability distribution of each activation value quantized bit, each second activation value quantized bit of each structural layer is randomly generated.

11. A model deployment device, characterized in that, The device includes: The determination module is used to randomly generate each first quantization bit of each structural layer of the neural network model to be deployed; and to determine the combination of each first quantization bit of the neural network model based on each first quantization bit of each structural layer. The quantization module is used to quantize and compress the neural network model according to the respective combinations of first quantization bits to obtain each first candidate quantization model. The testing module is used to test each of the first candidate quantization models through the target device to be deployed, and to obtain the prediction time and accuracy of each of the first candidate quantization models. The deployment module is used to determine the target quantization model based on the first candidate quantization models whose prediction time and accuracy both meet the requirements, and to deploy the target quantization model to the target device. The determining module is further configured to select a preset first number of first dominant quantization bit combinations based on the prediction time and accuracy of each first candidate quantization model; determine a first probability distribution of each quantization bit based on each first dominant quantization bit combination; and randomly generate each second quantization bit of each structural layer based on the first probability distribution of each quantization bit. The deployment module is also used to deploy a quantization model based on each of the second quantization bits of each of the structural layers. The deployment module is specifically configured to: determine the combinations of second quantization bits of the neural network model based on the respective second quantization bits of each structural layer; quantize and compress the neural network model according to the combinations of second quantization bits to obtain the second candidate quantization models; test the second candidate quantization models on the target device to be deployed to obtain the prediction time and accuracy of the second candidate quantization models; select a preset second number of second dominant quantization bit combinations based on the prediction time and accuracy of the second candidate quantization models; determine the second probability distribution of each quantization bit based on the combinations of second dominant quantization bits; and so on, until a candidate quantization model that meets the requirements for both prediction time and accuracy is determined.

12. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images