Model reasoning method, model quantification method and electronic equipment
By performing block quantization and weight block mapping on large models, the storage space and hardware requirements of large models when deployed on edge devices are solved, achieving more efficient model quantization compression and accuracy.
Patent Information
- Application Number
- CN202511075766.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-11
AI Technical Summary
Existing technologies require a large amount of storage space and have high hardware requirements when deploying large models to edge devices. Quantization technology has failed to effectively reduce the amount of storage space required, which affects the device and user experience.
By using the block quantization approach, the network layer weight blocks of a large model are segmented and quantized to determine the mapping relationship between the target weight block and its matching similar weight blocks, generating model quantization data and sharing the quantization weights of the target weight block to reduce storage space usage.
This approach achieves a further reduction in storage space and hardware requirements without compromising model accuracy, thereby improving model deployment efficiency and accuracy.
Smart Images
Figure CN120930694A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a model reasoning method, a model quantization method, and an electronic device. Background Technology
[0002] With the continuous enhancement of the capabilities of large-scale AI (Artificial Intelligence) models, AI models are being used more and more widely. In practical applications, considering issues such as network latency and data privacy, it is often necessary to deploy models on edge devices such as mobile phones and computers.
[0003] Because large AI models typically have an extremely large number of parameters, they are characterized by high storage space consumption and demanding hardware requirements. Therefore, when deploying them to edge devices, low-bit quantization is often required to reduce storage space consumption and lower hardware requirements. While some quantization techniques have reduced storage space consumption to some extent, the overall storage space usage remains relatively large, significantly impacting device performance and user experience. Therefore, how to further reduce storage space consumption and lower hardware requirements during model quantization has become a pressing technical problem to be solved in this field. Summary of the Invention
[0004] Therefore, this application discloses the following technical solution:
[0005] A model reasoning method, comprising:
[0006] Obtain the tasks to be processed and the model quantization data;
[0007] The model quantization data includes: at least one first quantized weight obtained by quantizing at least one first weight block corresponding to the first network layer in the first model; mapping relationship information between the target second weight block in at least one second weight block corresponding to the second network layer in the first model and the target first weight block in each first weight block; and the quantized weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions.
[0008] Based on the mapping relationship information, at least one of the first quantization weights is determined to be a target first quantization weight that satisfies the similarity condition with the target second quantization weight corresponding to the target second weight block;
[0009] The second network layer performs inference on the task to be processed based at least on the target first quantization weight to generate the corresponding task processing result.
[0010] Optionally, the weights corresponding to the first network layer are more important than the weights corresponding to the second network layer, and the first network layer is located before the second network layer in the first model;
[0011] The first weight block and the second weight block are blocks obtained by dividing the weights in the corresponding network layers.
[0012] Optionally, the task to be processed is inferred in the second network layer based at least on the target first quantization weights, including:
[0013] In the second network layer, inference is performed on the task to be processed based on the target first quantization weight and the non-target second quantization weight;
[0014] The model quantization data further includes the non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each of the second weight blocks.
[0015] The method further includes:
[0016] The task to be processed is inferred in the first network layer based on at least one of the first quantization weights.
[0017] Optionally, the model quantization data further includes quantization parameters corresponding to the first weight block, the target second weight block, and the non-target second weight block, respectively;
[0018] The method further includes at least some of the following:
[0019] Based on the first quantization parameters corresponding to each first weight block, the first quantization weights corresponding to each first weight block are dequantized to obtain the first dequantization weights.
[0020] Based on the second quantization parameter corresponding to the non-target second weight block, the non-target second quantization weight corresponding to the non-target second weight block is dequantized to obtain the second dequantization weight.
[0021] Based on the third quantization parameter corresponding to the second weight block of the target, the first quantization weight of the target is dequantized to obtain the third dequantization weight.
[0022] Optionally, the task to be processed is inferred in the second network layer based at least on the target first quantization weights, including:
[0023] The task to be processed is inferred in the second network layer based on the third inverse quantization weight;
[0024] Alternatively, the task to be processed can be inferred in the second network layer based on the second dequantization weight and the third dequantization weight.
[0025] Optionally, inference of the task to be processed in the first network layer based on at least one of the first quantization weights includes:
[0026] The task to be processed is inferred in the first network layer based on the first dequantization weight.
[0027] A model quantization method, comprising:
[0028] Obtain block quantization data corresponding to the first model; the block quantization data includes: at least one quantized weight obtained by quantizing at least one weight block corresponding to the network layer in the first model, wherein the weight block corresponding to the network layer is a block obtained by dividing the weight corresponding to the network layer;
[0029] Determine the target second weight block and the target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer.
[0030] Generate mapping information between the second weight block of the target and the first weight block of the target to obtain model quantization data;
[0031] The model quantization data includes: a first quantization weight and the mapping relationship information; the quantization weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions; the first quantization weight is the quantization weight obtained by quantizing at least one first weight block corresponding to the first network layer.
[0032] Optionally, the model quantization data further includes: non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each second weight block corresponding to the second network layer.
[0033] Optionally, the weights corresponding to the first network layer are more important than the weights corresponding to the second network layer.
[0034] Optionally, the model quantization data further includes: quantization parameters corresponding to each weight block of the network layer in the first model; the method further includes:
[0035] The quantization parameters contained in the model quantization data are adjusted.
[0036] Optionally, the quantization parameters included in the model quantization data may be adjusted, including:
[0037] The quantization parameter is adjusted based on the loss function and optimization objective through a quantization-aware training method until the termination condition is met.
[0038] The loss function is used to characterize the difference in weight distribution between the second model and the first model. The second model is a model obtained by performing model dequantization processing based on the model quantization data. The optimization objective is to minimize the value of the loss function.
[0039] An electronic device, comprising:
[0040] Memory, used to store at least one set of computer instructions;
[0041] A processor is configured to implement any of the model inference methods or model quantization methods provided above by executing the instruction set stored in the memory.
[0042] A model reasoning device, comprising:
[0043] The second acquisition module is used to obtain the tasks to be processed and the model quantization data;
[0044] The model quantization data includes: at least one first quantized weight obtained by quantizing at least one first weight block corresponding to the first network layer in the first model; mapping relationship information between the target second weight block in at least one second weight block corresponding to the second network layer in the first model and the target first weight block in each first weight block; and the quantized weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions.
[0045] The second determining module is used to determine, based on the mapping relationship information, at least one of the first quantization weights, a target second quantization weight corresponding to the target second weight block that satisfies the similarity condition;
[0046] The inference module is used to perform inference on the task to be processed in the second network layer at least based on the target first quantization weight, so as to generate the corresponding task processing result.
[0047] A model quantization device, comprising:
[0048] The first acquisition module is used to obtain block quantization data corresponding to the first model; the block quantization data includes: at least one quantized weight obtained by quantizing at least one weight block corresponding to the network layer in the first model, wherein the weight block corresponding to the network layer is a block obtained by dividing the weight corresponding to the network layer;
[0049] The first determining module is used to determine the target second weight block and the target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer.
[0050] The generation module is used to generate mapping relationship information between the target second weight block and the target first weight block to obtain model quantization data;
[0051] The model quantization data includes: a first quantization weight, a non-target second quantization weight, and the mapping relationship information; the quantization weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions; the first quantization weight is the quantization weight obtained by quantizing at least one first weight block corresponding to the first network layer, and the non-target second quantization weight is the quantization weight obtained by quantizing the non-target second weight blocks in each second weight block corresponding to the second network layer.
[0052] A storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement any of the model quantization methods or model inference methods provided above. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0054] Figure 1 This is a flowchart illustrating one of the model quantization methods provided in this application;
[0055] Figure 2 This is another flowchart illustrating the model quantization method provided in this application;
[0056] Figure 3 This is a flowchart illustrating the model inference method provided in this application;
[0057] Figure 4 This is a structural diagram of the model quantization device provided in this application;
[0058] Figure 5 This is a structural diagram of the model inference device provided in this application;
[0059] Figure 6 This is a structural diagram of the electronic device provided in this application. Detailed Implementation
[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0061] This application provides a model inference method, a model quantization method, and an electronic device, applicable to, but not limited to, scenarios where models are deployed and run on edge devices such as mobile phones and computers, or deployed in the cloud and run on edge devices. These methods are used to reduce the storage space occupied by AI models and lower hardware requirements for different scenarios.
[0062] The model inference method and model quantization method can be applied, but are not limited to, to electronic devices in a variety of general or special computing environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.
[0063] See Figure 1 The flowchart shown illustrates the model quantization method provided in this application embodiment, which may include the following steps 101 to 103, which are described in detail below.
[0064] Step 101: Obtain the block quantization data corresponding to the first model.
[0065] The block quantization data includes at least one quantized weight obtained by quantizing at least one weight block corresponding to a network layer in the first model. Optionally, in this implementation, each weight block corresponding to each network layer in the first model can be quantized. In this embodiment, the block quantization data includes each quantized weight obtained by quantizing each weight block corresponding to each network layer in the first model.
[0066] The weight block corresponding to the network layer is a block obtained by dividing the weights corresponding to the network layer. For example, it can be a block obtained by dividing the weight matrix corresponding to the network layer based on a set block size.
[0067] The first model can be an AI model, such as a large model or a small model. The large model can be a general large model, such as a general large language model (LLM), a visual large model, a multimodal large model, etc., or it can be an expert large model obtained by fine-tuning a general large model based on requirements. This application embodiment does not limit this.
[0068] Large models learn the features and patterns of input information by training on a large amount of diverse data. They typically have hundreds of millions to hundreds of billions of model parameters (such as the weights corresponding to the model network layers, used to control the model behavior). They can capture complex relationships and patterns in the input information. At the same time, in scenarios where large models are deployed and run on edge devices, they will bring problems such as large storage space consumption and high hardware requirements. This application uses the solution to this problem as an example to illustrate the problem. Moreover, this application has more obvious effects and advantages for the case where the first model is a large model (compared to a small model).
[0069] To address the aforementioned technical issues, this application quantizes the first model based on the idea of block quantization. Specifically, the weights (such as weight matrices) corresponding to each network layer of the first model can be first divided to obtain weight blocks corresponding to each network layer. Then, at least one weight block (such as all weight blocks corresponding to each network layer) in the first model can be further quantized to obtain the block quantized data.
[0070] In this step, the block quantization data can be obtained by performing block quantization processing on the first model in real time, or the block quantization data generated and saved by performing block quantization processing on the first model in advance can be read.
[0071] Step 102: Determine the target second weight block and the target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer.
[0072] The quantization weights corresponding to the second target weight block and the first target weight block respectively satisfy similarity conditions.
[0073] The similarity condition can be, but is not limited to, being set to the similarity of the quantized weights corresponding to different weight blocks reaching a similarity threshold.
[0074] Optionally, the weights corresponding to the first network layer are more important than the weights corresponding to the second network layer.
[0075] Before determining the matching target second weight block and target first weight block, the first network layer and the second network layer can be determined based on the importance of the weights corresponding to the network layers in the first model. For example, the network layers in the first model can be arranged in descending order according to the importance of their corresponding weights, and the first N% of the network layers in the resulting descending sequence can be determined as the first network layer, and the remaining network layers can be determined as the second network layer.
[0076] Optionally, the importance of the weights corresponding to each network layer in the first model can be determined using the signal-to-noise ratio (SNR) method. The higher the SNR of a network layer, the more important its corresponding weights are.
[0077] In practical applications, the earlier a network layer appears in a model, the more important its corresponding weights usually are. Based on this, the first network layer is located before the second network layer in the first model. To leverage this characteristic, in practice, the first N% of the network layers in the first model can be directly designated as the first network layer, and the remaining network layers as the second network layer. The value of N can be determined based on experience or experiments.
[0078] After determining the first network layer and the second network layer, the similarity between the second weight block corresponding to the second network layer and the first weight block corresponding to the first network layer after quantization processing can be further determined. The second weight block and the first weight block whose similarity of the corresponding quantized weight reaches the similarity threshold are determined as the matching target second weight block and target first weight block.
[0079] Step 103: Generate the mapping relationship information between the second target weight block and the first target weight block to obtain model quantization data.
[0080] Based on the determination of the matching target second weight block and target first weight block, step 103 further generates mapping relationship information between the target second weight block and the target first weight block. This mapping relationship information may include, but is not limited to, the correspondence between the position information of the target second weight block and the position information of the target first weight block.
[0081] Optionally, the location information of the weight block may include the layer number of the network layer to which the weight block corresponds, and the block number of the weight block in each weight block in the corresponding network layer.
[0082] After generating the mapping relationship information between the target second weight block and the target first weight block, the model quantization data corresponding to the first model can be obtained based on the mapping relationship information. This model quantization data is the quantization result corresponding to the first model.
[0083] The model quantization data includes a first quantization weight, a non-target second quantization weight, and the mapping relationship information.
[0084] The first quantization weight is the quantization weight obtained by quantizing at least one first weight block corresponding to the first network layer, and the non-target second quantization weight is the quantization weight obtained by quantizing the non-target second weight block in each second weight block corresponding to the second network layer.
[0085] It is easy to understand that in practical applications, if each second weight block corresponding to the second network layer is determined to be the target second weight block, that is, each second weight block is matched with the corresponding target first weight block in the first weight block corresponding to the first network layer, then in this case, the model quantization data only includes the first quantization weight and the mapping relationship information.
[0086] The mapping relationship information can be recorded and stored using, but is not limited to, a mapping table, which can also be called a weight-sharing mapping table. Based on the mapping relationship information recorded in the weight-sharing mapping table, the target second weight block can share the target first quantization weight corresponding to its matching target first weight block, so as to support the execution of related processing based on the shared target first quantization weight. For example, when it is necessary to perform dequantization processing on the model quantization data in order to perform task inference based on the model obtained after dequantization, it is supported to perform dequantization processing to recover the target second weight block based on the target first quantization weight corresponding to the target first weight block matched by the target second weight block.
[0087] When the first model needs to be deployed on terminal devices such as mobile phones and computers, the deployment of the first model can be achieved based on the model quantization data corresponding to the first model. In summary, the model quantization method provided in this application, in the model quantization process, by obtaining the block quantization data corresponding to the first model, determines the target second weight block and target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer, and generates the mapping relationship information between the target second weight block and the target first weight block. Based on the mapping relationship information, the model quantization data corresponding to the first model is obtained, which allows the target second weight block to share the target first quantization weight corresponding to its matching target first weight block, without having to retain the quantization weight corresponding to the target second weight block in the model quantization data. This achieves further quantization compression of the first model, facilitates the deployment and runtime of the model, further reduces the model's storage space occupation, and reduces hardware requirements.
[0088] Furthermore, this embodiment determines the first and second network layers based on the importance of the weights corresponding to the network layers in the first model. It retains the first quantized weights obtained by quantizing the first weight blocks corresponding to the first network layers with high weight importance. For the second network layers with low weight importance, it matches the second weight blocks corresponding to the first weight blocks corresponding to the first network layers with the similarity between their respective quantized weights, and generates mapping relationship information between the matching target second weight blocks and target first weight blocks. Based on this mapping relationship information, the target second weight blocks share the target first quantized weights corresponding to the target first weight blocks they match. This achieves further model compression to reduce storage space while preserving the quantization results of the weight blocks with high importance as much as possible, thereby ensuring the quantization quality / accuracy of the model.
[0089] In an optional embodiment, the model quantization data further includes quantization parameters corresponding to each weight block of the network layer in the first model.
[0090] Optionally, the quantization parameters corresponding to the weight block include scaling factors and zeros. The scaling factors represent the scaling ratio between floating-point numbers and integers during quantization, and are used to linearly map the floating-point range to the integer range; the zeros are the offset of the floating-point number during quantization, and are used to adjust the offset of the quantization center.
[0091] See Figure 2 The model quantization method provided in this application may further include the following processing:
[0092] Step 104: Adjust the quantization parameters contained in the model quantization data.
[0093] Optionally, the quantization parameter can be adjusted based on the loss function and optimization objective through a quantization-aware training method, and the process can be iterated until the termination condition is met.
[0094] The loss function is used to characterize the difference in weight distribution between the second model and the first model. The second model is a model obtained by performing model dequantization processing based on the model quantization data. The optimization objective is to minimize the value of the loss function.
[0095] The termination condition may be, but is not limited to, being set to the number of iterations for adjusting the quantization parameters reaching a threshold, or the value of the loss function being lower than a threshold.
[0096] In practical applications, when adjusting the value of the quantization parameter, the values of the weights corresponding to the network layers in the first model can also be adjusted simultaneously.
[0097] The applicant found through research that, for cases where the first model is a large model, adjusting the weights corresponding to its network layers has very little effect on improving the accuracy of the large model and can be ignored. In order to reduce the workload, the preferred approach is not to adjust the weights corresponding to the network layers in the model, but only to adjust the quantization parameters of the weights corresponding to the network layers.
[0098] This embodiment optimizes the quantization parameters by adjusting the quantization data contained in the model, thereby improving the accuracy of model inference. Furthermore, by adjusting only the quantization parameters of the weights corresponding to the model's network layers, without adjusting the weights themselves (especially for large models), model accuracy is maintained while avoiding excessive workload, thus improving model quantization efficiency.
[0099] See Figure 3 The flowchart shown illustrates the model reasoning method provided in this application embodiment, which may include the following steps 301 to 303, which are described in detail below.
[0100] Step 301: Obtain the task to be processed and the model quantization data.
[0101] The model quantization data includes: at least one first quantized weight obtained by quantizing at least one first weight block corresponding to the first network layer in the first model; mapping relationship information between the target second weight block in at least one second weight block corresponding to the second network layer in the first model and the target first weight block in each first weight block; and the quantized weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions.
[0102] The model inference method provided in this application embodiment can be applied to an edge device on which the first model is deployed or to an edge device that can call the first model in the cloud. For example, the edge device deploys the first model based on the model quantization data corresponding to the first model. After the deployment is completed, the electronic device stores the model quantization data corresponding to the first model.
[0103] The task to be processed is a related reasoning task that needs to be performed based on the first model. Specifically, it may be, but is not limited to, a natural language processing task, a computer vision-related task, or a multimodal task, depending on the actual application.
[0104] The natural language processing tasks include, but are not limited to, language understanding, information extraction, text generation, and knowledge question answering; the computer vision-related tasks include, but are not limited to, image processing, image generation, and object detection and recognition; and the multimodal tasks include, but are not limited to, cross-modal understanding and generation of images and text.
[0105] Step 302: Based on the mapping relationship information, determine at least one of the first quantization weights that, in relation to the target second weight block, satisfies the similarity condition for the target first quantization weight.
[0106] Specifically, based on the mapping relationship information, a target first weight block matching the target second weight block can be determined, and then the first quantization weight corresponding to the target first weight block can be determined as the target first quantization weight that satisfies the similarity condition with the target second quantization weight corresponding to the target second weight block. For example, based on the correspondence between the position information of the target second weight block and the position information of the target first weight block contained in the weight sharing mapping table, the position information of the target first weight block matching the target second weight block can be determined, and then the first quantization weight corresponding to the target first weight block indicated by the position information can be determined as the target first quantization weight.
[0107] Step 303: In the second network layer, inference is performed on the task to be processed based at least on the target first quantization weight to generate the corresponding task processing result.
[0108] After determining the target first quantization weight that satisfies the similarity condition for the target second quantization weight corresponding to the target second weight block, the task to be processed can be reasoned in the second network layer at least based on the target first quantization weight to generate the corresponding task processing result, such as generating the answer to the user's question in the knowledge question answering task, or generating an image that meets the requirements or an image processing result that meets the requirements.
[0109] In the inference phase, optionally, specifically when the first model infers to the second network layer, the first quantization weight that satisfies the similarity condition to the target second quantization weight corresponding to the target second weight block can be determined in real time based on the mapping relationship information, and the target first quantization weight can be dequantized so that the task to be processed can be inferred in the second network layer based on the dequantized target first quantization weight.
[0110] The model quantization data may also include non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each second weight block corresponding to the second network layer.
[0111] In practical applications, whether the model quantization data includes the non-target second quantization weights depends on the matching of each second weight block corresponding to the second network layer with each first weight block corresponding to the first network layer. If each second weight block matches the corresponding target first weight block in the first weight block corresponding to the first network layer, then the model quantization data does not include the non-target second quantization weights. Conversely, if there is a second weight block that does not match the corresponding target first weight block, then the model quantization data includes the non-target second quantization weights.
[0112] Specifically, if the model quantization data does not include the non-target second quantization weight, this step involves inferring the task to be processed based on the target first quantization weight at the second network layer; if the model quantization data includes the non-target second quantization weight, this step involves inferring the task to be processed based on both the target first quantization weight and the non-target second quantization weight at the second network layer.
[0113] In addition, the model inference method provided in this application embodiment also performs inference on the task to be processed based on at least one of the first quantization weights in the first network layer, so as to combine the inference on the task to be processed based on at least the target first quantization weights in the second network layer to generate the task processing result corresponding to the task to be processed.
[0114] In summary, the model inference method provided in this application, by enabling the second target weight block to share the first target quantization weight corresponding to its matching first target weight block (the quantization weights corresponding to the second target weight block and the first target weight block respectively satisfy similarity conditions), ensures the quality of model quantization and correspondingly guarantees the accuracy of task inference based on model quantization data, while further quantizing and compressing the first model. It also further reduces the storage space occupied by the model and the hardware requirements when deploying the model.
[0115] In an optional embodiment, the model quantization data further includes quantization parameters corresponding to the first weight block, the target second weight block, and the non-target second weight block, respectively.
[0116] The model inference method may further include at least a portion of the following 11)-13):
[0117] 11) Based on the first quantization parameters corresponding to each first weight block, perform dequantization processing on the first quantization weights corresponding to each first weight block to obtain the first dequantization weights.
[0118] Specifically, the first quantization weights corresponding to each first weight block can be dequantized based on the scaling factor and zero point corresponding to each first weight block.
[0119] 12) Based on the second quantization parameter corresponding to the non-target second weight block, perform dequantization processing on the non-target second quantization weight corresponding to the non-target second weight block to obtain the second dequantization weight.
[0120] Specifically, the non-target second quantization weights corresponding to the non-target second weight block can be dequantized based on the scaling factor and zero point corresponding to the non-target second weight block.
[0121] 13) Based on the third quantization parameter corresponding to the second weight block of the target, the first quantization weight of the target is dequantized to obtain the third dequantization weight.
[0122] Based on the scaling factor and zero point corresponding to the second weight block of the target, the first quantization weight of the target can be dequantized.
[0123] In this embodiment, the model inference method can be implemented by inferring the task to be processed based on the target first quantization weight at least in the second network layer, which can be achieved by inferring the task to be processed based on the third inverse quantization weight in the second network layer; or by inferring the task to be processed based on the second inverse quantization weight and the third inverse quantization weight in the second network layer.
[0124] Specifically, if the model quantization data does not include the non-target second quantization weight, then inference is performed on the task to be processed in the second network layer based on the third inverse quantization weight; conversely, if the model quantization data includes the non-target second quantization weight, then inference is performed on the task to be processed in the second network layer based on the second inverse quantization weight and the third inverse quantization weight.
[0125] Similarly, in the model inference method, inference of the task to be processed based on at least one of the first quantization weights in the first network layer can be implemented as follows: inference of the task to be processed based on the first inverse quantization weights in the first network layer.
[0126] In implementation, during the inference phase, optionally, specifically, when the model infers to the first network layer, the first quantization weights corresponding to each first weight block in the first network layer can be dequantized in real time based on the first quantization parameters corresponding to each first weight block in the first network layer to obtain the first dequantized weights. The first dequantized weights can be cached, such as caching the first dequantized weights in memory, so that the inference of the task to be processed can be performed in the first network layer based on the first dequantized weights. When the inference operation of the first network layer is completed, the cached first dequantized weights can be cleared in time to save the occupation of memory and other cache space.
[0127] Similarly, when the model infers to the second network layer, the target first quantization weight can be dequantized in real time based on the third quantization parameter corresponding to the target second weight block to obtain the third dequantized weight, and the third dequantized weight can be cached in memory or other cache areas. Alternatively, the non-target second quantization weight corresponding to the non-target second weight block can be dequantized in real time based on the second quantization parameter corresponding to the non-target second weight block to obtain the second dequantized weight, and the second dequantized weight can be cached in memory or other cache areas. The inference of the task to be processed is performed in the second network layer based on the corresponding cached dequantized weight. When the inference operation of the second network layer is completed, the cached third dequantized weight, second dequantized weight, and other corresponding dequantized weights can be cleared in a timely manner to save the amount of memory or other cache space occupied.
[0128] In summary, this embodiment achieves further quantization compression of the first model by enabling the target second weight block to share the target first quantization weight corresponding to its matching target first weight block (the quantization weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions). This reduces the storage space occupied during model deployment and lowers hardware requirements while ensuring the quantization quality of the model and the accuracy of task inference based on the quantized model data. Simultaneously, by performing dequantization processing on the quantization weights corresponding to the weight blocks of the network layers in the model during the inference phase, and performing inference operations based on the obtained dequantized weights, and by promptly clearing the cached dequantized weights after inference, the accuracy of the model's inference of the task to be processed is ensured, thereby guaranteeing the accuracy of the task processing results. This also reduces the occupation of memory and other cache space, further lowering hardware requirements.
[0129] Corresponding to the model quantization method described above, this application also provides a model quantization device, the structure of which is as follows: Figure 4 As shown, it includes:
[0130] The first acquisition module 401 is used to acquire block quantization data corresponding to the first model; the block quantization data includes: at least one quantized weight obtained by quantizing at least one weight block corresponding to the network layer in the first model, wherein the weight block corresponding to the network layer is a block obtained by dividing the weight corresponding to the network layer.
[0131] The first determining module 402 is used to determine the target second weight block and the target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer.
[0132] The generation module 403 is used to generate mapping relationship information between the target second weight block and the target first weight block to obtain model quantization data;
[0133] The model quantization data includes: a first quantization weight and the mapping relationship information; the quantization weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions; the first quantization weight is the quantization weight obtained by quantizing at least one first weight block corresponding to the first network layer.
[0134] In one alternative implementation, the weights corresponding to the first network layer are of higher importance than the weights corresponding to the second network layer.
[0135] The model quantization data further includes: non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each second weight block corresponding to the second network layer.
[0136] The model quantization data also includes: the quantization parameters corresponding to each weight block of the network layer in the first model.
[0137] The model quantization device further includes an adjustment module for adjusting the quantization parameters contained in the model quantization data.
[0138] In one optional implementation, when adjusting the quantization parameters contained in the model quantization data, the adjustment module is specifically used for:
[0139] The quantization parameter is adjusted based on the loss function and optimization objective through a quantization-aware training method until the termination condition is met.
[0140] The loss function is used to characterize the difference in weight distribution between the second model and the first model. The second model is a model obtained by performing model dequantization processing based on the model quantization data. The optimization objective is to minimize the value of the loss function.
[0141] Corresponding to the model reasoning method described above, this application also provides a model reasoning apparatus, the structure of which is as follows: Figure 5 As shown, it includes:
[0142] The second acquisition module 501 is used to acquire the task to be processed and the model quantization data;
[0143] The model quantization data includes: at least one first quantized weight obtained by quantizing at least one first weight block corresponding to the first network layer in the first model; mapping relationship information between the target second weight block in at least one second weight block corresponding to the second network layer in the first model and the target first weight block in each first weight block; and the quantized weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions.
[0144] The second determining module 502 is used to determine, based on the mapping relationship information, at least one of the first quantization weights, a target second quantization weight corresponding to the target second weight block that satisfies the similarity condition;
[0145] The inference module 503 is used to infer the task to be processed in the second network layer based at least on the target first quantization weight, so as to generate the corresponding task processing result.
[0146] In one alternative implementation, the weights corresponding to the first network layer are more important than the weights corresponding to the second network layer, and the first network layer is located before the second network layer in the first model;
[0147] The first weight block and the second weight block are blocks obtained by dividing the weights in the corresponding network layers.
[0148] In an optional implementation, when the inference module 503 performs inference on the task to be processed in the second network layer at least based on the target first quantization weight, it is specifically used for:
[0149] In the second network layer, inference is performed on the task to be processed based on the target first quantization weight and the non-target second quantization weight;
[0150] The model quantization data further includes the non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each of the second weight blocks.
[0151] The inference module 503 is further configured to: perform inference on the task to be processed in the first network layer based on at least one of the first quantization weights.
[0152] In an optional implementation, the model quantization data further includes quantization parameters corresponding to the first weight block, the target second weight block, and the non-target second weight block, respectively;
[0153] The model inference apparatus further includes a dequantization module for performing at least some of the following:
[0154] Based on the first quantization parameters corresponding to each first weight block, the first quantization weights corresponding to each first weight block are dequantized to obtain the first dequantization weights.
[0155] Based on the second quantization parameter corresponding to the non-target second weight block, the non-target second quantization weight corresponding to the non-target second weight block is dequantized to obtain the second dequantization weight.
[0156] Based on the third quantization parameter corresponding to the second weight block of the target, the first quantization weight of the target is dequantized to obtain the third dequantization weight.
[0157] In an optional implementation, when the inference module 503 performs inference on the task to be processed in the second network layer at least based on the target first quantization weight, it is specifically used for:
[0158] The task to be processed is inferred in the second network layer based on the third inverse quantization weight;
[0159] Alternatively, the task to be processed can be inferred in the second network layer based on the second dequantization weight and the third dequantization weight.
[0160] In an optional implementation, when the inference module 503 performs inference on the task to be processed based on at least one of the first quantization weights in the first network layer, it is specifically used for:
[0161] The task to be processed is inferred in the first network layer based on the first dequantization weight.
[0162] This application also discloses an electronic device, the composition and structure of which are as follows: Figure 6 As shown, it includes at least:
[0163] Memory 10 is used to store the computer instruction set;
[0164] Computer instruction sets can be implemented in the form of computer programs.
[0165] The processor 20 is configured to implement the model quantization method or model inference method provided in any of the above method embodiments by executing the computer instruction set in the memory.
[0166] The processor 20 can be a central processing unit (CPU), a graphics processor, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU), or other programmable logic devices.
[0167] Optionally, electronic devices may also include storage resources such as memory and cache.
[0168] Optionally, the electronic device may also include a camera assembly, and / or be connected to an external camera assembly.
[0169] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.
[0170] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.
[0171] This application also discloses a storage medium carrying one or more computer instruction sets, which, when executed by an electronic device, enable the electronic device to implement the model quantization method or model inference method provided in any of the above method embodiments.
[0172] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0173] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.
[0174] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence or the part that makes a creative contribution, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0175] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0176] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A model reasoning method, comprising: Obtain the tasks to be processed and the model quantization data; The model quantization data includes: at least one first quantized weight obtained by quantizing at least one first weight block corresponding to the first network layer in the first model; mapping relationship information between the target second weight block in at least one second weight block corresponding to the second network layer in the first model and the target first weight block in each first weight block; and the quantized weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions. Based on the mapping relationship information, at least one of the first quantization weights is determined to be a target first quantization weight that satisfies the similarity condition with the target second quantization weight corresponding to the target second weight block; The second network layer performs inference on the task to be processed based at least on the target first quantization weight to generate the corresponding task processing result.
2. In the model inference method according to claim 1, the importance of the weights corresponding to the first network layer is higher than the importance of the weights corresponding to the second network layer, and the first network layer is located before the second network layer in the first model; The first weight block and the second weight block are blocks obtained by dividing the weights in the corresponding network layers.
3. The model inference method according to claim 1 or 2, wherein in the second network layer, inference is performed on the task to be processed based at least on the target first quantization weight, comprising: In the second network layer, inference is performed on the task to be processed based on the target first quantization weight and the non-target second quantization weight; The model quantization data further includes the non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each of the second weight blocks. The method further includes: The task to be processed is inferred in the first network layer based on at least one of the first quantization weights.
4. The model inference method according to claim 3, wherein the model quantization data further includes quantization parameters corresponding to the first weight block, the target second weight block, and the non-target second weight block, respectively; The method further includes at least some of the following: Based on the first quantization parameters corresponding to each first weight block, the first quantization weights corresponding to each first weight block are dequantized to obtain the first dequantization weights. Based on the second quantization parameter corresponding to the non-target second weight block, the non-target second quantization weight corresponding to the non-target second weight block is dequantized to obtain the second dequantization weight. Based on the third quantization parameter corresponding to the second weight block of the target, the first quantization weight of the target is dequantized to obtain the third dequantization weight.
5. The model inference method according to claim 4, wherein in the second network layer, inference is performed on the task to be processed based at least on the target first quantization weight, comprising: The task to be processed is inferred in the second network layer based on the third inverse quantization weight; Alternatively, the task to be processed can be inferred in the second network layer based on the second dequantization weight and the third dequantization weight.
6. The model inference method according to claim 4, wherein inference is performed on the task to be processed in the first network layer based on at least one of the first quantization weights, comprising: The task to be processed is inferred in the first network layer based on the first dequantization weight.
7. A model quantization method, comprising: Obtain the block quantization data corresponding to the first model; The block quantization data includes: at least one quantized weight obtained by quantizing at least one weight block corresponding to the network layer in the first model, wherein the weight block corresponding to the network layer is a block obtained by dividing the weight corresponding to the network layer. Determine the target second weight block and the target first weight block that match the second weight block corresponding to the second network layer in the first model and the first weight block corresponding to the first network layer. Generate mapping information between the second weight block of the target and the first weight block of the target to obtain model quantization data; The model quantization data includes: a first quantization weight and the mapping relationship information; the quantization weights corresponding to the target second weight block and the target first weight block respectively satisfy similarity conditions; the first quantization weight is the quantization weight obtained by quantizing at least one first weight block corresponding to the first network layer.
8. The model quantization method according to claim 7, The weights corresponding to the first network layer are more important than the weights corresponding to the second network layer. The model quantization data further includes: non-target second quantization weights, which are quantization weights obtained by quantizing the non-target second weight blocks in each second weight block corresponding to the second network layer; The model quantization data further includes: quantization parameters corresponding to each weight block of the network layer in the first model; the method further includes: adjusting the quantization parameters contained in the model quantization data.
9. The model quantization method according to claim 8, wherein adjusting the quantization parameters included in the model quantization data includes: The quantization parameter is adjusted based on the loss function and optimization objective through a quantization-aware training method until the termination condition is met. The loss function is used to characterize the difference in weight distribution between the second model and the first model. The second model is a model obtained by performing model dequantization processing based on the model quantization data. The optimization objective is to minimize the value of the loss function.
10. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor configured to implement the model inference method as described in any one of claims 1-6, or the model quantization method as described in any one of claims 7-9, by executing the instruction set stored in the memory.