Method and device for deploying quantitative model for embedded neural network processor

Abstract

The invention discloses a method and a device for deploying a quantitative model for an embedded neural network processor, which are used for improving the adaptation degree of quantitative deployment of a neural network model on the embedded neural network processor, and comprise the following steps: obtaining topological structure information of an NPU, the topological structure information comprises a hardware connection relation of a function module in the NPU and a supported operator quantization attribute; generating a plurality of assembly line modules based on the topological structure information, wherein each assembly line module comprises a function module for realizing a model operator function and an operator quantization attribute of the function module in the assembly line module to which the function module belongs; determining a matching relationship between each node in the network structure of the to-be-deployed model and a functional module in the assembly line module, wherein a topological connection relationship of any node in the network structure is consistent with a hardware connection relationship of the matched functional module in the assembly line module to which the matched functional module belongs; and quantitatively deploying the to-be-deployed model to the NPU according to the operator quantification attribute of the assembly line module matched with each operator in the to-be-deployed model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural network models, in particular to a method and device for deploying a quantized model on an embedded neural processing unit. BACKGROUND

[0002] Quantization is a means to compress neural network models. Specifically, quantization is to use low-bit representation of weights and activations to reduce memory occupation and accelerate inference process.

[0003] In actual applications, a neural network model can be deployed on an embedded neural processing unit (NPU) for application. The functions of operators (OPs) of the neural network model can be implemented by multiple hardware modules in the NPU. The quantization of the neural network model can be limited by the functions of different hardware modules, and in actual applications, there can be a situation that the quantization parameters do not match the hardware functions, which causes the hardware to be unable to meet the model requirements, and thus the model output result is abnormal.

[0004] How to improve the adaptation degree of the deployment of a quantized neural network model on an embedded neural processing unit is a technical problem to be solved by the present application. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a method and device for deploying a quantized model on an embedded neural processing unit, so as to improve the adaptation degree of the deployment of a quantized neural network model on an embedded neural processing unit.

[0006] In a first aspect, a method for deploying a quantized model on an embedded neural processing unit is provided, comprising:

[0007] Obtaining topology structure information of an embedded neural processing unit (NPU), the topology structure information comprising hardware connection relationships between multiple functional modules in the NPU and operator quantization attributes supported by each functional module;

[0008] Generating multiple pipeline modules based on the topology structure information, any one of the pipeline modules comprising at least one functional module for implementing a function of a model operator and an operator quantization attribute of the functional module in the pipeline module to which the functional module belongs;

[0009] Determining a matching relationship between each node in a network structure of a to-be-deployed model and a functional module in a pipeline module, wherein a topology connection relationship of any one node in the network structure is consistent with a hardware connection relationship of the matched functional module in the pipeline module to which the functional module belongs;

[0010] According to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model, the to-be-deployed model is quantizationally deployed to the NPU.

[0011] In a second aspect, an apparatus for deploying a quantization model to an embedded neural network processing unit is provided, comprising:

[0012] An obtaining module is configured to obtain topology information of an embedded neural network processing unit (NPU), the topology information comprising a hardware connection relationship between a plurality of functional modules in the NPU and an operator quantization attribute supported by each functional module;

[0013] A generating module is configured to generate a plurality of pipeline modules based on the topology information, each of the pipeline modules comprising at least one functional module for implementing a model operator function and an operator quantization attribute of the functional module in the pipeline module;

[0014] A determining module is configured to determine a matching relationship between each node in a network structure of a to-be-deployed model and a functional module in a pipeline module, wherein a topology connection relationship of each node in the network structure is consistent with a hardware connection relationship of the matched functional module in the pipeline module;

[0015] A deploying module is configured to, according to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model, quantizationally deploy the to-be-deployed model to the NPU.

[0016] In a third aspect, an electronic device is provided, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps of the method of the first aspect are implemented.

[0017] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0018] In a fifth aspect, a computer program product is provided, which includes a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform some or all of the steps of the method of the first aspect.

[0019] In the embodiment of the present application, first, the topology structure information of the embedded neural network processor NPU is acquired, and the topology structure information includes the hardware connection relationship between a plurality of functional modules in the NPU and the operator quantization attribute supported by each functional module. Then, a plurality of pipeline modules are generated based on the topology structure information, and each pipeline module includes at least one functional module for implementing the function of a model operator and the operator quantization attribute of the functional module in the corresponding pipeline module. Among them, the pipeline module can represent the pipeline function that can be implemented by the hardware module, and the operator quantization attribute suitable for the functional module in the corresponding pipeline module. Subsequently, the matching relationship between each node in the network structure of the to-be-deployed model and the functional module in the pipeline module is determined, wherein the topological connection relationship of any node in the network structure is consistent with the hardware connection relationship of the matched functional module in the corresponding pipeline module; finally, the to-be-deployed model is quantitatively deployed to the NPU according to the operator quantization attribute of the pipeline module matched by each operator in the to-be-deployed model. Since the operator quantization attribute used in the quantization deployment is determined in advance according to the hardware function of the hardware module, and is suitable for the pipeline module to which the hardware module belongs, the model performing quantization deployment based on the operator quantization attribute can adapt to the actual function of the hardware module in the NPU, ensure that the node function of the model can be effectively supported by the hardware module, avoid the conflict between the model function and the hardware performance, and effectively improve the adaptation degree of the neural network model quantization deployment on the embedded neural network processor. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0021] Figure 1a is a module logic diagram of an NPU hardware according to an embodiment of the present application;

[0022] Figure 1b is a flowchart of a method for deploying a quantized model on an embedded neural network processor according to an embodiment of the present application;

[0023] Figure 1c is a flowchart of a method for deploying a quantized model on an embedded neural network processor according to an embodiment of the present application;

[0024] Figure 2a is a flowchart of a method for deploying a quantized model on an embedded neural network processor according to an embodiment of the present application;

[0025] Figure 2b is a pipeline template generated in a method for deploying a quantized model on an embedded neural network processor according to an embodiment of the present application;

[0026] Figure 3a is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0027] Figure 3b is a partial network topology diagram of a model to be deployed in a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0028] Figure 4a is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0029] Figure 4b is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0030] Figure 4c is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0031] Figure 5a is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0032] Figure 5b is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0033] Figure 5c is a flowchart of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application;

[0034] Figure 6 is a device structure diagram of a method for deploying a quantized model to an embedded neural network processor according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. The figure numbers in the present application are only used to distinguish each step in the scheme, and are not used to limit the execution order of each step, which is subject to the description in the specification.

[0036] In the field of neural network models, a deep learning framework is a running system that can run a deep learning network. In actual applications, a model can be quantized by integrating a basic quantization algorithm inside the framework, but this quantization method cannot guarantee the accuracy of the quantized model and the quantization effect is not good. Although some NPU manufacturers provide quantization schemes that can be applied, these quantization schemes are often limited by the quantization algorithm capabilities of the manufacturer.

[0037] In addition, the quantization effect of the model is also affected by the hardware functional structure. For example, in the application scenario of deploying a model to an NPU with graph fusion, the quantization attribute will flexibly change on the hardware graph fusion template. For another example, the function of the same operator in the model is implemented by different hardware modules of the NPU, and the quantization scheme set for the layer type of the operator may not be applicable to different hardware modules. These factors often make the quantization deployment effect of the model unstable. Among them, an operator (OP) is a basic component node of a deep learning network, and different operators can have different functions, such as convolution (Convolution), pooling (Pooling), etc.

[0038] The following will illustrate the technical problems existing in the process of deploying a quantized model to an NPU with an example.

[0039] Figure 1a A module logic diagram of an NPU hardware is shown. For the "Convolution+Relu+Convolution" structure in the network topology structure of the model, the output quantization bit width of Convolution and Relu is S32, that is, int32 type. For the structure containing only "Convolution" single OP in the network topology structure of the model, the input and output quantization bit width of Convolution only supports s8, that is, int8 type.

[0040] In this example, both Convolution 0 and Convolution 1 in the NPU hardware can be used to implement the function of the Convolution operator. According to the above example, the quantization strategy suitable for the Convolution operator in different topological structures is different, which may cause the situation that the quantization strategy does not match the function of the hardware module.

[0041] As can be seen, for an NPU hardware platform, especially a hardware platform with graph fusion attributes for the bottom OP operator, the quantization bit width of the OP will be adaptively adjusted by whether it is in the hardware pipeline, which may cause the situation that the quantization attribute of the model after quantization does not match the actual function limitation of the hardware when the model is deployed to the hardware.

[0042] To address the problems existing in the prior art, embodiments of this application provide a method for deploying a quantization model on an embedded neural network processor, such as... Figure 1b As shown, it includes the following steps:

[0043] S11: Obtain the topology information of the embedded neural network processor (NPU), which includes the hardware connection relationships between multiple functional modules in the NPU and the operator quantization attributes supported by each functional module.

[0044] In this step, the topology information of the NPU hardware for the model to be deployed is obtained. This topology information represents the hardware functional modules contained in the NPU, the connection relationships between the hardware functional modules, and the operator quantization attributes supported by the hardware functional modules. In practical applications, the topology information can be displayed in various forms, such as hardware pipeline information and hardware execution information.

[0045] Among them, operator quantization attributes refer to parameter information related to operator quantization, such as the quantization bit width of the input or output, quantization strategy, quantization rounding method, etc.

[0046] For any functional module in the NPU, the supported operator quantization attributes may not be unique. Figure 1a Taking the NPU shown as an example, the input bit width of Convolution 0 can support, for example, s8 and s16. The quantization strategy of Convolution 0 can support per-channel quantization, per-tensor quantization, per-tensor asymmetric quantization, etc. The operator quantization attributes obtained in this step are the operator quantization attributes supported by the functional modules of the NPU at the hardware level, that is, the operator quantization attributes that can be implemented in hardware functionality.

[0047] S12: Generate multiple pipeline modules based on the topology information. Each pipeline module includes at least one functional module for implementing model operator functions and the operator quantization attribute of the functional module in its respective pipeline module.

[0048] In practical applications, NPUs can often flexibly implement a variety of different pipelines. In this example, the pipelines that can be implemented by each functional module in the NPU are expressed by generating pipeline modules.

[0049] by Figure 1aAs shown in the NPU, it is assumed that the input and output quantization attributes of the Convolution function module in the NPU support s8 and s32. Then, the generated pipeline module may include, for example, “input → Convolution 0 → output”, which only contains the Convolution 0 function module, and has a single-OP structure directly connecting the input and the output. The input quantization bit width and the output quantization bit width in the operator quantization attribute can be determined as s8.

[0050] For another example, the generated pipeline module may include, for example, “input → Convolution 0 → Convolution 1 → output”, in which, in the case that the output quantization attribute of Convolution 0 supports s8 and s32, s32 can be determined as the output quantization attribute of Convolution 0 in this pipeline module.

[0051] In this example, the operator quantization attribute of the function module in the pipeline module to which the function module belongs can be determined according to the model performance or the preset priority. For example, for the quantization strategy in the operator quantization attribute, a quantization strategy with better performance and suitable for the pipeline can be selected from the multiple quantization strategies supported by the function module as the quantization strategy of the function module in the pipeline module to which the function module belongs, so as to improve the quantization effect. For another example, for the input or output quantization bit width in the operator quantization attribute, the widest quantization bit width suitable for the pipeline can be selected from the multiple quantization bit widths supported by the function module, so as to not only ensure that the quantization bit width is suitable for the hardware performance, but also improve the quantization model precision as much as possible and ensure the model effect after quantization.

[0052] Since the operator quantization attribute supported by the function module determined in the above step is the operator quantization attribute supported by the hardware, the operator quantization attribute suitable for the pipeline module to which the function module belongs is determined from the hardware-supported operator quantization attribute according to the model function and the pipeline requirement in this step, so that the determined operator quantization attribute can be supported by the hardware and has better quantization effect.

[0053] S13: Determine the matching relationship between each node in the network structure of the model to be deployed and the function module in the pipeline module, wherein the topological connection relationship of any node in the network structure is consistent with the hardware connection relationship of the matched function module in the pipeline module to which the function module belongs.

[0054] The model to be deployed in this embodiment is a model that needs to be quantized and deployed to NPU hardware. This model can be used to implement detection, classification, or other algorithm functions. The network structure of the model to be deployed can also be called the network topology structure, which is used to represent the topological connection relationship between deep learning operators (OPs) in the model. It can be visualized in the form of a network topology graph.

[0055] In this step, the multiple pipeline modules generated in the previous steps are matched with each node in the network structure of the model to be deployed. Each node is a deep learning operator (OP), and the functionality of the OP needs to be implemented by the corresponding functional modules of the NPU.

[0056] In this step, any node in the network structure of the model to be deployed needs to be matched with a functional module in a pipeline module to ensure that the operator function corresponding to this node is implemented by a hardware functional module of an NPU.

[0057] Any pipeline module can be matched multiple times within the network structure of the model to be deployed. For example, if the network structure of the model to be deployed contains "Convolution→Relu→Convolution→Relu", based on... Figure 1a The NPU shown can match the pipeline module "Convolution 0→Relu"→"Convolution 0→Relu" for the four nodes in this example. Through this step, any one of the four nodes has matched a functional module in the pipeline module. The pipeline module "Convolution 0→Relu" was matched twice during the matching process.

[0058] During the matching process, the topological connectivity of any node in the network structure is consistent with the hardware connectivity of the matched functional module in its respective pipeline module. This will continue below with the above... Figure 1a The NPU shown is used for illustration. Assuming the network structure of the model to be deployed includes "Convolution→Convolution→Relu", then the pipeline module "Convolution 0→Relu" can be matched, but the pipeline module "Convolution 0→Relu→Convolution1" cannot be matched. That is, an incomplete pipeline module cannot be used to match nodes in the network structure. Specifically, for the network structure "Convolution→Convolution→Relu", the matching can be achieved by selecting the two pipeline modules "Convolution 0" → "Convolution0→Relu", ensuring that the topological connections of nodes in the network structure are consistent with the hardware connections of the matched functional modules in their respective pipeline modules.

[0059] S14: quantize and deploy the to-be-deployed model to the NPU according to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model.

[0060] In this step, the to-be-deployed model is deployed to the NPU based on the matched operator quantization attribute. In actual application, the quantization mode may be binary quantization, linear quantization, or pointer quantization, where the linear quantization may specifically include symmetric quantization and asymmetric quantization, and the symmetric quantization has a lower calculation amount than the asymmetric quantization.

[0061] The model quantization may be implemented through post-training quantization or quantization-aware training. The post-training quantization is to quantize a trained model, and very little or even no additional data is needed in the quantization process, the quantization process is simple and efficient, and the calculation cost is low. The quantization-aware training is to simulate quantization during the training process, that is, to retrain the model, and the hyperparameters of the model may need to be adjusted during the process, and the quantization-aware training has a better quantization effect than the post-training quantization in some application scenarios.

[0062] Through the scheme provided in the embodiments of the present application, since the operator quantization attribute of the pipeline module can be supported by the hardware of the NPU, and the operator quantization attribute is determined for the pipeline module to which the functional module belongs, the effect of quantizing and deploying the model can be optimized on the basis of the hardware support of the operator quantization attribute. Each node in the network structure of the to-be-deployed model can be matched to the functional module of the NPU through the matching mode, the association and matching of the operator function and the hardware function are realized, the model is quantized and deployed according to the matched operator quantization attribute, the operator function of the model can be supported by the hardware functional module after quantization and deployment, and the adaptation degree of the neural network model quantization deployment on the embedded neural network processor is improved.

[0063] The scheme provided in the embodiments of the present application can solve the problem of poor quantization algorithm effect of the hardware platform. To ensure the quantization algorithm effect, the operator quantization attribute is determined in advance based on the hardware support of the NPU, and then the NPU hardware quantization attribute adaptation is performed on the quantized model. On the premise that the algorithm quantization effect is basically unchanged, the deployment and landing of the general quantized model on the NPU hardware based on the layer type can be realized. The scheme provided in the embodiments of the present application can be based on the NPU hardware graph fusion or pipeline information, and the NPU-based adaptive quantization is performed to optimize the quantization effect. The quantization scheme does not have to be based on the original factory quantization, and the stability of the algorithm effect can be ensured. The scheme does not destroy the NPU graph fusion pipeline, and the efficiency is optimal. The model can be quantized and optimized based on the hardware adaptation. Moreover, the scheme is applicable to various to-be-deployed models, and has universality for the models to be quantized and deployed.

[0064] In the following, the application will be described in conjunction with Figure 1c The flow of the present application is further described.

[0065] In Figure 1c In the flow shown, the floating point model refers to a model to be deployed, and the quantization data refers to data required for performing quantization deployment on the floating point model, and can specifically include a preset data set. For the floating point model, the present application obtains topology structure information of an NPU through NPU hardware quantization attribute analysis configuration, and generates a plurality of pipeline modules based on the topology structure information, and further determines a matching relationship between each node in the network structure of the model to be deployed and the functional modules in the pipeline modules. Then, based on the matching relationship, the preset hardware is configured with OP operator quantization attribute information layer by layer, and the floating point model is quantized and deployed to the NPU by using the quantization data according to the operator quantization attribute by using a quantization tool, so as to realize hardware adaptive quantization model. After quantization deployment, the NPU hardware inference is used to obtain the network output result.

[0066] The scheme provided by the embodiments of the present application can deploy the quantized model to the required NPU hardware platform. For the hardware NPU with graph fusion, the general quantization model or quantization scheme will often encounter a situation of mismatching quantization attributes when deployed on flexible NPU graph fusion hardware, resulting in failure to deploy or error. To solve this technical problem, the scheme provided by the embodiments of the present application pre-divides different models according to the actual hardware supported functions to generate hardware pipelines, selects the optimal quantization attribute information for each OP operator, performs quantization and optimization based on the NPU hardware pipeline, and finally obtains a quantization model completely matched with the hardware, so as to completely adapt the model to be deployed to the NPU hardware through quantization deployment.

[0067] In the embodiment of the present application, first, the topology structure information of the embedded neural network processor NPU is acquired, and the topology structure information includes the hardware connection relationship between the plurality of functional modules in the NPU and the operator quantization attribute supported by each functional module. Then, a plurality of pipeline modules are generated based on the topology structure information, and each pipeline module includes at least one functional module for implementing the function of a model operator and the operator quantization attribute of the functional module in the corresponding pipeline module. Wherein, the pipeline module can represent the pipeline function that can be implemented by the hardware module, and the operator quantization attribute applicable to the functional module in the corresponding pipeline module. Subsequently, the matching relationship between each node in the network structure of the to-be-deployed model and the functional module in the pipeline module is determined, wherein the topology connection relationship of any node in the network structure is consistent with the hardware connection relationship of the matched functional module in the corresponding pipeline module; finally, the to-be-deployed model is quantitatively deployed to the NPU according to the operator quantization attribute of the pipeline module matched by each operator in the to-be-deployed model. Since the operator quantization attribute used in the quantization deployment is determined in advance according to the hardware function of the hardware module, and is applicable to the pipeline module to which the hardware module belongs, the model performing quantization deployment based on the operator quantization attribute can adapt to the actual function of the hardware module in the NPU, ensure that the node function of the model can be effectively supported by the hardware module, avoid the conflict between the model function and the hardware performance, and effectively improve the adaptation degree of the quantization deployment of the neural network model on the embedded neural network processor.

[0068] Based on the above-mentioned embodiments, optionally, as shown in Figure 2a In step S12, a plurality of pipeline modules are generated based on the topology structure information, including:

[0069] S21: determining a pipeline connected from the input end of the NPU to the output end of the NPU based on the topology structure information, and the pipeline includes at least one linearly connected functional module.

[0070] In the step of generating a plurality of pipeline modules based on the topology structure information, since the topology structure information includes the hardware connection relationship between the functional modules in the NPU, the pipeline connected from the input end of the NPU to the output end of the NPU is determined in the present example. In actual execution, the steps of S21-S23 of the present application can be repeatedly executed to generate corresponding pipeline modules for all the pipelines available in the NPU. Alternatively, a plurality of pipeline modules can also be generated in parallel based on the topology structure information of the NPU through distributed tasks.

[0071] In this step, according to the hardware connection relationship between the functional modules contained in the topology structure information, the pipeline connected to the NPU output end through at least one functional module is determined from the NPU input end as the starting point. Any pipeline contains at least one functional module, and if the pipeline contains multiple functional modules, these functional modules are linearly connected in the pipeline.

[0072] Optionally, any pipeline does not contain a feedback loop, that is, each functional module belonging to a pipeline is linearly connected in the pipeline.

[0073] Optionally, any functional module can belong to multiple pipelines.

[0074] S22: Determine the operator quantization attribute of the functional module in the pipeline based on the hardware connection relationship.

[0075] In practical applications, the operator quantization attribute can specifically include multiple attributes. In order to illustrate the present scheme, the quantization bit width in the operator quantization attribute is taken as an example for illustration.

[0076] Taking the NPU structure shown in FIG. 7 as an example, the pipeline determined based on the topology structure information includes, for example, 7 pipelines shown in FIG. 8, which correspond to pipeline template 1-pipeline template 7 respectively. Among them, pipeline template 5 is "input end→Convolution 0→output end", pipeline template 6 is "input end→Relu→output end", and pipeline template 7 is "input end→Convolution 1→output end". Figure 1a Figure 2b For example, the quantization bit width of the input end and the output end of any pipeline is specified as s8. It is assumed that the quantization bit width supported by the Convolution functional module in this example includes s8, s16 and s32, and the quantization bit width supported by the Relu functional module includes s8 and s32. In this step, the operator quantization attribute of the functional module in the pipeline is determined based on the hardware connection relationship. The quantization bit width determined for each functional module in the pipeline is marked at the arrow connected to the functional module.

[0077]

[0078] ​​Taking the pipeline template 1 as an example, the input bit width and the output bit width of Convolution 0 support s8, s16 and s32, but the input end is the input end of the pipeline, so the input quantization bit width of Convolution 0 is determined as s8 according to the quantization bit width of the input end, and the output quantization bit width of Relu is determined as s8 according to the quantization bit width of the output end. The bit width supported by the Convolution 0 output and the Relu input in the pipeline includes s8 and s32. In this case, the wider s32 can be determined as the output bit width of Convolution 0 and the input bit width of Relu, so as to improve the data precision in the pipeline and improve the overall quantization effect of the model after deployment.

[0079] Optionally, the quantization bit width applicable to each functional module in the corresponding pipeline can be efficiently determined by constructing a quantization bit width set. Specifically, the supported operator quantization attributes of each functional module in a pipeline can be determined by calling the topology information. The operator quantization attributes may, for example, include input bit width, output bit width, quantization rounding method, quantization algorithm strategy, etc. Taking the output bit width as an example, the maximum bit width available to the functional module is determined in this example by the set. Specifically, for any functional module, the quantization bit width set B supported by the functional module is constructed by calling the operator quantization attributes supported by the functional module. For example, the quantization bit width set of the Convolution functional module can include three elements of s8, s16 and s32, and the maximum bit width in the set is determined as the maximum output bit width bitdepth that the Convolution functional module can support: bitdepth = max(B)

[0080] In addition, the operator quantization attributes of the functional modules in the pipeline can also be determined based on preset priorities. For example, for the quantization algorithm strategy in the operator quantization attributes, different functional modules can support multiple different quantization algorithm strategies, and the highest priority quantization algorithm strategy can be selected as the corresponding quantization algorithm strategy of the functional module in the pipeline based on the priorities of the multiple quantization algorithm strategies.

[0081] For example, for the quantization algorithm strategy, it is assumed that Convolution 0 supports Per Channel quantization, Per Tensor quantization and Per Tensor asymmetric quantization. Among them, the preset priority of Per Tensor quantization is the highest, and Per Tensor quantization can be selected as the corresponding quantization algorithm strategy of the Convolution 0 functional module in the pipeline.

[0082] In some application scenarios, the same quantization algorithm strategy can also be set for multiple hardware function modules with the same function. For example, in the case where Convolution 0 supports Per Channel quantization, Per Tensor quantization, Per Tensor asymmetric quantization, and Convolution 1 supports Per Tensor quantization and Per Tensor asymmetric quantization, the quantization algorithm strategies commonly supported by the two Convolution function modules can be determined first, including Per Tensor quantization and Per Tensor asymmetric quantization. Then, if there are multiple commonly supported operator quantization attributes, the quantization algorithm strategy with the highest priority is selected as the quantization algorithm strategy corresponding to Convolution 0 and Convolution 1 in the pipeline.

[0083] In actual applications, different quantization algorithm strategies have their own advantages and disadvantages, and the optimal operator quantization attribute can be determined according to the OP function of the function module. When setting the priority of the operator quantization attribute, the priority of the operator quantization attribute can also be set for different functions respectively. For example, Per Tensor mentioned above refers to quantizing the entire tensor, that is, mapping the entire tensor to a fixed integer value interval. This method is suitable for the case where the value range of all elements in the tensor is similar, such as convolution kernel and fully connected layer weight. Based on this, for the related function modules of the convolution kernel and the fully connected layer weight, Per Tensor mentioned above can be set as high priority. Per Channel refers to quantizing each channel Channel in the tensor separately, that is, each channel is mapped to a fixed integer value interval. This method is suitable for the case where the value range of different channels in the tensor is greatly different, such as the feature map of the convolution layer. Based on this, for the related function modules of the feature map of the convolution layer, Per Channel mentioned above can be set as high priority.

[0084] In addition, the output value of the activation Relu is often unevenly distributed, and if symmetric quantization is used, it may cause precision loss and information loss, etc. Therefore, symmetric quantization can be set as low priority, and asymmetric quantization can be set as high priority. The value distribution of the weight is relatively uniform, so symmetric quantization can be set as high priority, and asymmetric quantization can be set as low priority.

[0085] In actual applications, the optimal operator quantization attribute can be determined according to the actual function of the function module to improve the quantization effect after model deployment.

[0086] S23: generating a pipeline module with the function modules in the pipeline and the operator quantization attributes of the function modules in the pipeline.

[0087] In this step, the pipeline module is generated based on the operator quantization attributes of each functional module and operator in the above pipeline, and the network structure of the model to be deployed is obtained based on the pipeline module. Figure 2b As shown in FIG. 7, seven pipeline modules can be generated, and each pipeline module includes at least one linearly connected functional module connected between the input end and the output end, and the quantization bit width is marked at the input end and the output end of each functional module. It should be understood that the above operator quantization attributes can also include quantization rounding rules, quantization algorithm strategies, quantization offsets, quantization coefficients and other attribute parameters. These operator quantization attributes are not shown in FIG. 7, and can be associated with the functional modules to generate the pipeline modules in actual applications. Figure 2b

[0088] Through the scheme provided by the embodiments of the present application, the hardware-supported pipeline module can be generated according to the topology structure information. The pipeline module can realize the connection between the input end and the output end of the NPU, and the optimal operator quantization attributes of each functional module in the pipeline are determined, thereby providing an effective information basis for subsequent network matching.

[0089] Based on the scheme provided in the above embodiments, as shown in FIG. 8, in step S13, the matching relationship between each node in the network structure of the model to be deployed and the functional module in the pipeline module is determined, including: Figure 3a

[0090] S31: The plurality of pipeline modules are sorted in descending order of the number of functional modules contained.

[0091] In this step, based on the pipeline modules generated in the above steps, the number of functional modules contained in each pipeline module is counted, and each pipeline module is sorted in descending order. For a plurality of pipeline modules containing the same number of functional modules, they can be sorted randomly to ensure that each pipeline module has a unique sorting order.

[0092] S32: The pipeline modules are searched in the network structure of the model to be deployed in sequence according to the sorting, and each structure subgraph matched is searched, wherein the topology structure relationship of each structure subgraph is consistent with the hardware connection relationship of the matched pipeline module.

[0093] ​​In this step, the network structure of the model to be deployed is matched and searched according to the above-mentioned sorting order from the pipeline module with the most functional modules. The network structure of the model to be deployed is composed of a plurality of functional nodes connected in a mesh. In this step, the structure subgraph in the network structure and the pipeline module at the hardware level are associated by matching and searching. In the matched structure subgraph and pipeline module, the number, function, and connection mode of each node in the structure subgraph correspond to the number, function, and hardware connection relationship of each functional module in the pipeline module, thereby associating the model network node with the hardware module actually executed.

[0094] In the present application, each pipeline module is sorted according to the number of functional modules, and a greedy algorithm is used to start matching and searching the network structure of the model to be deployed from the longest pipeline module. Figure 2b As shown in each pipeline template, the pipeline template 4 containing 3 functional modules is used to match and search the network structure of the model to be deployed, and the nodes of "Convolution→Relu→Convolution" in the network structure are matched and associated. Then, the pipeline template 1, the pipeline template 2, and the pipeline template 3 containing 2 functional modules are used to perform matching and searching. Finally, the pipeline template 5, the pipeline template 6, and the pipeline template 7 containing only 1 functional module are used to perform matching and searching. Through the matching and searching in this step, each node in the network structure of the model to be deployed can be matched with a hardware functional module, and the network structure can be deployed on the hardware functional module to realize the function required by the model. Moreover, the present scheme uses a greedy algorithm to preferentially match the pipeline template with a long pipeline, so that after the model is quantized and deployed, the plurality of functional nodes belonging to one pipeline can improve the processing efficiency and reduce the read-write frequency in units of pipeline modules, thereby improving the overall processing efficiency of the deployed model.

[0095] Optionally, the matching and searching efficiency of this step can be improved based on a preset formula. Assuming that M i is a pipeline module, i is the identification of different pipeline modules, |M i | represents the length of the pipeline module, p0 is an OP node to be matched, and m(p0) is the pipeline module matched by p0 through matching and searching. The pipeline module matched by p0 through matching and searching can be determined by searching p0 as follows:

[0096] S33: Determine the matching relationship between each node in the network structure and the functional module in the pipeline module according to the matching relationship between the pipeline module and the structure subgraph.

[0097] Next, the matching relationship between the nodes in the network topology and the functional modules in the pipeline module will be described with reference to an example.Figure 3b .

[0098] Figure 3b A part of network topology diagram of the model to be deployed is shown, which includes sequentially connected Conv_0 node, Conv_1 node, Relu_2 node, Conv_3 node, MaxPool_4 node and Relu_5 node. In the names of the nodes in the diagram, the part before the underscore is used to identify the function corresponding to the node, and the part after the underscore is used to distinguish the nodes.

[0099] For each node in the part of network topology shown in Figure 3b , retrieval matching is performed according to each pipeline template shown in Figure 2b . First, matching is performed using the pipeline template 4 containing the largest number of functional modules, in which the sequentially connected Conv_1 node, Relu_2 node and Conv_3 node correspond to the "Convolution 0→Relu→Convolution 1" matching pair in the pipeline template 4, and the topological relationship of the three nodes in the topology structure is consistent with the hardware connection relationship of the pipeline template 4. Therefore, it can be determined that the structure subgraph containing the three nodes matches the pipeline template 4.

[0100] Based on the scheme provided in the above embodiment, optionally, as shown in Figure 4a , in the step S32, the pipeline module is sequentially searched for a matching structure subgraph in the network structure of the model to be deployed according to the sorting, including:

[0101] S41: selecting a target pipeline module to be matched according to the sorting.

[0102] S42: sequentially searching for a matching node of a functional module in the network structure according to the pipeline order of the target pipeline module.

[0103] S43: if the topological relationship of at least one target structure subgraph in the network structure is consistent with the hardware connection relationship of the target pipeline module, then constructing a matching relationship between each target structure subgraph and the target pipeline module.

[0104] In actual application, in order to ensure that each node in the model to be deployed corresponds to a matching functional module, matching search can be performed on the full network structure of the model to be deployed. As shown in Figure 4b , based on the pre-acquired topological structure information, possible matching templates are constructed according to the NPU hardware, wherein the possible matching templates are the above-mentioned pipeline modules. Then, the matching templates are arranged in order from large to small according to the number of nodes in each matching template subgraph.

[0105] Based on the sorting result, the target pipeline module to be matched is selected as a matching template in sequence from the longest path template containing the most functional modules to perform subsequent matching search steps.

[0106] Subsequently, subgraph isomorphism is performed based on the acquired matching template. In the subgraph isomorphism step, a brute force search matching, VF2 algorithm, or other matching retrieval method can be used to perform traversal on the network structure graph of the model to be deployed, search whether the unmarked matching OP matches the currently selected matching template, and thereby build the association between the nodes of the network topology and the templates of the hardware.

[0107] Then, for the matched subgraph in the network topology graph, the matched OP is marked, and the operator quantization attribute and other parameters in the matching template are encapsulated as a matching template type, so that each node in the encapsulated subgraph is associated with the operator quantization attribute of the functional module in the corresponding pipeline. Thus, the retrieval and matching of the selected one pipeline module in the network topology of the model to be deployed is completed.

[0108] Next, the next matching template is selected according to the sorting order, and the matching of all matching templates is performed until the traversal and matching of all matching templates is completed, and the full retrieval and matching of the network topology of the model to be deployed is completed, so that any node in the network topology matches a corresponding hardware functional module.

[0109] In this example, the matching search of a single candidate template on the full graph can select a suitable search algorithm according to actual needs. For example, Figure 4cA brute force search flowchart is shown. On the basis of a selected matching template, a full graph traversal is performed on the topology of the model to be deployed to search for nodes matching the root node of the matching template, which is stored as match_patterns. Then, a full graph traversal is performed on the topology of the model to be deployed starting from the second node connected to the root node to determine whether the matching template has completed the traversal, i.e., whether the matching template contains more nodes that have not been matched. If the traversal has been completed (there are no more nodes), the answer is yes, and if the traversal has not been completed (there are more nodes that have not been matched), the answer is no. In the case of no, a full graph traversal is performed on the topology of the model to be deployed starting from the node matched by the root node to search for nodes matching the second node, which are stored as candidate search nodes. Next, it is necessary to select from the candidate search nodes the nodes connected to the match_patterns, i.e., to determine whether the input of the candidate node is in the match_patterns. If the answer is yes, it means that this candidate node is connected after the match_patterns, and the match_patterns is updated by adding this matched candidate node to the match_patterns. If the answer is no, it means that this candidate node is not connected to the match_patterns, and the match_patterns is updated by deleting the unmatched match_patterns. By traversing the candidate search nodes, it is ensured that a matching determination is performed for each candidate search node, and the nodes of the matching template are traversed until the traversal is completed, so that each node in the network topology of the model to be deployed is matched with a hardware function module.

[0110] Optionally, in the step of searching for a matching subgraph based on the candidate template on the full graph, for ambiguous subgraph patterns, the matching complexity often grows exponentially. In order to limit the execution time of the algorithm and improve the matching efficiency, a search upper limit max_candidates can be set in advance. When more than max_candidates pattern subgraphs are matched, the algorithm is forced to stop and an error is returned.

[0111] Based on the scheme provided in the above embodiment, optionally, as shown in Figure 5a After the step S14, i.e., after quantizing and deploying the model to be deployed to the NPU according to the operator quantization attributes of the pipeline module matched with each operator in the model to be deployed, the method further includes:

[0112] S51: inputting a test sample in a test set into the deployed model to determine the quantization effect of the deployed model according to the matching degree between the output first test result and the test label corresponding to the test sample in the test set; and / or,

[0113] S52: input the test samples in the test set into the to-be-deployed model and the deployed model respectively to determine the quantization effect of the deployed model according to the matching degree of the second test result output by the to-be-deployed model and the third test result output by the deployed model.

[0114] In the scheme provided by the embodiments of the present application, after the model is quantized and deployed, the test set can be used to test the quantization deployment effect of the deployed model. The test set is a data set containing multiple test samples. Assuming that the image classification model is quantized and deployed, the test samples in the test set include sample images and test labels corresponding to the sample images, and the test labels represent the image classification to which the corresponding sample images belong.

[0115] In step S51, the sample images in the test set are first input into the deployed model to obtain the first test result output by the deployed model for the sample images. Then, the first test result is matched with the test label corresponding to the sample image, and the higher the matching degree, the better the quantization deployment effect.

[0116] In step S52, the sample images in the test set are input into the model quantized and deployed on the hardware and the original model before quantization and deployment, respectively. The second test result output by the to-be-deployed model is the model classification result without quantization and deployment, and the third test result output by the deployed model is the model classification result after quantization and deployment. The difference between the second test result and the third test result can represent the influence of quantization and deployment on the model classification result. The matching degree between the second test result and the third test result is determined, and the higher the matching degree, the smaller the influence of quantization and deployment on the model classification result, and the better the quantization deployment effect.

[0117] In actual application, after the model is quantized and deployed on the NPU hardware, the above test steps of the present application can be realized through hardware bittrue simulation. The quantization deployment effect is determined by analyzing the error between the original floating-point model before quantization and deployment and the real hardware after quantization. If the quantization deployment effect does not meet the expectation, the effect tuning scheme of quantization algorithm can be used for tuning, such as correction based on the real calculation result of the hardware or reverse tuning through quantization-aware-training (QAT). After the effect tuning is completed, the quantization model is converted into the format supported by the NPU hardware, so as to perform NPU hardware inference, and finally obtain the inference result of the network based on the NPU hardware.

[0118] The embodiments of the present application will be further described below. Figure 5b The embodiments of the present application will be further described below.

[0119] First, according to the topology information of the NPU, a pipeline module based on the current NPU hardware is constructed as a candidate template. The hardware quantization attribute of each OP on the candidate template is determined, that is, the hardware quantization attribute applicable in the corresponding pipeline of each functional module in the pipeline module is determined.

[0120] Then, starting from the longest template where the current OP is located, the matching relationship between each node in the network structure of the model to be deployed and the functional modules in the pipeline module is determined by retrieval.

[0121] Next, based on the matching relationship, the per-OP hardware configuration quantization attribute information based on the current model is marked, and the operator quantization attribute corresponding to the functional module is configured to the corresponding node in the model.

[0122] Subsequently, the model is quantized and deployed based on the per-OP hardware attribute.

[0123] For the model after quantization and deployment, the quantization effect is tested by the scheme provided in the examples of the present application to determine whether the quantization effect meets the expectation. If it meets the expectation, the process ends. If it does not meet the expectation, quantization optimization is performed to optimize the quantization and deployment effect of the model.

[0124] Optionally, referring to Figure 5c In the step of marking the per-OP hardware adaptation quantization attribute information based on the current model, the configuration uniqueness of the model nodes can be specifically checked to avoid contradictions caused by matching multiple operator quantization attributes to one node.

[0125] First, input the model and perform DFS (Depth First Search) depth-first traversal sorting on the nodes in the model. The depth-first search is a graph algorithm that is used to go as deep as possible along each branch of the tree and to go as far as possible along each level.

[0126] Then, based on the topology information of the NPU hardware, a pipeline module containing operator quantization attributes is constructed as a hardware candidate template. Template matching is performed on each node in the input model to obtain the matching topology information of the current model based on the NPU hardware candidate template. The matching topology information can represent the association relationship between each node in the model and the hardware functional module.

[0127] Next, the entire model is traversed in order per OP, and the uniqueness of the template to which each OP node belongs is checked. Specifically, starting from the first OP of the model, it is determined whether the hardware candidate template where the current OP is located is unique.

[0128] If it is not unique, the longest candidate template in the candidate template is found as the hardware candidate template where the current OP is located, and other candidate templates are discarded to determine that the candidate template to which the OP belongs is unique.

[0129] If the candidate matching template is unique, the NPU hardware-based quantization attribute information is preset for the current OP with respect to the candidate template.

[0130] Until all OPs of the current input model are labeled with hardware preset quantization attribute information, the labeling of hardware quantization preset attribute information of the current model is completed.

[0131] Finally, the model with hardware quantization attribute information preset is subjected to quantization algorithm level optimization, and after the effect meets the standard, the quantization algorithm level optimization is stopped, and the model based on the current NPU hardware adaptive quantization optimization is obtained.

[0132] Based on the scheme provided in the above embodiments, optionally, the operator quantization attribute includes at least one of the following:

[0133] quantization bit width, quantization rounding rule, quantization algorithm strategy, quantization offset, and quantization coefficient.

[0134] The quantization bit width includes input quantization bit width and output quantization bit width, and the input quantization bit width and the output quantization bit width of one functional module can be different. The quantization rounding rule can specifically include a round rounding mode, for example, can include rounding up, rounding down, rounding to the nearest integer, etc. The quantization algorithm strategy can specifically include Per Channel quantization, Per Tensor quantization, etc., and the quantization algorithm strategy has been described in detail in the foregoing. The quantization offset can specifically include a zero point zeropoint, which is used to move the scaled data to another position, or in other words, to move from one interval to another interval, and is usually used for asymmetric quantization. The quantization coefficient can specifically include scale, etc., and the quantization coefficient corresponding to different quantization algorithm strategies is often different, and the quantization coefficient can be determined according to the actually determined quantization algorithm strategy.

[0135] The scheme provided by the embodiments of the present application aims at the defect that a model of a general layer type quantization scheme cannot be completely adapted when deployed on NPU hardware, and provides a scheme for deploying a quantization model on NPU, which can be used for NPU hardware adaptive quantization attribute deployment with graph fusion. For NPU hardware with graph fusion, the scheme constructs an NPU hardware graph fusion quantization attribute candidate template according to the topology structure information of the NPU hardware, performs hardware quantization template matching on the model to be deployed, selects a candidate template with a limited path length to perform matching, determines the optimal quantization attribute template for each OP, and determines the optimal operator quantization attribute for each OP. Then, the quantization deployment is performed based on the optimal hardware quantization attribute information of each OP, and then the quantization algorithm level can be optimized without changing the labeled hardware quantization attribute, and finally the hardware adaptive quantization model is obtained. Since the scheme provided by the embodiments of the present application does not affect the NPU hardware pipeline and does not interrupt the hardware graph fusion information, the efficiency can be ensured. Under the premise of ensuring the efficiency, the hardware adaptive quantization and optimization of the model are performed to ensure the optimal efficiency, which can help the algorithm effect to meet the demand.

[0136] In order to solve the problems in the prior art, the embodiments of the present application also provide a device 60 for deploying a quantization model on an embedded neural network processor, as shown in Figure 6 The device 60 comprises:

[0137] An acquisition module 61 acquires topology structure information of an embedded neural network processor NPU, wherein the topology structure information comprises a hardware connection relationship between a plurality of functional modules in the NPU and operator quantization attributes supported by each functional module;

[0138] A generation module 62 generates a plurality of pipeline modules based on the topology structure information, wherein any one of the pipeline modules comprises at least one functional module for implementing a model operator function and an operator quantization attribute of the functional module in the corresponding pipeline module;

[0139] A determination module 63 determines a matching relationship between each node in a network structure of a model to be deployed and a functional module in a pipeline module, wherein a topology connection relationship of any one node in the network structure is consistent with a hardware connection relationship of the matched functional module in the corresponding pipeline module;

[0140] A deployment module 64 quantizes and deploys the model to be deployed to the NPU according to the operator quantization attribute of the pipeline module matched with each operator in the model to be deployed.

[0141] The device provided by the embodiment of the present application firstly acquires the topology structure information of the embedded neural network processor NPU, and the topology structure information includes the hardware connection relationship between the plurality of function modules in the NPU and the operator quantization attribute supported by each function module. Then, the plurality of pipeline modules are generated based on the topology structure information, and any one of the pipeline modules includes at least one function module for implementing the function of the model operator and the operator quantization attribute of the function module in the corresponding pipeline module. Wherein, the pipeline module can represent the pipeline function that can be implemented by the hardware module, and the operator quantization attribute applicable to the function module in the corresponding pipeline module. Subsequently, the matching relationship between each node in the network structure of the to-be-deployed model and the function module in the pipeline module is determined, wherein the topology connection relationship of any one node in the network structure is consistent with the hardware connection relationship of the matched function module in the corresponding pipeline module. Finally, the to-be-deployed model is quantitatively deployed to the NPU according to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model. Since the operator quantization attribute used in the quantitative deployment is determined in advance according to the hardware function of the hardware module, and is applicable to the pipeline module to which the hardware module belongs, the model performing the quantitative deployment based on the operator quantization attribute can adapt to the actual function of the hardware module in the NPU, ensure that the node function of the model can be effectively supported by the hardware module, avoid the conflict between the model function and the hardware performance, and effectively improve the adaptation degree of the quantitative deployment of the neural network model on the embedded neural network processor.

[0142] In the device provided by the embodiment of the present application, the above-mentioned modules can also realize the method steps provided by the method embodiments. Alternatively, the device provided by the embodiment of the present application can also include other modules in addition to the above-mentioned modules to realize the method steps provided by the method embodiments. And the device provided by the embodiment of the present application can achieve the technical effects that the method embodiments can achieve.

[0143] Preferably, the embodiment of the present application further provides an electronic device, including a processor, a memory, a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the processes of the above-mentioned method embodiments for deploying the quantized model of the embedded neural network processor are realized, and the same technical effects can be achieved. To avoid repetition, it will not be repeated here.

[0144] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the method for deploying a quantized model on an embedded neural network processor, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The computer readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0145] The embodiment of the present application further provides a computer program product, which comprises a non-transitory computer readable storage medium storing a computer program. The computer program is operable to cause a computer to perform some or all of the steps of the method for deploying a quantized model on an embedded neural network processor, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0146] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. In addition, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0147] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0148] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0149] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0150] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0151] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0152] Computer readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that is accessible to a computing device. According to the definition provided herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," and any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the identified element.

[0154] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0155] The above descriptions are only some embodiments of the present application and are not intended to limit the present application. Various modifications and changes can be made by those skilled in the art to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for deploying a quantized model to an embedded neural network processor, the method comprising: The method comprises the following steps: obtaining topology information of an embedded neural network processor NPU, wherein the topology information comprises a hardware connection relationship between a plurality of functional modules in the NPU and an operator quantization attribute supported by each functional module; generating a plurality of pipeline modules based on the topology information, wherein any one of the pipeline modules comprises at least one functional module for implementing a model operator function and an operator quantization attribute of the functional module in the corresponding pipeline module; determining a matching relationship between each node in a network structure of a to-be-deployed model and a functional module in a pipeline module, wherein a topological connection relationship of any one node in the network structure is consistent with a hardware connection relationship of the matched functional module in the corresponding pipeline module; quantizing and deploying the to-be-deployed model to the NPU according to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model.

2. The method of claim 1, wherein, Generating a plurality of pipeline modules based on the topology information comprises: determining a pipeline connected from an input end of the NPU to an output end of the NPU based on the topology information, wherein the pipeline comprises at least one linearly connected functional module; determining an operator quantization attribute of the functional module in the pipeline based on the hardware connection relationship; generating a pipeline module based on the functional module in the pipeline and the operator quantization attribute of the functional module in the pipeline.

3. The method of claim 2, wherein, Determining a matching relationship between each node in a network structure of a to-be-deployed model and a functional module in a pipeline module comprises: sorting a plurality of pipeline modules in an order from more to less according to the number of contained functional modules; sequentially searching for a matched structure subgraph of the pipeline module in the network structure of the to-be-deployed model according to the sorting, wherein a topological structure relationship of any one structure subgraph is consistent with a hardware connection relationship of the matched pipeline module; determining a matching relationship between each node in the network structure and the functional module in the pipeline module according to the matching relationship between the pipeline module and the structure subgraph.

4. The method of claim 3, wherein, Sequentially searching for a matched structure subgraph of the pipeline module in the network structure of the to-be-deployed model according to the sorting comprises: selecting a target pipeline module to be matched according to the sorting; sequentially searching for a matched node of the functional module in the network structure according to a pipeline order of the target pipeline module; if a topological structure relationship of at least one target structure subgraph in the network structure is consistent with a hardware connection relationship of the target pipeline module, then constructing a matching relationship between each target structure subgraph and the target pipeline module.

5. The method according to any one of claims 1 to 4, characterized in that, After quantizing and deploying the to-be-deployed model to the NPU according to the operator quantization attribute of the pipeline module matched with each operator in the to-be-deployed model, the method further comprises: inputting a test sample in a test set into the deployed model to determine a quantization effect of the deployed model according to a matching degree of a first test result output by the deployed model and a test label corresponding to the test sample in the test set; and / or, The test samples in the test set are respectively input into the to-be-deployed model and the deployed model to determine the quantization effect of the deployed model according to the matching degree of the second test result output by the to-be-deployed model and the third test result output by the deployed model.

6. The method according to any one of claims 1 to 4, characterized in that, The operator quantization attribute includes at least one of: a quantization bit width, a quantization rounding rule, a quantization algorithm strategy, a quantization offset, and a quantization coefficient.

7. An apparatus for deploying a quantized model to an embedded neural network processor, the apparatus comprising: The method comprises: an acquisition module, which acquires topology structure information of an embedded neural network processor (NPU), the topology structure information including hardware connection relationships between a plurality of functional modules in the NPU and operator quantization attributes supported by each functional module; a generation module, which generates a plurality of pipeline modules based on the topology structure information, any one of the pipeline modules including at least one functional module for implementing model operator functions and operator quantization attributes of the functional modules in the corresponding pipeline module; a determination module, which determines matching relationships between each node in a network structure of a to-be-deployed model and functional modules in pipeline modules, wherein the topology connection relationships of any one node in the network structure are consistent with the hardware connection relationships of the matched functional modules in the corresponding pipeline module; a deployment module, which quantizes and deploys the to-be-deployed model to the NPU according to the operator quantization attributes of the pipeline modules matched with each operator in the to-be-deployed model.

8. An electronic device, comprising: The method comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises a non-transitory computer readable storage medium storing a computer program, the computer program being operable to cause a computer to perform some or all of the steps of the method according to any one of claims 1 to 6.

Images