Model optimization method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a model optimization method, device, electronic device and storage medium, which relate to the field of machine learning technology. The method includes: obtaining the operator specifications of each operator contained in the deep learning model before optimization; based on the obtained operator specifications, determining the operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed from the operators contained in the deep learning model before optimization as over-specification operators; according to the operator specifications of each over-specification operator, generating the optimization operator corresponding to the over-specification operator; the optimization operator corresponding to an over-specification operator includes the public operators and extended operators supported by the neural processing unit; replacing each over-specification operator with the corresponding optimization operator to obtain the optimized deep learning model; the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model. In this way, the normal operation of the deep learning model on the hardware platform can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model optimization method, device, electronic device and storage medium. Background Art

[0002] In the field of machine learning technology, a deep learning model can be represented as multiple operators and the execution order of the multiple operators. Among them, the operator represents the basic unit for building a deep learning model. For example, operators include: convolution operators, pooling operators, fully connected operators, etc. Accordingly, developers can build a deep learning model that meets the requirements by setting operators and the connection relationship between operators, and then deploy the built deep learning model to the hardware platform.

[0003] However, due to the resource limitations of the hardware platform, the NPU (Neural Network Processing Unit) in the hardware platform can usually only support some fixed-specification operators. Therefore, the deep learning model may involve operators that are not supported by the NPU (which can be called over-specification operators), which in turn makes it impossible for the deep learning model containing over-specification operators to run normally on the hardware platform. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a model optimization method, device, electronic device and storage medium to ensure the normal operation of the deep learning model on the hardware platform. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a model optimization method is first provided, the method comprising:

[0006] Obtain the operator specifications of each operator included in the deep learning model before optimization;

[0007] Based on the obtained operator specifications, determine, from the operators included in the deep learning model before the optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators;

[0008] According to the operator specification of each over-specification operator, an optimization operator corresponding to the over-specification operator is generated; wherein the optimization operator corresponding to an over-specification operator includes public operators and extended operators supported by the neural processing unit, and the overall operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator;

[0009] Each over-specification operator in the deep learning model before optimization is replaced with a corresponding optimized operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

[0010] In some embodiments, based on the acquired operator specifications, determining, from the operators included in the deep learning model before optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators, includes:

[0011] For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, determining that the operator is an over-specification operator;

[0012] If the operator type of the operator exists among the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist among the operator specifications supported by the neural processing unit, then the operator is determined to be an over-specification operator.

[0013] In some embodiments, generating an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator includes:

[0014] For each over-specification operator, if the operator specification supported by the neural processing unit is inconsistent with the operator specification of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels, then according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, a first extended operator is generated; wherein the number of output channels of the first extended operator is the difference, and the first extended operator is consistent with the over-specification operator in other specification parameters except the number of output channels;

[0015] Generate a first public operator and a second extended operator; wherein the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the over-specification operator;

[0016] The first extended operator, the first public operator and the second extended operator are combined to obtain an optimized operator corresponding to the super-specification operator.

[0017] In some embodiments, combining the first extended operator, the first public operator, and the second extended operator to obtain an optimized operator corresponding to the over-specification operator includes:

[0018] Merging the first expansion operator and the second expansion operator to obtain a merged expansion operator;

[0019] The combined extended operator and the first public operator are combined to obtain an optimized operator corresponding to the over-specification operator.

[0020] In some embodiments, generating an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator includes:

[0021] For each over-specification operator, determine the operator specification closest to the operator specification of the over-specification operator from the operator specifications supported by the neural processing unit; wherein the operator specification closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in all specification parameters, and relative to other operator specifications supported by the neural processing unit, the operator specification of the over-specification operator has the largest number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator;

[0022] Generate a second public operator corresponding to the over-specification operator according to the operator specification that is closest to the operator specification of the over-specification operator;

[0023] Generate a third extended operator corresponding to the over-specification operator according to a difference between the operator specification of the over-specification operator and the operator specification of the second public operator corresponding to the over-specification operator;

[0024] The generated second public operator and third extended operator are combined to obtain an optimized operator corresponding to the super-specification operator.

[0025] In some embodiments, for each out-of-specification operator, determining, from operator specifications supported by the neural processing unit, an operator specification that is closest to the operator specification of the out-of-specification operator includes:

[0026] For each over-specification operator, if the over-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the neural processing unit, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, then the product of the step size of the over-specification operator and the current coefficient is calculated; wherein the initial value of the coefficient is 1;

[0027] Calculate the difference between the convolution kernel size of the oversized operator and the current product;

[0028] Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference value among the operator specifications of the convolution operator supported by the neural processing unit;

[0029] If it exists, the target operator specification is determined to be the operator specification that is closest to the operator specification of the super-specification operator;

[0030] If it does not exist, the current coefficient is increased by 1, and the step of calculating the product of the step size of the over-specification operator and the current coefficient is returned.

[0031] In some embodiments, for each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels, convolution kernel size, step size, padding size, and dilation rate;

[0032] If the operator is a fully connected operator, the operator specification of the operator includes the following parameters: the dimension of the weight matrix and the number of output channels;

[0033] If the operator is a pooling operator, the operator specification of the operator includes the following parameters: pooling kernel size, step size, and padding size.

[0034] In some embodiments, the method further comprises:

[0035] receiving data to be processed;

[0036] The data to be processed is input into the optimized deep learning model, so as to run the extended operator in the optimized deep learning model in the central processing unit, and the public operator in the optimized deep learning model in the neural processing unit, so as to process the data to be processed and obtain a processing result.

[0037] A second aspect of the embodiments of the present application provides a model optimization device, the device comprising:

[0038] A specification acquisition module is used to obtain the operator specifications of each operator included in the deep learning model before optimization;

[0039] An over-specification operator determination module, for determining, based on the acquired operator specifications, from the operators included in the deep learning model before optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators;

[0040] An optimization operator generation module, used to generate an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator; wherein the optimization operator corresponding to an over-specification operator includes public operators and extended operators supported by the neural processing unit, and the overall operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator;

[0041] An operator replacement module is used to replace each over-specification operator in the deep learning model before optimization with a corresponding optimized operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

[0042] In some embodiments, the over-specification operator determination module is specifically used to:

[0043] For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, determining that the operator is an over-specification operator;

[0044] If the operator type of the operator exists among the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist among the operator specifications supported by the neural processing unit, then the operator is determined to be an over-specification operator.

[0045] In some embodiments, the optimization operator generation module includes:

[0046] A first generating submodule is used for generating, for each over-specification operator, a first extended operator according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, if the operator specifications supported by the neural processing unit are inconsistent with the operator specifications of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels; wherein the number of output channels of the first extended operator is the difference, and the first extended operator is consistent with the over-specification operator in other specification parameters except the number of output channels;

[0047] A second generating submodule, configured to generate a first public operator and a second extended operator; wherein the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the over-specification operator;

[0048] The first optimization operator acquisition submodule is used to combine the first extended operator, the first public operator and the second extended operator to obtain the optimization operator corresponding to the super-specification operator.

[0049] In some embodiments, the first optimization operator acquisition submodule is specifically used to:

[0050] Merging the first expansion operator and the second expansion operator to obtain a merged expansion operator;

[0051] The combined extended operator and the first public operator are combined to obtain an optimized operator corresponding to the over-specification operator.

[0052] In some embodiments, the optimization operator generation module includes:

[0053] a specification determination submodule, for determining, for each over-specification operator, from the operator specifications supported by the neural processing unit, an operator specification that is closest to the operator specification of the over-specification operator; wherein the operator specification that is closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in terms of each specification parameter, and relative to other operator specifications supported by the neural processing unit, the operator specification of the over-specification operator has the largest number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator;

[0054] A third generating submodule, configured to generate a second public operator corresponding to the over-specification operator according to an operator specification that is closest to the operator specification of the over-specification operator;

[0055] A fourth generating submodule, configured to generate a third extended operator corresponding to the over-specification operator according to a difference between an operator specification of the over-specification operator and an operator specification of a second public operator corresponding to the over-specification operator;

[0056] The second optimization operator acquisition submodule is used to combine the generated second public operator and the third extended operator to obtain the optimization operator corresponding to the super-specification operator.

[0057] In some embodiments, the specification determination submodule is specifically used to:

[0058] For each over-specification operator, if the over-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the neural processing unit, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, then the product of the step size of the over-specification operator and the current coefficient is calculated; wherein the initial value of the coefficient is 1;

[0059] Calculate the difference between the convolution kernel size of the oversized operator and the current product;

[0060] Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference value among the operator specifications of the convolution operator supported by the neural processing unit;

[0061] If it exists, the target operator specification is determined to be the operator specification that is closest to the operator specification of the super-specification operator;

[0062] If it does not exist, the current coefficient is increased by 1, and the step of calculating the product of the step size of the over-specification operator and the current coefficient is returned.

[0063] In some embodiments, for each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels, convolution kernel size, step size, padding size, and dilation rate;

[0064] If the operator is a fully connected operator, the operator specification of the operator includes the following parameters: the dimension of the weight matrix and the number of output channels;

[0065] If the operator is a pooling operator, the operator specification of the operator includes the following parameters: pooling kernel size, step size, and padding size.

[0066] In some embodiments, the apparatus further comprises:

[0067] A data receiving module, used for receiving data to be processed;

[0068] A data processing module is used to input the data to be processed into the optimized deep learning model, so as to run the extended operators in the optimized deep learning model in the central processing unit, and to run the public operators in the optimized deep learning model in the neural processing unit, so as to process the data to be processed and obtain processing results.

[0069] According to a third aspect of the embodiments of the present application, an electronic device is provided, including:

[0070] Memory, used to store computer programs;

[0071] The processor is used to implement any of the above-mentioned model optimization methods when executing the program stored in the memory.

[0072] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned model optimization methods is implemented.

[0073] According to another aspect of the embodiments of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned model optimization methods.

[0074] Beneficial effects of the embodiments of the present application:

[0075] An embodiment of the present application provides a model optimization method, the method comprising: obtaining the operator specification of each operator contained in a deep learning model before optimization; based on the obtained operator specifications, determining, from the operators contained in the deep learning model before optimization, operators that are not supported by a neural processing unit in a hardware platform of the model to be deployed as over-specification operators; generating an optimization operator corresponding to the over-specification operator according to the operator specification of each over-specification operator; wherein the optimization operator corresponding to an over-specification operator includes public operators and extended operators supported by the neural processing unit, and the overall operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator; replacing each over-specification operator in the deep learning model before optimization with the corresponding optimization operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

[0076] Based on the above processing, for the deep learning model that needs to be optimized currently (i.e., the deep learning model before optimization), the electronic device can obtain the operator specifications of each operator contained in the deep learning model before optimization. Since the neural processing unit (i.e., NPU) in the hardware platform of the model to be deployed usually can only support some fixed-specification operators, it is possible to determine the over-specification operators contained in the deep learning model before optimization based on the operator specifications of each operator obtained, that is, the operators not supported by the NPU in the above hardware platform. Accordingly, for each over-specification operator, the optimized operator corresponding to the over-specification operator (including the public operators supported by the NPU and the extended operators running on the central processing unit) can be generated according to the operator specifications of the over-specification operator. It is understandable that the operator specifications of the public operators contained in the optimization operator corresponding to an over-specification operator must be different from the operator specifications of the over-specification operator, that is, there is a difference between the operation logic of the public operator and the operation logic of the over-specification operator. Therefore, in order to ensure that the operation logic of the over-specification operator is fully implemented during the execution of the subsequent model, an extended operator can be generated to compensate for the difference to ensure that the total operation logic of the optimization operator corresponding to an over-specification operator is consistent with the operation logic of the over-specification operator. Furthermore, the electronic device can replace each over-specification operator in the deep learning model before optimization with the corresponding optimization operator to obtain the optimized deep learning model. Subsequently, the hardware platform can deploy the optimized deep learning model, and during the operation of the optimized deep learning model, the central processing unit (CPU, Central Processing Unit) in the hardware platform is used to run the extended operator in the optimized deep learning model, and the NPU in the hardware platform is used to run other operators in the optimized deep learning model except the extended operator. In this way, the normal operation of the optimized deep learning model on the hardware platform can be guaranteed.

[0077] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0079] Figure 1 A first flow chart of the model optimization method provided in the embodiment of the present application;

[0080] Figure 2 An example diagram of a deep learning model calculation diagram provided in an embodiment of the present application;

[0081] Figure 3 A second flow chart of the model optimization method provided in the embodiment of the present application;

[0082] Figure 4 A third flow chart of the model optimization method provided in the embodiment of the present application;

[0083] Figure 5 A schematic diagram of the operation logic of the N2 operator provided in this application;

[0084] Figure 6 A schematic diagram of the operation logic of the second public operator corresponding to the N2 operator provided in this application;

[0085] Figure 7 A schematic diagram of the operation logic of the NX operator provided in this application;

[0086] Figure 8 Schematic diagram of the difference between the hardware platform provided in the embodiment of the present application executing the over-specification operator;

[0087] Fig. 9 A schematic diagram of replacing an over-specification operator provided in an embodiment of the present application;

[0088] Fig.10 A schematic diagram of a process for determining an over-specification operator in a deep learning model provided in an embodiment of the present application;

[0089] Fig.11 A schematic diagram of a process for determining an optimization operator corresponding to an over-specification operator provided in an embodiment of the present application;

[0090] Fig.12 A structural diagram of a model optimization device provided in an embodiment of the present application;

[0091] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0093] In the field of machine learning technology, a deep learning model can be represented as multiple operators and the execution order of the multiple operators. Among them, the operator represents the basic unit for building a deep learning model. For example, operators include: convolution operators, pooling operators, fully connected operators, etc. Accordingly, developers can build a deep learning model that meets the requirements by setting operators and the operator connection relationship between operators, and then deploy the built deep learning model to the hardware platform.

[0094] Developers can use preset deep learning frameworks (for example, PyTorch, TensorFlow, etc.) to build deep learning models. The deployment file of the built deep learning model contains the code corresponding to each operator in the deep learning model.

[0095] Correspondingly, the constructed deep learning model can be represented in a graphical way, and the graphical representation result of a deep learning model can be called the calculation graph of the deep learning model. In other words, the calculation graph of a deep learning model is composed of multiple operators with operator connection relationships. Among them, an operator corresponds to an operation in the model execution process; the operator connection relationship between operators represents: the data processing flow between the two operators.

[0096] In the present application, the deep learning model may be a target detection model. Accordingly, after the target detection model is deployed on the hardware platform, the acquired image can be processed by running the target detection model to detect the target in the image.

[0097] Alternatively, the deep learning model can also be a classification model. Correspondingly, after the hardware platform deploys the classification model, the input data can be classified by running the classification model to obtain the classification result of the input data. In this application, the format of the input data is not limited, such as the input data can be an image, text, or audio.

[0098] However, due to the resource limitations of the hardware platform, the NPU (Neural Network Processing Unit) in the hardware platform can usually only support some fixed-specification operators. Therefore, the deep learning model may involve operators that are not supported by the NPU (which can be called over-specification operators), which in turn makes it impossible for the deep learning model containing over-specification operators to run normally on the hardware platform.

[0099] In order to solve the above problems, an embodiment of the present application provides a model optimization method, which can be applied to electronic devices.

[0100] For example, the electronic device may be a hardware platform for the model to be deployed. Accordingly, after obtaining the deep learning model constructed by the developer (i.e., the deep learning model before optimization), the hardware platform may optimize the deep learning model before optimization according to the model optimization method provided in this application to obtain the optimized deep learning model, and thus, the optimized deep learning model may be run in the hardware platform.

[0101] Alternatively, the electronic device can also be the server of the aforementioned model development tool. After obtaining the calculation graph drawn by the developer, the server can optimize the deep learning model represented by the calculation graph according to the model optimization method provided in this application to obtain an optimized deep learning model, and subsequently deploy the optimized deep learning model on the hardware platform.

[0102] See also Figure 1 , Figure 1 The first flow chart of the model optimization method provided in the embodiment of the present application comprises the following steps:

[0103] S101: Obtain operator specifications of each operator included in the deep learning model before optimization.

[0104] S102: Based on the obtained operator specifications, determine, from the operators included in the deep learning model before optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators.

[0105] S103: Generate an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator.

[0106] Among them, the optimization operator corresponding to an over-specification operator includes public operators and extended operators supported by the neural processing unit, and the overall operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator.

[0107] S104: Replace each over-specification operator in the deep learning model before optimization with a corresponding optimization operator to obtain an optimized deep learning model.

[0108] Among them, the central processing unit in the hardware platform is used to run the extended operators in the optimized deep learning model.

[0109] Based on the above processing, for the deep learning model that needs to be optimized currently (i.e., the deep learning model before optimization), the electronic device can obtain the operator specifications of each operator contained in the deep learning model before optimization. Since the neural processing unit (i.e., NPU) in the hardware platform of the model to be deployed can usually only support some fixed-specification operators, it is possible to determine the over-specification operators contained in the deep learning model before optimization based on the operator specifications of each operator obtained, that is, the operators not supported by the NPU in the above hardware platform. Accordingly, for each over-specification operator, the optimized operator corresponding to the over-specification operator (including the public operators supported by the NPU and the extended operators running on the CPU) can be generated according to the operator specifications of the over-specification operator. It is understandable that the operator specifications of the public operators contained in the optimization operator corresponding to an over-specification operator must be different from the operator specifications of the over-specification operator, that is, there is a difference between the operation logic of the public operator and the operation logic of the over-specification operator. Therefore, in order to ensure that the operation logic of the over-specification operator is fully implemented during the execution of the subsequent model, an extended operator can be generated to compensate for the difference to ensure that the total operation logic of the optimization operator corresponding to an over-specification operator is consistent with the operation logic of the over-specification operator. Furthermore, the electronic device can replace each over-specification operator in the deep learning model before optimization with the corresponding optimization operator to obtain the optimized deep learning model. Subsequently, the hardware platform can deploy the optimized deep learning model, and during the operation of the optimized deep learning model, the central processing unit (i.e., CPU) in the hardware platform is used to run the extended operator in the optimized deep learning model, and the NPU in the hardware platform is used to run other operators in the optimized deep learning model except the extended operator. In this way, the normal operation of the optimized deep learning model on the hardware platform can be guaranteed.

[0110] With respect to step S101, the deep learning model before optimization can be any model that needs to be deployed on a hardware platform, wherein the specific type of the deep learning model is determined by the actual needs of the user.

[0111] For example, during the production process of a product, if the user needs to use a model to process a production video of the product to detect defective products, the deep learning model may be a target detection model. Specifically, the deep learning model may be a YOLO (You Only Look Once) model.

[0112] For another example, in an environment classification scenario, if the user needs to use a model to classify the acquired images according to the scene, the deep learning model may be a classification model. Specifically, the deep learning model may be a Residual Neural Network (ResNet) model.

[0113] A deep learning model usually contains operators of multiple operator types, such as convolution operators, pooling operators, fully connected operators, activation operators, etc. There is an operator connection relationship between the multiple operators, and two operators with an operator connection relationship will be executed in sequence. That is, for any model, the model can be represented by determining the operators contained in the model and the operator connection relationship between the operators.

[0114] For example, see Figure 2 , Figure 2 An example diagram of a deep learning model calculation graph provided in an embodiment of the present application. Figure 2 The deep learning model before optimization shown in the figure contains: 3 convolution operators (i.e., N1, N2, and N3), 1 pooling operator (i.e., N4), 1 activation operator (i.e., N5), and 1 fully connected operator (i.e., N6). Among them, an operator (called the first operator) points to another operator (called the second operator), which means that there is an operator connection relationship between the first operator and the second operator, and the output data of the first operator is the input data of the second operator.

[0115] Correspondingly, for each operator, various specification parameters (also called hyperparameters) of the operator can be used to represent the operator specification of the operator.

[0116] In one implementation, for operators of the same operator type, the specification parameters corresponding to the operators of the operator type are fixed. Accordingly, for each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels (num_output), convolution kernel size (kernel), stride, padding size (pad), and dilation rate (dilation rate).

[0117] If the operator is a fully connected operator, the specification parameters in the operator specification of the operator include: the dimension of the weight matrix and the number of output channels.

[0118] If the operator is a pooling operator, the operator specification of the operator includes the following parameters: pooling kernel size (kernel), stride (stride), and padding size (pad).

[0119] In an embodiment of the present application, for any operator in a deep learning model (i.e., the deep learning model before optimization and the deep learning model after optimization), the operator specification of the operator is represented as the parameter values ​​of each specification parameter corresponding to the operator.

[0120] Correspondingly, if an operator is a convolution operator, the operator specification of the operator can be expressed as: {number of output channels: ...; convolution kernel size: ...; step size: ...; padding size: ...; expansion rate: ...}.

[0121] For example, the above Figure 2 The operator specification of the N1 operator in can be expressed as:

[0122] N1{output channels: 2500; convolution kernel size: 6×6; step size: 2; padding size: 0; dilation rate: 1};

[0123] Above Figure 2 The operator specification of the N2 operator in can be expressed as:

[0124] N2{number of output channels: 2048; convolution kernel size: 6×6; step size: 2; padding size: 3; dilation rate: 1}.

[0125] Above Figure 2 The operator specification of the N3 operator in can be expressed as:

[0126] N3{Number of output channels: 2048; Convolution kernel size: 4×4; Step size: 2; Padding size: 0; Dilation rate: 1}.

[0127] If an operator is a fully connected operator, the operator specification of the operator can be expressed as: {dimension of the weight matrix: ...; number of output channels: ...}.

[0128] For example, the above Figure 2 The operator specification of the N6 operator in can be expressed as: N4{dimension of weight matrix: 8×8; number of output channels: 8}.

[0129] If an operator is a pooling operator, the operator specification of the operator can be expressed as: {pooling kernel size: ...; step size: ...}.

[0130] For example, the above Figure 2 The operator specification of the N4 operator in can be expressed as:

[0131] N4{pooling kernel size: 4×4; stride: 2; padding size: 0}.

[0132] Based on the above processing, for each operator, the operator specification of the operator can be represented by each specification parameter of the operator. Subsequently, the electronic device can compare the operator specification of each operator included in the deep learning model before optimization with the operator specification supported by the neural processing unit in the hardware platform to determine the over-specification operator.

[0133] It is understandable that, for each operator included in the deep learning model before optimization, the operator specification of the operator is pre-set by the developer in the process of building the deep learning model. Accordingly, the electronic device can obtain the operator specification of each operator included in the deep learning model before optimization.

[0134] For example, for the above Figure 2 The deep learning model shown in the figure, the operator specification of each operator contained in the deep learning model before optimization can be recorded as:

[0135] N1{output channels: 2500; convolution kernel size: 6×6; step size: 2; padding size: 0; dilation rate: 1};

[0136] N2{output channels: 2048; convolution kernel size: 6×6; step size: 2; padding size: 3; dilation rate: 1};

[0137] N3{output channels: 2048; convolution kernel size: 4×4; step size: 2; padding size: 0; dilation rate: 1};

[0138] N4{pooling kernel size: 4×4; stride: 2; padding size: 0};

[0139] N6{Dimension of weight matrix: 8×8; Number of output channels: 8}.

[0140] For step S102, the NPU in the hardware platform supports an operator, which means that the NPU in the hardware platform supports the operator type of the operator, and for the operator type, the NPU in the hardware platform also supports the operator specification of the operator. The operator supported by the NPU in the hardware platform can be called a public operator.

[0141] The electronic device may record in advance the operator types supported by the neural processing unit in the hardware platform of the model to be deployed, and the operator specifications supported by the neural processing unit for each operator type. For example, the electronic device may record the operator specifications supported by the neural processing unit in the hardware platform of the model to be deployed in a database (which may be referred to as a hardware operator library).

[0142] It is understandable that with the development of computer technology, the hardware platform can also be updated, such as replacing the processing unit in the hardware platform with the latest model of processing unit. In this way, the operators currently supported by the hardware platform will also be updated. Correspondingly, the operator specifications supported by the neural processing unit in the hardware platform of the model to be deployed recorded by the electronic device can also be updated synchronously.

[0143] For example, the hardware operator library can also be updated as the hardware platform is updated. The device used to update the hardware operator library can be an electronic device for model optimization, in which case the electronic device can directly update the local hardware operator library; or, the device used to update the hardware operator library can also be other devices, and after the hardware operator library is updated by other devices, the electronic device directly obtains the latest hardware operator library from other devices. This application does not limit this.

[0144] Correspondingly, after obtaining the operator specifications of each operator contained in the deep learning model before optimization, the operator specifications of each operator contained in the deep learning model before optimization can be compared with the operator specifications supported by the neural processing unit in the hardware platform, so as to determine the operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed from the operators contained in the deep learning model before optimization as over-specification operators.

[0145] In some embodiments, see Figure 3 , Figure 3 A second flow chart of the model optimization method provided in the embodiment of the present application, in Figure 1 On the basis of, step S102 comprises:

[0146] S1021: For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, the operator is determined to be an out-of-specification operator.

[0147] S1022: For each operator, if the operator type of the operator exists in the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist in the operator specifications supported by the neural processing unit, then the operator is determined to be an over-specification operator.

[0148] In an embodiment of the present application, for each operator included in the deep learning model before optimization, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, the electronic device may determine that the operator is an over-specification operator.

[0149] For example, for the above Figure 2The deep learning model shown and the operator specifications of each operator obtained above, if there is no pooling operator in the operator types supported by the NPU in the hardware platform, the electronic device can determine the pooling operator (N3) in the deep learning model as an over-specification operator.

[0150] For another example, if a deep learning model before optimization contains a custom operator set by a developer, and the operator type of the custom operator is inconsistent with the operator type supported by the NPU, the custom operator can be determined as an over-specification operator.

[0151] For each operator included in the deep learning model before optimization, if the NPU supports the operator type of the operator, but the NPU in the hardware platform does not support the operator specification of the operator for the operator type, then the operator is determined to be an over-specification operator.

[0152] For example, for the above Figure 2 The deep learning model shown and the operator specifications of each operator obtained above, if the NPU in the hardware platform supports: public operator 1, public operator 2, public operator 3.

[0153] Among them, public operator 1 represents: the convolution operator with operator specifications of {output channels: 2048; convolution kernel size: 4×4; step size: 2; padding size: 0; dilation rate: 1}. Public operator 2 represents: the convolution operator with operator specifications of {output channels: 2048; convolution kernel size: 2×2; step size: 2; padding size: 2; dilation rate: 1}. Public operator 3 represents: the convolution operator with operator specifications of {output channels: 2048; convolution kernel size: 4×4; step size: 2; padding size: 0; dilation rate: 1}.

[0154] It can be seen that for the above Figure 2 For N1 in the deep learning model shown, the operator type (convolution) of this operator exists in the operator types supported by the NPU, and for the convolution operator, the operator specification of N1 does not exist in the operator specifications supported by the neural processing unit. Therefore, the electronic device can determine that N1 is an over-specification operator.

[0155] Regarding the above Figure 2 For N2 in the deep learning model shown, the operator type (convolution) of this operator exists in the operator types supported by the NPU, and for the convolution operator, the operator specification of N2 does not exist in the operator specifications supported by the neural processing unit. Therefore, the electronic device can determine that N2 is an over-specification operator.

[0156] Regarding the above Figure 2For N3 in the deep learning model shown, the operator type (convolution) of this operator exists in the operator types supported by the NPU, and for the convolution operator, the operator specification of N3 exists in the operator specifications supported by the neural processing unit, that is, N3 is an operator supported by the NPU in the hardware platform.

[0157] Based on the above processing, for each operator contained in the deep learning model before optimization, it is possible to determine whether the operator is an operator supported by the NPU in the hardware platform through the two aspects of the operator type and operator specification of the operator. When the operator type of the operator is not an operator type supported by the NPU, it can be directly determined that the operator is an over-specification operator. When the operator type of the operator is an operator type supported by the NPU, it is further determined whether the operator specification of the operator is supported by the NPU. In this way, the accuracy of the determined over-specification operator can be guaranteed and the efficiency of determining the over-specification operator can be improved.

[0158] In this application, the hardware platform can adopt a heterogeneous multi-core architecture of CPU (i.e., soft core) and NPU (i.e., hard core). Among them, the CPU computing power is usually lower than the NPU computing power. For example, the NPU computing power can reach several to dozens of times the CPU computing power. The CPU can be used to run operators that are not supported by the NPU in the deep learning model. Correspondingly, for each operator contained in a deep learning model, the operators supported by the NPU can be run through the NPU, and the operators not supported by the NPU can be run through the CPU. In this way, the efficiency of the operator operation can be improved on the basis of the normal execution of the deep learning model, which can also improve the efficiency of the model execution.

[0159] It is understandable that, for each over-specification operator included in the deep learning model before optimization, the over-specification operator can also be run by the CPU in the hardware platform.

[0160] However, since the CPU computing power is usually lower than the NPU computing power, if the over-specification operator is run through the CPU in the hardware platform, the operation efficiency of the operator will be low, and thus the efficiency of model execution will be reduced. In particular, when the over-specification operator is a computationally intensive operator such as a convolution operator or a fully connected operator, since a large amount of calculation is required during the operation of the computationally intensive operator, running the over-specification operator directly through the CPU will cause the operator to run much faster than other operators running on the NPU, resulting in a waste of NPU resources and may even cause the deep learning model to fail to execute.

[0161] Therefore, for each over-specification operator included in the deep learning model, the model optimization method provided in the present application can optimize the over-specification operator and reduce the part of the operation logic of the over-specification operator that needs to be processed by the CPU.

[0162] For step S103-step S104, for each over-specification operator included in the deep learning model before optimization, the electronic device can generate an optimization operator corresponding to the over-specification operator according to the operator specification of the over-specification operator. The optimization operator corresponding to an over-specification operator includes: public operators and extended operators supported by the NPU.

[0163] In this application, an extended operator refers to an operator that is run by a CPU. Generating an operator means that an electronic device generates code corresponding to the operator, and when the hardware platform runs the operator (i.e., executes the code corresponding to the operator) through an NPU or CPU, it can implement the operation logic represented by the operator type and operator specification of the operator.

[0164] In one implementation, when generating the code corresponding to the extension operator, the developer can customize the code corresponding to the extension operator through the feature extension interface reserved by the model development tool to implement the operation logic of the extension operator when running the extension operator on the hardware platform.

[0165] The generation process of the optimization operator corresponding to an over-specification operator will be described in subsequent embodiments.

[0166] In the process of the electronic device generating the optimization operator corresponding to each over-specification operator, it is necessary to make the overall operation logic of the optimization operator corresponding to the over-specification operator consistent with the operation logic of the over-specification operator.

[0167] In this way, it is possible to achieve equivalent replacement between the optimization operator corresponding to an over-specification operator and the over-specification operator, ensuring that for the same input data, the processing results obtained by the CPU executing the over-specification operator to process the input data are consistent with the processing results obtained by the NPU running the public operators included in the optimization operator corresponding to the over-specification operator and the CPU running the extended operators included in the optimization operators corresponding to each over-specification operator to process the input data.

[0168] Correspondingly, after generating the optimization operator corresponding to each over-specification operator, the electronic device can replace each over-specification operator in the deep learning model before optimization with the corresponding optimization operator to obtain the optimized deep learning model.

[0169] Subsequently, during the execution of the optimized deep learning model, the public operators included in the optimized operators corresponding to each over-specification operator can be run through the NPU in the hardware platform, and the extended operators included in the optimized operators corresponding to each over-specification operator can be run through the CPU.

[0170] In this way, it can be ensured that the execution effect of the optimized deep learning model is consistent with the deep learning model before optimization, thereby ensuring the realization of the model function of the deep learning model.

[0171] In this application, the process of generating an optimization operator corresponding to each over-specification operator may also be referred to as the over-specification operator segmentation process. Figure 4 , Figure 4 This is a third flow chart of the model optimization method provided in the embodiment of the present application. Figure 1 On the basis of, step S103 comprises:

[0172] S1031: For each over-specification operator, determine the operator specification that is closest to the operator specification of the over-specification operator from the operator specifications supported by the neural processing unit.

[0173] Among them, the operator specification closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in all specification parameters, and relative to other operator specifications supported by the neural processing unit, the number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator is the largest.

[0174] S1032: Generate a second public operator corresponding to the over-specification operator according to the operator specification that is closest to the operator specification of the over-specification operator.

[0175] S1033: Generate a third extended operator corresponding to the over-specification operator according to the difference between the operator specification of the over-specification operator and the operator specification of the second public operator corresponding to the over-specification operator.

[0176] S1034: Combine the generated second public operator and the third extended operator to obtain an optimized operator corresponding to the super-specification operator.

[0177] In the embodiment of the present application, for step S1031, for each over-specification operator, the electronic device can determine the operation logic of the over-specification operator according to the operator type and operator specification of the over-specification operator. Accordingly, the process of segmenting the over-specification operator is actually the process of splitting the operation logic of the over-specification operator according to mathematical rules.

[0178] It can be understood that, for the public operator and the third extended operator included in the optimization operator corresponding to an over-specification operator, the smaller the difference between the operation logic of the public operator included in the optimization operator and the operation logic of the over-specification operator, the lower the calculation amount corresponding to the third extended operator used to compensate for the difference in the operation logic.

[0179] Correspondingly, for an over-specification operator, an operator specification that is closest to the operator specification of the over-specification operator can be determined from operator specifications supported by the neural processing unit according to one of the following methods.

[0180] Method 1: Reduce the operator specifications while keeping the operator type unchanged.

[0181] For an over-specification operator and the operator type of the over-specification operator, the electronic device can determine, among the operator specifications supported by the NPU, the operator specification that is not greater than the operator specification of the over-specification operator in all specification parameters and has the largest number of identical specification parameters with the operator specification of the over-specification operator, as the operator specification that is closest to the operator specification of the over-specification operator.

[0182] In the embodiment of the present application, Figure 2 Take the over-specification operator N2 in the deep learning model shown as an example.

[0183] In this example, N2 is a convolution operator, and the operator specification of N2 is: N2{number of output channels: 2048; convolution kernel size: 6×6; step size: 2; padding size: 0; expansion rate: 1}.

[0184] For the convolution operator, the operator specifications supported by NPU include: {number of output channels: 2048; convolution kernel size: 5×5; step size: 2; padding size: 0; dilation rate: 1} (recorded as specification 1), {number of output channels: 2048; convolution kernel size: 4×4; step size: 2; padding size: 0; dilation rate: 1} (recorded as specification 2), {number of output channels: 2048; convolution kernel size: 3×3; step size: 1; padding size: 0; dilation rate: 1} (recorded as specification 3), and {number of output channels: 2048; convolution kernel size: 8×8; step size: 2; padding size: 3; dilation rate: 1} (recorded as specification 4).

[0185] For N2, in the above specifications 1-4, the convolution kernel size in the specification parameters of specification 4 is larger than the convolution kernel size in the specification parameters of the operator specifications of N2. Therefore, specification 4 can be excluded as the operator specification closest to the operator specification of N2. Furthermore, in specifications 1-3, the same specification parameters between the operator specifications of specification 1 and N2 include: the number of output channels, step size, padding size and dilation rate, that is, the number of the same specification parameters between the operator specifications of specification 1 and N2 is 4. The same specification parameters between the operator specifications of specification 2 and N2 include: the number of output channels, step size, padding size and dilation rate, that is, the number of the same specification parameters between the operator specifications of specification 1 and N2 is 4. The same specification parameters between the operator specifications of specification 3 and N2 include: the number of output channels and dilation rate, that is, the number of the same specification parameters between the operator specifications of specification 3 and N2 is 2. Therefore, an operator specification closest to the operator specification of N2 can be selected from specifications 1 and 2.

[0186] Method 2: The operator types are different, but the logical operation processes are similar or identical.

[0187] For an over-specification operator, the electronic device may determine the operator specification of an operator whose operation logic is similar to or the same as that of the over-specification operator from other operator types other than the operator type of the over-specification operator.

[0188] For example, if an over-specification operator is a fully connected operator, the operator specification of a convolution operator having a similar operation logic to the fully connected operator may be used as the operator specification that is closest to the operator specification of the over-specification operator.

[0189] For another example, if an over-specification operator is a global pooling operator, the operator specification of a local pooling operator having a similar operation logic to the global pooling operator may be used as the operator specification that is closest to the operator specification of the global pooling operator.

[0190] Accordingly, for an over-specification operator, the optimization operator corresponding to the over-specification operator generated according to the above method 1 may be inconsistent with the optimization operator corresponding to the over-specification operator generated according to the above method 2. Furthermore, the electronic device can calculate the computational amount of the extended operator running on the CPU in different optimization schemes, and generate the optimization operator corresponding to the over-specification operator according to the optimization scheme with a smaller computational amount. In this way, the amount of computation required to be performed by the CPU can be further reduced, and the execution efficiency of the operator can be improved.

[0191] In one implementation, step S1031 includes:

[0192] Step 1: For each over-specification operator, if the over-specification operator is a convolution operator, and there is no operator specification whose convolution kernel size is the same as the convolution kernel size of the over-specification operator among the operator specifications of the convolution operator supported by the neural processing unit, then calculate the product of the step size of the over-specification operator and the current coefficient. The initial value of the coefficient is 1.

[0193] Step 2: Calculate the difference between the convolution kernel size of the oversized operator and the current product.

[0194] Step 3: Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference among the operator specifications of the convolution operator supported by the neural processing unit.

[0195] Step 4: If it exists, determine the target operator specification as the operator specification that is closest to the operator specification of the super-specification operator.

[0196] Step 5: If it does not exist, add 1 to the current coefficient and return to step 1.

[0197] In the embodiment of the present application, the super-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the NPU, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the super-specification operator.

[0198] Accordingly, for an over-specification operator that meets the above conditions, the operator specification that is closest to the operator specification of the over-specification operator can be determined in a loop (performing steps 1 to 3 once can be called a loop). For an over-specification operator, when step 1 is executed for the first time, the current coefficient is 1.

[0199] During one loop, the electronic device can calculate the product of the step size of the over-specification operator and the current coefficient, and then calculate the difference between the convolution kernel size of the over-specification operator and the current product. Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference among the operator specifications of the convolution operators supported by the neural processing unit. If so, determine that the target operator specification is the operator specification that is closest to the operator specification of the over-specification operator. If not, add 1 to the current coefficient and return to execute step one (i.e., enter the next loop).

[0200] Continuing with the above example, N2 is a convolution operator, and the operator specification of N2 is: N2{output channels: 2048; convolution kernel size: 6×6; step size: 2; padding size: 3; dilation rate: 1}. Among the above specifications 1 to 4, there is no operator specification whose convolution kernel size is the same as the convolution kernel size of N2 (6×6).

[0201] When executing step 1 for the first time, the electronic device can calculate the product of the step size of N2 and the current coefficient (ie, 2×1=2). Then, the difference between the convolution kernel size of N2 and the current product is calculated, that is, 6-2=4.

[0202] Among the operator specifications of the convolution operator supported by the NPU, there is a target operator specification whose convolution kernel size is consistent with the current difference (4), and the target operator specification is specification 2 {output channel number: 2048; convolution kernel size: 4×4; step size: 2; padding size: 2; expansion rate: 1}. Accordingly, the electronic device can determine specification 2 as the operator specification closest to the operator specification of N2.

[0203] Based on the above processing, when the over-specification operator is a convolution operator, and in the operator specifications of the convolution operator supported by the NPU, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, a loop can be used to reduce the convolution kernel size according to the step size. In this way, the amount of calculation for subsequently generating the third extended operator corresponding to the over-specification operator can be reduced.

[0204] With respect to steps S1032 to S1033, after determining the operator specification closest to the operator specification of each over-specification operator, the electronic device may generate a public operator corresponding to the over-specification operator according to the operator specification closest to the operator specification of the over-specification operator. Furthermore, according to the difference between the operator specification of the over-specification operator and the operator specification of the second public operator corresponding to the over-specification operator, a third extended operator corresponding to the over-specification operator is generated.

[0205] Accordingly, the total operation logic of the optimization operator corresponding to an over-specification operator is consistent with the operation logic of the over-specification operator. That is, the total operation logic of the second public operator and the third extended operator included in the optimization operator corresponding to an over-specification operator is consistent with the operation logic of the over-specification operator.

[0206] The operation logic of an operator is determined based on the operator type and operator specification of the operator. Figure 2 The deep learning model contains N2, the operator type of N2 is convolution, and the operator specification of N2 is N2{number of output channels: 2048; convolution kernel size: 6×6; step size: 2; padding size: 0; expansion rate: 1}.

[0207] See also Figure 5 , Figure 5 A schematic diagram of the operation logic of the N2 operator provided in this application. Figure 5 In the example, the operation logic of a channel in the N2 operator is taken as an example (the operation logic in different channels is the same). The input feature map (InputFeaturemap) represents the input data of a channel in N2. The size of the input feature map is 8×8, and a11 to a88 correspond to the elements of a position in the input feature map. A11, A12, A21, and A22 in the input feature map correspond to a 6×6 rectangular pixel area. A11 represents: a rectangular pixel area with a16 and a61 as diagonal elements and a size of 6×6. A12 represents: a rectangular pixel area with a18 and a63 as diagonal elements and a size of 6×6. A21 represents: a rectangular pixel area with a36 and a81 as diagonal elements and a size of 6×6. A22 represents: a rectangular pixel area with a38 and a83 as diagonal elements and a size of 6×6.

[0208] The rectangular pixel area corresponding to B11 represents the complete convolution kernel, and b11 to b66 correspond to the elements of a position in the convolution kernel. The convolution kernel size is 6×6, and when performing convolution operations, the stride is 2; the padding size is 0; and the dilation rate is 1.

[0209] Figure 5In the figure, the “×” between the input feature map and the convolution kernel represents the convolution operation; “=” represents the result of the convolution operation (i.e., the output feature map).

[0210] The output feature map represents the output data of this channel of N2, and the size of the output feature map is 2×2. c11 to c22 correspond to the elements of a position in the output feature map. Among them, c11 is the multiplication and accumulation of each pixel value of A11 and B11, c12 is the multiplication and accumulation of each pixel value of A12 and B11, c21 is the multiplication and accumulation of each pixel value of A21 and B11; c22 is the multiplication and accumulation of each pixel value of A22 and B11.

[0211] For the second public operator corresponding to the generated N2, if the operator type of the second public operator is convolution, the operator specification of the second public operator is {number of output channels: 2048; convolution kernel size: 4×4; step size: 2; padding size: 0; expansion rate: 1}.

[0212] At this time, the electronic device can Figure 5 The pixels above and to the left of the convolution kernel shown are cropped with a cropping size of 2 (actually a positive integer multiple of the stride can be selected), resulting in a convolution kernel with a size of 4×4.

[0213] See also Figure 6 , Figure 6 A schematic diagram of the operation logic of the second public operator corresponding to the N2 operator provided in this application. Figure 6 In the figure, the operation logic of a channel in the second public operator corresponding to the N2 operator is taken as an example (the operation logic in different channels is consistent). The input feature map represents the input data of a channel. The size of the input feature map is 8×8, and a11 to a88 correspond to elements at a position in the input feature map respectively.

[0214] Represents: a rectangular pixel area of ​​size 2×2 with a12 and a21 as diagonal elements. Represents: a rectangular pixel area of ​​size 2×4 with a32 and a61 as diagonal elements. Represents: a rectangular pixel area of ​​size 4×2 with a16 and a23 as diagonal elements. Represents: a rectangular pixel area of ​​size 4×4 with a36 and a63 as diagonal elements.

[0215] b11 to b66 correspond to an element at a position in the complete convolution kernel respectively. The corresponding rectangular pixel area represents the cropped convolution kernel, and b33 to b66 correspond to the elements of a position in the cropped convolution kernel. The convolution kernel size is 4×4, and when performing convolution operations, the stride is 2; the padding size is 0; and the dilation rate is 1.

[0216] Represents: a rectangular pixel area of ​​size 2×2 with b12 and b21 as diagonal elements. Represents: a rectangular pixel area of ​​size 2×4 with b32 and b61 as diagonal elements. Represents: a rectangular pixel area of ​​size 4×2 with b16 and b23 as diagonal elements. Represents: a rectangular pixel area of ​​size 4×4 with b36 and b63 as diagonal elements.

[0217] Figure 6 In the figure, the “×” between the Input Featuremap and the Kernel represents the convolution operation; the “=” represents the result of the convolution operation (i.e., the output feature map).

[0218] The output feature map represents the output data of the second public operator for this channel. The size of the output feature map is 3×3. d11 to d33 correspond to the elements of a position in the output feature map. Among them, d22 is and The multiplication and accumulation of each pixel value and the convolution calculation process of the values ​​at other positions are the same as d22, which will not be repeated here.

[0219] exist Figure 6 and Figure 5 Based on this, it can be seen that the output feature map of the second public operator output corresponding to N2 has one more row in the h (height) direction compared to the output feature map of N2, and is located at the top of the output feature map of the second public operator output corresponding to N2, and has one more column in the w (width) direction, and is located at the leftmost side of the output feature map of the second public operator output corresponding to N2.

[0220] According to the principle of convolution operation:

[0221]

[0222] in, express: and Multiplication and accumulation of each pixel value; express: and Multiplication and accumulation of each pixel value; express: and Multiply and accumulate pixel values.

[0223] Similarly,

[0224]

[0225] in, for and The multiplication and accumulation of each pixel value, To express Figure 6 A rectangular pixel area of ​​size 4×4 with a38 and a65 as diagonal elements.

[0226] express Figure 6 A rectangular pixel area of ​​2×2 with a14 and a23 as diagonal elements; express Figure 6 A rectangular pixel area of ​​2×4 with a18 and a25 as diagonal elements; express Figure 6 A rectangular pixel area of ​​size 4×2 with a34 and a63 as diagonal elements.

[0227] express: and Multiplication and accumulation of each pixel value; express: and Multiplication and accumulation of each pixel value; express: and Multiply and accumulate pixel values.

[0228] Similarly,

[0229]

[0230] in, for and The multiplication and accumulation of each pixel value, To express Figure 6 A rectangular pixel area of ​​size 4×4 with a56 and a83 as diagonal elements.

[0231] express Figure 6 A rectangular pixel area of ​​size 2×2 with a32 and a41 as diagonal elements; express Figure 6 A rectangular pixel area of ​​2×4 with a36 and a43 as diagonal elements; express Figure 6 A rectangular pixel area of ​​size 4×2 with a52 and a81 as diagonal elements.

[0232] express: and Multiplication and accumulation of each pixel value; express: and Multiplication and accumulation of each pixel value; express: and Multiply and accumulate pixel values.

[0233] Similarly,

[0234]

[0235] in, for and The multiplication and accumulation of each pixel value, To express Figure 6 A rectangular pixel area of ​​size 4×4 with a58 and a85 as diagonal elements.

[0236] express Figure 6 A rectangular pixel area of ​​2×2 with a34 and a43 as diagonal elements; express Figure 6 A rectangular pixel area of ​​2×4 with a38 and a45 as diagonal elements; express Figure 6 A rectangular pixel area of ​​size 4×2 with a54 and a83 as diagonal elements.

[0237] express: and Multiplication and accumulation of each pixel value; express: and Multiplication and accumulation of each pixel value; express: and Multiply and accumulate pixel values.

[0238] That is, the electronic device can split the 6x6 convolution operation performed on each channel in N2 into a 4x4 convolution operation and a part of the multiplication-accumulation operation. Accordingly, the operation logic corresponding to the 4x4 convolution operation can be implemented by the second public operator corresponding to N2, and the other part of the multiplication-accumulation operation can be implemented by the extended operator corresponding to N2.

[0239] It is understandable that if N2 is run through the CPU, the amount of calculation corresponding to N2 is: .

[0240] If the second public operator corresponding to N2 is run through the NPU, the computation amount corresponding to the second public operator is: The third extended operator corresponding to N2 is run by the CPU, and the amount of calculation corresponding to the third extended operator is: .

[0241] Since N2 is an over-specification operator, the hardware platform needs to run the over-specification operator through the CPU. Due to the low CPU computing power, the execution efficiency of the operator is poor. After replacing the over-specification operator with the corresponding optimization operator, that is, replacing N2 with the second public operator and the third extended operator corresponding to N2, the hardware platform can run the second public operator through the NPU and the third extended operator through the CPU. Moreover, the amount of calculation corresponding to the CPU after the replacement only accounts for 80 / 144 (0.556) of the amount of calculation before the replacement. In this way, the amount of calculation required to be processed by the CPU can be reduced, and the overall efficiency of the deep learning model operation can be improved.

[0242] For another example, for convolution operators, the hardware platform only supports operator specifications with a convolution kernel size of 4×4 and a padding size of 0, and an operator specification with a convolution kernel size of 4×4 and a padding size of 2. Correspondingly, for an operator in a deep learning model (which can be denoted as NX), if the operator specification of NX is {convolution kernel size 4x4, step size: 1, padding size: 3, dilation rate: 1, number of output channels: 32}, then NX is an over-specification operator.

[0243] See also Figure 7 , Figure 7 A schematic diagram of the operation logic of the NX operator provided in this application. Figure 7 In the example, the operation logic of a channel in the NX operator is taken as an example (the operation logic in different channels is the same). The input feature map (InputFeaturemap) represents the input data of a channel. The size of the input feature map (size) is 6×6, and the input feature map corresponds to a rectangular pixel area with a0409 and a0904 as diagonal elements. Since the padding size (pad) in each specification parameter corresponding to NX is 3, the input feature map after NX padding corresponds to a rectangular pixel area with a0112 and a1201 as diagonal elements and a size of 12×12. Among them, A11 represents: a rectangular pixel area with a0104 and a0401 as diagonal elements and a size of 4×4.

[0244] The rectangular pixel area B11 represents the complete convolution kernel, and b11 to b44 correspond to the elements of a position in the convolution kernel. The convolution kernel size is 4×4, and the stride is 2 when performing the convolution operation; the dilation rate is 1.

[0245] Figure 7 In the figure, the “×” between the Input Featuremap and the Kernel represents the convolution operation; the “=” represents the result of the convolution operation (i.e., the output feature map).

[0246] The output feature map represents the output data of the channel, and the size of the output feature map is 9×9. e11 to e99 correspond to the elements of a position in the output feature map. Among them, e11 is the multiplication and accumulation of the pixel values ​​of A11 and B11, and the element operation process at other positions is not repeated here.

[0247] However, since the NPU in the hardware platform supports operator specifications with a convolution kernel size of 4×4 and a padding size of 2, that is, the operator specification of the second public operator corresponding to NX is {convolution kernel size 4x4, step size: 1, padding size: 2, expansion rate: 1, number of output channels: 32}. Accordingly, after padding the 6×6 input feature map, only a rectangular pixel area with a0211 and a1102 as diagonal elements can be obtained.

[0248] Furthermore, the size of the Output Featuremap output by the second public operator corresponding to NX is 8×8, that is, a rectangular pixel area with e28 and e82 as diagonal elements.

[0249] That is to say, the output feature map of the second public operator output corresponding to NX lacks two rows in the h (height) direction relative to the output feature map of NX output, namely, the top row (e11, e12...e19) and the bottom row (e91, e92...e99) in the output feature map of NX output. Moreover, the output feature map of the second public operator output corresponding to NX lacks two columns in the w (width) direction relative to the output feature map of NX output, namely, the leftmost column (e11, e21...e91) and the rightmost column (e19, e29...e99) in the output feature map of NX output.

[0250] Accordingly, ,in, express and Multiply and accumulate pixel values.

[0251] Similarly, .in, express Figure 7 A rectangular pixel area of ​​size 4×4 with a0104 and a0401 as diagonal elements. express and Multiply and accumulate pixel values.

[0252] By analogy, the values ​​of the elements in the top row, bottom row, leftmost column, and rightmost column of the output feature graph output by NX can all be calculated in the above way.

[0253] That is, the electronic device can split the convolution operation of the pixel area filled with 3 in NX into the convolution operation of the pixel area filled with 2 and a part of the multiplication-accumulation operation. Accordingly, the operation logic corresponding to the convolution operation of the pixel area filled with 2 can be implemented by the second public operator corresponding to NX, and the other part of the multiplication-accumulation operation can be implemented by the extended operator corresponding to NX.

[0254] With respect to step S1034, for each over-specification operator, obtaining the optimized operator corresponding to the over-specification operator includes: generating a second public operator and a third extended operator corresponding to the over-specification operator.

[0255] Based on the above processing, an operator specification that is not supported by an NPU can be split into an operator specification that is supported by the NPU, and then a public operator (i.e., a second public operator) and an extended operator of the operator specification supported by the NPU are generated. Among them, the second public operator retains most of the computational workload of the original over-specification operator, while the extended operator is used to complete the remaining computational workload (ensuring end-to-end consistency before and after replacement). In this way, it is possible to avoid executing all the computational workload corresponding to the over-specification operator through the CPU, thereby improving the efficiency of operator execution.

[0256] In some embodiments, step S103 includes:

[0257] Step 1: For each over-specification operator, if the operator specifications supported by the neural processing unit are inconsistent with the operator specifications of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels, then generate a first extended operator according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator.

[0258] The number of output channels of the first extension operator is a difference, and the first extension operator is consistent with the super-specification operator in other specification parameters except the number of output channels.

[0259] Step 2: Generate a first public operator and a second extended operator.

[0260] Among them, the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the super-specification operator.

[0261] Step 3: Combine the first extended operator, the first public operator and the second extended operator to obtain an optimized operator corresponding to the super-specification operator.

[0262] In an embodiment of the present application, for each over-specification operator, if the operator specifications supported by the neural processing unit are inconsistent with the operator specifications of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels, then according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, a first extended operator corresponding to the over-specification operator is generated.

[0263] Furthermore, the electronic device may determine to generate the first public operator and the second extended operator corresponding to the over-specification operator according to the difference between the operation logic of the over-specification operator and the operation logic of the first extended operator.

[0264] Correspondingly, in the operator specification of the first public operator, the number of output channels is consistent with the maximum number of output channels supported by the NPU for the operator type of the super-specification operator. The method for determining the first public operator can refer to the method for determining the second public operator in the above embodiment, which will not be repeated here.

[0265] Furthermore, the operation logic corresponding to the difference between the operation logic of the super-specification operator and the operation logic of the first extended operator is consistent with the total operation logic of the first public operator and the second extended operator.

[0266] It is understandable that, during the subsequent operation of the deep learning model, the hardware platform can run the first extended operator and the second extended operator corresponding to an over-specification operator through the CPU, and run the first public operator corresponding to an over-specification operator through the NPU.

[0267] For example, the maximum number of output channels supported by the hardware platform NPU for the convolution operator is 2048, while the deep learning model created by the developer may include a convolution operator with an output channel number of 2500. Accordingly, the operator is an over-specification operator, and for the over-specification operator, the electronic device can generate a first extension operator corresponding to the over-specification operator, and the first extension operator is used to output the convolution result of 452 (2500 minus 2048) channels.

[0268] If the convolution kernel size (6×6) of the super-specification operator is not the convolution kernel size of the convolution operator supported by the NPU, the first public operator (the convolution kernel size is 4×4) and the second extended operator corresponding to the super-specification operator can be generated, where the first public operator and the second extended operator are used to output the convolution result of 2048 channels. The first public operator runs on the NPU, and the first extended operator and the second extended operator run on the CPU.

[0269] In one implementation, step 3 includes:

[0270] Step 3-1: Merge the first expansion operator and the second expansion operator to obtain a merged expansion operator.

[0271] Step 3-2: Combine the merged extended operator and the first public operator to obtain the optimized operator corresponding to the over-specification operator.

[0272] In the embodiment of the present application, for an over-specification operator, when the operator specification of the over-specification operator is inconsistent with the operator specification of the first public operator in multiple specification parameters, the electronic device needs to generate multiple extended operators.

[0273] Since the running process of the extended operator is on the CPU, the running of the extended operator will increase the number of I / O (i.e., input / output) of the CPU. In order to reduce the number of I / O of the CPU during the running of the extended operator, multiple extended operators corresponding to an over-specification operator can be merged to obtain a merged extended operator.

[0274] For example, when a convolution operator does not support both the convolution kernel size and the number of output channels, the electronic device can generate two extended operators for the convolution operator (one to compensate for the change in the convolution kernel size of the public operator, and the other to compensate for the change in the number of output channels of the public operator). Accordingly, the electronic device can merge the two extended operators into one extended operator, thereby reducing the I / O occupation of data transmission between operators and further improving the efficiency of operator execution.

[0275] In some embodiments, the method further comprises:

[0276] Step 1: Receive data to be processed;

[0277] Step 2: Input the data to be processed into the optimized deep learning model to run the extended operators in the optimized deep learning model in the central processing unit, and run the public operators in the optimized deep learning model in the neural processing unit to process the data to be processed and obtain the processing results.

[0278] In an embodiment of the present application, the data to be processed is the data required to be processed by the deep learning model.

[0279] For example, in the production process of a product, if the user needs to use a model to process images containing the product to detect defective products, the deep learning model can be a target detection model. In this case, the data to be processed can be images containing the product.

[0280] Accordingly, the electronic device can input the data to be processed into the optimized deep learning model, and the hardware platform can run the optimized deep learning model. During the model operation, the CPU in the hardware platform is used to run the extended operator in the optimized deep learning model, and the NPU in the hardware platform is used to run other operators in the optimized deep learning model except the extended operator. In this way, the normal operation of the optimized deep learning model on the hardware platform can be guaranteed.

[0281] See also Figure 8 , Figure 8 Schematic diagram of the differences in executing over-specification operators on the hardware platform provided in the embodiment of the present application.

[0282] For each over-specification operator contained in the deep learning model before optimization, the over-specification operator can be run as an extended operator through the CPU of the hardware platform. However, since the CPU computing power is usually lower than the NPU computing power, if the over-specification operator is run through the CPU in the hardware platform, the operation efficiency of the operator will be low, and thus, the efficiency of model execution will be reduced.

[0283] Accordingly, the electronic device can generate the public operator (i.e., the approximate operator supported by the NPU) and the extended operator corresponding to each over-specification operator according to the model optimization method provided in this application. In this way, the part of the operation logic of the over-specification operator that needs to be processed by the CPU can be reduced.

[0284] See also Fig. 9 , Fig. 9 A schematic diagram of replacing an over-specification operator provided in an embodiment of the present application.

[0285] Among them, in the deep learning model before replacement, the convolution operator (Convolution) is an over-specification operator in the deep learning model, and in the operator specification of the over-specification operator, the convolution kernel size (i.e., kernel size) is 6×6. The pre-layer (Pre layer) means that in the data processing flow, the previous operator of the over-specification operator in the deep learning model, that is, the output data of the Prelayer is the input data of the over-specification operator. The next layer (Next layer) means that in the data processing flow, the next operator of the over-specification operator in the deep learning model, that is, the output data of the over-specification operator is the input data of the Pre layer.

[0286] For the over-specification operator, an optimized operator corresponding to the over-specification operator can be generated, that is, a public operator supported by the NPU with a convolution kernel size of 4×4 (i.e., Convolution, kernel size (convolution kernel size) is 4×4), and an extended operator (also called Custom layer). Accordingly, the over-specification operator in the deep learning model before optimization can be replaced with the corresponding optimized operator to obtain the optimized deep learning model. That is, in the replaced deep learning model (i.e., the optimized deep learning model), the output data of the Pre layer in the deep learning model can be input to the public operator and the extended operator, and the Next layer in the deep learning model can obtain the output data of the public operator and the extended operator.

[0287] See also Fig.10 , Fig.10 A schematic diagram of a process for determining over-specification operators in a deep learning model provided in an embodiment of the present application.

[0288] S1001: Start.

[0289] S1002: Obtain a computational graph of the model.

[0290] That is, the electronic device can obtain the computational graph of the deep learning model. The computational graph is composed of multiple operators, and the operator connection relationship between operators can represent the data processing flow between operators.

[0291] S1003: Determine whether the computation graph traversal is complete. If yes, execute step S1010; if no, execute step S1004.

[0292] S1004: Read the next operator.

[0293] That is, according to the data processing flow between the operators in the calculation graph, the operator that needs to be judged currently is determined.

[0294] S1005: Obtain the operator type of the operator.

[0295] That is, for the operator that needs to be judged currently, the operator type of the operator is obtained.

[0296] S1006: Determine whether the operator type is supported. If yes, execute step S1007; if no, execute step S1009.

[0297] That is, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, the operator is determined to be an out-of-specification operator.

[0298] S1007: Read the operator specification of the operator.

[0299] That is, if the operator type of the operator exists in the operator types supported by the neural processing unit, the operator specification of the operator is further determined.

[0300] S1008: Determine whether the operator specification is supported. If yes, execute step S1003; if no, execute step S1009.

[0301] S1009: Mark the operator as an over-specification operator.

[0302] That is, if the operator type of the operator exists in the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist in the operator specifications supported by the neural processing unit, then the operator is determined to be an out-of-specification operator.

[0303] S1010: End.

[0304] See also Fig.11 , Fig.11 A schematic diagram of a process for determining an optimization operator corresponding to an over-specification operator provided in an embodiment of the present application.

[0305] S1101: Start.

[0306] S1102: Clarify the original over-specification operator calculation process.

[0307] That is, for an over-specification operator, the operation logic of the operator is determined according to the operator type and operator specification of the over-specification operator.

[0308] S1103: Derive a public operator that satisfies partial equivalence.

[0309] That is, for an over-specification operator, an operator specification that is closest to the operator specification of the over-specification operator is determined from operator specifications supported by the neural processing unit.

[0310] For example, the operator specification that is closest to the operator specification of the over-specification operator can be determined by reducing the operator specification while keeping the operator type unchanged. Accordingly, the operator to which the operator specification belongs is the operator of the same type after the specification is reduced.

[0311] Alternatively, the operator specification closest to the operator specification of the over-specification operator can be determined in a manner that the operator types are different and the logical operation processes are similar or identical. Accordingly, the operators to which the operator specification belongs are operators of different types with similar calculation processes.

[0312] Alternatively, the electronic device may determine the operator specification that is closest to the operator specification of the over-specification operator in other ways. The specific process of determining the operator specification that is closest to the operator specification of the over-specification operator can refer to the above embodiment, which will not be described in detail here.

[0313] S1104: Screening the best partially equivalent public operator.

[0314] That is, for an over-specification operator, the optimization operators corresponding to the over-specification operator generated in the above two ways may be inconsistent. Furthermore, the electronic device can calculate the computational amount of the extended operator running on the CPU in different optimization schemes, and generate the optimization operator corresponding to the over-specification operator according to the optimization scheme with a smaller computational amount. Accordingly, the public operator in the scheme is the optimal partially equivalent public operator selected.

[0315] S1105: Use the extended operator to compensate the optimal public operator.

[0316] That is, an extended operator is generated to compensate for the difference between the optimal public operator and the over-spec operator.

[0317] S1106: Perform fusion optimization on multiple extended operators.

[0318] That is, when the optimization operator corresponding to an over-specification operator includes multiple expansion operators, the multiple expansion operators can be merged to obtain a merged expansion operator.

[0319] S1107: End.

[0320] Based on the same inventive concept, the present application embodiment provides a model optimization device. Fig.12 , Fig.12 A structural diagram of a model optimization device provided in an embodiment of the present application, the device comprising:

[0321] A specification acquisition module 1201 is used to obtain the operator specification of each operator included in the deep learning model before optimization;

[0322] An over-specification operator determination module 1202 is used to determine, based on the acquired operator specifications, from the operators included in the deep learning model before optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators;

[0323] The optimization operator generation module 1203 is used to generate an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator; wherein the optimization operator corresponding to an over-specification operator includes a public operator and an extended operator supported by the neural processing unit, and the overall operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator;

[0324] The operator replacement module 1204 is used to replace each over-specification operator in the deep learning model before optimization with a corresponding optimized operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

[0325] In some embodiments, the over-specification operator determination module 1202 is specifically used to:

[0326] For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, determining that the operator is an over-specification operator;

[0327] If the operator type of the operator exists among the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist among the operator specifications supported by the neural processing unit, then the operator is determined to be an over-specification operator.

[0328] In some embodiments, the optimization operator generation module 1203 includes:

[0329] A first generating submodule is used for generating, for each over-specification operator, a first extended operator according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, if the operator specifications supported by the neural processing unit are inconsistent with the operator specifications of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels; wherein the number of output channels of the first extended operator is the difference, and the first extended operator is consistent with the over-specification operator in other specification parameters except the number of output channels;

[0330] A second generating submodule, configured to generate a first public operator and a second extended operator; wherein the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the over-specification operator;

[0331] The first optimization operator acquisition submodule is used to combine the first extended operator, the first public operator and the second extended operator to obtain the optimization operator corresponding to the super-specification operator.

[0332] In some embodiments, the first optimization operator acquisition submodule is specifically used to:

[0333] Merging the first expansion operator and the second expansion operator to obtain a merged expansion operator;

[0334] The combined extended operator and the first public operator are combined to obtain an optimized operator corresponding to the over-specification operator.

[0335] In some embodiments, the optimization operator generation module 1203 includes:

[0336] a specification determination submodule, for determining, for each over-specification operator, from the operator specifications supported by the neural processing unit, an operator specification that is closest to the operator specification of the over-specification operator; wherein the operator specification that is closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in terms of each specification parameter, and relative to other operator specifications supported by the neural processing unit, the operator specification of the over-specification operator has the largest number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator;

[0337] A third generating submodule, configured to generate a second public operator corresponding to the over-specification operator according to an operator specification that is closest to the operator specification of the over-specification operator;

[0338] A fourth generating submodule, configured to generate a third extended operator corresponding to the over-specification operator according to a difference between an operator specification of the over-specification operator and an operator specification of a second public operator corresponding to the over-specification operator;

[0339] The second optimization operator acquisition submodule is used to combine the generated second public operator and the third extended operator to obtain the optimization operator corresponding to the super-specification operator.

[0340] In some embodiments, the specification determination submodule is specifically used to:

[0341] For each over-specification operator, if the over-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the neural processing unit, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, then the product of the step size of the over-specification operator and the current coefficient is calculated; wherein the initial value of the coefficient is 1;

[0342] Calculate the difference between the convolution kernel size of the oversized operator and the current product;

[0343] Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference value among the operator specifications of the convolution operator supported by the neural processing unit;

[0344] If it exists, the target operator specification is determined to be the operator specification that is closest to the operator specification of the super-specification operator;

[0345] If it does not exist, the current coefficient is increased by 1, and the step of calculating the product of the step size of the over-specification operator and the current coefficient is returned.

[0346] In some embodiments, for each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels, convolution kernel size, step size, padding size, and dilation rate;

[0347] If the operator is a fully connected operator, the specification parameters in the operator specification of the operator include: the dimension of the weight matrix and the number of output channels.

[0348] In some embodiments, the apparatus further comprises:

[0349] A data receiving module, used for receiving data to be processed;

[0350] A data processing module is used to input the data to be processed into the optimized deep learning model, so as to run the extended operators in the optimized deep learning model in the central processing unit, and to run the public operators in the optimized deep learning model in the neural processing unit, so as to process the data to be processed and obtain processing results.

[0351] The present application also provides an electronic device, such as Fig.13 As shown, including:

[0352] Memory 1301, used for storing computer programs;

[0353] The processor 1302 is used to implement the steps of any of the above-mentioned model optimization methods when executing the program stored in the memory 1301.

[0354] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1302, the communication interface, and the memory 1301 communicate with each other via the communication bus.

[0355] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0356] The communication interface is used for communication between the above electronic device and other devices.

[0357] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0358] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0359] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model optimization methods are implemented.

[0360] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any model optimization method in the above embodiments.

[0361] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0362] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0363] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0364] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model optimization method, characterized in that: The method comprises: Obtain the operator specifications of each operator included in the deep learning model before optimization; Based on the obtained operator specifications, determine, from the operators included in the deep learning model before the optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators; According to the operator specification of each over-specification operator, an optimization operator corresponding to the over-specification operator is generated; wherein the optimization operator corresponding to an over-specification operator includes a public operator and an extended operator supported by the neural processing unit, and the total operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator; the public operator corresponding to an over-specification operator is generated according to the operator specification that is closest to the operator specification of the over-specification operator; there is a difference between the operation logic of the public operator and the operation logic of the over-specification operator, and the generated extended operator is used to compensate for the difference; Each over-specification operator in the deep learning model before optimization is replaced with a corresponding optimized operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

2. The method according to claim 1, characterized in that The step of determining, based on the obtained operator specifications, operators not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators from among the operators included in the deep learning model before optimization, includes: For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, determining that the operator is an over-specification operator; If the operator type of the operator exists among the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist among the operator specifications supported by the neural processing unit, then the operator is determined to be an over-specification operator.

3. The method according to claim 1, characterized in that The step of generating an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator includes: For each over-specification operator, if the operator specification supported by the neural processing unit is inconsistent with the operator specification of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels, then according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, a first extended operator is generated; wherein the number of output channels of the first extended operator is the difference, and the first extended operator is consistent with the over-specification operator in other specification parameters except the number of output channels; Generate a first public operator and a second extended operator; wherein the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the over-specification operator; The first extended operator, the first public operator and the second extended operator are combined to obtain an optimized operator corresponding to the super-specification operator.

4. The method according to claim 3, characterized in that: The combining the first extended operator, the first public operator and the second extended operator to obtain the optimized operator corresponding to the super-specification operator includes: Merging the first expansion operator and the second expansion operator to obtain a merged expansion operator; The combined extended operator and the first public operator are combined to obtain an optimized operator corresponding to the over-specification operator.

5. The method according to claim 1, characterized in that The step of generating an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator includes: For each over-specification operator, determine the operator specification closest to the operator specification of the over-specification operator from the operator specifications supported by the neural processing unit; wherein the operator specification closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in all specification parameters, and relative to other operator specifications supported by the neural processing unit, the operator specification of the over-specification operator has the largest number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator; Generate a second public operator corresponding to the over-specification operator according to the operator specification that is closest to the operator specification of the over-specification operator; Generate a third extended operator corresponding to the over-specification operator according to a difference between the operator specification of the over-specification operator and the operator specification of the second public operator corresponding to the over-specification operator; The generated second public operator and the third extended operator are combined to obtain the optimized operator corresponding to the super-specification operator.

6. The method according to claim 5, characterized in that The step of determining, for each over-specification operator, an operator specification closest to the operator specification of the over-specification operator from operator specifications supported by the neural processing unit, includes: For each over-specification operator, if the over-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the neural processing unit, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, then the product of the step size of the over-specification operator and the current coefficient is calculated; wherein the initial value of the coefficient is 1; Calculate the difference between the convolution kernel size of the oversized operator and the current product; Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference value among the operator specifications of the convolution operator supported by the neural processing unit; If it exists, the target operator specification is determined to be the operator specification that is closest to the operator specification of the super-specification operator; If it does not exist, the current coefficient is increased by 1, and the step of calculating the product of the step size of the over-specification operator and the current coefficient is returned.

7. The method according to claim 3 or 5, characterized in that: For each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels, convolution kernel size, step size, padding size, and dilation rate; If the operator is a fully connected operator, the operator specification of the operator includes the following parameters: the dimension of the weight matrix and the number of output channels; If the operator is a pooling operator, the operator specification of the operator includes the following parameters: pooling kernel size, step size, and padding size.

8. The method according to claim 1, characterized in that The method further comprises: receiving data to be processed; The data to be processed is input into the optimized deep learning model, so as to run the extended operator in the optimized deep learning model in the central processing unit, and the public operator in the optimized deep learning model in the neural processing unit, so as to process the data to be processed and obtain a processing result.

9. A model optimization device, characterized in that: The device comprises: A specification acquisition module is used to obtain the operator specifications of each operator included in the deep learning model before optimization; An over-specification operator determination module, for determining, based on the acquired operator specifications, from the operators included in the deep learning model before optimization, operators that are not supported by the neural processing unit in the hardware platform of the model to be deployed as over-specification operators; An optimization operator generation module is used to generate an optimization operator corresponding to each over-specification operator according to the operator specification of the over-specification operator; wherein the optimization operator corresponding to an over-specification operator includes a public operator and an extended operator supported by the neural processing unit, and the total operation logic of the optimization operator corresponding to the over-specification operator is consistent with the operation logic of the over-specification operator; the public operator corresponding to an over-specification operator is generated according to the operator specification that is closest to the operator specification of the over-specification operator; there is a difference between the operation logic of the public operator and the operation logic of the over-specification operator, and the generated extended operator is used to compensate for the difference; An operator replacement module is used to replace each over-specification operator in the deep learning model before optimization with a corresponding optimized operator to obtain an optimized deep learning model; wherein the central processing unit in the hardware platform is used to run the extended operator in the optimized deep learning model.

10. The device according to claim 9, characterized in that The over-specification operator determination module is specifically used for: For each operator, if the operator type of the operator does not exist in the operator types supported by the neural processing unit, determining that the operator is an over-specification operator; If the operator type of the operator exists in the operator types supported by the neural processing unit, and for the operator type, the operator specification of the operator does not exist in the operator specifications supported by the neural processing unit, then determining that the operator is an over-specification operator; and / or, The optimization operator generation module comprises: A first generating submodule is used for generating, for each over-specification operator, a first extended operator according to the difference between the number of output channels of the over-specification operator and the maximum number of output channels supported by the neural processing unit for the operator type of the over-specification operator, if the operator specifications supported by the neural processing unit are inconsistent with the operator specifications of the over-specification operator in multiple specification parameters, and the multiple specification parameters include the number of output channels; wherein the number of output channels of the first extended operator is the difference, and the first extended operator is consistent with the over-specification operator in other specification parameters except the number of output channels; A second generating submodule, configured to generate a first public operator and a second extended operator; wherein the first public operator is an operator supported by the neural processing unit, and the total operation logic of the first public operator and the second extended operator is consistent with the operation logic corresponding to the maximum number of output channels in the over-specification operator; A first optimization operator acquisition submodule, used to combine the first extended operator, the first public operator and the second extended operator to obtain an optimization operator corresponding to the over-specification operator; and / or, The first optimization operator acquisition submodule is specifically used for: Merging the first expansion operator and the second expansion operator to obtain a merged expansion operator; Combining the merged extended operator and the first public operator to obtain an optimized operator corresponding to the over-specification operator; and / or, The optimization operator generation module comprises: a specification determination submodule, for determining, for each over-specification operator, from the operator specifications supported by the neural processing unit, an operator specification that is closest to the operator specification of the over-specification operator; wherein the operator specification that is closest to the operator specification of an over-specification operator is not greater than the operator specification of the over-specification operator in terms of each specification parameter, and relative to other operator specifications supported by the neural processing unit, the operator specification of the over-specification operator has the largest number of identical specification parameters between the closest operator specification and the operator specification of the over-specification operator; A third generating submodule, configured to generate a second public operator corresponding to the over-specification operator according to an operator specification that is closest to the operator specification of the over-specification operator; a fourth generating submodule, configured to generate a third extended operator corresponding to the over-specification operator according to a difference between an operator specification of the over-specification operator and an operator specification of a second public operator corresponding to the over-specification operator; A second optimization operator acquisition submodule, used to combine the generated second public operator and the third extended operator to obtain an optimization operator corresponding to the over-specification operator; and / or, The specification determination submodule is specifically used for: For each over-specification operator, if the over-specification operator is a convolution operator, and among the operator specifications of the convolution operator supported by the neural processing unit, there is no operator specification whose convolution kernel size is consistent with the convolution kernel size of the over-specification operator, then the product of the step size of the over-specification operator and the current coefficient is calculated; wherein the initial value of the coefficient is 1; Calculate the difference between the convolution kernel size of the oversized operator and the current product; Determine whether there is a target operator specification whose convolution kernel size is consistent with the current difference value among the operator specifications of the convolution operator supported by the neural processing unit; If it exists, the target operator specification is determined to be the operator specification that is closest to the operator specification of the super-specification operator; If it does not exist, then add 1 to the current coefficient and return to the step of calculating the product of the step size of the over-specification operator and the current coefficient; and / or, For each operator included in the deep learning model before optimization, if the operator is a convolution operator, the specification parameters in the operator specification of the operator include: number of output channels, convolution kernel size, step size, padding size, and dilation rate; If the operator is a fully connected operator, the operator specification of the operator includes the following parameters: the dimension of the weight matrix and the number of output channels; If the operator is a pooling operator, the operator specification of the operator includes the following parameters: pooling kernel size, step size, and padding size; and / or, The device also includes: A data receiving module, used for receiving data to be processed; A data processing module is used to input the data to be processed into the optimized deep learning model, so as to run the extended operators in the optimized deep learning model in the central processing unit, and to run the public operators in the optimized deep learning model in the neural processing unit, so as to process the data to be processed and obtain processing results.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images