Model conversion methods and systems for deep learning model inference hardware acceleration

Abstract

The application provides a model conversion method, system, storage medium and electronic equipment for deep learning model inference hardware acceleration, and relates to the technical field of deep learning.The application comprises the following steps: replacing a large convolution kernel in a deep learning model which cannot be directly deployed on hardware with a plurality of small convolution kernels; determining the weights of the small convolution kernels in each layer according to the weight differences of the large convolution kernel and the small convolution kernels in each layer by using a genetic algorithm; and deploying the converted deep learning model on hardware to obtain a convolution result of an input feature map, thereby obtaining a final inference result of the deep learning model. By replacing the large convolution kernel with a plurality of small convolution kernels which are executed in series, the weights of the small convolution kernels are directly calculated from the weights of the large convolution kernel, the influence of the replacement process on the output is minimized, the network model does not need to be retrained, the calculation amount is small, and the network model can be directly deployed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a model conversion method and system for deep learning model inference hardware acceleration, a storage medium and an electronic device. BACKGROUND

[0002] In order to more efficiently complete the inference of a deep learning model, artificial intelligence chips specialized in deep learning calculation are gradually emerging. Such artificial intelligence chips contain acceleration cores specialized in deep learning inference calculation, which are used to perform deep learning model inference with high energy efficiency. These deep learning acceleration cores generally perform calculations in parallel using many computing devices and utilize the multiplexing of data to minimize data movement as much as possible.

[0003] Convolution calculation is one of the most common calculations in a deep learning model. According to statistics, the amount of calculation occupied by convolution calculation in a convolutional neural network is more than 90%. Due to the limitations of hardware implementation, the number of parallel calculations that can be simultaneously supported by any deep learning hardware architecture is limited. For example, if the hardware is designed to efficiently support convolution operations of 3x3 size convolution kernels, once this hardware architecture is determined, it cannot directly support the calculation of larger (such as 5x5 or 7x7) convolution kernels. If the neural network structure is designed to include larger convolution kernels, such as 5x5 size convolution kernels, it cannot be directly deployed on this hardware architecture.

[0004] Currently, there are two methods to solve the above problem:

[0005] (1) One method is to split the 5x5 convolution kernel into four 3x3 size convolution kernels, respectively perform convolution calculation, and then add the convolution results to obtain the final convolution calculation result. The relationship between the weights of the new four 3x3 convolution kernels and the weights of the original 5x5 size convolution kernel under this splitting method is shown in Figure 1 . Under this splitting method, the weights of the new small convolution kernels can be directly obtained from the weights of the original large convolution kernel, without the need to retrain the network. Under this splitting method, the original 5x5 convolution kernel can be considered to be split into four 3x3 parallel executed convolution kernels, and the relationship between the new network architecture and the original network architecture is shown in Figure 2 . Under this splitting method, it should be ensured that all the weights of the original convolution kernel are split into small convolution kernels that adapt to the hardware size. For the part of the original convolution kernel that is less than the size of the small convolution kernel after splitting, 0 is filled.

[0006] (2) Another method is to split a layer of large convolution kernel neural network layer into multiple small convolution kernel layers that are executed in series. It should be ensured that after splitting into serial convolution layers, the receptive field of each output of the next layer remains unchanged, so as to avoid information loss caused by the replacement process. For example, as shown in Figure 3As shown, for a 5x5 convolution kernel, the receptive field of one output is a 5x5 window, so two layers of 3x3 small convolution kernel layers are needed to replace. For example, for a 7x7 convolution kernel, the receptive field of each output is a 7x7 window, and three layers of 3x3 small convolution layers are needed to replace. After the convolution kernel replacement is performed by this method, the original large convolution kernel layer does not exist, and several layers of serially executed small convolution kernel layers are added, and the network needs to be retrained to obtain the weights of the several layers of small convolution layers after replacement. That is, this method needs to redesign the network structure and retrain. The training process often requires a large amount of data and a long time. This method cannot achieve the effect of rapid deployment.

[0007] Therefore, the above method either faces the problem of large amount of inference calculation or the problem of needing to retrain and being unable to quickly deploy. SUMMARY

[0008] (I) Technical problems solved

[0009] In view of the deficiencies of the prior art, the present application provides a model conversion method, system, storage medium and electronic device for deep learning model inference hardware acceleration, which solves the technical problems of large amount of data calculation or the need for retraining and being unable to quickly deploy.

[0010] (II) Technical solutions

[0011] To achieve the above purpose, the present application is implemented by the following technical solutions:

[0012] A model conversion method for deep learning model inference hardware acceleration, comprising:

[0013] S1, for the original large convolution kernel in the deep learning model that cannot be directly deployed on hardware, a plurality of layers of small convolution kernels are used to replace the large convolution kernel;

[0014] S2, according to the weight difference of the large convolution kernel and each layer of small convolution kernel, a genetic algorithm is used to determine the weight of each layer of small convolution kernel;

[0015] S3, the converted deep learning model is deployed on hardware to obtain the convolution result of the input feature map.

[0016] Preferably, when the number of rows and the number of columns of the original large convolution kernel are both odd, the S2 specifically comprises:

[0017] S21, set the coordinates of the center position of the receptive field of the input feature map on the original large convolution kernel as (0, 0), then the coordinates of the top left corner of the receptive field are the coordinates of the top right corner are the coordinates of the bottom left corner are the coordinates of the bottom right corner are The coordinates of the remaining positions are sequentially similar; wherein, r, c respectively represent the number of rows, the number of columns of the large convolution kernel, the left half of each coordinate represents the row index number, and the right half represents the column index number; and the coordinates of each weight in the large convolution kernel are conventionally in the same way;

[0018] Before the replacement using the plurality of small convolution kernels, the original convolution calculation result is as follows:

[0019]

[0020] Wherein, x j，k represents the value of the input feature map at the coordinate (j, k) in the original large convolution kernel receptive field; u j，k represents the weight at the coordinate (j, k) in the large convolution kernel;

[0021] S22, set to replace the original large convolution kernel using n layers of small convolution kernels, the center position coordinate of the i-th layer of small convolution kernel is (0, 0), the coordinate of the upper left corner is The coordinate of the upper right corner is The coordinate of the lower left corner is The coordinate of the lower right corner is The coordinates of the remaining positions are sequentially similar; wherein, r i , c i respectively represent the number of rows, the number of columns of the i-th layer of small convolution kernel, both are odd numbers, and should satisfy: And

[0022] Substitute the above calculation process into the convolution calculation formula layer by layer until the feature map in the original receptive field is replaced, and perform the same item merging for all items of the feature map in the original receptive field participating in the calculation, extract the value of the feature map respectively, and obtain the final calculation result The expression is as follows:

[0023]

[0024] Wherein, u i,s,t represents the weight at the coordinate (s, t) in the i-th layer of small convolution kernel; for any one item of x j,k participating in the calculation before merging, s i is the row index number of the i-th layer of small convolution kernel weight participating in the calculation; t i is the column index number of the i-th layer of small convolution kernel weight participating in the calculation, which should satisfy i s i =j and i t i =k, the coefficient of this item is After merging the same items, the coefficient of x j,k is

[0025] S23, based on the two formulas in steps S21 and S22, taking the input feature map xj,k as a variable, the difference between the coefficients of the two formulas is expressed as:

[0026]

[0027] S24, setting the value of u in the formula of S23 unchanged, taking all u as variables to be solved, and taking the minimum value of diff in the formula of S23 as the target, the formula in S23 is solved by using a genetic algorithm to obtain the weight u of each layer of small convolution kernel. j,k i,s,t i,s,t

[0028] Preferably, when either the number of rows or the number of columns of the original large convolution kernel is even, the S2 specifically comprises:

[0029] (1) If the number of rows r is even and the number of columns c is odd

[0030] In the replaced multiple small convolution kernels, a convolution kernel with a height of 2 and a width of 1 is used first, so that the number of rows of the feature map after the convolution operation is r-1, which becomes odd, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th layer of small convolution kernel in the following n layers of small convolution kernel, which are both odd numbers and should satisfy: and Then, the weight of each layer of small convolution kernel is obtained by genetic algorithm according to the weight difference between each layer of small convolution kernel and the original large convolution kernel.

[0031] (2) If the number of columns c is even and the number of rows r is odd

[0032] In the replaced multiple small convolution kernels, a convolution kernel with a width of 2 and a height of 1 is used first, so that the number of columns of the feature map after the convolution operation is c-1, which becomes odd, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th layer of small convolution kernel in the following n layers of small convolution kernel, which are both odd numbers and should satisfy: and Then, the weight of each layer of small convolution kernel is obtained by genetic algorithm according to the weight difference between each layer of small convolution kernel and the original large convolution kernel.

[0033] (3) If the number of rows r is even and the number of columns c is also even

[0034] ​​​First, a 2x2 convolution kernel is used in the replaced multiple small convolution kernels, so that the number of rows and columns of the feature map after performing this convolution operation is r-1 and c-1, both of which are odd numbers, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and columns of the i-th layer of small convolution kernels in the following n layers of small convolution kernels, both of which are odd numbers, and should satisfy: and Then, according to the weight difference of each layer of small convolution kernel and the original large convolution kernel, the weight of each layer of small convolution kernel is obtained by genetic algorithm.

[0035] A model conversion system for deep learning model inference hardware acceleration, comprising:

[0036] A replacement module is used to replace the original large convolution kernel in the deep learning model which cannot be directly deployed on hardware with multiple layers of small convolution kernels.

[0037] A determination module is used to determine the weight of each layer of small convolution kernel by genetic algorithm according to the weight difference of the large convolution kernel and each layer of small convolution kernel.

[0038] A convolution module is used to deploy the converted deep learning model on hardware to obtain the convolution result of the input feature map.

[0039] Preferably, when the number of rows and columns of the original large convolution kernel are both odd numbers, the determination module specifically comprises:

[0040] S21, set the coordinates of the center position of the receptive field of the input feature map in the original large convolution kernel as (0, 0), then the coordinates of the top left corner of the receptive field are the coordinates of the top right corner are the coordinates of the bottom left corner are the coordinates of the bottom right corner are the coordinates of the remaining positions are sequentially deduced; wherein r and c represent the number of rows and columns of the large convolution kernel respectively, the left half of each coordinate represents the row index number, and the right half represents the column index number; and the coordinates of each weight in the large convolution kernel are determined in the same way;

[0041] Before the multiple small convolution kernels are used for replacement, the original convolution calculation result is as follows:

[0042]

[0043] Wherein, x j，k represents the value of the input feature map at the coordinate (j, k) in the receptive field of the original large convolution kernel; u j，k represents the weight at the coordinate (j, k) in the large convolution kernel;

[0044] S22, set to replace the original large convolution kernel using n-layer small convolution kernel, the center position coordinates of the i-th layer small convolution kernel is (0, 0), the coordinates of the upper left corner position The coordinates of the upper right corner position are The coordinates of the lower left corner position are The coordinates of the lower right corner position are The coordinates of the remaining positions are sequentially similar; wherein, r i , c i respectively represent the number of rows and columns of the i-th layer small convolution kernel, both are odd numbers, and should satisfy: and

[0045] Substitute the above calculation process into the convolution calculation formula layer by layer until the feature map in the original receptive field is replaced, and perform the same item merging for all items participating in the calculation of the feature map in the original receptive field, extract the value of the feature map respectively, and obtain the final calculation result The expression is as follows:

[0046]

[0047] Wherein, u i，s，t represents the weight of the i-th layer small convolution kernel at the internal coordinate (s, t); for any one item participating in the calculation before merging, s j，k is the row index number of the i-th layer small convolution kernel weight participating in the calculation; t i is the column index number of the i-th layer small convolution kernel weight participating in the calculation, which should satisfy i i s i=j and i t i =k, the coefficient of this item is After merging the same items, the coefficient of x j，k is

[0048] S23, based on the two formulas in steps S21 and S22, with the minimum error between the final convolution result after replacement and the original convolution result as the target, with the input feature map x j，k as a variable, then the difference between the coefficients of the two formulas is expressed as:

[0049]

[0050] S24, set the value of u j，k in the formula of S23 unchanged, take all u i，s，t as the variable to be solved, take the minimum value of diff in the formula of S23 as the target, and solve the formula in S23 using genetic algorithm to obtain the weight u i，s，t of each layer small convolution kernel respectively. ​

[0051] Preferably, when either the number of rows or the number of columns of the original large convolution kernel is even, the determining module is specifically configured to:

[0052] (1) If the number of rows r is even and the number of columns c is odd

[0053] In the replaced multiple small convolution kernels, a convolution kernel with a height of 2 and a width of 1 is used first, so that the number of rows of the feature map after the convolution operation is performed this time is r-1, which becomes odd, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th small convolution kernel of the n layers of small convolution kernels, which are both odd, and should satisfy: and Then, according to the weight difference between each layer of small convolution kernel and the original large convolution kernel, the weights of each layer of small convolution kernel are obtained through a genetic algorithm.

[0054] (2) If the number of columns c is even and the number of rows r is odd

[0055] In the replaced multiple small convolution kernels, a convolution kernel with a width of 2 and a height of 1 is used first, so that the number of columns of the feature map after the convolution operation is performed this time is c-1, which becomes odd, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th small convolution kernel of the n layers of small convolution kernels, which are both odd, and should satisfy: and Then, according to the weight difference between each layer of small convolution kernel and the original large convolution kernel, the weights of each layer of small convolution kernel are obtained through a genetic algorithm.

[0056] (3) If the number of rows r is even and the number of columns c is also even

[0057] In the replaced multiple small convolution kernels, a convolution kernel with a width of 2 and a height of 2 is used first, so that the number of rows and the number of columns of the feature map after the convolution operation is performed this time are both r-1 and c-1, which become odd, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th small convolution kernel of the n layers of small convolution kernels, which are both odd, and should satisfy: and Then, according to the weight difference between each layer of small convolution kernel and the original large convolution kernel, the weights of each layer of small convolution kernel are obtained through a genetic algorithm.

[0058] A storage medium storing a computer program for model conversion for deep learning model inference hardware acceleration, wherein the computer program causes a computer to execute the model conversion method for deep learning model inference hardware acceleration as described above.

[0059] An electronic device comprising:

[0060] one or more processors;

[0061] a memory; and

[0062] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for performing the model conversion method for deep learning model inference hardware acceleration as described above

[0063] (iii) Advantages

[0064] The present application provides a model conversion method, system, storage medium and electronic device for deep learning model inference hardware acceleration. Compared with the prior art, the following advantages are achieved:

[0065] The present application comprises: replacing a large convolution kernel in a deep learning model that cannot be directly deployed on hardware with a plurality of small convolution kernels; using a genetic algorithm to determine the weights of each layer of small convolution kernels according to the weight differences of the large convolution kernel and each layer of small convolution kernels; deploying the converted deep learning model on hardware to obtain the convolution result of the input feature map, thereby obtaining the final inference result of the deep learning model. By replacing the large convolution kernel with a plurality of serially executed small convolution kernels, the weights of the plurality of small convolution kernels are directly calculated from the weights of the large convolution kernel, thereby minimizing the impact of the replacement process on the output, eliminating the need to retrain the network model, and reducing the computational complexity, thereby achieving direct deployment of the network model. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0067] Figure 1 A schematic diagram of a method for decomposing a 5x5 large convolution kernel into a hardware-supported 3x3 small convolution kernel in the prior art;

[0068] Figure 2 A schematic diagram of a method for decomposing a 5x5 large convolution kernel into a hardware-supported 3x3 small convolution kernel in the prior art; Figure 1A schematic diagram of the prior art method of replacing a 5x5 large convolution kernel with 4 3x3 small convolution kernels is shown.

[0069] Figure 3 A schematic diagram of the prior art method of replacing a 5x5 large convolution kernel with 2 layers of 3x3 small convolution kernels is shown.

[0070] Figure 4 A flowchart of a model conversion method for deep learning model inference hardware acceleration provided by an embodiment of the present application is shown.

[0071] Figure 5 A weight diagram of a 5x5 convolution kernel provided by an embodiment of the present application is shown.

[0072] Figure 6 A schematic diagram of an input feature map representation in a receptive field of a 5x5 convolution kernel provided by an embodiment of the present application is shown.

[0073] Figure 7 A schematic diagram of a 2-layer 3x3 convolution layer representation provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0074] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0075] The embodiments of the present application provide a model conversion method, system, storage medium and electronic device for deep learning model inference hardware acceleration, which solves or faces the problem of large inference calculation amount, or faces the technical problem of needing retraining and being unable to be quickly deployed.

[0076] The technical solutions in the embodiments of the present application have the following general ideas to solve the above technical problems:

[0077] The original large convolution kernel in the deep learning model that cannot be directly deployed on hardware is replaced with multiple layers of small convolution kernels. According to the weight differences of the large convolution kernel and the small convolution kernels in each layer, a genetic algorithm is used to determine the weights of the small convolution kernels in each layer. The converted deep learning model is deployed on hardware to obtain the convolution result of the input feature map, thereby obtaining the final inference result of the deep learning model. By replacing the large convolution kernel with small convolution kernels of smaller size that are suitable for hardware execution, the size of the receptive field of each output is kept unchanged, and for any input, the output after replacement is as close as possible to the original output.

[0078] In addition, it needs to be pointed out that the method proposed in the embodiment of the application is aimed at two-dimensional convolution calculation with any step, also includes dilated convolution calculation, and also includes deformable convolution calculation and other convolution calculations. Although the input feature map receptive field aimed at by these convolution calculations has different shapes in the original input feature map, in the application, the input feature map in the receptive field participating in the convolution calculation is regarded as a two-dimensional matrix in a unified manner, and the coordinates are arranged in the form of a two-dimensional matrix, and then processed in a unified manner. The method proposed in the application can be applied to the deployment of deep learning models in the field of computer vision on artificial intelligence chips, one of the application fields being the field of industrial color sorters. For example, in the field of color sorters, the device needs to identify the type of grain in the image collected by the color sorter within a limited number of milliseconds, such as whether the grain is normal grain or defective grain. Obviously, such a device has strict requirements on the inference time of the deep learning model. On a traditional CPU-based device, for a deep learning model with a very large amount of calculation, the inference of the deep learning model cannot be completed within a limited time. On an artificial intelligence chip specially designed for deep learning model inference, a large number of hardware units can be used to perform calculations in deep learning model inference at the same time, so the inference of the deep model can be efficiently completed. However, once the hardware architecture of an artificial intelligence chip based on a hardware acceleration core is fixed, it cannot directly support the calculation of different sizes of convolution kernels.

[0079] In summary, the method proposed in the embodiment of the application can be applied to the field of deep learning model inference including color sorter image processing.

[0080] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0081] Embodiment:

[0082] In the first aspect, as shown in the drawings, Figure 4 The embodiment of the application provides a model conversion method for hardware acceleration of deep learning model inference, which comprises the following steps:

[0083] S1, for the original large convolution kernel in the deep learning model which cannot be directly deployed on hardware, a multi-layer small convolution kernel is used to replace the large convolution kernel;

[0084] S2, according to the weight difference of the large convolution kernel and each layer of the small convolution kernel, a genetic algorithm is used to determine the weight of each layer of the small convolution kernel;

[0085] S3, deploying the converted deep learning model on hardware to obtain the convolution result of the input feature map.

[0086] The embodiment of the application replaces a large convolution kernel with a plurality of small convolution kernels which are executed in series, the weights of the plurality of small convolution kernels being directly calculated from the weights of the large convolution kernel, so as to reduce the influence of the replacement process on the output as much as possible, without the need to retrain the network model and with small calculation amount, thereby realizing direct deployment of the network model.

[0087] The various steps of the above technical solution will be described in detail below.

[0088] In step S1, the original large convolution kernel in the deep learning model which cannot be directly deployed on hardware is replaced with a plurality of small convolution kernels.

[0089] In step S2, the weights of the small convolution kernels are determined respectively by using a genetic algorithm according to the weight differences of the large convolution kernel and the small convolution kernels.

[0090] When the number of rows and the number of columns of the original large convolution kernel are both odd numbers, the S2 specifically includes:

[0091] S21, the coordinates of the center position of the input feature map in the receptive field of the original large convolution kernel are set as (0, 0), the coordinates of the top-left corner position of the receptive field are the coordinates of the top-right corner position are the coordinates of the bottom-left corner position are the coordinates of the bottom-right corner position are the coordinates of the remaining positions are sequentially deduced; wherein r and c respectively represent the number of rows and the number of columns of the large convolution kernel, the left half of each coordinate represents the row index number, and the right half represents the column index number; and the coordinates of each weight in the large convolution kernel are determined by the same method;

[0092] Before the original convolution calculation result is replaced by the plurality of small convolution kernels, the original convolution calculation result is as follows:

[0093]

[0094] wherein x j，k represents the value of the input feature map at the coordinate (j, k) in the receptive field of the original large convolution kernel; u j，k represents the weight at the coordinate (j, k) in the large convolution kernel;

[0095] S22, the original large convolution kernel is replaced by n layers of small convolution kernels, the center position coordinates of the i-th layer of small convolution kernels are set as (0, 0), the coordinates of the top-left corner position are the coordinates of the top-right corner position are the coordinates of the bottom-left corner position are the coordinates of the bottom-right corner position are the coordinates of the remaining positions are sequentially deduced; wherein r i , c irespectively represent the number of rows and the number of columns of the i-th layer small convolution kernel, both are odd numbers, and should satisfy: and

[0096] The above calculation process is substituted into the convolution calculation formula layer by layer until the feature map in the original receptive field is replaced, and the like terms are combined for all terms in which the feature map in the original receptive field participates in the calculation, the values of the feature map are extracted respectively, and the final calculation result is obtained. The expression is as follows:

[0097]

[0098] Wherein, u i，s，t represents the weight of the i-th layer small convolution kernel at the internal coordinates (s, t); for any one of the combined x j，k participates in the calculation, s i is the row index number of the i-th layer small convolution kernel weight participating in the calculation; t i is the column index number of the i-th layer small convolution kernel weight participating in the calculation, and should satisfy ∑ i s i =j and ∑ i t i =k, the coefficient of this term is After combining the like terms, the coefficient of x j，k is

[0099] S23, based on the two formulas in steps S21 and S22, taking the minimum error between the final convolution result after replacement and the original convolution result as the target, taking the input feature map x j，k as a variable, then the difference between the coefficients of the two formulas is represented as:

[0100]

[0101] S24, setting the value of u j，k in the formula of S23 unchanged, taking all u i，s，t as variables to be solved, taking the minimum value of diff in the formula of S23 as the target, using genetic algorithm to solve the formula in S23, and respectively obtaining the weights u i，st of each layer small convolution kernel.

[0102] When the number of rows or columns of the original large convolution kernel is even, that is,

[0103] (1) if the number of rows r is even and the number of columns c is odd

[0104] First, a convolution kernel with a height of 2 and a width of 1 is used in the replaced multiple small convolution kernels, so that the number of rows of the feature map after the convolution operation is r-1, which is an odd number, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the fth small convolution kernel in the following n layers of small convolution kernels, both of which are odd numbers and should satisfy: and Then, the weight of each layer of small convolution kernel is obtained through a genetic algorithm according to the weight difference between each layer of small convolution kernel and the original large convolution kernel.

[0105] (2) If the number of columns c is even and the number of rows r is odd

[0106] First, a convolution kernel with a width of 2 and a height of 1 is used in the replaced multiple small convolution kernels, so that the number of columns of the feature map after the convolution operation is c-1, which is an odd number, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the fth small convolution kernel in the following n layers of small convolution kernels, both of which are odd numbers and should satisfy: and Then, the weight of each layer of small convolution kernel is obtained through a genetic algorithm according to the weight difference between each layer of small convolution kernel and the original large convolution kernel.

[0107] (3) If the number of rows r is even and the number of columns c is also even

[0108] First, a convolution kernel with a width of 2 and a height of 2 is used in the replaced multiple small convolution kernels, so that the number of rows and the number of columns of the feature map after the convolution operation are r-1 and c-1 respectively, both of which are odd numbers, and then n layers of small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the fth small convolution kernel in the following n layers of small convolution kernels, both of which are odd numbers and should satisfy: and Then, the weight of each layer of small convolution kernel is obtained through a genetic algorithm according to the weight difference between each layer of small convolution kernel and the original large convolution kernel.

[0109] In step S3, the converted deep learning model is deployed on hardware to obtain the convolution result of the input feature map.

[0110] The model conversion (convolution kernel replacement) process introduced in steps S1 and S2 is only performed once before deployment to obtain the weights of the replaced convolution layers. After deployment on hardware, the convolution kernel weights do not need to be calculated again at each inference.

[0111] The transformation method provided by the embodiment of the application:

[0112] In one aspect, the problem of the need for retraining when large convolution kernels are split into multiple layers of small convolution kernels when deploying a deep learning model to a specific hardware architecture can be solved.

[0113] In another aspect, the problem of large computation when large convolution kernels are split into multiple parallel small convolution kernels when deploying a deep learning model to a specific hardware architecture can be solved. Using the method, only the convolution kernel coefficients of each layer need to be obtained once before deploying the neural network model to hardware, and then the convolution kernel coefficients no longer need to be obtained again each time the hardware is inferred, and each inference is reduced in computation relative to the traditional method.

[0114] The detailed proof process is as follows:

[0115] Let the output feature map height be H and the width be W, then the computation of the layer is approximately H*W*r*c. After using the method provided by the embodiment of the application, the computation is The computation is greatly reduced.

[0116] In the embodiment of the application, the large convolution kernel that cannot be directly deployed on hardware is targeted, and r and c should be at least greater than 5. When r=5 and c=5, 2 layers of 3x3 small convolution kernels can be replaced. The original computation is H*W*25, and the computation after replacement is H*W*(3*3+3*3), which is obviously smaller. For larger convolution kernels, it can be proved by mathematical induction that the computation after replacement is always smaller than the original computation of the convolution layer before replacement.

[0117] Still taking a 5x5 convolution kernel as an example, the weight expression is as shown in Figure 5 The input feature map expression corresponding to an output is as shown in Figure 6 For convenience of understanding, the row index number and the column index of the weight and the input feature map both start from 1.

[0118] When the original convolution kernel is used, the calculation method of the output is as follows:

[0119] result=x 1，1 *i 1，1 +x 1，2 *u 1，2 +x 1，3 *u 1，3 +x 1，4 *u 1，4 +x 1，5 *u 1，5 +x 2，1 *u 2，1 +x 2，2 *u2，2 +x 2，3 *u 2，3 +x 2，4 *u 2，4 +x 2，5 *u 2，5 +x 3，1 *u 3，1 +x 3，2 *u 3，2 +x 3，3 *u 3，3 +x 3，4 *u 3，4 +x 3，5 *u 3，5 +x 4，1 *u 4，1 +x 4，2 *u 4，2 +x 4，3 *u 4，3 +x 4，4 *u 4，4 +x 4，5 *u 4，5 +x 5，1 *u 5，1 +x 5，2 *u 5，2 +x 5，3 *u 5，3 +x 5，4 *u 5，4 +x 5，5 *u 5，5 (1)

[0120] If replaced by 2 layers of 3x3 convolutional layers, the expression of the weights of the two convolutional layers is shown as Figure 7 , where W represents the first layer of convolution kernel weights, and V represents the second layer of convolution kernel weights. For easy understanding, the row index number and the column index of the small convolution kernel after replacement start from 1.

[0121] After replacing by 2 layers of 3x3 convolutional layers, the calculation method of the output at the same position is as follows:

[0122] result=v 1，1 *(x 1，1 *w 1，1 +x 1，2 *w 1，2 +x 1，3 *w 1，3 +x 2，1 *w 2，1 +x 2，2 *w 2，2 +x 2，3 *w 2，3 +x 3，1 *w 3，1 +x3，2 *w 3，2 +x 3，3 *w 3，3 )+v 1，2 *(x 1，2 *w 1，1 +x 1，3 *w 1，2 +x 1，4 *w 1，3 +x 2，2 *w 2，1 +x 2，3 *w 2，2 +x 2，4 *w 2，3 +x 3，2 *w 3，1 +x 3，3 *w 3，2 +x 3，4 *w 3，3 )+v 1，3 *(x 1，3 *w 1，1 +x 1，4 *w 1，2 +x 1，5 *w 1，3 +x 2，3 *w 2，1 +x 2，4 *w 2，2 +x 2，5 *w 2，3 +x 3，3 *w 3，1 +x 3，4 *w 3，2 +x 3，5 *w 3，3 )+v 2，1 *(x 2，1 *w 1，1 +x 2，2 *w 1，2 +x 2，3 *w 1，3 +x 3，1 *w 2，1 +x 3，2 *w 2，2 +x 3，3 *w 2，3 +x 4，1 *w 3，1 +x 4，2 *w 3，2 +x 4，3 *w 3，3 )+v 2，2 *(x 2，2 *w 1，1 +x 2，3 *w 1，2 +x2，4 *w 1，3 +x 3，2 *w 2，1 +x 3，3 *w 2，2 +x 3，4 *w 2，3 +x 4，2 *w 3，1 +x 4，3 *w 3，2 +x 4，4 *w 3，3 )+v 2，3 *(x 2，3 *w 1，1 +x 2，4 *w 1，2 +x 2，5 *w 1，3 +x 3，3 *w 2，1 +x 3，4 *w 2，2 +x 3，5 *w 2，3 +x 4，3 *w 3，1 +x 4，4 *w 3，2 +x 4，5 *w 3，3 )+v 3，1 *(x 3，1 *w 1，1 +x 3，2 *w 1，2 +x 3，3 *w 1，3 +x 4，1 *w 2，1 +x 4，2 *w 2，2 +x 4，3 *w 2，3 +x 5，1 *w 3，1 +x 5，2 *w 3，2 +x 5，3 *w 3，3 )+v 3，2 *(x 3，2 *w 1，1 +x 3，3 *w 1，2 +x 3，4 *w 1，3 +x 4，2 *w 2，1 +x 4，3 *w 2，2 +x 4，4 *w 2，3 +x 5，2 *w3，1 +x 5，3 *w 3，2 +x 5，4 *w 3，3 )+v 3，3 *(x 3，3 *w 1，1 +x 3，4 *w 1，2 +x 3，5 *w 1，3 +x 4，3 *w 2，1 +x 4，4 *w 2，2 +x 4，5 *w 2，3 +x 5，3 *w 3，1 +x 5，4 *w 3，2 +x 5，5 *w 3，3 ) (2)

[0123] The terms in Equation (2) that contain the same input feature map element are merged for the input feature map element, which can be rewritten as:

[0124] result = x 1，1 *v 1，1 *w 1，1 +x 1，2 *(v 1，1 *w 1，2 +v 1，2 *w 1，1 )+x 1，3 *(v 1，1 *w 1，3 +v 1，2 *w 1，2 +v 1，3 *w 1，1 )+x 1，4 *(v 1，2 *w 1，3 +v 1，3 *w 1，2 )+x 1，5 *(v 1，3 *w 1，3 )+x 2，1 *(v 1，1 *w 2，1 +v 2，1 *w 1，1 )+x 2，2 *(v 1，1 *w 2，2 +v 1，2 *w 2，1 +v 2，1 *w 1，2+ v 2，2 * w 1，1 + x 2，3 * (v 1，1 * w 2，3 + v 1，2 * w 2，2 + v 1，3 * w 2，1 + v 2，1 * w 1，3 + v 2，2 * w 1，2 + v 2，3 * w 1，1 + x 2， 4 * (v 1，2 * w 2，3 + v 1，3 * w 2，2 + v 2，2 * w 1，3 + v 2，3 * w 1，2 + x 2，5 * (v 1，3 * w 2，3 + v 2，3 * w 1，3 + x 3，1 * (v 1，1 * w 3，1 + v 2，1 * w 2，1 + v 3，1 * w 1，1 + x 3，2 * (v 1，1 * w 3，2 + v 1，2 * w 3，1 + v 2，1 * w 2，2 + v 2，2 * w 2，1 + v 3，1 * w 1，2 + v 3，2 * w 1，1 + x 3，3 * (v 1，1 * w 3，3 + v 1，2 * w 3，2 + v 1，3 * w 3，1 + v 2，1 * w 2，3 + v 2，2 * w 2，2 + v 2，3 * w 3，1 + v 3，1 * w 1，3 + v 3，2 * w 1，2+ v 3，3 * w 1，1 + x 3，4 * (v 1，2 * w 3，3 + v 1，3 * w 3，2 + v 2，2 * w 2，3 + v 2，3 * w 2，2 + v 3，2 * w 1，3 + v 3，3 * w 1，2 + x 3，5 * (v 1，3 * w 3，3 + v 2，3 * w 2，3 + v 3，3 * w 1，3 + x 4，1 * (v 2，1 * w 3，1 + v 3，1 * w 2，1 + x 4，2 * (v 2，1 * w 3，2 + v 2，2 * w 3，1 + v 3，1 * w 2，2 + v 3，2 * w 2，1 + x 4，3 * (v 2，1 * w 3，3 + v 2，2 * w 3，2 + v 2，3 * w 3，1 + v 3，1 * w 2，3 + v 3，2 * w 2，2 + v 3，3 * w 2，1 + x 4，4 * (v 2，2 * w 3，3 + v 2，3 * w 3，2 + v 3，2 * w 2，3 + v 3，3 * w 2，2 + x 4，5 * (v 2，3 * w 3，3 + v 3，3 * w 2，3 + x 5，1 v 3，1 * w3，1 +x 5，2 *(v 3，1 *w 3，2 +v 3，2 *w 3，1 )+x 5，3 *(v 3，1 *w 3，3 +v 3，2 *w 3，2 +v 3，3 *w 3，1 )+x 5，4 *(v 3，2 *w 3，3 +v 3，3 *w 3，2 )+x 5，5 *v 3，3 *w 3，3 (3)

[0125] The purpose of replacing the 2-layer 3x3 convolution layer is to make the results generated by the same output position after replacement as possible as the original 5x5 convolution kernel, so as to reduce the influence on the accuracy.

[0126] When the 5x5 size convolution kernel is replaced by 2-layer 3x3 size convolution kernel, the weight has been fixed. When deploying the network model, it is faced with 25 inputs that can take any value in the receptive field. That is, the variable in formula (3) is 25 inputs, and the weight can be considered as an invariable coefficient.

[0127] In order to make the output after the replacement of the convolution kernel as possible as the output before the replacement, it is required that the coefficients of formula (3) and formula (1) are as equal as possible.

[0128] The coefficients in formula (3) are denoted as:

[0129] u′ 1，1 =v 1，1 *w 1，1

[0130] u′ 1，2 =v 1，1 *w 1，2 +v 1，2 *w 1，1

[0131] u′ 1，3 =v 1，1 *w 1，3 +v 1，2 *w 1，2 +v 1，3 *w 1，1

[0132] u′ 1，4 =v1，2 *w 1，3 +v 1，3 *w 1，2

[0133] u′ 1，5 =v 1，3 *w 1，3

[0134] u′ 2，1 =v 1，1 *w 2，1 +v 2，1 *w 1，1

[0135] u′ 2，2 =v 1，1 *w 2，2 +v 1，2 *w 2，1 +v 2，1 *w 1，2 +v 2，2 *w 1，1

[0136] u′ 2，3 =v 1，1 *w 2，3 +v 1，2 *w 2，2 +v 1，3 *w 2，1 +v 2，1 *w 1，3 +v 2，2 *w 1，2 +v 2，3 *w 1，1

[0137] u′ 2，4 =v 1，2 *w 2，3 +v 1，3 *w 2，2 +v 2，2 *w 1，3 +v 2，3 *w 1，2

[0138] u′ 2，5 =v 1，3 *w 2，3 +v 2，3 *w 1，3

[0139] u′ 3，1 =v 1，1 *w 3，1 +v 2，1 *w 2，1 +v 3，1 *w 1，1

[0140] u' 3，2 = v 1，1 * w 3，2 + v 1，2 * w 3，1 + v 2，1 * w 2，2 + v 2，2 * w 2，1 + v 3，1 * w 1，2 + v 3，2 * w 1，1

[0141] u' 3，3 = v 1，1 * w 3，3 + v 1，2 * w 3，2 + v 1，3 * w 3，1 + v 2，1 * w 2，3 + v 2，2 * w 2，2 + v 2，3 * w 3，1 + v 3，1 * w 1，3 + v 3，2 * w 1，2 + v 3，3 * w 1，1

[0142] u' 3，4 = v 1，2 * w 3，3 + v 1，3 * w 3，2 + v 2，2 * w 2，3 + v 2，3 * w 2，2 + v 3，2 * w 1，3 + v 3，3 * w 1，2

[0143] u' 3，5 = v 1，3 * w 3，3 + v 2，3 * w 2，3 + v 3，3 * w 1，3

[0144] u' 4，1 = v 2，1 * w 3，1 + v 3，1 * w 2，1

[0145] u' 4，2 = v 2，1 * w 3，2 + v 2，2 * w 3，1 + v 3，1 * w 2，2 + v 3，2 * w 2，1

[0146] u' 4，3 = v 2，1 * w 3，3 + v 2，2 * w 3，2 + v 2，3 * w 3，1 + v 3，1 * w 2，3 + v 3，2 * w 2，2 + v 3，3 * w 2，1

[0147] u' 4，4 = v 2，2 * w 3，3 + v 2，3 * w 3，2 + v 3，2 * w 2，3 + v 3，3 * w 2，2

[0148] u' 4，5 = v 2，3 * w 3，3 + v 3，3 * w 2，3

[0149] u' 5，1 = v 3，1 * w 3，1

[0150] u' 5，2 = v 3，1 * w 3，2 + v 3，2 * w 3，1

[0151] u' 5，3 = v 3，1 * w 3，3 + v 3，2 * w 3，2 + v 3，3 * w 3，1

[0152] u' 5，4 = v 3，2 * w 3，3+ v 3，3 * w 3，2

[0153] u' 5，5 = v 3，3 * w 3，3

[0154] After replacing the 2-layer 3x3 size convolution kernel, the output calculation formula can be rewritten as:

[0155] result = x 1，1 * u' 1，1 + x 1，2 * u' 1，2 + x 1，3 * u' 1，3 + x 1，4 * u' 1，4 + x 1，5 * u' 1，5 + x 2，1 * u' 2，1 + x 2，2 * u' 2，2 + x 2，3 * u' 2，3 + x 2，4 * u' 2，4 + x 2，5 * u' 2，5 + x 3，1 * u' 3，1 + x 3，2 * u' 3，2 + x 3，3 * u' 3，3 + x 3，4 * u' 3，4 + x 3，5 * u' 3，5 x 4，1 * u' 4，1 + x 4，2 * u' 4，2 + x 4，3 * u' 4，3 + x 4，4 * u' 4，4 + x 4，5 * u' 4，5 + x 5，1 * u' 5，1 + x 5，2 * u' 5，2 + x 5，3 * u' 5，3 + x 5，4 * u' 5，4 + x 5，5 * u' 5，5

[0156] If the output at this time is to be as close as possible to the output before replacement, i.e., the calculation results of formula (1) and formula (3), the coefficients of the two formulas should be as close as possible. Using the L2 norm to describe the difference between the two sets of coefficients, the target is converted to making the following function as small as possible:

[0157] diff = (u 1，1 -u' 1，1 ) 2 +(u 1，2 -u' 1，2 ) 2 +(u 1，3 -u' 1，3 ) 2 +(u 1，4 -u' 1，4 ) 2 +(u 1，5 -u' 1，5 ) 2 +(u 2，1 -u' 2，1 ) 2 +(u 2，2 -u' 2，2 ) 2 +(u 2，3 -u' 2，3 ) 2 +(u 2，4 -u' 2，4 ) 2 +(u 2，5 -u' 2，5 ) 2 +(u 3，1 -u' 3，1 ) 2 +(u 3，2 -u' 3，2 ) 2 +(u 3，3 -u' 3，3 ) 2 +(u 3，4 -u' 3，4 ) 2 +(u 3，5 -u' 3，5 ) 2 +(u 4，1 -u' 4，1 ) 2 +(u 4，2 -u' 4，2 ) 2 +(u 4，3 -u' 4，3 ) 2 +(u 4，4 -u' 4，4 ) 2 +(u4，5 - u' 4，5 ) 2 + u 5，1 - u' 5，1 ) 2 + u 5，2 - u' 5，2 ) 2 + u 5，3 - u' 5，3 ) 2 + u 5，4 - u' 5，4 ) 2 + u 5，5 - u' 5，5 ) 2 (4)

[0158] In the above formula, the symbols without the superscript "'" represent the weights of the 5x5 convolution kernel before replacement, and the symbols with the superscript "'" represent the coefficients corresponding to the same input after the replacement of the 2-layer 3x3 convolution kernel. In this formula, the symbols without the superscript "'" are fixed weights, and the symbols with the superscript "'" are coefficients composed of the weight combination of the two 3x3 convolution kernels that need to be solved.

[0159] Therefore, in formula (4), the symbols with the superscript "'" are variable, but their changes are controlled by the weights of the two 3x3 convolution kernels. The problem is transformed into finding the weights of the two 3x3 convolution kernels that make formula (4) take the minimum value, and the variables are 2*3*3 = 18 in total, that is, finding the 18 values of formula (4) represented by 18 variables that make formula (4) take the minimum value.

[0160] After solving the 18 weights that make formula (4) take the minimum value by the genetic algorithm, the weights of the 2-layer 3x3 convolution kernel that need to be replaced are obtained. These 18 weights can make formula (4) take the minimum value, that is, formula (3) and formula (1) can be as close as possible to the value of any input feature map. Then, the replacement is directly performed. The purpose of the replacement is to replace the original one layer of 5x5 size convolution kernel with two layers of 3x3 size convolution kernel, so that the change of the output after the replacement is as small as possible for any input. The first convolution layer after the replacement does not need to add bias and activation function.

[0161] Similarly, for a 7x7 size convolution kernel, the replacement can also be performed in the above similar method, and the replacement is a convolution layer composed of 3 layers of 3x3 size convolution kernel.

[0162] Similarly, if the maximum convolution kernel supported by the hardware is not 3x3, but 5x5, the method proposed in the embodiment of the application can also be used to replace the 1-layer 9x9 convolution layer with 2 layers of 5x5 convolution kernel. Or using the method proposed in the embodiment of the application, replace the 1-layer 7x7 convolution layer with 1-layer 5x5 convolution kernel and 1-layer 3x3 convolution kernel.

[0163] Compared with Figure 2 the large convolution kernel splitting method shown in the figure, the advantage of the embodiment of the application is small calculation amount. For example, for a 5x5 large convolution kernel, if it is to be split into 3x3 convolution operation, using the method shown in the figure, 4 times of 3x3 convolution operation need to be performed, and the calculation amount for each output is 4*3*3=36. Using the method proposed in the application, only 2 times of 3x3 convolution operation need to be performed, and the average calculation amount for each output is 2*3*3=18, which is half of the method shown in the figure. Figure 2 For another example, for a 7x7 large convolution kernel, if it is to be split into 3x3 convolution operation, using the method shown in the figure, it needs to be split into 9 3x3 convolution kernels, and 9 times of 3x3 convolution operation needs to be performed, and 9*3*3=81 times of multiplication and accumulation calculation is needed for each output. Using the method proposed in the application, only 3 layers of 3x3 convolution layers are replaced, 3 times of 3x3 convolution operation is performed, and 3*3*3=27 times of multiplication and accumulation calculation is needed for each output on average, which is only 1 / 3 of the method shown in the figure. Figure 2 Figure 2 Compared with Figure 2 the large convolution kernel replacement method shown in the figure, the advantage of the embodiment of the application is that it does not need to retrain the network model and has small calculation amount, and can achieve the goal of fast deployment.

[0164] Compared with Figure 3 the large convolution kernel replacement method shown in the figure, the advantage of the embodiment of the application is that it does not need to retrain the network model and has small calculation amount, and can achieve the goal of fast deployment.

[0165] In a second aspect, the embodiment of the application provides a model conversion system for deep learning model inference hardware acceleration, comprising:

[0166] A replacement module is configured to replace the original large convolution kernel in the deep learning model which cannot be directly deployed on the hardware with multiple layers of small convolution kernels.

[0167] A determination module is configured to determine the weights of each layer of small convolution kernels respectively by using a genetic algorithm according to the weight difference between the large convolution kernel and each layer of small convolution kernel.

[0168] A convolution module is configured to deploy the converted deep learning model on the hardware and obtain the convolution result of the input feature map.

[0169] In an embodiment, when the number of rows and the number of columns of the original large convolution kernel are both odd, the determination module specifically comprises:​

[0170] S21, set the coordinates of the input feature map at the center position of the receptive field of the original large convolution kernel as (0, 0), the coordinates of the top left corner of the receptive field are the coordinates of the top right corner are the coordinates of the bottom left corner are the coordinates of the bottom right corner are the coordinates of the remaining positions are sequentially similar; wherein r and c respectively represent the number of rows and columns of the large convolution kernel, the left half of each coordinate represents the row index number, and the right half represents the column index number; and the coordinates of each weight in the large convolution kernel are conventionally in the same way;

[0171] Before using multiple small convolution kernels for replacement, the original convolution calculation result is as follows:

[0172]

[0173] wherein x j，k represents the value of the input feature map at the coordinate (j, k) in the receptive field of the original large convolution kernel; u j，k represents the weight at the coordinate (j, k) in the large convolution kernel;

[0174] S22, set to replace the original large convolution kernel with n layers of small convolution kernels, the center position coordinates of the i-th layer of small convolution kernel are (0, 0), the coordinates of the top left corner are the coordinates of the top right corner are the coordinates of the bottom left corner are the coordinates of the bottom right corner are the coordinates of the remaining positions are sequentially similar; wherein r i and c i respectively represent the number of rows and columns of the i-th layer of small convolution kernel, both are odd numbers, and should satisfy: and

[0175] Substitute the above calculation process into the convolution calculation formula layer by layer until the feature map in the original receptive field is replaced, and perform the same item merging for all items of the feature map in the original receptive field participating in the calculation, extract the value of the feature map respectively, and obtain the final calculation result The expression is as follows:

[0176]

[0177] wherein u i，s，t represents the weight at the coordinate (s, t) in the i-th layer of small convolution kernel; for any one item of x j，k participating in the calculation before merging, s i is the row index number of the i-th layer of small convolution kernel weight participating in the calculation; t iis the column index number of the weight of the small convolution kernel of the i-th layer participating in this calculation, which should satisfy ∑ i s i =j and∑ i t i =k, the coefficient of this term is After combining like terms, x j，k The coefficient of

[0178] S23, based on the two formulas in steps S21 and S22, with the goal of minimizing the error between the final convolution result after replacement and the original convolution result, taking the input feature map x j，k As a variable, the difference between the coefficients of the two equations is expressed as:

[0179]

[0180] S24, set u in the formula of S23 j，k The value of u remains unchanged. i，s，t As the variable to be solved, with the goal of minimizing the diff value in formula S23, a genetic algorithm is used to solve the formula in S23 to obtain the weights u of the small convolution kernels in each layer. i，s，t .

[0181] In one embodiment, when either the number of rows or the number of columns of the original large convolution kernel is an even number, the determining module is specifically configured to:

[0182] (1) If the number of rows r is even and the number of columns c is odd

[0183] Among the multiple small convolution kernels after replacement, a convolution kernel with a height of 2 and a width of 1 is first used to make the number of rows of the feature map after the convolution operation r-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel.

[0184] (2) If the number of columns c is even and the number of rows r is odd

[0185] Among the multiple small convolution kernels after replacement, a convolution kernel with a width of 2 and a height of 1 is first used, so that the number of columns of the feature map after the convolution operation is c-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels are used to replace the original large convolution kernel, r i 、c irespectively represent the number of rows and the number of columns of the i-th small convolution kernel in the n small convolution kernels, and both are odd numbers and should satisfy: and Then, the weights of the small convolution kernels are obtained by a genetic algorithm according to the weight differences between the small convolution kernels and the original large convolution kernel.

[0186] (3) if the number of rows r is even and the number of columns c is also even

[0187] In the replaced small convolution kernels, a convolution kernel with a width of 2 and a height of 2 is used first, so that the number of rows and the number of columns of the feature map after the convolution operation are both odd numbers, and then n small convolution kernels are used to replace the original large convolution kernel, r i , c i respectively represent the number of rows and the number of columns of the i-th small convolution kernel in the n small convolution kernels, and both are odd numbers and should satisfy: and Then, the weights of the small convolution kernels are obtained by a genetic algorithm according to the weight differences between the small convolution kernels and the original large convolution kernel.

[0188] In a third aspect, an embodiment of the present application provides a storage medium storing a computer program for model conversion for deep learning model inference hardware acceleration, wherein the computer program causes a computer to execute the model conversion method for deep learning model inference hardware acceleration.

[0189] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0190] one or more processors;

[0191] a memory; and

[0192] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include a program for executing the model conversion method for deep learning model inference hardware acceleration.

[0193] It can be understood that the model conversion system for deep learning model inference hardware acceleration, the storage medium and the electronic device provided by the embodiments of the present application correspond to the model conversion method for deep learning model inference hardware acceleration provided by the embodiments of the present application, and the related content explanation, examples and beneficial effects of the model conversion method for deep learning model inference hardware acceleration can be referred to the corresponding part of the model conversion method for deep learning model inference hardware acceleration, which will not be repeated here.

[0194] In summary, compared with the prior art, the present application has the following beneficial effects:

[0195] 1. In one aspect, the problem of retraining when deploying a deep learning model to a specific hardware architecture by splitting a large convolution kernel into multiple layers of small convolution kernels can be solved.

[0196] 2. In another aspect, the problem of large computation when deploying a deep learning model to a specific hardware architecture by splitting a large convolution kernel into multiple parallel small convolution kernels can be solved. Using the method, only once the convolution kernel coefficients of each layer are obtained before deploying the neural network model to hardware, and then each time the inference on the hardware no longer needs to reacquire the convolution kernel coefficients, and then each time the inference is reduced in computation relative to the traditional method.

[0197] It should be noted that the terms such as first and second, etc., are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0198] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model conversion method for hardware acceleration of deep learning model inference, characterized in that: include: S1. For the original large convolution kernels in the deep learning model that cannot be directly deployed on hardware, multiple layers of small convolution kernels are used to replace the large convolution kernels. S2. Based on the weight differences between the large convolution kernel and the small convolution kernels in each layer, a genetic algorithm is used to determine the weights of the small convolution kernels in each layer. S3. Deploy the converted deep learning model on hardware to obtain the convolution result of the input feature map of the image to be processed; When the number of rows and columns of the original large convolution kernel are both odd numbers, S2 specifically includes: S21. Set the coordinates of the center of the receptive field of the input feature map in the original large convolution kernel to (0,0), then the coordinates of the upper left corner of the receptive field are The coordinates of the upper right corner are The coordinates of the lower left corner are The coordinates of the lower right corner are The coordinates of the remaining positions are deduced in the same way; where r and c represent the number of rows and columns of the large convolution kernel, respectively. The left half of each coordinate represents the row index number, and the right half represents the column index number. The coordinates of each weight in the large convolution kernel are agreed upon in the same way. Before using multiple small convolution kernels for replacement, the original convolution calculation results are as follows: Among them, x j,k Represents the value of the input feature map at the coordinate (j, k) in the receptive field of the original large convolution kernel; u j,k Represents the weight at coordinate (j, k) in the large convolution kernel; S22, set to use n layers of small convolution kernel to replace the original large convolution kernel, the center position coordinates of the i-th layer of small convolution kernel are (0,0), the coordinates of the upper left corner position are The coordinates of the upper right corner are The coordinates of the lower left corner are The coordinates of the lower right corner are The coordinates of the remaining positions are deduced in the same way; among them, r i 、c i They represent the number of rows and columns of the small convolution kernel in the i-th layer, both of which are odd numbers and should satisfy: and Substitute the above calculation process into the convolution calculation formula layer by layer until it is replaced with the feature map in the original receptive field, and merge similar items for all items involved in the calculation of the feature maps in the original receptive field, extract the values ​​of the feature maps respectively, and obtain the final calculation result. The expression is as follows: Among them, u i,s,t Represents the weight at the internal coordinate (s, t) of the small convolution kernel of layer i; for x before merging j,k Any item involved in the calculation, s i is the row index number of the i-th layer small convolution kernel weight participating in this calculation; t i is the column index number of the weight of the small convolution kernel of the i-th layer participating in this calculation, which should satisfy ∑ i s i =j and∑ i t i =k, the coefficient of this term is After combining like terms, x j,k The coefficient of S23, based on the two formulas in steps S21 and S22, with the goal of minimizing the error between the final convolution result after replacement and the original convolution result, taking the input feature map x j,k As a variable, the difference between the coefficients of the two equations is expressed as: S24, set u in the formula of S23 j,k The value of u remains unchanged. i,s,t As the variable to be solved, with the goal of minimizing the diff value in formula S23, a genetic algorithm is used to solve the formula in S23 to obtain the weights u of the small convolution kernels in each layer. i,s,t ; or When either the number of rows or the number of columns of the original large convolution kernel is an even number, S2 specifically includes: (1) If the number of rows r is even and the number of columns c is odd Among the multiple small convolution kernels after replacement, a convolution kernel with a height of 2 and a width of 1 is first used to make the number of rows of the feature map after the convolution operation r-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel. (2) If the number of columns c is even and the number of rows r is odd Among the multiple small convolution kernels after replacement, a convolution kernel with a width of 2 and a height of 1 is first used, so that the number of columns of the feature map after the convolution operation is c-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels are used to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel. (3) If the number of rows r is even and the number of columns c is also even Among the multiple small convolution kernels after replacement, a convolution kernel with a width of 2 and a height of 2 is first used, so that after the convolution operation, the number of rows of the feature map is r-1 and the number of columns is c-1, both of which become odd numbers. Then, n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels are used to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel.

2. A model conversion system for hardware acceleration of deep learning model inference, characterized in that: include: The replacement module is used to replace the original large convolution kernels in the deep learning model that cannot be directly deployed on the hardware with multi-layer small convolution kernels; Determine the module, based on the weight differences between the large convolution kernel and the small convolution kernels in each layer, and use the genetic algorithm to determine the weights of the small convolution kernels in each layer; The convolution module is used to deploy the converted deep learning model on the hardware and obtain the convolution result of the input feature map of the image to be processed; When the number of rows and columns of the original large convolution kernel are both odd numbers, the determination module specifically includes: S21. Set the coordinates of the center of the receptive field of the input feature map in the original large convolution kernel to (0,0), then the coordinates of the upper left corner of the receptive field are The coordinates of the upper right corner are The coordinates of the lower left corner are The coordinates of the lower right corner are The coordinates of the remaining positions are deduced in the same way; where r and c represent the number of rows and columns of the large convolution kernel, respectively. The left half of each coordinate represents the row index number, and the right half represents the column index number. The coordinates of each weight in the large convolution kernel are agreed upon in the same way. Before using multiple small convolution kernels for replacement, the original convolution calculation results are as follows: Among them, x j,k Represents the value of the input feature map at the coordinate (j, k) in the receptive field of the original large convolution kernel; u j,k Represents the weight at coordinate (j, k) in the large convolution kernel; S22, set to use n layers of small convolution kernel to replace the original large convolution kernel, the center position coordinates of the i-th layer of small convolution kernel are (0,0), the coordinates of the upper left corner position are The coordinates of the upper right corner are The coordinates of the lower left corner are The coordinates of the lower right corner are The coordinates of the remaining positions are deduced in the same way; among them, r i 、c i They represent the number of rows and columns of the small convolution kernel in the i-th layer, both of which are odd numbers and should satisfy: and Substitute the above calculation process into the convolution calculation formula layer by layer until it is replaced with the feature map in the original receptive field, and merge similar items for all items involved in the calculation of the feature maps in the original receptive field, extract the values ​​of the feature maps respectively, and obtain the final calculation result. The expression is as follows: Among them, u i,s,t Represents the weight at the internal coordinate (s, t) of the small convolution kernel of layer i; for x before merging j,k Any item involved in the calculation, s i is the row index number of the i-th layer small convolution kernel weight participating in this calculation; t i is the column index number of the weight of the small convolution kernel of the i-th layer participating in this calculation, which should satisfy ∑ i s i =j and∑ i t i =k, the coefficient of this term is After combining like terms, x j,k The coefficient of S23, based on the two formulas in steps S21 and S22, with the goal of minimizing the error between the final convolution result after replacement and the original convolution result, taking the input feature map x j,k As a variable, the difference between the coefficients of the two equations is expressed as: S24, set u in the formula of S23 j,k The value of u remains unchanged. i,s,t As the variable to be solved, with the goal of minimizing the diff value in formula S23, a genetic algorithm is used to solve the formula in S23 to obtain the weights u of the small convolution kernels in each layer. i,s,t ; or When either the number of rows or the number of columns of the original large convolution kernel is an even number, the replacement module specifically includes: (1) If the number of rows r is even and the number of columns c is odd Among the multiple small convolution kernels after replacement, a convolution kernel with a height of 2 and a width of 1 is first used to make the number of rows of the feature map after the convolution operation r-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel. (2) If the number of columns c is even and the number of rows r is odd Among the multiple small convolution kernels after replacement, a convolution kernel with a width of 2 and a height of 1 is first used, so that the number of columns of the feature map after the convolution operation is c-1, which becomes an odd number. Then n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels are used to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel. (3) If the number of rows r is even, the number of columns c is also even Among the multiple small convolution kernels after replacement, a convolution kernel with a width of 2 and a height of 2 is first used, so that after the convolution operation, the number of rows of the feature map is r-1 and the number of columns is c-1, both of which become odd numbers. Then, n layers of small convolution kernels are used, a total of n+1 layers of small convolution kernels are used to replace the original large convolution kernel, r i 、c i They represent the number of rows and columns of the i-th small convolution kernel in the next n layers, both of which are odd numbers and should satisfy: and Afterwards, the weights of the small convolution kernels in each layer are obtained through a genetic algorithm according to the weight differences between the small convolution kernels in each layer and the original large convolution kernel.

3. A storage medium, characterized in that: It stores a computer program for model conversion for hardware acceleration of deep learning model reasoning, wherein the computer program enables a computer to execute the model conversion method for hardware acceleration of deep learning model reasoning as claimed in claim 1.

4. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the model conversion method for hardware acceleration of deep learning model inference as described in claim 1.

Images