Processing method of neural network model, electronic device and readable medium

Abstract

This application relates to the field of deep learning technology and discloses a method for processing neural network models, an electronic device, and a readable medium. The method for processing neural network models includes: obtaining a first neural network model; determining that the first neural network model includes convolution operators and memory operators; correspondingly determining that the memory operators include breadth-to-depth operators, deleting the breadth-to-depth operators, and folding the channel dimension data in the convolution operators to the height and width dimensions to obtain a second neural network model; correspondingly determining that the memory operators include depth-to-breadth operators, deleting the depth-to-breadth operators, and folding the height and width dimension data in the convolution operators to the channel dimensions to obtain the second neural network model. The above scheme adjusts the data structure of the convolution kernel of the convolution operator based on the dimension transformation method of the memory operator, so that even if the memory operator is deleted, convolution calculations performed only through the convolution operator can obtain the same data processing results as those obtained by the combination operator.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method for processing neural network models, an electronic device, and a readable medium. Background Technology

[0002] Neural network models generally include two types of operators: computational operators (many computational operations but few memory access operations, such as the convolution operator (Conv)) and memory-intensive operators (many memory access operations but few computational operations, such as the transpose operator, depth-to-space (D2S) operator, space-to-depth (S2D) operator, and the concat function operator. In some cases, when a neural network model is deployed in an electronic device, if the model contains a large number of memory-intensive operators, it will consume a significant amount of the device's memory resources. Summary of the Invention

[0003] To reduce the memory space occupied by neural network models, this application provides a method for processing neural network models, an electronic device, and a readable medium.

[0004] In a first aspect, embodiments of this application provide a method for processing a neural network model, applied to an electronic device. The method includes: obtaining a first neural network model; determining that the first neural network model includes a convolution operator and a memory operator; corresponding to determining that the memory operator includes a breadth-to-depth operator, deleting the breadth-to-depth operator, and folding the channel dimension data in the convolution operator to the height and width dimensions to obtain a second neural network model, wherein the breadth-to-depth operator is used to fold the height and width dimensions data in the input data to the channel dimension; corresponding to determining that the memory operator includes a depth-to-breadth operator, deleting the depth-to-breadth operator, and folding the height and width dimensions data in the convolution operator to the channel dimension to obtain a second neural network model, wherein the depth-to-breadth operator is used to fold the channel dimension data in the input data to the height and width dimensions.

[0005] In this embodiment, the data structure of the convolution kernel of the convolution operator can be adjusted based on the dimension transformation method of the memory operator, so that even if the breadth-to-depth operator or the depth-to-breadth operator is deleted, the same data processing result as the combination operator can be obtained by performing convolution calculation only through the convolution operator, which reduces the memory space occupied by the neural network model and improves the running efficiency of the neural network model.

[0006] In one possible implementation, corresponding to determining that the memory operator includes a breadth-to-depth operator, removing the breadth-to-depth operator, and folding the channel dimension data in the convolution operator to the height and width dimensions to obtain a second neural network model, the steps include: removing the breadth-to-depth operator when its output is connected to the input of the convolution operator; determining the adjustment coefficient of the convolution kernel of the convolution operator based on the memory size of the breadth-to-depth operator and the size of the convolution kernel of the convolution operator; adjusting the height, width, and channel dimensions of the convolution kernel of the convolution operator based on the adjustment coefficient to fold the channel dimension data in the convolution operator to the height and width dimensions to obtain the second neural network model.

[0007] In one possible implementation, adjusting the dimensions of the height, width, and channel dimensions of the convolution kernel of the convolution operator based on adjustment coefficients includes: increasing the dimensions of the height and width dimensions of the convolution kernel of the convolution operator by a multiple of the adjustment coefficient; and decreasing the dimensions of the channel dimension of the convolution kernel of the convolution operator by a multiple of a first preset value, where the first preset value is the square of the adjustment coefficient.

[0008] In one possible implementation, corresponding to determining that the memory operator includes a breadth-to-depth operator, removing the breadth-to-depth operator, and folding the channel dimension data in the convolution operator to the height and width dimensions to obtain a second neural network model, the following steps are taken: when the output of the convolution operator is connected to the input of the breadth-to-depth operator, the breadth-to-depth operator is removed; based on the memory size of the breadth-to-depth operator and the size of the convolution kernel of the convolution operator, the adjustment coefficient of the convolution kernel of the convolution operator is determined; based on the adjustment coefficient, the height and width dimensions of the convolution kernel of the convolution operator and the number of convolution kernels of the convolution operator are expanded to obtain the second neural network model.

[0009] In one possible implementation, expanding the height and width dimensions of the convolution kernel of the convolution operator and the number of convolution kernels of the convolution operator based on adjustment coefficients includes: expanding the height and width dimensions of the convolution kernel of the convolution operator by a multiple of a second preset value, where the second preset value is the adjustment coefficient; expanding the number of convolution kernels of the convolution operator by a multiple of a third preset value, where the third preset value is the square of the adjustment coefficient; wherein the convolution kernels of the convolution operator are located at different positions of the convolution kernels of the expanded third preset value convolution operators.

[0010] In one possible implementation, corresponding to determining that the memory operator includes a depth-to-breadth operator, deleting the depth-to-breadth operator, and folding the height and width dimensions of the convolution operator to the channel dimension to obtain a second neural network model, the following steps are taken: when the output of the depth-to-breadth operator is connected to the input of the convolution operator, the depth-to-breadth operator is deleted; the memory size of the depth-to-breadth operator is determined to be equal to the size of the convolution kernel of the convolution operator, and the size of the convolution kernel of the convolution operator is not 1*1; a compression coefficient is determined based on the size of the convolution kernel of the convolution operator, wherein the fourth preset value is an integer greater than 1; the size of the convolution kernel of the convolution operator is adjusted based on the compression coefficient to obtain the second neural network model.

[0011] In one possible implementation, the compression factor can be determined by: determining the height dimension of the convolution kernel of the convolution operator as the compression factor; or determining the width dimension of the convolution kernel of the convolution operator as the compression factor.

[0012] In one possible implementation, the adjustment factor of the convolution kernel of the convolution operator is determined in the following ways: based on the size of the height dimension of the memory operator and the size of the height dimension of the convolution kernel of the convolution operator; or based on the size of the width dimension of the memory operator and the size of the width dimension of the convolution kernel of the convolution operator.

[0013] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for implementing the processing method of any neural network model provided by the first aspect and various possible implementations of the first aspect.

[0014] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to implement the processing method of any neural network model provided by the first aspect and various possible implementations of the first aspect. Attached Figure Description

[0015] Figure 1 According to some embodiments of this application, a schematic diagram of a scenario in which a neural network model 11 is deployed on a computer 20 is shown;

[0016] Figure 2 According to some embodiments of this application, a schematic diagram of a method for calculating a convolution operator is shown;

[0017] Figure 3 According to some embodiments of this application, a schematic diagram of an S2D operator data processing method is shown;

[0018] Figure 4 According to some embodiments of this application, a schematic diagram of a D2S operator data processing method is shown;

[0019] Figure 5 A schematic diagram of a data processing method using a combination operator is shown according to some embodiments of this application;

[0020] Figure 6 A schematic diagram of a data processing method using a combination operator is shown according to some embodiments of this application;

[0021] Figure 7 A schematic diagram of a data processing method using a combination operator is shown according to some embodiments of this application;

[0022] Figure 8 A schematic diagram of a data processing method using a combination operator is shown according to some embodiments of this application;

[0023] Figure 9 A schematic diagram of a data processing method using a combination operator is shown according to some embodiments of this application;

[0024] Figure 10 According to some embodiments of this application, a flowchart of a method for processing a neural network model is shown;

[0025] Figure 11 According to some embodiments of this application, a flowchart of a method for processing a neural network model is shown;

[0026] Figure 12 According to some embodiments of this application, a flowchart of a method for processing a neural network model is shown;

[0027] Figure 13 According to some embodiments of this application, a flowchart of a method for processing a neural network model is shown;

[0028] Figure 14 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 100 is shown. Detailed Implementation

[0029] The illustrative embodiments of this application include, but are not limited to, a method for processing a neural network model, an electronic device, and a readable medium.

[0030] It is understood that the neural network model mentioned in the technical solution of this application can be a neural network model used in tasks such as image recognition, object detection, reinforcement learning, and semantic analysis. The types of neural network models are not limited to convolutional neural networks, Transformer (a model based on multi-head attention mechanism), recurrent neural networks (RNN), long short-term memory neural networks (LSTM), and any other neural network model.

[0031] like Figure 1 As shown, in one specific implementation, server 10 can deploy a neural network model 11 for image detection, processed by the neural network model processing method of this application, to computer 20, so that computer 20 can perform image recognition based on neural network model 11. In some embodiments, computer 20 can also be any implementable electronic device such as a tablet, camera, or mobile phone.

[0032] It is understood that in some embodiments, the neural network model 11 may include computational operators, such as the Conv operator, and memory operators, such as the S2D operator or the D2S operator. It is understood that, to a certain extent, due to the presence of memory operators, such as the D2S operator or the S2D operator, in the neural network model 11, the neural network model 11 will consume a significant amount of memory resources from the computer 20.

[0033] The Conv operator, D2S operator, and S2D operator will be introduced below.

[0034] The Conv operator is used to perform convolution calculations on the input data of the Conv operator and obtain the output data.

[0035] like Figure 2 As shown, the Conv operator can be used to perform convolution calculation on input data A01 and convolution kernel C01 to obtain output data D0, where D0 = 2*2 + 1*1 + 3*0 + 2*1 = 7.

[0036] The S2D operator is used to transform the H (height) and W (width) dimensions of the input data to the C (channel) dimension.

[0037] like Figure 3 As shown, for a 2*2*3 data block A11, the S2D operator can expand the data in the H and W dimensions of the input data A11 to the C dimension, resulting in a 1*1*12 data block A12.

[0038] The D2S operator is used to transform data in the C dimension of the input data to the H and W dimensions.

[0039] like Figure 4As shown, for a 1*1*12 data block A12, the D2S operator can fold the data in dimension C of the input data A12 to dimensions H and W, resulting in a 2*2*3 data block A11.

[0040] The following sections will introduce the processing methods of the three combined operators—S2D and Conv operators, Conv and S2D operators, and D2S and Conv operators—in the neural network model.

[0041] Figure 5 This illustrates the process of convolution processing of input data using the S2D and Conv operators. After the S2D operator receives the input data, it expands the H and W dimensions of the input data to the C dimension. Then, the Conv operator performs a convolution operation between the output data of the S2D operator and the convolution kernel to obtain the calculation result D1.

[0042] The above method is illustrated below using a 2x2x1 data block B11 as an example. After the S2D operator obtains data block B11, it expands the H and W dimensions of data block B11 to the C dimension, resulting in a 1x1x4 data block B12. When performing convolution calculations using the Conv operator, for ease of calculation, the C dimension of the 1x1 convolution kernel K1 of the Conv operator can be equivalent to the C dimension of data block B12; that is, the 1x1 convolution kernel K1 of the Conv operator can also be considered a 1x1x1 convolution kernel. Multiplying the 1x1x4 data block B12 by the 1x1x1 convolution kernel K1 yields a 1x1x1 calculation result D1.

[0043] Figure 6 The diagram illustrates the process of convolution processing of input data using the Conv and S2D operators. After receiving the input data, the Conv operator performs convolution with the convolution kernel to obtain the result D2. The S2D operator reads the result D2 and expands the data in dimensions H and W of D2 to dimension C.

[0044] The above method is illustrated below using a 2x2x1 data block B21 as input data. The C-dimensional size of the 1x1 convolution kernel K1 of the Conv operator can be equivalent to the C-dimensional size of data block B21; that is, the 1x1 convolution kernel K1 of the Conv operator can be considered equivalent to a 1x1x1 convolution kernel. After convolving the 2x2x1 data block B21 with the 1x1x1 convolution kernel K1, a 2x2x1 calculation result D2 is obtained. After obtaining the calculation result D2, the S2D operator expands the H-dimensional and W-dimensional data in the 2x2x1 calculation result D2 along the C-dimensional plane to obtain a 1x1x4 data block B22.

[0045] Figure 7 The diagram illustrates the process of convolution processing of input data using the D2S and Conv operators. After the D2S operator receives the input data, it folds the C-dimensional data into H and W dimensions. Then, the Conv operator performs convolution calculations on the D2S operator's output data and the input data with the convolution kernel, yielding the result D3.

[0046] The above method is illustrated below using a 1*1*4 data block B31 as an example. After the D2S operator obtains data block B31, it folds the C-dimensional data in data block B31 to the H and W dimensions, resulting in a 2*2*1 data block B32. The C-dimensional size of the 2*2 convolution kernel K1 of the Conv operator can be equivalent to the C-dimensional size of data block B32; that is, the 2*2 convolution kernel K1 of the Conv operator can also be equivalent to a 2*2*1 convolution kernel. Multiplying the 2*2*1 data block B32 by the 2*2*1 convolution kernel K1 yields a 1*1*1 calculation result D3.

[0047] As can be understood from the calculation process of the above combined operators, the role of the S2D or D2S operators is merely to transform the dimension of the input data. Therefore, if the S2D or D2S operators in the above combined operators can be removed through a certain method, the original function of the combined operators can still be achieved, and the same calculation results can be obtained, which will effectively reduce the memory usage of the neural network model.

[0048] To address the aforementioned issues, this application provides a model processing method. For neural network models comprising a combination of Conv and memory operators (i.e., S2D or D2S operators), the data structure of the Conv operator's convolution kernel can be adjusted based on the dimension transformation method of the memory operator. For example, if the memory operator is an S2D operator, the corresponding dimension transformation method is to transform the H and W dimensions of the input data to the C dimension. Therefore, the C dimension data of the Conv operator's convolution kernel can be folded to the H and W dimensions. If the memory operator is a D2S operator, the corresponding dimension transformation method is to transform the C dimension data of the input data to the H and W dimensions. Therefore, the H and W dimensions data of the Conv operator can be folded to the C dimension. This ensures that even if the S2D or D2S operator is removed, the same data processing result as with the combination operator can be obtained by performing convolution calculations solely through the Conv operator.

[0049] The following sections explain the data format (size) adjustment methods for the Conv operators corresponding to the three combined operators mentioned above: S2D operator and Conv operator, Conv operator and S2D operator, and D2S operator and Conv operator.

[0050] If the combined operators are S2D and Conv, the data structure of the convolution kernel of the Conv operator can be adjusted in the opposite way to the dimension transformation of the memory operator. For example, the dimension transformation method corresponding to the S2D operator is to transform the H and W dimensions of the data to the C dimension, then the C dimension data of the convolution kernel of the Conv operator can be folded to the H and W dimensions.

[0051] For example, such as Figure 8 As shown, the input data is a 2*2*1 data block B11. The original Conv operator's convolution kernel K1 is 1*1*4. To transform the H-dimensional and W-dimensional data to the C-dimensional data, the H-dimensional and W-dimensional dimensions of the original convolution kernel can be doubled, while the C-dimensional dimension can be reduced by a factor of 4. That is, the convolution kernel K1 is updated to a 2*2*1 convolution kernel K2. The 2*2*1 convolution kernel K2 performs convolution calculation on the data block B11, resulting in a 1*1*1 calculation result D1.

[0052] If the combined operator is a Conv operator and an S2D operator, and the S2D operator is (BS_X1)*(BS_Y1), then the size of the H dimension of the convolution kernel K1 of the Conv operator (BS_X2)*(BS_Y2) can be updated to (BS_X2)(BS_X1), and the size of the W dimension can be updated to (BS_Y2)(BS_Y1). The stride and the number of expanded convolution kernels are then updated based on (BS_X1) or (BS_Y1) of the S2D operator. Here, BS represents the operator's memory size (blocksize), BS_X represents the operator's width, and BS_Y represents the operator's height.

[0053] like Figure 9 As shown, the input data is a 2*2*C data block B21, and the memory size of the S2D operator is 2*2. We can first expand the 1*1*C convolution kernel K1 of the Conv operator into four 2*2*C convolution kernels, updating the stride to 2. Specifically, the 1*1*C convolution kernel K1 is positioned sequentially at each position of the four 2*2*C convolution kernels, with the remaining positions filled with 0. That is, convolving the 2*2*C data block B21 with the four 2*2*C convolution kernels yields four 1*1*1 convolution results, and outputting these four 1*1*1 convolution results sequentially yields the result. Figure 6 The output results shown are the same as the output data. Here, C represents the number of channels in the K1 convolution kernel, which is not specifically limited here.

[0054] If the combined operator is a D2S operator and a Conv operator, and the size of the convolution kernel K1 of the Conv operator is the same as the memory size of the D2S operator, and the Conv operator is not a 1*1 convolution kernel K1, then based on the size of H dimension = 1 and the size of W dimension = 1, the data of the H dimension and W dimension of the convolution kernel K1 of the Conv operator are folded to the C dimension to obtain the convolution kernel K2.

[0055] In some embodiments, the size of the H dimension or the size of the W dimension can be used as a scaling factor to reduce the size of the H dimension of the convolution kernel K1 by a scaling factor multiple, reduce the size of the W dimension of the convolution kernel K1 by a scaling factor multiple, and increase the size of the C dimension of the convolution kernel K1 by a scaling factor squared multiple, so as to obtain a 1*1*C convolution kernel K2.

[0056] For example, given a 1*1*4 data block B31, a D2S memory size of 2*2, and a 2*2*1 convolution kernel K1 for the Conv operator, it can be determined that the H and W dimensions of the original convolution kernel K1 should be reduced by a factor of 2, while the C dimension should be increased by a factor of 4. This means the convolution kernel K1 should be updated to a 1*1*4 convolution kernel K2. The 1*1*4 convolution kernel K2 performs convolution calculations on data block B31, resulting in a 1*1*1 calculation result D3.

[0057] The processing method of the neural network model provided in the embodiments of this application will be described in detail below. The processing method of the neural network model in the embodiments of this application is applied to electronic devices. Figure 10 The diagram illustrates a flowchart of a neural network model processing method according to an embodiment of this application. The neural network model processing method includes:

[0058] 1001: Obtain the first neural network model.

[0059] In this embodiment, the first neural network model can be a neural network model used in tasks such as image recognition, object detection, reinforcement learning, and semantic analysis. The neural network model processing method provided in this embodiment can be performed offline before deploying the neural network model, avoiding impact on its operation.

[0060] 1002: The first neural network model is determined to include convolution operators and memory operators.

[0061] It is understood that the convolution operators in the first neural network model may include Conv operators, and the memory operators may include S2D operators and / or D2S operators.

[0062] 1003: Determine the dimension transformation method for memory operators.

[0063] It's understandable that memory operators are used to read input data. The S2D operator can transform the H and W dimensions of the input data to the C dimension. The D2S operator can transform the C dimension of the input data to the H and W dimensions.

[0064] 1004: Adjust the data structure of the corresponding convolution operator according to the dimension transformation method of the memory operator.

[0065] Based on the dimensionality transformation method of the memory operator, the first neural network model can be identified as including three combination operators: S2D operator and Conv operator, and Conv operator and S2D operator. The processing methods for the Conv operator in these three combination operators are described below. Specifically, the combination of S2D and Conv operators means that the output of the S2D operator is connected to the input of the Conv operator. The combination of Conv and S2D operators means that the output of the Conv operator is connected to the input of the S2D operator. The combination of D2S and Conv operators means that the output of the D2S operator is connected to the input of the Conv operator.

[0066] Figure 11 This illustrates the processing method corresponding to step 1004 when the combination operators are the S2D operator and the Conv operator. The method includes:

[0067] 1101: Determine the memory size of the S2D operator and the size of the convolution kernel K1 of the Conv operator.

[0068] The memory size of the S2D operator includes the size of the S2D operator in the H dimension and the size in the W dimension, and the size of the convolution kernel K1 of the Conv operator includes the size of the convolution kernel K1 in the H dimension and the size in the W dimension.

[0069] 1102: Determine the adjustment coefficients based on the memory size of the S2D operator and the size of the convolution kernel K1 of the Conv operator.

[0070] Wherein, the adjustment coefficient = the C-dimensional size of the S2D operator / the C-dimensional size of the convolution kernel K1 of the Conv operator = the H-dimensional size of the S2D operator / the H-dimensional size of the convolution kernel K1 of the Conv operator.

[0071] 1103: Adjust the data structure of the convolution kernel K1 of the Conv operator according to the adjustment coefficient to obtain the convolution kernel K2.

[0072] In this embodiment, the size of convolution kernel K2 in dimension H = the size of convolution kernel K1 in dimension H * adjustment factor. The size of convolution kernel K2 in dimension W = the size of convolution kernel K1 in dimension W * adjustment factor. The size of convolution kernel K2 in dimension C = the size of convolution kernel K1 in dimension C / (adjustment factor * adjustment factor).

[0073] For example, based on the 2*2*1 S2D operator and the 1*1*4 convolution kernel K1 of the Conv operator, the adjustment coefficient can be determined as 2 / 1 = 2. The size of the H dimension of the adjusted convolution kernel K2 is 1*2 = 2. The size of the W dimension of the convolution kernel K2 is 1*2 = 2. The size of the C dimension of the convolution kernel K2 is 4 / (2*2) = 1. The convolution kernel K1 of the Conv operator is then updated to a 2*2*1 convolution kernel K2.

[0074] like Figure 8 As shown, when the 2*2*1 convolution kernel K2 obtains the 2*2*1 data block B11, the 2*2*1 convolution kernel K2 and the 2*2*1 data block B11 are convolved to obtain the 1*1*1 calculation result D1.

[0075] Figure 12 This illustrates the processing method corresponding to step 1004 when the combination operators are the Conv operator and the S2D operator. The method includes:

[0076] 1201: Determine the memory size of the S2D operator and the size of the convolution kernel K1 of the Conv operator.

[0077] The memory size of the S2D operator includes the size of the S2D operator in the H dimension and the size in the W dimension, and the size of the convolution kernel K1 of the Conv operator includes the size of the convolution kernel K1 in the H dimension and the size in the W dimension.

[0078] 1202: Expand the H-dimensional size of the convolution kernel K1 of the Conv operator according to the H-dimensional size of the S2D operator, and expand the W-dimensional size of the convolution kernel K1 of the Conv operator according to the W-dimensional size of the S2D operator.

[0079] In this embodiment, the size of the S2D operator in dimension H can be multiplied by the size of the convolution kernel K1 in dimension H to obtain the size of the expanded convolution kernel in dimension H. Similarly, the size of the S2D operator in dimension W can be multiplied by the size of the convolution kernel K1 in dimension W to obtain the size of the expanded convolution kernel in dimension W.

[0080] For example, if the S2D operator is (BS_X1)*(BS_Y1), then the size of the H dimension of the (BS_X2)*(BS_Y2) convolution kernel K1 of the Conv operator can be updated to (BS_X2)(BS_X1), and the size of the W dimension can be updated to (BS_Y2)(BS_Y1).

[0081] 1203: Based on the size of the H dimension or the W dimension of the S2D operator, expand the number of convolution kernels to obtain K*K convolution kernels; where convolution kernel K1 is located at different positions of the K*K convolution kernels.

[0082] It can be understood that K = adjustment coefficient = size of C dimension of S2D operator / size of C dimension of convolution kernel K1 of Conv operator = size of H dimension of S2D operator / size of H dimension of convolution kernel K1 of Conv operator.

[0083] For example, such as Figure 9 As shown, the 1*1*C convolution kernel K1 of the Conv operator is expanded into four 2*2*C convolution kernels. The convolution kernel K1 of the Conv operator is located at the first, second, third, and fourth positions of the four 2*2*C convolution kernels, respectively.

[0084] like Figure 9 As shown, when the adjusted convolution kernels of the four 2*2*C Conv operators obtain a 2*2*1 data block B21, the convolution kernels of the four 2*2*C Conv operators are convolved with the 2*2*1 data block B11 respectively, resulting in four 1*1*1 convolution results. These four 1*1*1 convolution results are then output sequentially, thus obtaining the result... Figure 6 The output data shown is the same as the output result.

[0085] Figure 13 This illustrates the processing method corresponding to step 1004 when the combination operators are the D2S operator and the Conv operator. The method includes:

[0086] 1301: Determine if the size of the convolution kernel K1 of the Conv operator is equal to the memory size of the D2S operator. If the result is yes, proceed to 1302: Determine if the size of the convolution kernel K1 of the Conv operator is 1*1*C. If the result is no, end.

[0087] The memory size of the D2S operator includes the size of the H dimension and the size of the W dimension, and the size of the convolution kernel K1 of the Conv operator includes the size of the H dimension and the size of the W dimension.

[0088] 1302: Determine if the size of the convolution kernel K1 of the Conv operator is 1*1*C. If the result is no, proceed to 1303: Based on the size of H dimension = 1 and the size of W dimension = 1, adjust the size of the convolution kernel K1 of the Conv operator to obtain the convolution kernel K2. If the result is yes, end.

[0089] It is understandable that C in 1*1*C can be any positive integer.

[0090] 1303: Based on the size of H dimension = 1 and W dimension = 1, the size of the convolution kernel K1 of the Conv operator is adjusted to obtain the convolution kernel K2.

[0091] In this embodiment of the application, based on the size of H dimension = 1 and the size of W dimension = 1, the data of H dimension and W dimension in convolution kernel K1 can be folded to C dimension to obtain convolution kernel K2. Then, the input data is processed by dimensionality reduction (reducing the size of C dimension) through convolution kernel K2.

[0092] In some embodiments, the size of the H dimension or the size of the W dimension can be used as a scaling factor to reduce the size of the H dimension of the convolution kernel K1 by a scaling factor multiple, reduce the size of the W dimension of the convolution kernel K1 by a scaling factor multiple, and increase the size of the C dimension of the convolution kernel K1 by a scaling factor squared multiple, so as to obtain a 1*1*C convolution kernel K2.

[0093] For example, given a 1*1*4 data block B31 as input, the data structure of the convolution kernel K1 for both the D2S and Conv operators is 2*2*1. Based on this data, it can be determined that the dimensions of the original convolution kernel K1 in the H and W dimensions should be reduced by a factor of 2, while the dimension in the C dimension should be increased by a factor of 4. That is, the convolution kernel K1 should be updated to a 1*1*4 convolution kernel K2. Then, the data block B31 is convolved using the 1*1*4 convolution kernel K2 to obtain a 1*1*1 calculation result D3.

[0094] It is understandable that when the size of the Conv operator is adjusted, the NHWC index of the convolution kernel of the convolution operator will also be updated.

[0095] 1005: Delete the memory operators corresponding to the convolution operators after adjusting the data structure to obtain the second neural network model.

[0096] In this embodiment, when the combined operator is an S2D operator and a Conv operator, or a Conv operator and an S2D operator, the memory operator corresponding to the convolution operator after adjusting the data structure is the S2D operator, i.e., the S2D operator is deleted. When the combined operator is a D2S operator and a Conv operator, the memory operator corresponding to the convolution operator after adjusting the data structure is the D2S operator, i.e., the D2S operator is deleted.

[0097] It is understood that the second neural network model obtained in the embodiments of this application can be a neural network model used in tasks such as image recognition, object detection, reinforcement learning, and semantic analysis, and the input data can be image, audio, video data, etc.

[0098] This application embodiment obtains a second neural network model by deleting the memory operators corresponding to the convolution operators after data structure adjustment in the first neural network model, thereby reducing the memory space occupied by the second neural network model. Furthermore, it reduces the operations and time required to deploy the second neural network model, improving the deployment efficiency of the neural network model.

[0099] In some embodiments, in order to address the problem of neural network models consuming a large amount of memory resources in electronic devices and to improve the operating efficiency of neural network models in electronic devices, memory operators with smaller memory footprints and faster data reading speeds can be selected based on the characteristics of memory operators, such as burst length, cache line, and interleaved memory banks, in order to reduce the memory resources occupied by neural network models and improve their operating efficiency.

[0100] Among them, the memory burst length refers to the number of storage units (columns) involved in the continuous transmission of adjacent storage units in the same row.

[0101] In some embodiments, memory operators can be allocated to input data based on the memory burst length of the memory operator to reduce the waiting time for input data transmission.

[0102] A cache line is a 64-byte data line that the central processing unit (CPU) reads from memory.

[0103] In some embodiments, memory operators with small cache lines can be selected. It is understood that the smaller the cache line, the lower the local space efficiency and the faster the read time.

[0104] Interleaved memory storage refers to the characteristic that each memory operator can access independently without any dependency on the others.

[0105] It is understandable that when memory operators have the characteristic of interleaved memory storage, during the process of CPU reading data, since multiple memory operators take turns providing input data, the CPU can access the input data immediately, reducing the input data reading time.

[0106] However, in the above methods, the computational operators of the neural network model still occupy memory operators, and the memory access operations of memory operators will inevitably consume time. That is, during the operation of the neural network model, a large amount of memory resources of electronic devices will still be occupied due to memory access data.

[0107] The neural network model processing method provided in this application can adjust the data structure of the convolution kernel of the computation operator based on the dimension transformation method of the memory operator, so that even if the memory operator is deleted, the same data processing result as the combination operator can be obtained by performing convolution calculation only through the computation operator, which significantly reduces the memory resources occupied by the neural network model.

[0108] Figure 14 According to some embodiments of this application, a schematic diagram of the structure of an electronic device 100 is shown. For example... Figure 14As shown, the electronic device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output (I / O) devices 105, and system control logic 106 for coupling the processor 101, system memory 102, non-volatile memory 103, communication interface 104, and input / output (I / O) devices 105. Wherein:

[0109] Processor 101 can be used to control an electronic device to execute the processing method of the neural network model of this application. Processor 101 may include one or more processing units, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, or a processing module or circuit of a field-programmable gate array (FPGA). It may include one or more single-core or multi-core processors. System memory 102 is volatile memory, such as random-access memory (RAM), double-data-rate synchronous dynamic random-access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions.

[0110] The non-volatile memory 103 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the non-volatile memory 103 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as a hard disk drive (HDD), a compact disc (CD), a digital versatile disc (DVD), a solid-state drive (SSD), etc. In some embodiments, the non-volatile memory 103 may also be a removable storage medium, such as a Secure Digital (SD) memory card, etc.

[0111] Specifically, system memory 102 and non-volatile memory 103 may each include a temporary copy and a permanent copy of instruction 107. Instruction 107 may include, when executed by processor 101, a processing method that causes electronic device 100 to implement the neural network model provided in the embodiments of this application.

[0112] The communication interface 104 may include a transceiver for providing a wired or wireless communication interface for the electronic device 100, thereby enabling communication with any other suitable device via one or more networks. In some embodiments, the communication interface 104 may be integrated into other components of the electronic device 100, for example, the communication interface 104 may be integrated into the processor 101. In some embodiments, the electronic device 100 may communicate with other devices through the communication interface 104; for example, the electronic device 100 may obtain the processing method of a neural network model to be run from other electronic devices through the communication interface 104.

[0113] Input / output (I / O) device 105 may include input devices such as keyboard, mouse, etc., and output devices such as monitor, etc. Users can interact with electronic device 100 through input / output (I / O) device 105.

[0114] System control logic 106 may include any suitable interface controller to provide any suitable interface to other modules of electronic device 100. For example, in some embodiments, system control logic 106 may include one or more memory controllers to provide an interface to system memory 102 and non-volatile memory 103.

[0115] In some embodiments, at least one of the processors 101 may be packaged together with the logic of one or more controllers for system control logic 106 to form a system-in-package (SiP). In other embodiments, at least one of the processors 101 may also be integrated on the same chip with the logic of one or more controllers for system control logic 106 to form a SoC.

[0116] It is understood that electronic device 100 can be any electronic device capable of running neural networks, including but not limited to mobile phones, wearable devices (such as smartwatches), tablets, desktops, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc., and this application embodiment does not limit it.

[0117] Understandable. Figure 14 The structure of the electronic device 100 shown is merely an example. In other embodiments, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0118] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0119] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0120] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0121] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0122] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0123] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0124] It should be noted that in the examples and description of this patent, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of this application.

Claims

1. A processing method of a neural network model, characterized by, Applied to electronic devices, the method includes: Obtain the first neural network model; The first neural network model was determined to include convolution operators and memory operators; Corresponding to the determination that the memory operator includes a breadth-to-depth operator, when the output of the breadth-to-depth operator is connected to the input of the convolution operator, the breadth-to-depth operator is deleted. Based on the memory size of the breadth-to-depth operator and the size of the convolution kernel of the convolution operator, the adjustment coefficient of the convolution kernel of the convolution operator is determined. Based on the adjustment coefficient, the height dimension, width dimension and channel dimension of the convolution kernel of the convolution operator are adjusted to fold the channel dimension data in the convolution operator to the height dimension and width dimension, thereby obtaining a second neural network model. The breadth-to-depth operator is used to fold the height dimension and width dimension data in the input data to the channel dimension. Corresponding to the determination that the memory operator includes a depth-to-breadth operator, when the output of the convolution operator is connected to the input of the breadth-to-depth operator, the breadth-to-depth operator is deleted. Based on the memory size of the breadth-to-depth operator and the size of the convolution kernel of the convolution operator, the adjustment coefficient of the convolution kernel of the convolution operator is determined. Based on the adjustment coefficient, the height and width dimensions of the convolution kernel of the convolution operator and the number of convolution kernels of the convolution operator are expanded to obtain the second neural network model, wherein the depth-to-breadth operator is used to fold the channel dimension data in the input data to the height and width dimensions; The second neural network model is used for image recognition, and the input data includes image data.

2. The method for processing neural network models according to claim 1, characterized in that, The adjustment of the kernel dimensions (height, width, and channel dimensions) of the convolution operator based on the adjustment coefficients includes: The height and width dimensions of the convolution kernel of the convolution operator are increased by a factor of the adjustment coefficient; The size of the channel dimension in the convolution kernel of the convolution operator is reduced by a first preset value, where the first preset value is the square of the adjustment coefficient.

3. The method for processing neural network models according to claim 1, characterized in that, The process of expanding the height and width dimensions of the convolution kernel of the convolution operator based on the adjustment coefficient, as well as the number of convolution kernels of the convolution operator, includes: The dimensions of the height and width of the convolution kernel of the convolution operator are expanded by a multiple of a second preset value, where the second preset value is the adjustment coefficient; The number of convolution kernels of the convolution operator is increased by a third preset value, where the third preset value is the square of the adjustment coefficient; The convolution kernel of the convolution operator is located at different positions of the convolution kernel of the expanded third preset number of convolution operators.

4. The method for processing neural network models according to claim 1, characterized in that, The step of determining that the memory operator includes a depth-to-breadth operator, deleting the depth-to-breadth operator, and folding the height and width dimensions of the convolution operator to the channel dimension to obtain the second neural network model includes: When the output of the depth-to-breadth operator is connected to the input of the convolution operator, the depth-to-breadth operator is deleted; The memory size from depth to breadth is determined to be equal to the size of the convolution kernel of the convolution operator, and the size of the convolution kernel of the convolution operator is not 1*1. The compression coefficient is determined based on the size of the convolution kernel of the convolution operator. The size of the convolution kernel of the convolution operator is adjusted based on the compression coefficient to obtain the second neural network model.

5. The method for processing neural network models according to claim 4, characterized in that, The method for determining the compression factor includes: The height dimension of the convolution kernel of the convolution operator is determined as the compression factor; or The width dimension of the convolution kernel of the convolution operator is determined as the compression factor.

6. The method for processing the neural network model according to claim 2 or 3, characterized in that, The methods for determining the adjustment coefficients of the convolution kernel of the convolution operator include: The adjustment coefficient of the convolution kernel of the convolution operator is determined based on the height dimension of the memory operator and the height dimension of the convolution kernel of the convolution operator; or The adjustment factor of the convolution kernel of the convolution operator is determined based on the size of the width dimension of the memory operator and the size of the width dimension of the convolution kernel of the convolution operator.

7. An electronic device, characterized in that, It includes: a memory for storing instructions executed by one or more processors of the electronic device, and the processor being one of the one or more processors of the electronic device for executing a processing method of the neural network model according to any one of claims 1 to 6.

8. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the processing method of the neural network model according to any one of claims 1 to 6.

Images