Compiling method and device of neural network model, storage medium and electronic equipment

Abstract

A method and apparatus for compiling a neural network model, a storage medium, and an electronic device are disclosed. The method includes obtaining a neural network model to be compiled, the neural network model to be compiled including a first network layer corresponding to a first operation type being a variant of a rectified linear unit (ReLU) operation; transforming, based on a layer parameter of the first network layer, a first operation of an input feature map of the first network layer into a second operation to obtain a second network layer, the second operation being an operation supported by a neural network accelerator; and generating, based on network layers other than the first network layer in the neural network model to be compiled and the second network layer, a target neural network model through a compilation process. Embodiments of the present disclosure can implement the variant of the ReLU operation without data migration, thereby effectively saving bandwidth and time, reducing the implementation cost of the variant of the ReLU operation, and avoiding occupying additional resources.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence technology, and in particular to a method, apparatus, storage medium, and electronic device for compiling a neural network model. Background Technology

[0002] In some cases, on-chip neural network accelerators do not support variants of the Rectified Linear Unit (ReLU) operation, while in practical applications there are variants of neural network models that require ReLU operations during runtime. The current solution for this situation is to move the data of the variants that require ReLU operations to the Central Processing Unit (CPU) for processing, and then move the results obtained by the CPU back to the neural network accelerator. However, this solution has the problems of wasted bandwidth and time and high cost due to the need for data relocation. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, storage medium, and electronic device for compiling a neural network model.

[0004] According to one aspect of the present disclosure, a method for compiling a neural network model is provided, comprising: Obtain a neural network model to be compiled, wherein the neural network model to be compiled includes a first network layer whose corresponding operation type is a first operation, and the first operation is a variant of the modified linear unit operation; Based on the layer parameters of the first network layer, the first operation of the input feature map of the first network layer is transformed into a second operation to obtain the second network layer. The second operation is an operation supported by the neural network accelerator. Based on the network layers other than the first network layer in the neural network model to be compiled, and the second network layer, a target neural network model is generated through compilation processing.

[0005] According to another aspect of the present disclosure, a neural network model compilation apparatus is provided, comprising: The first acquisition module is used to acquire a neural network model to be compiled, wherein the neural network model to be compiled includes a first network layer whose corresponding operation type is a first operation, and the first operation is a variant of the modified linear unit operation; The second acquisition module is used to transform the first operation of the input feature map of the first network layer into a second operation based on the layer parameters of the first network layer in the neural network model to be compiled obtained by the first acquisition module, so as to obtain a second network layer. The second operation is an operation supported by the neural network accelerator. The generation module is used to generate a target neural network model by compiling the network layers other than the first network layer in the neural network model to be compiled, obtained by the first acquisition module, and the second network layer obtained by the second acquisition module.

[0006] According to another aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing the compilation method of the neural network model described above.

[0007] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the compilation method of the neural network model described above.

[0008] Based on the neural network model compilation method, apparatus, storage medium, and electronic device provided in the above embodiments of this disclosure, during the compilation of the neural network model, variants of the ReLU operation can be transformed into operations supported by the neural network accelerator. In this way, even if the neural network accelerator does not support variants of the ReLU operation, variants of the ReLU operation can be indirectly implemented through operations equivalent to variants of the ReLU operation. Therefore, the embodiments of this disclosure can implement variants of the ReLU operation without data migration, thereby effectively saving bandwidth and time, reducing the implementation cost of variants of the ReLU operation, and avoiding the occupation of additional resources.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is a schematic diagram of a variant of the ReLU operation implemented in an embodiment of this disclosure.

[0012] Figure 2 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure.

[0013] Figure 3 This is a flowchart illustrating a method for compiling a neural network model provided in another exemplary embodiment of this disclosure.

[0014] Figure 4-1 This is a schematic diagram of the first set of convolution kernels in an exemplary embodiment of this disclosure.

[0015] Figure 4-2 This is a schematic diagram of the operational logic of the second operation in an exemplary embodiment of this disclosure.

[0016] Figure 4-3 This is a schematic diagram of the operational logic of the second operation in another exemplary embodiment of this disclosure.

[0017] Figure 5 This is a schematic diagram of the structure of a compilation device for a neural network model provided in an exemplary embodiment of this disclosure.

[0018] Figure 6 This is a schematic diagram of the structure of a compilation device for a neural network model provided in another exemplary embodiment of this disclosure.

[0019] Figure 7 This is a schematic diagram of the structure of a compilation apparatus for a neural network model provided in another exemplary embodiment of the present disclosure.

[0020] Figure 8 This is a schematic diagram of the structure of a compilation device for a neural network model provided in yet another exemplary embodiment of this disclosure.

[0021] Figure 9 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0024] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0025] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0026] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0027] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0028] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0029] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0030] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0031] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0032] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0033] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0034] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0035] Application Overview Some chips can have neural network accelerators deployed on them; for example, artificial intelligence (AI) chips can have brain processing units (BPUs) deployed on them. It should be noted that neural network accelerators can be used to implement neural network models, such as neural network models for object detection.

[0036] In the process of realizing this disclosure, the inventors discovered that in some cases, the neural network accelerators deployed on the chip do not support variants of ReLU operation, while in practical applications there are variants of neural network models that require ReLU operation during runtime; among them, variants of ReLU operation include but are not limited to Leaky ReLUs operation, PreLU operation, and RreLU operation.

[0037] The ReLU operation can be used to set all negative values ​​in an input matrix to 0, while leaving the rest of the values ​​unchanged. The mathematical expression for the ReLU operation is:

[0038] The Leaky ReLUs operation can be used to assign a non-zero slope to negative values ​​in an input matrix, while leaving the other values ​​in the input matrix unchanged. The mathematical expression for the Leaky ReLUs operation is as follows:

[0039] in, i is a fixed parameter within a predefined interval (1, +∞); To assign a non-zero slope to negative values.

[0040] The PreLU operation and the Leaky ReLU operation have similar operational logic. The main difference is that when using the PreLU operation, a non-zero slope is assigned to negative values ​​(i.e., ...). These are not predefined parameters, but rather parameters determined based on data.

[0041] For variants of neural network accelerators deployed on-chip that do not support ReLU operations, but where the neural network model requires ReLU operations during runtime, the current solution is to move the data of the variant requiring ReLU operations to the CPU for ReLU execution, and then move the result obtained by the CPU back to the neural network accelerator. However, this solution wastes bandwidth and time due to the data movement, resulting in high costs. Furthermore, it consumes CPU resources. Therefore, a more reasonable solution is needed to address this situation.

[0042] Exemplary System It should be noted that even if the neural network accelerators deployed on the chip do not support variants of the ReLU operation, they often support some common operation types, such as convolution, ReLU, and elementwise operations.

[0043] In view of this, in the embodiments of this disclosure, such as Figure 1 As shown, the neural network model involves two stages: compilation and execution. During compilation, variants of the ReLU operation can be transformed into operations supported by the neural network accelerator. During execution, variants of the ReLU operation are indirectly implemented through the actual execution of operations supported by the neural network accelerator. Thus, even if the neural network accelerator does not support variants of the ReLU operation, variants of the ReLU operation can still be implemented without data migration or additional resource consumption.

[0044] Exemplary methods Figure 2 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure. Figure 2 The method shown can be applied to compilers. Figure 2 The method shown includes steps 210, 220 and 230, which are explained below.

[0045] Step 210: Obtain the neural network model to be compiled. The neural network model to be compiled includes a first network layer with the corresponding operation type of the first operation, which is a variant of the ReLU operation.

[0046] Optionally, the first operation includes, but is not limited to, Leaky ReLU operations, PreLU operations, RreLU operations, etc. When the first operation is a Leaky ReLU operation, the specific operational logic of the first operation can be found in the mathematical expression of the Leaky ReLU operation above.

[0047] It should be noted that the neural network model to be compiled refers to the neural network model that needs to be compiled. The neural network model to be compiled may include multiple network layers, and among the multiple network layers, there may be at least one network layer whose corresponding operation type is the first operation. Any one of the at least one network layer can be used as a first network layer in step 210.

[0048] Step 220: Based on the layer parameters of the first network layer, the first operation of the input feature map of the first network layer is transformed into a second operation to obtain the second network layer. The second operation is an operation supported by the neural network accelerator.

[0049] It should be noted that the layer parameters of the first network layer may include parameters required to implement variants of the ReLU operation, such as those involved in the mathematical expression for the Leaky ReLU operation mentioned above. i.

[0050] Optionally, the second operation includes, but is not limited to, common operation types such as convolution, ReLU, and element-wise operations.

[0051] It should be noted that in step 220, transforming the first operation into the second operation essentially means transforming the first operation into an equivalent operation. In other words, the result obtained by executing the equivalent operation is the same as or basically the same as the result obtained by executing the first operation.

[0052] Step 230: Based on the network layers other than the first network layer and the second network layer in the neural network model to be compiled, generate the target neural network model through compilation processing.

[0053] Optionally, the network layers in the neural network model to be compiled, excluding the first network layer, include, but are not limited to, network layers whose corresponding operation type is convolution (i.e., convolutional layer), network layers whose corresponding operation type is pooling (i.e., pooling layer), network layers whose corresponding operation type is ReLU (i.e., ReLU layer), etc.

[0054] In step 230, the compiler backend can perform compilation processing based on the network layers in the neural network model to be compiled, excluding the first network layer, and the second network layer obtained through operation transformation, thereby generating a binary target neural network model. The specific compilation processing method can be any feasible method according to actual needs, and this disclosure will not elaborate on it.

[0055] Based on the neural network model compilation method provided in the above embodiments of this disclosure, during the compilation of the neural network model, the variant of ReLU operation can be transformed into an operation supported by the neural network accelerator. In this way, even if the neural network accelerator does not support the variant of ReLU operation, the variant of ReLU operation can be indirectly implemented through an operation equivalent to the variant of ReLU operation. Therefore, the embodiments of this disclosure can implement the variant of ReLU operation without data migration, thereby effectively saving bandwidth and time, reducing the implementation cost of the variant of ReLU operation, and avoiding the occupation of additional resources.

[0056] In an optional example, the first network layer includes C parameters, where C is the number of channels in the input feature map. Here, there can be a one-to-one correspondence between the C parameters and the C channels of the input feature map. Assuming the first operation is a Leaky ReLU operation, the parameter corresponding to any channel can be 1 / [the number of channels used when performing the Leaky ReLU operation on that channel]. i.

[0057] exist Figure 2 Based on the illustrated embodiments, as Figure 3 As shown, step 220 includes steps 2201, 2203 and 2205.

[0058] Step 2201: Determine the smaller value between each of the C parameters and the first preset value to obtain the C smaller values ​​corresponding to the C parameters.

[0059] Optionally, the first preset value can be 1.

[0060] In step 2201, the C parameters can be compared one by one with 1, which is the first preset value, to obtain C smaller values ​​corresponding to the C parameters.

[0061] Step 2203: Determine the first difference between each of the C smaller values ​​and the first preset value, and obtain the C first differences corresponding to the C smaller values.

[0062] In step 2203, the C smaller values ​​can be subtracted one by one from 1, which is the first preset value, to obtain C first difference values ​​corresponding to the C smaller values.

[0063] Step 2205: Based on C first differences, transform the first operation of the input feature map of the first network layer into a second operation.

[0064] In one specific embodiment, step 2205 includes: Determine the larger value between each of the C parameters and the first preset value to obtain the C larger values ​​corresponding to the C parameters; Determine the second difference between the first preset value and each of the C larger values ​​to obtain the C second differences corresponding to the C larger values; Based on C first differences and C second differences, the first operation of the input feature map of the first network layer is transformed into the second operation.

[0065] Here, the C parameters can be compared one by one with 1, which is the first preset value, to obtain the C larger values ​​corresponding to the C parameters. Then, the difference between 1, which is the first preset value, and the C larger values ​​can be obtained to obtain the C second difference values ​​corresponding to the C larger values. Then, based on the C first difference values ​​and the C second difference values, the first operation of the input feature map of the first network layer can be transformed into the second operation.

[0066] Optionally, based on C first differences and C second differences, the first operation of the input feature map of the first network layer is transformed into a second operation, including: Construct the first set of convolutional kernels based on C first differences; Construct a second set of convolutional kernels based on C second differences; Based on C second preset values, construct a third set of convolutional kernels; Based on C third preset values, construct a fourth set of convolution kernels; Based on the first set of convolutional kernels, the second set of convolutional kernels, the third set of convolutional kernels, and the fourth set of convolutional kernels, the first operation of the input feature map of the first network layer is transformed into the second operation.

[0067] Optionally, the second preset value can be 1, the third preset value can be -1, and the set of convolutional kernels on which the first operation transforms the input feature map into the second operation is based can satisfy at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences, and the remaining elements in the first set of convolutional kernels are all 0. The second convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The j-th element of the j-th convolutional kernel in the second convolutional kernel set is the j-th second difference among the C second differences, and the remaining elements in the second convolutional kernel set are all 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements in the third convolutional kernel set are all 0. The fourth convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The t-th element of the t-th convolutional kernel in the fourth convolutional kernel set is -1, and the remaining elements in the fourth convolutional kernel set are all 0.

[0068] Assuming C is 3, and the C first differences are z1, z2, and z3, then the first set of convolutional kernels can include three convolutional kernels with a width of 1, a height of 1, and 3 channels. Furthermore, the three convolutional kernels in the first set satisfy the following: the first element of the first convolutional kernel is z1, the second element of the second convolutional kernel is z2, and the third element of the third convolutional kernel is z3. All other elements are 0. Thus, the first set of convolutional kernels can be seen in [reference needed]. Figure 4-1 , Figure 4-1 kernel0, kernel1, ..., kernelli can each be used as a convolution kernel in the first set of convolution kernels. Figure 4-1 x0, x1, ..., x(C-1) indicate the positions of the first differences.

[0069] In this way, by determining the shape of the convolution kernel set based on C and 1, and by placing the first difference at a specific position in the convolution kernel set and placing 0 at the remaining positions in the convolution kernel set, the first convolution kernel set can be constructed efficiently and reliably.

[0070] It should be noted that the construction methods of the second, third, and fourth convolutional kernel sets are similar to those of the first convolutional kernel set. The only difference is that the second convolutional kernel set places the second difference at a specific position, the third convolutional kernel set places 1 at a specific position, and the fourth convolutional kernel set places -1 at a specific position. In this way, the second, third, and fourth convolutional kernel sets can also be constructed efficiently and reliably.

[0071] When the first, second, third, and fourth convolution kernel sets are all known, these sets can be used for the transformation from the first operation to the second operation. Optionally, the second operation may include: The first feature map is obtained by performing a convolution operation between the first set of convolution kernels and the input feature map; Perform a ReLU operation on the first feature map to obtain the second feature map; The third feature map is obtained by performing a convolution operation between the second set of convolution kernels and the input feature map; Perform a ReLU operation on the third feature map to obtain the fourth feature map; The third set of convolution kernels is convolved with the second feature map to obtain the fifth feature map. The fourth convolution kernel set and the fourth feature map are convolved to obtain the sixth feature map; The input feature map, the fifth feature map, and the sixth feature map are added element-wise to obtain the output feature map of the second network layer.

[0072] It should be noted that the output feature map of the second network layer obtained after the operation transformation can be the same as the output feature map of the first network layer before the operation transformation.

[0073] Let `factor` represent an array of C parameters in the first network layer, `a` represent an array of C smaller values ​​obtained by comparing each of the C parameters in `factor` with 1 (a first preset value), and `b` represent an array of C larger values ​​obtained by comparing each of the C parameters in `factor` with 1 (a first preset value). Then, the first convolutional kernel set can be represented as `a-1`, and the second convolutional kernel set can be represented as `1-b`. Furthermore, the input feature map can be represented as `F0`, and the output feature map of the second network layer can be represented as `F2`. Therefore: F2 = F0 + ReLU[(a-1)] F0]-ReLU[(1-b) F0] a = min(factor, 1) b = max(factor, 1) By transforming the above expression, we can obtain the following expression (for ease of explanation, it will be referred to as the target expression below): F2 = F0 + { 1 ReLU[(a-1) F0]}+{(-1】 ReLU[(1-b) F0]} in," The plus sign (") indicates a convolution operation, and the plus sign (") indicates an element-wise add operation. Thus, (a-1) F0 can be used as the first feature map, ReLU[(a-1)] F0] can be used as the second feature map, (1-b) F0 can be used as the third feature map, ReLU[(1-b)] F0] can be used as the fourth feature map, 1 ReLU[(a-1) F0] in " "The '1' at the beginning can represent the set of the third convolution kernel, 1" ReLU[(a-1) F0] can be used as the fifth feature map, (-1) ReLU[(1-b) F0] in " "The -1 at the beginning can represent the set of the fourth convolution kernel, (-1)" ReLU[(1-b) F0] can be used as the sixth feature map.

[0074] It should be noted that the specific operational logic for the second operation can be found in [link to relevant documentation]. Figure 4-2 , Figure 4-2 F in a = ReLU[(a-1) F0], Figure 4-2 F in b = ReLU[(1-b) F0], Figure 4-2 In this context, weight 'a' is equivalent to the first set of convolution kernels mentioned above. Figure 4-2 In this context, weight b is equivalent to the set of the second convolution kernel mentioned above. Figure 4-2 In this context, weight 1 is equivalent to the third convolution kernel set mentioned above. Figure 4-2 In this context, weight -1 is equivalent to the fourth set of convolution kernels mentioned above. Figure 4-2 In this context, F1 = F0 + ReLU[(a-1)] F0].

[0075] In one example, a pixel in F0 has a value of F. 0（i） The pixel value corresponding to that pixel value in F2 is F. 2（i） In factor and F 0（i） The parameter corresponding to the channel is factor. i Then, combining the target expression above, we can determine: F 2（i） = F 0（i） +{ 1 ReLU[(a i -1) F 0（i） ]}+ {(-1) ReLU[(1-b i ) F 0（i） ]} Among them, a i For factor i Compared to the smaller value in 1, bi For factor i The larger value among 1.

[0076] Assuming factor i If a > 1, then a i =1, b i =factor i a i -1=0, 1-b i 0. We will discuss two cases below: Case 1: F 0（i） 0, then (1-b) i ) F 0（i） 0, ReLU[(1-b i ) F 0（i） If ] = 0, then F 2（i） = F 0（i） +0+0= F 0（i） ; Scenario 2: F 0（i） 0, then (1-b) i ) F 0（i） >0, ReLU[(1-b i ) F 0（i） ] = (1 - b i ) F 0（i） So, F 2（i） = F 0（i） +0+(-1) (1-b) i ) F 0（i） = b i F 0（i） = factor i F 0（i） .

[0077] Assuming factor i 1, then a i = factor i b i =1, a i -1 0,1-b i If the value is 0, we will discuss two cases below: Case 1: F0（i） 0, then (a) i -1) F 0（i） 0, ReLU[(a i -1) F 0（i） If ] = 0, then F 2（i） = F 0（i） +0+0= F 0（i） ; Scenario 2: F 0（i） 0, then (a) i -1) F 0（i） >0, ReLU[(a i -1) F 0（i） ] = (a i -1) F 0（i） So, F 2（i） = F 0（i） +(a i -1) F 0（i） +0= a i F 0（i） = factor i F 0（i） .

[0078] As shown in the examples above, when a pixel value in the input feature map is non-negative, the result of the second operation is the pixel value itself. When the pixel value is negative, the result is equivalent to assigning a non-zero slope to that pixel value. Combining this with the mathematical expression for ReLU operations mentioned above, the logic of Leaky ReLU operations (which belong to the first operation) is to assign a non-zero slope to negative values ​​in the input feature map while keeping the other values ​​in the input feature map unchanged. Clearly, the result obtained using the second operation is consistent with the result obtained using the first operation. Therefore, it can be proven that the output feature map of the second network layer obtained after the operation transformation is the same as the output feature map of the first network layer before the operation transformation.

[0079] It is easy to see that the second operation specifically includes convolution operation, ReLU operation, and element-wise addition operation. Since neural network accelerators usually support convolution operation, ReLU operation, and element-wise addition operation, this can ensure the normal and fast implementation of the second operation, which is conducive to ensuring the normal implementation of the variant of ReLU operation.

[0080] In this implementation, by comparing the magnitudes of the values, C larger values ​​can be determined efficiently and reliably. By performing subtraction, C second differences can be determined efficiently and reliably. With reference to C first differences and C second differences, various variants of the conversion from the first operation to the ReLU operation can be realized efficiently and reliably.

[0081] In another specific implementation, the first operation is a Leaky ReLU operation, and step 2205 includes: Construct the first set of convolutional kernels based on C first differences; Based on C second preset values, construct a third set of convolutional kernels; Based on the first set of convolutional kernels and the third set of convolutional kernels, the first operation of the input feature map of the first network layer is transformed into the second operation.

[0082] Optionally, the second preset value is 1, and the set of convolutional kernels on which the first operation transforming the input feature map into the second operation is based satisfies at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences, and the remaining elements in the first set of convolutional kernels are all 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements in the third convolutional kernel set are all 0.

[0083] It should be noted that the construction methods of the first convolution kernel set and the third convolution kernel set can refer to the relevant description of the previous specific implementation method in step 2205, and will not be repeated here.

[0084] Optionally, the second operation may include: The first feature map is obtained by performing a convolution operation between the first set of convolution kernels and the input feature map; Perform a ReLU operation on the first feature map to obtain the second feature map; The third set of convolution kernels is convolved with the second feature map to obtain the fifth feature map. The input feature map and the fifth feature map are added element-wise to obtain the output feature map of the second network layer.

[0085] Thus, the second operation can be expressed by the following formula: F2 = F0 + ReLU[(a-1)] F0] a = min(factor, 1) It should be noted that each parameter in the factor of Leaky ReLUs is between 0 and 1. Therefore, for the case where the first operation is a Leaky ReLUs operation, each larger value in the array represented by b is 1. Correspondingly, in the previous specific embodiment of step 2205, each element in the second convolution kernel set represented by 1-b is 0. Therefore, it can be considered that in the above target expression, (-1) ReLU[(1-b) Since F0]=0, the objective expression can be simplified to: F2= F0+{ 1 ReLU[(a-1) F0]}.

[0086] It should be noted that the specific operational logic for the second operation can be found in [link to relevant documentation]. Figure 4-3 ,and Figure 4-2 compared to, Figure 4-3 Remove F b The relevant parts. It is easy to see that, for the case where the first operation is Leaky ReLU operation, the second operation specifically includes convolution operation, ReLU operation, and element-wise addition operation. Since neural network accelerators usually support product operation, ReLU operation, and element-wise addition operation, this can ensure the normal and fast implementation of the second operation, thereby helping to ensure the normal implementation of Leaky ReLU operation.

[0087] In this implementation, based on C first differences and second preset values, the construction of the first convolution kernel set can be achieved efficiently and reliably. Based on C and the second preset values, the construction of the third convolution kernel set can be achieved efficiently and reliably. In order to refer to the first convolution kernel set and the third convolution kernel set, the transformation from the first operation to the Leaky ReLU operation can be achieved efficiently and reliably.

[0088] In the embodiments of this disclosure, by comparing the magnitudes of the numerical values, C smaller values ​​can be determined efficiently and reliably. By performing subtraction, C first differences can be determined efficiently and reliably. These C first differences can then be used as a reference to efficiently and reliably convert from the first operation to the second operation. It should be noted that the two specific implementations of step 2205 above describe two conversion methods. The former implementation is applicable to various variations of the second operation being a ReLU operation, while the latter implementation is only applicable to cases where the second operation is a Leaky ReLU operation.

[0089] The compilation method for any neural network model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the compilation method for any neural network model provided in this disclosure can be executed by a processor, such as by a processor executing the compilation method for any neural network model mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0090] Exemplary device Figure 5 This is a schematic diagram of the structure of a compilation device for a neural network model provided in an exemplary embodiment of this disclosure. Figure 5 The apparatus shown includes a first acquisition module 510, a second acquisition module 520, and a generation module 530.

[0091] The first acquisition module 510 is used to acquire the neural network model to be compiled. The neural network model to be compiled includes a first network layer whose corresponding operation type is a first operation, and the first operation is a variant of the ReLU operation. The second acquisition module 520 is used to transform the first operation of the input feature map of the first network layer into a second operation based on the layer parameters of the first network layer in the neural network model to be compiled obtained by the first acquisition module 510, so as to obtain the second network layer. The second operation is an operation supported by the neural network accelerator. The generation module 530 is used to generate a target neural network model by compiling the network layers other than the first network layer in the neural network model to be compiled, which are obtained by the first acquisition module 510, and the second network layer obtained by the second acquisition module 520.

[0092] In one optional example, the layer parameters of the first network layer include C parameters, where C is the number of channels in the input feature map; like Figure 6 As shown, the second acquisition module 520 includes: The first acquisition submodule 5201 is used to determine the smaller value between each of the C parameters and the first preset value, so as to obtain the C smaller values ​​corresponding to the C parameters; The second acquisition submodule 5203 is used to determine the first difference between each of the C smaller values ​​obtained by the first acquisition submodule 5201 and the first preset value, so as to obtain C first differences corresponding to the C smaller values. The transformation submodule 5205 is used to transform the first operation of the input feature map of the first network layer into the second operation based on the C first differences obtained by the second acquisition submodule 5203.

[0093] In an optional example, such as Figure 7 As shown, the transformation submodule 5205 includes: The first acquisition unit 52051 is used to determine the larger value between each of the C parameters and the first preset value, so as to obtain the C larger values ​​corresponding to the C parameters; The second acquisition unit 52053 is used to determine the second difference between the first preset value and each of the C larger values ​​obtained by the first acquisition unit 52051, so as to obtain C second differences corresponding to the C larger values. The first transformation unit 52055 is used to transform the first operation of the input feature map of the first network layer into the second operation based on the C first differences obtained by the second acquisition submodule 5203 and the C second differences obtained by the second acquisition unit 52053.

[0094] In one optional example, the first transformation unit 52055 includes: The first construction subunit is used to construct a first set of convolutional kernels based on the C first differences obtained by the second acquisition submodule 5203; The second construction subunit is used to construct a second set of convolutional kernels based on the C second differences obtained by the second acquisition unit 52053; The third construction subunit is used to construct a third set of convolutional kernels based on C second preset values; The fourth construction subunit is used to construct a fourth set of convolutional kernels based on C third preset values; The transformation subunit is used to transform the first operation of the input feature map of the first network layer into the second operation based on the first set of convolutional kernels constructed by the first construction subunit, the second set of convolutional kernels constructed by the second construction subunit, the third set of convolutional kernels constructed by the third construction subunit, and the fourth set of convolutional kernels constructed by the fourth construction subunit.

[0095] In one optional example, the second preset value is 1, the third preset value is -1, and the set of convolutional kernels on which the first operation transforming the input feature map into the second operation is based satisfies at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences, and the remaining elements in the first set of convolutional kernels are all 0. The second convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The j-th element of the j-th convolutional kernel in the second convolutional kernel set is the j-th second difference among the C second differences, and the remaining elements in the second convolutional kernel set are all 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements in the third convolutional kernel set are all 0. The fourth convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The t-th element of the t-th convolutional kernel in the fourth convolutional kernel set is -1, and the remaining elements in the fourth convolutional kernel set are all 0.

[0096] In one optional example, the second operation includes: The first feature map is obtained by performing a convolution operation between the first set of convolution kernels and the input feature map; Perform a ReLU operation on the first feature map to obtain the second feature map; The third feature map is obtained by performing a convolution operation between the second set of convolution kernels and the input feature map; Perform a ReLU operation on the third feature map to obtain the fourth feature map; The third set of convolution kernels is convolved with the second feature map to obtain the fifth feature map. The fourth convolution kernel set and the fourth feature map are convolved to obtain the sixth feature map; The input feature map, the fifth feature map, and the sixth feature map are added element-wise to obtain the output feature map of the second network layer.

[0097] In one optional example, the first operation is the Leaky ReLUs operation; like Figure 8 As shown, the transformation submodule 5205 includes: The first construction unit 52057 is used to construct a first set of convolutional kernels based on C first differences; The second construction unit 52058 is used to construct a third set of convolutional kernels based on C second preset values; The second transformation unit 52059 is used to transform the first operation of the input feature map of the first network layer into the second operation based on the first set of convolutional kernels and the third set of convolutional kernels.

[0098] In one optional example, the second preset value is 1, and the set of convolutional kernels on which the first operation transforming the input feature map into the second operation is based satisfies at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences, and the remaining elements in the first set of convolutional kernels are all 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements in the third convolutional kernel set are all 0.

[0099] In one optional example, the second operation includes: The first feature map is obtained by performing a convolution operation between the first set of convolution kernels and the input feature map; Perform a ReLU operation on the first feature map to obtain the second feature map; The third set of convolution kernels is convolved with the second feature map to obtain the fifth feature map. The input feature map and the fifth feature map are added element-wise to obtain the output feature map of the second network layer.

[0100] Exemplary electronic devices Below, for reference Figure 9 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0101] Figure 9 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0102] like Figure 9 As shown, the electronic device 900 includes one or more processors 901 and memory 902.

[0103] The processor 901 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 900 to perform desired functions.

[0104] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the compilation methods of the neural network models of the various embodiments of this disclosure described above, and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0105] In one example, the electronic device 900 may also include an input device 903 and an output device 904, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0106] For example, when the electronic device is a first device or a second device, the input device 903 can be the aforementioned microphone or microphone array for capturing the input signal from the sound source. When the electronic device is a standalone device, the input device 903 can be a communication network connector for receiving the acquired input signals from the first device and the second device.

[0107] In addition, the input device 903 may also include, for example, a keyboard, a mouse, etc.

[0108] The output device 904 can output various information to the outside, including determined distance information, direction information, etc. The output device 904 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0109] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 900 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 900 may include any other suitable components depending on the specific application.

[0110] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for compiling neural network models according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0111] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0112] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the compilation methods of neural network models according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0113] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0114] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0115] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0116] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0117] The above description of the disclosed aspects is provided to enable those skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0118] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for compiling a neural network model, comprising: Obtain a neural network model to be compiled, wherein the neural network model to be compiled includes a first network layer whose corresponding operation type is a first operation, and the first operation is a variant of the modified linear unit operation; Based on the layer parameters of the first network layer, the first operation of the input feature map of the first network layer is transformed into a second operation to obtain the second network layer. The second operation is an operation supported by the neural network accelerator. The second operation includes: performing a convolution operation on the first set of convolution kernels and the input feature map to obtain a first feature map; performing a modified linear unit operation on the first feature map to obtain a second feature map; performing a convolution operation on the second set of convolution kernels and the input feature map to obtain a third feature map; performing a modified linear unit operation on the third feature map to obtain a fourth feature map; performing a convolution operation on the third set of convolution kernels and the second feature map to obtain a fifth feature map; performing a convolution operation on the fourth set of convolution kernels and the fourth feature map to obtain a sixth feature map; and performing a convolution operation on the input feature map. The fifth feature map and the sixth feature map are element-wise added together to obtain the output feature map of the second network layer; wherein, the layer parameters of the first network layer include C parameters, where C is the number of channels of the input feature map, the first convolutional kernel set is constructed based on C first differences, the second convolutional kernel set is constructed based on C second differences, the C first differences and the C second differences are determined based on the C parameters and a first preset value, the third convolutional kernel set is constructed based on C second preset values, and the fourth convolutional kernel set is constructed based on C third preset values; Based on the network layers other than the first network layer in the neural network model to be compiled, and the second network layer, a target neural network model is generated through compilation processing.

2. The method according to claim 1, wherein, The transformation of the first operation of the input feature map of the first network layer into a second operation based on the layer parameters of the first network layer includes: Determine the smaller value between each of the C parameters and the first preset value to obtain the C smaller values ​​corresponding to the C parameters; Each of the C smaller values ​​is determined to have a first difference between it and the first preset value, thus obtaining C first differences corresponding to the C smaller values; Based on the C first differences, the first operation of the input feature map of the first network layer is transformed into a second operation.

3. The method according to claim 2, wherein, The step of transforming the first operation of the input feature map of the first network layer into a second operation based on the C first differences includes: Determine the larger value between each of the C parameters and the first preset value to obtain the C larger values ​​corresponding to the C parameters; The second difference between the first preset value and each of the C larger values ​​is determined to obtain C second differences corresponding to the C larger values; Based on the C first differences and the C second differences, the first operation of the input feature map of the first network layer is transformed into a second operation.

4. The method according to claim 1, wherein, The second preset value is 1, the third preset value is -1, and the set of convolutional kernels on which the first operation of the input feature map is transformed into the second operation satisfies at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences. All other elements in the first set of convolutional kernels are 0. The second convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The j-th element of the j-th convolutional kernel in the second convolutional kernel set is the j-th second difference among the C second differences. All other elements in the second convolutional kernel set are 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements of the third convolutional kernel set are all 0. The fourth convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The t-th element of the t-th convolutional kernel in the fourth convolutional kernel set is -1, and the remaining elements in the fourth convolutional kernel set are all 0.

5. The method according to claim 2, wherein, The first operation is a leakage correction linear unit operation; The step of transforming the first operation of the input feature map of the first network layer into a second operation based on the C first differences includes: Based on the C first differences, construct a first set of convolutional kernels; Based on the C second preset values, a third set of convolutional kernels is constructed; Based on the first set of convolutional kernels and the third set of convolutional kernels, the first operation of the input feature map of the first network layer is transformed into a second operation.

6. The method according to claim 5, wherein, The second preset value is 1, and the set of convolutional kernels on which the first operation of the input feature map is transformed into the second operation satisfies at least one of the following: The first set of convolutional kernels includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The i-th element of the i-th convolutional kernel in the first set of convolutional kernels is the i-th first difference among the C first differences. All other elements in the first set of convolutional kernels are 0. The third convolutional kernel set includes C convolutional kernels with a width of 1, a height of 1, and a number of channels of C. The s-th element of the s-th convolutional kernel in the third convolutional kernel set is 1, and the remaining elements of the third convolutional kernel set are all 0.

7. The method according to claim 6, wherein, The second operation includes: The first set of convolutional kernels is convolved with the input feature map to obtain the first feature map; The first feature map is subjected to a modified linear unit operation to obtain the second feature map; The third set of convolution kernels is convolved with the second feature map to obtain the fifth feature map; The input feature map and the fifth feature map are added element-wise to obtain the output feature map of the second network layer.

8. A compilation device for a neural network model, comprising: The first acquisition module is used to acquire a neural network model to be compiled, wherein the neural network model to be compiled includes a first network layer whose corresponding operation type is a first operation, and the first operation is a variant of the modified linear unit operation; The second acquisition module is used to transform the first operation of the input feature map of the first network layer into a second operation based on the layer parameters of the first network layer in the neural network model to be compiled obtained by the first acquisition module, so as to obtain a second network layer. The second operation is an operation supported by the neural network accelerator. The second operation includes: performing a convolution operation on the first set of convolution kernels and the input feature map to obtain a first feature map; performing a modified linear unit operation on the first feature map to obtain a second feature map; performing a convolution operation on the second set of convolution kernels and the input feature map to obtain a third feature map; performing a modified linear unit operation on the third feature map to obtain a fourth feature map; performing a convolution operation on the third set of convolution kernels and the second feature map to obtain a fifth feature map; performing a convolution operation on the fourth set of convolution kernels and the fourth feature map to obtain a sixth feature map; and performing a convolution operation on the input feature map. The fifth feature map and the sixth feature map are element-wise added together to obtain the output feature map of the second network layer; wherein, the layer parameters of the first network layer include C parameters, where C is the number of channels of the input feature map, the first convolutional kernel set is constructed based on C first differences, the second convolutional kernel set is constructed based on C second differences, the C first differences and the C second differences are determined based on the C parameters and a first preset value, the third convolutional kernel set is constructed based on C second preset values, and the fourth convolutional kernel set is constructed based on C third preset values; The generation module is used to generate a target neural network model by compiling the network layers other than the first network layer in the neural network model to be compiled, obtained by the first acquisition module, and the second network layer obtained by the second acquisition module.

9. A computer-readable storage medium storing a computer program for executing a method for compiling a neural network model according to any one of claims 1-7.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the compilation method of the neural network model according to any one of claims 1-7.

Images