Compilation method, device and computer-readable storage medium of neural network model

Abstract

Disclosed are a method, device and computer-readable storage medium for compiling a neural network model. The method includes: determining a target network layer included in the neural network model to be compiled; adjusting the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by a neural network accelerator based on the layer parameters of the target network layer; compiling and generating a target neural network model based on the network layers other than the target network layer in the neural network model to be compiled, and the target network layer after the association operation is adjusted to the operation logic. Since the embodiments of the present disclosure do not need to move data to a general-purpose processor for processing, the association operation can be implemented at a relatively low cost without occupying additional resources.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, and in particular to a method and device for compiling a neural network model and a computer-readable storage medium. Background Art

[0002] In some cases, the neural network accelerator deployed on the chip does not support correlation operations, while the actual operation of the neural network model requires correlation operations. In response to the above situation, the current measures are: moving the data that needs to be correlated to the general processor to perform the correlation operation, and then moving the results from the general processor back. Summary of the invention

[0003] In order to solve the technical problems of the above measures that the data transfer cost is high and additional resources are required, the present disclosure is proposed. The embodiments of the present disclosure provide a compilation method, device and computer-readable storage medium for a neural network model.

[0004] According to one aspect of the present disclosure, a method for compiling a neural network model is provided, comprising:

[0005] Determine the target network layer included in the neural network model to be compiled;

[0006] Based on the layer parameters of the target network layer, adjusting the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by the neural network accelerator;

[0007] Based on the network layers other than the target network layer in the neural network model to be compiled, and the target network layer after the associated operation is adjusted to the operation logic, the target neural network model is compiled and generated.

[0008] According to another aspect of the present disclosure, a compiling device for a neural network model is provided, comprising:

[0009] A first acquisition module is used to determine the target network layer included in the neural network model to be compiled;

[0010] An adjustment module, configured to adjust the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by a neural network accelerator based on the layer parameters of the target network layer acquired by the first acquisition module;

[0011] A generation module is used to compile and generate a target neural network model based on the network layers in the neural network model to be compiled except the target network layer obtained by the first acquisition module, and the target network layer after the association operation is adjusted to the operation logic.

[0012] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the compilation method of the above-mentioned neural network model.

[0013] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0014] processor;

[0015] a memory for storing instructions executable by the processor;

[0016] The processor is used to read the executable instructions from the memory and execute the instructions to implement the compilation method of the above-mentioned neural network model.

[0017] Based on the compilation method, device, computer-readable storage medium and electronic device of the neural network model provided by the above-mentioned embodiments of the present disclosure, in the process of compiling the neural network model, the association operation between the first input feature map and the second input feature map of the target network layer can be adjusted to the operation logic supported by the neural network accelerator. In this way, even if the neural network accelerator does not support the association operation, the association operation can be implemented by executing an operation equivalent to the association operation. Since the embodiments of the present disclosure do not need to move data to a general-purpose processor for processing, the association operation can be implemented at a lower cost and without occupying additional resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 It is a schematic diagram of implementing the association operation in the embodiment of the present disclosure.

[0020] Figure 2 It is a flowchart of a method for compiling a neural network model provided by an exemplary embodiment of the present disclosure.

[0021] Figure 3 It is a flowchart of a method for compiling a neural network model provided by another exemplary embodiment of the present disclosure.

[0022] Figure 4 It is a schematic diagram of the first convolution kernel in the compilation method of the neural network model provided by an exemplary embodiment of the present disclosure.

[0023] Figure 5 It is a schematic diagram of the second convolution kernel in the compilation method of the neural network model provided by an exemplary embodiment of the present disclosure.

[0024] Figure 6 It is a schematic diagram of the fourth feature map in the compilation method of the neural network model provided by an exemplary embodiment of the present disclosure.

[0025] Figure 7 The sliding window Figure 6 Sliding on the fourth feature map shown, a schematic diagram of the area covered by the sliding window.

[0026] Figure 8 It is a flowchart for implementing the association operation in the embodiment of the present disclosure.

[0027] Fig. 9 It is a structural diagram of a compiling device for a neural network model provided by an exemplary embodiment of the present disclosure.

[0028] Fig.10 It is a structural diagram of a compiling device for a neural network model provided by another exemplary embodiment of the present disclosure.

[0029] Fig.11 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0031] Application Overview

[0032] Some chips can be deployed with neural network accelerators, for example, artificial intelligence (AI) chips can be deployed with brain processors (Brain Processing Unit, BPU). It should be noted that the neural network accelerator can be used to implement a neural network model, for example, a neural network model for object detection.

[0033] In the process of implementing the present disclosure, the inventors found that in some cases, the neural network accelerator deployed on the chip does not support association operations, while the actual operation of the neural network model requires association operations. Among them, the input required for the association operation is generally two feature maps. Optionally, these two feature maps can be images of the previous and next moments in the Flow (i.e., optical flow) task, or images of the left and right cameras in the Stereo (i.e., stereo) task. In actual operation, it is generally necessary to take out a patch (i.e., a small block) from each of the two feature maps, and then calculate the correlation coefficient of the two patches as the output value of the current position. In this way, the role of the association operation is essentially to calculate the correlation of the local areas of the two inputs. In the Flow task and the Stereo task, the association operation can be used to construct the cost volume (which is a left-right disparity search space in the stereo matching problem in computer vision).

[0034] In the case where the neural network accelerator deployed on the chip does not support associative operations, but the actual operation of the neural network model requires associative operations, the current measures are: move the data that needs to be associatively operated to the general processor to perform the associative operation, such as moving it to the central processing unit (CPU) to perform the associative operation, and then move the results from the general processor back. However, the above measures have high data moving costs and require additional resources, so it is necessary to take a more reasonable measure to deal with the above situation.

[0035] Exemplary Systems

[0036] It should be noted that even if the neural network accelerator deployed on the chip does not support associative operations, it often supports some common types of operations, such as convolution operations, channel sum operations, element-wise operations, etc.

[0037] In view of this, in the embodiments of the present disclosure, Figure 1 As shown in the figure, the neural network model involves two stages, namely the compilation stage and the execution stage. In the compilation stage, the associated operation can be adjusted to the operation logic supported by the neural network model, and the compilation stage does not actually execute the operation logic supported by the neural network model. In the execution stage, the associated operation can be indirectly realized through the actual execution of the operation logic supported by the neural network model. In this way, even if the neural network accelerator does not support the associated operation, the associated operation can be realized without moving data and occupying additional resources.

[0038] Exemplary Methods

[0039] Figure 2 It is a flowchart of a method for compiling a neural network model provided by an exemplary embodiment of the present disclosure. Figure 2 The method shown can be specifically applied to compilers, Figure 2 The method shown includes step 201, step 202 and step 203, and each step is described below.

[0040] Step 201, determining the target network layer included in the neural network model to be compiled.

[0041] It should be noted that the neural network model to be compiled refers to a neural network model that needs to be compiled. The neural network model to be compiled may include multiple network layers. The operation type corresponding to at least one of the multiple network layers is an associative operation. Any network layer of the at least one network layer can be used as the target network layer in step 201.

[0042] Step 202: Based on the layer parameters of the target network layer, the association operation between the first input feature map and the second input feature map of the target network layer is adjusted to the operation logic supported by the neural network accelerator.

[0043] Here, the layer parameters of the target network layer may include various parameters required to implement the association operation. For a clear layout, the layer parameters of the target network layer will be specifically introduced later.

[0044] Here, the first input feature map can be represented as data1, the second input feature map can be represented as data2, and the first input feature map and the second input feature map have the same number of channels.

[0045] Here, the operation logic supported by the neural network accelerator includes but is not limited to common operation types such as number acquisition operation, channel summation operation, element-by-element operation, and division operation.

[0046] It should be noted that in step 202, the associated operation is adjusted to the operation logic supported by the neural network accelerator, which actually transforms the associated operation into an equivalent operation. That is, the result obtained by executing the equivalent operation is the same as the result obtained by executing the associated operation.

[0047] Step 203, based on the network layers other than the target network layer in the neural network model to be compiled, and the target network layer after the associated operation is adjusted to the operation logic supported by the neural network, compile and generate the target neural network model.

[0048] It should be noted that the network layers other than the target network layers in the neural network model to be compiled refer to network layers whose corresponding operation types are not associative operations, including but not limited to network layers whose corresponding operation types are convolution operations (i.e., convolution layers), network layers whose corresponding operation types are pooling operations (i.e., pooling layers), network layers whose corresponding operation types are rectified linear unit (ReLU) operations (i.e., ReLU layers), etc.

[0049] In step 203, the compiler backend may perform compilation processing based on the network layers other than the target network layer in the neural network model to be compiled, and the target network layer after the associated operations are adjusted to the operation logic supported by the neural network accelerator, so as to generate a binary target neural network model. The specific compilation processing method may adopt any feasible method according to actual needs, which is not elaborated in the present disclosure.

[0050] Based on the compilation method of the neural network model provided by the above-mentioned embodiment of the present disclosure, in the process of compiling the neural network model, the association operation between the first input feature map and the second input feature map of the target network layer can be adjusted to the operation logic supported by the neural network accelerator. In this way, even if the neural network accelerator does not support the association operation, the association operation can be implemented by executing an operation equivalent to the association operation. Since the embodiment of the present disclosure does not need to move data to a general-purpose processor for processing, the association operation can be implemented at a lower cost and without occupying additional resources.

[0051] exist Figure 2 Based on the embodiment shown, Figure 3 As shown, step 202 includes step 2021, step 2022 and step 2023.

[0052] Step 2021, based on the layer parameters of the target network layer, determine a first data extraction method for the first input feature map of the target network layer, a second data extraction method for the second input feature map of the target network layer, and a convolution kernel size.

[0053] Here, the layer parameters of the target network layer may include a first data extraction method, a second data extraction method, and a convolution kernel size; wherein the first data extraction method is used to indicate how to extract data from the first input feature map, and the second data extraction method is used to indicate how to extract data from the second input feature map; the convolution kernel size may include width and height. Thus, in step 2021, the first data extraction method, the second data extraction method, and the convolution kernel size may be directly extracted from the layer parameters of the target network layer.

[0054] Optionally, the first data extraction method may indicate an extraction starting position, an extraction step length, etc. for performing data extraction. When data extraction is performed according to the first data extraction method, the data extracted each time may be fixed.

[0055] Optionally, the second data extraction method can indicate the extraction starting position, extraction step length, etc. for data extraction. The second data extraction method can also indicate the size, sliding range, moving step length, etc. of the sliding window used for data extraction. The second data extraction method can specify a maximum offset (max_displacement), through which the sliding range of the sliding window used for data extraction can be limited. Since the second data extraction method involves a sliding window, when data extraction is performed according to the second data extraction method, the data extracted each time can be different. Among them, the extraction step lengths indicated by the first data extraction method and the second data extraction method can be expressed as stride1; the moving step length indicated by the second data extraction method can be expressed as stride2.

[0056] Optionally, the convolution kernel size may also be referred to as kernel size. When the width and height of the convolution kernel size are both 3, the convolution kernel size may be expressed as kernel=(3,3).

[0057] Step 2022, generate target data based on the convolution kernel size.

[0058] In step 2022, target data may be generated based on the width and height of the convolution kernel size.

[0059] Step 2023, based on the first data extraction method, the second data extraction method, and the target data, adjust the association operation between the first input feature map and the second input feature map to the operation logic supported by the neural network accelerator.

[0060] Optionally, the operation logic supported by the neural network includes:

[0061] Performing a data extraction operation on the first input feature map in a first data extraction manner;

[0062] Performing a data extraction operation on the second input feature map in a second data extraction manner;

[0063] Performing element-by-element operations on corresponding positions of a first feature map as a result of a data extraction operation on the first input feature map and a second feature map as a result of a data extraction operation on the second input feature map;

[0064] A channel summation operation is performed on a third feature map which is a result of element-by-element operation of corresponding positions of the first feature map and the second feature map;

[0065] The target operations are respectively associated with the target data and the fourth feature map, where the fourth feature map is the channel summation operation result of the third feature map.

[0066] Here, the element-by-element operation of the corresponding positions of the first feature map and the second feature map includes but is not limited to the multiplication operation of the corresponding positions, the absolute value operation of the difference of the corresponding positions, etc. In order to ensure that the element-by-element operation of the corresponding positions can be performed between the first feature map and the second feature map, the first data extraction method and the second data extraction method need to ensure that the first feature map and the second feature map have the same feature map size.

[0067] It should be noted that there are many possible situations for the content of the target data, and accordingly, there are also many possible situations for the type of the target operation, which are introduced below with examples.

[0068] In the first case, the target data includes: a first convolution kernel having a convolution kernel size, and each element in the first convolution kernel is 1 / N, where N is the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map:

[0069] Target operations, including:

[0070] Convolution operation of the fourth feature map and the first convolution kernel.

[0071] Assuming that the convolution kernel size is expressed in the form of kernel = (3,3), and the number of channels of the first input feature map is 10, then N = 3 × 3 × 10 = 90. The first convolution kernel with the convolution kernel size can be found in Figure 4 .

[0072] In this case, the operation type of the target operation is specifically a convolution operation. Since the neural network accelerator usually supports convolution operations, this can ensure the normal and rapid implementation of the target operation, which is conducive to ensuring the normal implementation of the associated operation.

[0073] In the second case, the target data includes: a second convolution kernel having a convolution kernel size, and each element in the second convolution kernel is 1:

[0074] Target operations, including:

[0075] Convolution operation of the fourth feature map and the second convolution kernel;

[0076] A division operation is performed on the fifth feature map which is the convolution operation result of the fourth feature map and the second convolution kernel, with N being the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map.

[0077] Assuming that the convolution kernel size is expressed in the form of kernel = (3,3), the second convolution kernel with the convolution kernel size can be found in Figure 5 Assuming that the number of channels of the first input feature map is 10, then N=3×3×10=90. Then, in the execution stage of the target neural network model, when the fifth feature map is obtained as the convolution operation result of the fourth feature map and the second convolution kernel, each pixel value in the fifth feature map can be divided by 90, and each pixel value in the fifth feature map can be updated to the division result of the pixel value and 90.

[0078] In this case, the operation types of the target operation specifically include convolution operation and division operation. Since the neural network accelerator usually supports convolution operation and division operation, this can ensure the normal and rapid implementation of the target operation, which is conducive to ensuring the normal implementation of the associated operation.

[0079] In a third case, the target data includes a window width determined based on a width of the third feature map and a width in a convolution kernel size, and a window height determined based on a height of the third feature map and a height in a convolution kernel size;

[0080] The target operation is associated with a sliding window having a window width and a window height.

[0081] Optionally, the window width is the sum of the first difference and 1, the window height is the sum of the second difference and 1, the first difference is the difference between the width of the third feature map and the width in the convolution kernel size, and the second difference is the difference between the height of the third feature map and the height in the convolution kernel size;

[0082] The target operation includes the following steps:

[0083] When the sliding window slides on the fourth feature map with a step size of 1, the area covered by the sliding window each time is stacked along the channel direction;

[0084] Channel summation operation of the sixth feature map formed by stacking all regions;

[0085] A division operation is performed on the seventh feature map as the channel summation operation result of the sixth feature map, with N being the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first feature map.

[0086] Assuming that the convolution kernel size is expressed as kernel=(3,3), the width of the third feature map is expressed as w, and the height of the third feature map is expressed as h, then the first difference can be expressed as w-3, the second difference can be expressed as h-3, the window width can be expressed as w-3+1, and the window height can be expressed as h-3+1. Assuming that w is 6 and h is 8, the window width is 4 and the window height is 6. Assuming that the fourth feature map is as follows Figure 6As shown in the tensor, when the sliding window of (4,6) slides on the tensor with a step size of 1, the area covered by the sliding window each time is a sub-tensor, and a total of 9 sub-tensors can be obtained. These 9 sub-tensors are as follows Figure 7 As shown, the size of the sixth feature map formed by stacking these 9 sub-tensors is 6×4×9. In the neural network model execution stage, after obtaining the sixth feature map, the channel summation operation can be performed on the sixth feature map, and the channel summation operation result is the seventh feature map. The size of the seventh feature map is 6×4×1. It is easy to see that the pixel value at the first row and first column of the seventh feature map = 1+2+3+9+10+11+17+18+19, and the pixel value at the first row and second column of the seventh feature map = 2+3+4+10+11+12+18+19+20. Assuming that the number of channels of the first input feature map is 10, then N = 3×3×10 = 90. Then, after obtaining the seventh feature map as the channel summation operation of the sixth feature map, each pixel value in the seventh feature map can be divided by 90, and each pixel value in the seventh feature map can be updated to the result of dividing the pixel value by 90.

[0087] In this case, the operation types of the target operation specifically include stacking operation, channel summation operation and division operation. Since the neural network accelerator usually supports stacking operation, channel summation operation and division operation, this can ensure the normal and rapid implementation of the target operation, which is conducive to ensuring the normal implementation of the associated operation. It should be pointed out that this case essentially uses stacking operation and channel summation operation to replace convolution operation, which can improve the parallelism of operation by moving data and can ensure the accuracy of operation as much as possible.

[0088] like Figure 8 As shown, assuming that there are two feature maps F1 (equivalent to the first input feature map in the above text) and F2 (equivalent to the second input feature map in the above text) that need to be associated, the data extraction operation of F1, the data extraction operation of F2, the corresponding position multiplication operation of the data extraction operation result of F1 and the data extraction operation result of F2, and the channel summation operation of the feature map as the result of the corresponding position multiplication operation can be performed in sequence to obtain F3 (equivalent to the fourth feature map in the above text). Afterwards, a kernel summation operation can be performed on F3 to obtain O (the specific operation method refers to any of the three target operations involved in the three situations of the above target data).

[0089] In the embodiments of the present disclosure, with reference to the layer parameters of the target network layer, the first data extraction method of the first input feature map of the target network layer, the second data extraction method of the second input feature map of the target network layer, and the convolution kernel size can be efficiently and reliably determined. With reference to the convolution kernel size, target data can be generated so that the association operation between the first input feature map and the second input feature map can be efficiently and reliably adjusted to an equivalent operation based on the first data extraction method, the second data extraction method, and the target data, thereby indirectly realizing the association operation.

[0090] In an optional example, the convolution operation of the fourth feature map and any convolution kernel includes the following steps:

[0091] A depthwise convolution operation of at least two fourth feature maps and at least two of the convolution kernels.

[0092] Here, the depthwise convolution operation can also be called the depthwise operation.

[0093] It should be noted that, in the execution stage of the target neural network model, when the first input feature map is subjected to data extraction operation in the first data extraction method, only one first feature map can be obtained by extraction, and when the second input feature map is subjected to data extraction in the second data extraction method, at least two second feature maps can be obtained by extraction, and each second feature map can be subjected to element-by-element operation at corresponding positions with the first feature map, so that at least two second feature maps can be obtained. Figure 1 At least two third feature maps corresponding to one, accordingly, based on the channel summation operation of the at least two third feature maps, the at least two second feature maps can be obtained. Figure 1 A corresponding at least two fourth characteristic maps.

[0094] After obtaining a fourth feature map each time, the convolution operation may not be performed temporarily, but the obtained fourth feature maps may be stacked in the channel direction to obtain at least two stacked fourth feature maps. Afterwards, for the first case of the above target data, the first convolution kernel may be used to perform a Depthwise operation on the at least two stacked fourth feature maps as a whole, instead of performing a convolution operation on each fourth feature map separately; similarly, for the second case of the above target data, the second convolution kernel may be used to perform a Depthwise operation on the at least two stacked fourth feature maps as a whole, instead of performing a convolution operation on each fourth feature map separately. In this way, by using the Depthwise operation instead of the convolution operation performed on each fourth feature map separately, it is beneficial to improve the code parallelism, reduce the amount of calculation, and save the calculation time.

[0095] Exemplary Devices

[0096] Fig. 9 It is a structural diagram of a compiling device for a neural network model provided by an exemplary embodiment of the present disclosure. Fig. 9 The device shown includes a first acquisition module 901 , an adjustment module 902 , and a generation module 903 .

[0097] A first acquisition module 901 is used to determine a target network layer included in the neural network model to be compiled;

[0098] An adjustment module 902, configured to adjust the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by the neural network accelerator based on the layer parameters of the target network layer acquired by the first acquisition module 901;

[0099] The generation module 903 is used to compile and generate the target neural network model based on the network layers in the neural network model to be compiled except the target network layer obtained by the first acquisition module 901, and the target network layer after the associated operation is adjusted to the operation logic.

[0100] In an alternative example, Fig.10 As shown, the adjustment module 902 includes:

[0101] A determination submodule 9021 is used to determine a first data extraction method for a first input feature map of the target network layer, a second data extraction method for a second input feature map of the target network layer, and a convolution kernel size based on a layer parameter of the target network layer;

[0102] A generating submodule 9022, used to generate target data based on the convolution kernel size determined by the determining submodule 9021;

[0103] The adjustment submodule 9023 is used to adjust the association operation between the first input feature map and the second input feature map to the operation logic supported by the neural network accelerator based on the first data extraction method determined by the determination submodule 9021, the second data extraction method determined by the determination submodule 9021, and the target data generated by the generation submodule 9022.

[0104] In an optional example, the operation logic supported by the neural network includes:

[0105] Performing a data extraction operation on the first input feature map in a first data extraction manner;

[0106] Performing a data extraction operation on the second input feature map in a second data extraction manner;

[0107] Performing element-by-element operations on corresponding positions of a first feature map as a result of a data extraction operation on the first input feature map and a second feature map as a result of a data extraction operation on the second input feature map;

[0108] A channel summation operation is performed on a third feature map which is a result of element-by-element operation of corresponding positions of the first feature map and the second feature map;

[0109] The target operations are respectively associated with the target data and the fourth feature map, where the fourth feature map is the channel summation operation result of the third feature map.

[0110] In an optional example, the target data includes: a first convolution kernel having a convolution kernel size, and each element in the first convolution kernel is 1 / N, where N is the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map:

[0111] Target operations, including:

[0112] Convolution operation of the fourth feature map and the first convolution kernel.

[0113] In an optional example, the target data includes: a second convolution kernel having a convolution kernel size, and each element in the second convolution kernel is 1:

[0114] Target operations, including:

[0115] Convolution operation of the fourth feature map and the second convolution kernel;

[0116] A division operation is performed on the fifth feature map which is the convolution operation result of the fourth feature map and the second convolution kernel, with N being the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map.

[0117] In an optional example, the convolution operation of the fourth feature map and any convolution kernel includes:

[0118] A depthwise convolution operation of at least two fourth feature maps and at least two of the convolution kernels.

[0119] In an alternative example,

[0120] The target data includes a window width determined based on a width of the third feature map and a width in a convolution kernel size, and a window height determined based on a height of the third feature map and a height in a convolution kernel size;

[0121] The target operation is associated with a sliding window having a window width and a window height.

[0122] In an optional example, the window width is the sum of the first difference and 1, the window height is the sum of the second difference and 1, the first difference is the difference between the width of the third feature map and the width in the convolution kernel size, and the second difference is the difference between the height of the third feature map and the height in the convolution kernel size;

[0123] Target operations, including:

[0124] When the sliding window slides on the fourth feature map with a step size of 1, the area covered by the sliding window each time is stacked along the channel direction;

[0125] Channel summation operation of the sixth feature map formed by stacking all regions;

[0126] A division operation is performed on the seventh feature map as the channel summation operation result of the sixth feature map, with N being the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first feature map.

[0127] Exemplary Electronic Devices

[0128] Below, reference Fig.11 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0129] Fig.11 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0130] like Fig.11 As shown, the electronic device 1100 includes one or more processors 1101 and a memory 1102 .

[0131] The processor 1101 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.

[0132] The memory 1102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1101 may run the program instructions to implement the compilation method of the neural network model of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0133] In one example, the electronic device 1100 may further include: an input device 1103 and an output device 1104 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0134] For example, when the electronic device is the first device or the second device, the input device 1103 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 1103 may be a communication network connector, for receiving collected input signals from the first device and the second device.

[0135] In addition, the input device 1103 may also include, for example, a keyboard, a mouse, and the like.

[0136] The output device 1104 can output various information to the outside, including the determined distance information, direction information, etc. The output device 1104 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0137] Of course, to simplify, Fig.11 Only some of the components related to the present disclosure in the electronic device 1100 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1100 may also include any other appropriate components.

[0138] Exemplary computer program products and computer-readable storage media

[0139] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method of compiling a neural network model according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0140] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0141] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for compiling a neural network model according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0142] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0144] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0145] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0147] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for compiling a neural network model, comprising: Determine the target network layer included in the neural network model to be compiled; Based on the layer parameters of the target network layer, the association operation between the first input feature map and the second input feature map of the target network layer is adjusted to the operation logic supported by the neural network accelerator; wherein the layer parameters of the target network layer include a first data extraction method for the first input feature map, a second data extraction method for the second input feature map, and a convolution kernel size; the operation logic supported by the neural network accelerator includes: a data extraction operation on the first input feature map in the first data extraction method; a data extraction operation on the second input feature map in the second data extraction method; an element-by-element operation on the corresponding positions of the first feature map as the result of the data extraction operation of the first input feature map and the second feature map as the result of the data extraction operation of the second input feature map; a channel summation operation on the third feature map as the result of the element-by-element operation on the corresponding positions of the first feature map and the second feature map; a target operation associated with the target data and the fourth feature map respectively, the fourth feature map being the channel summation operation result of the third feature map, wherein the target data is generated based on the convolution kernel size; Based on the network layers other than the target network layer in the neural network model to be compiled, and the target network layer after the associated operation is adjusted to the operation logic, the target neural network model is compiled and generated.

2. The method according to claim 1, wherein: The step of adjusting the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by the neural network accelerator based on the layer parameters of the target network layer includes: Determine, based on the layer parameters of the target network layer, a first data extraction method of a first input feature map of the target network layer, a second data extraction method of a second input feature map of the target network layer, and a convolution kernel size; Based on the convolution kernel size, generating the target data; Based on the first data extraction method, the second data extraction method, and the target data, the association operation between the first input feature map and the second input feature map is adjusted to an operation logic supported by a neural network accelerator.

3. The method according to claim 2, wherein: The target data includes: a first convolution kernel having the convolution kernel size, and each element in the first convolution kernel is 1 / N, where N is the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map: The target operation includes: A convolution operation is performed on the fourth feature map and the first convolution kernel.

4. The method according to claim 2, wherein: The target data includes: a second convolution kernel having the convolution kernel size, and each element in the second convolution kernel is 1: The target operation includes: A convolution operation of the fourth feature map and the second convolution kernel; A division operation is performed on a fifth feature map which is a result of the convolution operation of the fourth feature map and the second convolution kernel, with N being the product of the width in the convolution kernel size, the height in the convolution kernel size, and the number of channels of the first input feature map.

5. The method according to claim 3 or 4, wherein: The convolution operation of the fourth feature map and any convolution kernel includes: A depth convolution operation of at least two of the fourth feature maps and at least two of the convolution kernels.

6. The method according to claim 2, wherein: The target data includes a window width determined based on a width of the third feature map and a width in the convolution kernel size, and a window height determined based on a height of the third feature map and a height in the convolution kernel size; The target operation is associated with a sliding window having the window width and the window height.

7. A compiling device for a neural network model, comprising: A first acquisition module is used to determine the target network layer included in the neural network model to be compiled; An adjustment module, configured to adjust the association operation between the first input feature map and the second input feature map of the target network layer to an operation logic supported by the neural network accelerator based on the layer parameters of the target network layer obtained by the first acquisition module; wherein the layer parameters of the target network layer include a first data extraction method for the first input feature map, a second data extraction method for the second input feature map, and a convolution kernel size; the operation logic supported by the neural network accelerator includes: a data extraction operation on the first input feature map in the first data extraction method; a data extraction operation on the second input feature map in the second data extraction method; an element-by-element operation on the corresponding positions of the first feature map as the result of the data extraction operation of the first input feature map and the second feature map as the result of the data extraction operation of the second input feature map; a channel summation operation on the third feature map as the result of the element-by-element operation on the corresponding positions of the first feature map and the second feature map; a target operation associated with the target data and the fourth feature map respectively, the fourth feature map being the channel summation operation result of the third feature map, wherein the target data is generated based on the convolution kernel size; A generation module is used to compile and generate a target neural network model based on the network layers in the neural network model to be compiled except the target network layer obtained by the first acquisition module, and the target network layer after the association operation is adjusted to the operation logic.

8. A computer-readable storage medium storing a computer program for executing the compilation method of the neural network model described in any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the compilation method of the neural network model described in any one of claims 1-6 above.

Images