Method and apparatus for processing feature maps by artificial intelligence accelerator

Abstract

A method and device for processing a feature map by an artificial intelligence accelerator are disclosed. The method for processing the feature map by the artificial intelligence accelerator comprises: determining a merged convolution layer in a target neural network model, layer parameters of the merged convolution layer comprising depth convolution parameters and point convolution parameters; determining a cache area corresponding to the merged convolution layer; performing a depth convolution operation on an input feature map of the merged convolution layer based on the depth convolution parameters; sequentially caching at least two depth convolution results obtained by the depth convolution operation to the cache area, any two of the at least two depth convolution results corresponding to different local regions in the input feature map; performing a point convolution operation on the depth convolution results cached by the cache area based on the point convolution parameters; and determining an output feature map of the merged convolution layer based on a point convolution result obtained by the point convolution operation. The disclosed embodiments can effectively improve the operation speed of the depth separable convolution.

Description

Technical Field

[0001] This disclosure relates to the field of integrated circuit technology, and in particular to a method and apparatus for processing feature maps using an artificial intelligence accelerator. Background Technology

[0002] Depthwise separable convolution, which consists of two parts, depthwise convolution and pointwise convolution, is currently widely used.

[0003] In practice, the feature map to be depthwise separable convolution is usually first performed with depthwise convolution. After all the data in the feature map has been depthwise convolutioned and all the depthwise convolution results are stored in Double Data Rate Synchronous Dynamic Random Access Memory (DDR), pointwise convolution is then performed using the data stored in DDR to achieve depthwise separable convolution of the feature map. Summary of the Invention

[0004] To address the technical problem that depthwise separable convolution operations are slow and cannot adequately meet practical needs, this disclosure is proposed. Embodiments of this disclosure provide a method and apparatus for processing feature maps using an artificial intelligence accelerator.

[0005] According to one aspect of the present disclosure, a method for processing feature maps using an artificial intelligence accelerator is provided, comprising:

[0006] Determine the merged convolutional layer in the target neural network model, wherein the layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters;

[0007] Determine the cache area corresponding to the merged convolutional layer;

[0008] Based on the depth convolution parameters, a depth convolution operation is performed on the input feature map of the merged convolutional layer;

[0009] At least two depthwise convolution results obtained from the depthwise convolution operation are sequentially cached in the cache area, and any two depthwise convolution results correspond to different local regions in the input feature map.

[0010] Based on the point convolution parameters, point convolution operations are performed on the depth convolution results cached in the buffer.

[0011] Based on the point convolution result obtained from the point convolution operation, the output feature map of the merged convolutional layer is determined.

[0012] According to another aspect of the present disclosure, a method for compiling a neural network model is provided, comprising:

[0013] Identify the neural network model to be compiled;

[0014] Determine the paired first and second convolutional layers from the neural network model to be compiled;

[0015] The first convolutional layer and the second convolutional layer are merged to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer.

[0016] Allocate a buffer area for the merged convolutional layer;

[0017] Based on the merged convolutional layer, the buffer, and the network layers in the neural network model to be compiled, excluding the first and second convolutional layers, a target neural network model is compiled and generated. The target neural network model includes instructions for executing the above-described method of processing feature maps using an artificial intelligence accelerator.

[0018] According to another aspect of the present disclosure, an apparatus for processing feature maps via an artificial intelligence accelerator is provided, comprising:

[0019] The first determining module is used to determine the merged convolutional layer in the target neural network model, wherein the layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters.

[0020] The second determining module is used to determine the cache area corresponding to the merged convolutional layer determined by the first determining module;

[0021] The first calculation module is used to perform depth convolution operation on the input feature map of the merged convolutional layer based on the depth convolution parameters of the merged convolutional layer determined by the first determining module.

[0022] A caching module is used to sequentially cache at least two depth convolution results obtained by the depth convolution operation of the first operation module into the cache area determined by the second determination module, wherein any two depth convolution results correspond to different local regions in the input feature map;

[0023] The second operation module is used to perform point convolution operation on the depth convolution result cached in the buffer area determined by the second determination module, based on the point convolution parameters of the merged convolution layer determined by the first determination module.

[0024] The third determining module is used to determine the output feature map of the merged convolutional layer determined by the first determining module based on the point convolution result obtained by the point convolution operation of the second operation module.

[0025] According to another aspect of the present disclosure, a neural network model compilation apparatus is provided, comprising:

[0026] The fourth determination module is used to determine the neural network model to be compiled;

[0027] The fifth determining module is used to determine the paired first and second convolutional layers from the neural network model to be compiled determined by the fourth determining module;

[0028] The merging module is used to merge the first convolutional layer and the second convolutional layer determined by the fifth determining module to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer.

[0029] An allocation module is used to allocate a buffer area for the merged convolutional layer obtained by the merging module;

[0030] A generation module is used to compile and generate a target neural network model based on the merged convolutional layer obtained by the merging module, the buffer allocated by the allocation module, and the network layers in the neural network model to be compiled determined by the fourth determining module, excluding the first and second convolutional layers determined by the fifth determining module. The target neural network model includes instructions for executing the above-described method of processing feature maps through an artificial intelligence accelerator.

[0031] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for executing the above-described method for processing feature maps via an artificial intelligence accelerator or the compilation method for a neural network model.

[0032] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0033] processor;

[0034] Memory used to store the processor's executable instructions;

[0035] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described method for processing feature maps via an artificial intelligence accelerator or the compilation method for neural network models.

[0036] Based on the methods and apparatus for processing feature maps using an artificial intelligence accelerator, the methods and apparatus for compiling neural network models, the computer-readable storage medium, and the electronic device provided in the above embodiments of this disclosure, the fusion of depthwise convolution and pointwise convolution operations is effectively achieved by introducing a cache area based on the depthwise convolution parameters and pointwise convolution parameters of the merged convolutional layer, thereby achieving efficient and fast depthwise separable convolution. In addition, since the embodiments of this disclosure do not use the DDR located outside the artificial intelligence (AI) accelerator, but only use the cache inside the artificial intelligence accelerator to achieve depthwise separable convolution, the operation speed of depthwise separable convolution can be effectively improved. Attached Figure Description

[0037] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0038] Figure 1-1 This is a schematic diagram of the shape of the convolution kernel in ordinary convolution in related technologies.

[0039] Figure 1-2 This is a schematic diagram of the kernel shape of depthwise convolution in related technologies.

[0040] Figure 1-3 This is a schematic diagram of the shape of the convolution kernel in point convolution in related technologies.

[0041] Figure 2-1 This is a schematic diagram illustrating the calculation method of ordinary convolution in related technologies.

[0042] Figure 2-2 This is a schematic diagram illustrating the calculation method of depthwise separable convolution, which combines depthwise convolution and pointwise convolution in related technologies.

[0043] Figure 3 This is a schematic diagram illustrating the implementation principle of depthwise separable convolution in an exemplary embodiment of this disclosure.

[0044] Figure 4 This is a flowchart illustrating a method for processing feature maps using an artificial intelligence accelerator, provided in an exemplary embodiment of this disclosure.

[0045] Figure 5 This is a flowchart illustrating a method for processing feature maps using an artificial intelligence accelerator, provided in another exemplary embodiment of this disclosure.

[0046] Figure 6This is a flowchart illustrating a method for processing feature maps using an artificial intelligence accelerator, provided in yet another exemplary embodiment of this disclosure.

[0047] Figure 7 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure.

[0048] Figure 8 This is a flowchart illustrating a method for compiling a neural network model provided in another exemplary embodiment of this disclosure.

[0049] Figure 9 This is a schematic diagram illustrating the implementation principle of depthwise separable convolution in an exemplary embodiment of this disclosure.

[0050] Figure 10 This is a schematic diagram of the structure of an apparatus for processing feature maps by an artificial intelligence accelerator, provided in an exemplary embodiment of this disclosure.

[0051] Figure 11 This is a schematic diagram of the structure of an apparatus for processing feature maps via an artificial intelligence accelerator, provided in another exemplary embodiment of this disclosure.

[0052] Figure 12 This is a schematic diagram of the structure of an apparatus for processing feature maps by an artificial intelligence accelerator, provided in yet another exemplary embodiment of this disclosure.

[0053] Figure 13 This is a schematic diagram of the structure of a compilation device for a neural network model provided in an exemplary embodiment of this disclosure.

[0054] Figure 14 This is a schematic diagram of the structure of a compilation device for a neural network model provided in another exemplary embodiment of this disclosure.

[0055] Figure 15 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0056] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present disclosure, and not all embodiments, and the present disclosure is not limited to the exemplary embodiments.

[0057] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0058] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent specific technical meanings or necessary logical sequences. "Multiple" can refer to two or more, and "at least one" can refer to one, two, or more.

[0059] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0060] In this disclosure, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. In this disclosure, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0061] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0062] It should be understood that the dimensions of the various parts shown in the attached figures are not drawn to actual scale.

[0063] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0064] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0065] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0066] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0067] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0068] Application Overview

[0069] Depthwise convolution and pointwise convolution are two special types of convolution; depthwise convolution can also be called DW convolution, and pointwise convolution can also be called PW convolution. For example... Figure 1-1 As shown, the kernel shape in ordinary convolution is N*Dk*Dk*M; Figure 1-2 As shown, the kernel shape in depthwise convolution is M*Dk*Dk*1; as Figure 1-3 As shown, the kernel shape in point convolution is N*1*1*M.

[0070] Depthwise separable convolution, which consists of both depthwise convolution and pointwise convolution, is currently widely used. For the calculation method of ordinary convolution, please refer to [link to relevant documentation]. Figure 2-1 The calculation method for depthwise separable convolution (which combines depthwise convolution and pointwise convolution) can be found in [link to documentation]. Figure 2-2 .

[0071] It should be noted that in the actual implementation of depthwise separable convolution, depthwise convolution is generally performed on the feature map to be subjected to depthwise separable convolution first. After all data in the feature map has undergone depthwise convolution and all depthwise convolution results are stored in the DDR inside the AI ​​accelerator, pointwise convolution is then performed using the data stored in the DDR. In this way, completing the depthwise separable convolution of the feature map takes a very long time, the computation speed is slow, and it is difficult to meet practical needs.

[0072] Exemplary System

[0073] To address the technical problem that depthwise separable convolution operations are slow and cannot adequately meet practical needs, such as... Figure 3 As shown, this disclosure can be improved from the compilation stage and the execution stage of the neural network model respectively; wherein, in the compilation stage, the depthwise convolutional layer and the pointwise convolutional layer can be merged to obtain a merged convolutional layer, a buffer is allocated for the merged convolutional layer, and the merged convolutional layer and the buffer are used for instruction generation; in the execution stage, depthwise separable convolution can be efficiently and quickly implemented by executing the instructions generated in the compilation stage.

[0074] Exemplary methods

[0075] Figure 4 This is a flowchart illustrating a method for processing feature maps using an artificial intelligence accelerator, provided in an exemplary embodiment of this disclosure. Figure 4 The method shown includes steps 410, 420, 430, 440, 450 and 460, which are explained below.

[0076] Step 410: Determine the merged convolutional layer in the target neural network model. The layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters.

[0077] It should be noted that the target neural network model can be any neural network model compiled using the compilation method described below. The target neural network model may include merged convolutional layers, as well as other network layers besides merged convolutional layers.

[0078] The layer parameters of a merged convolutional layer can include depthwise convolution parameters and pointwise convolution parameters. Depthwise convolution parameters refer to the parameters required to perform depthwise convolution operations, while pointwise convolution parameters refer to the parameters required to perform pointwise convolution operations. Optionally, depthwise convolution parameters can include a depthwise convolution kernel, a stride (dw stride), paddings (dw paddings), etc., with the depthwise convolution kernel carrying information such as kernel shape (dw kernel shape) and weights (dw weights). Pointwise convolution parameters can include pointwise convolution kernels, which carry information such as the number of kernels (pw kernel num) and weights (pw weights).

[0079] Other network layers besides merged convolutional layers include, but are not limited to, deep convolutional layers, point convolutional layers, pooling layers, and rectified linear unit (ReLU) layers.

[0080] Step 420: Determine the buffer corresponding to the merged convolutional layer.

[0081] It should be noted that the cache corresponding to the merged convolutional layer can be the cache allocated by the compiler to the merged convolutional layer during the compilation phase. Optionally, the cache can be a region in the chip's static random-access memory (SRAM); the cache can also be called a temporary buffer.

[0082] Step 430: Perform depthwise convolution operation on the input feature map of the merged convolutional layer based on the depthwise convolution parameters.

[0083] In step 430, depth convolution operations can be performed on the input feature map of the merged convolutional layer based on the depth convolution kernel, stride value, and padding value in the depth convolution parameters, that is, convolution operations are performed on each channel of the input feature map separately.

[0084] If the padding value is not 0, the input feature map of the merged convolutional layer can be padded with edges based on the padding value to obtain the input feature map after edge padding. For example, if the padding value is 1, one pixel can be padded to the top, bottom, left and right sides of the input feature map.

[0085] Next, based on the stride value, the deep convolution kernel can be slid across the input feature map after edge padding. While the deep convolution kernel is sliding, element-wise multiplication and summation are performed on each channel of the region covered by the deep convolution kernel and the corresponding channel in the deep convolution kernel.

[0086] Step 440: At least two depth convolution results obtained from the depth convolution operation are cached sequentially in the cache area, and any two depth convolution results correspond to different local regions in the input feature map.

[0087] It should be noted that any depthwise convolution result corresponds to a local region in the input feature map. The relationship between the depthwise convolution result and this local region can be understood as follows: either the region covered by the depthwise convolution kernel only includes the local region, or, when the region covered by the depthwise convolution kernel only includes the local region and some edge-filled regions in the input feature map, each channel of the region covered by the depthwise convolution kernel is element-wise multiplied with the corresponding channel in the depthwise convolution kernel, and then summed to obtain the operation result value corresponding to each channel of the region covered by the depthwise convolution kernel. These operation result values ​​can be used to form the depthwise convolution result. That is, the width and height of each depthwise convolution result are both 1, and the number of channels is the same as the number of channels in the input feature map.

[0088] As the depth convolution kernel slides, the area covered by the depth convolution kernel changes continuously. Therefore, multiple depth convolution results of different local regions in the corresponding input feature map can be obtained sequentially. After obtaining a depth convolution result each time, the obtained depth convolution result can be cached in the cache area.

[0089] Step 450: Perform point convolution operation on the depth convolution result cached in the buffer based on the point convolution parameters.

[0090] In step 450, point convolution operations can be performed on the depth convolution results cached in the buffer based on the point convolution kernel in the point convolution parameters, that is, the channel-direction weighted summation of the depth convolution results cached in the buffer.

[0091] Step 460: Based on the point convolution result obtained from the point convolution operation, determine the output feature map of the merged convolutional layer.

[0092] Since all depthwise convolution results obtained from the depthwise convolution operation are cached sequentially in the cache area, by executing step 450, the point convolution results corresponding to each depthwise convolution result can be obtained. Thus, in step 460, by summarizing the point convolution results corresponding to each depthwise convolution result, the output feature map of the merged convolutional layer can be obtained.

[0093] In the embodiments of this disclosure, based on the depth convolution parameters and point convolution parameters of the merged convolutional layer, and by introducing a buffer, the fusion of depth convolution and point convolution operations is effectively achieved, thereby achieving efficient and fast depth-separable convolution. In addition, since the embodiments of this disclosure do not use DDR located outside the artificial intelligence accelerator, but only use the buffer inside the artificial intelligence accelerator to achieve depth-separable convolution, the operation speed of depth-separable convolution can be effectively improved.

[0094] exist Figure 4 Based on the illustrated embodiments, as Figure 5 As shown, step 450 includes steps 4505, 4507, 4509 and 4511.

[0095] Step 4505: Determine the first size relationship between the space occupancy rate of the cache area and the preset space occupancy rate.

[0096] Optionally, the preset space occupancy rate can be 85%, 90%, 95%, etc., which will not be listed here.

[0097] In step 4505, the current space occupancy rate of the buffer can be determined, and the current space occupancy rate of the buffer can be compared with the preset space occupancy rate to obtain the first size relationship.

[0098] Step 4507: Determine the second size relationship between the number of depthwise convolution results cached in the buffer and the first preset number.

[0099] Optionally, the first preset quantity can be 2, 3, 4, 5, etc., which will not be listed here.

[0100] In step 4507, the number of depthwise convolution results currently cached in the buffer can be determined, and the number of depthwise convolution results currently cached in the buffer can be compared with a first preset number to obtain a second size relationship.

[0101] Step 4509: Determine the time relationship between the current time and the preset time.

[0102] Here, the preset time can be a pre-defined time for performing point convolution operations. Optionally, a specific time can be used as the starting time, and each time interval of a preset duration can also be used as a preset time. For example, if t1 is the starting time and the preset duration is T, then t1+T, t1+2T, ..., t1+NT can all be used as preset times.

[0103] In step 4509, the current time can be determined by calling the system clock function. Then, the time relationship between the current time and the preset time can be determined. The time relationship can be used to characterize the chronological order between the current time and the preset time.

[0104] Step 4511: In response to at least one of the first size relationship, the second size relationship, and the time relationship, point convolution operation is triggered, and point convolution operation is performed on the depth convolution result cached in the buffer based on the point convolution parameters.

[0105] Assuming that the first size relationship represents the current space occupancy rate of the cache as greater than or equal to the preset space occupancy rate as the first condition, if the first condition is met, it can be determined that the first size relationship is used to trigger point convolution operation; if the first condition is not met, it can be determined that the first size relationship is not used to trigger point convolution operation.

[0106] Assuming that the second size relationship represents the number of depthwise convolution results currently cached in the cache as a second condition, if the second condition is met, it can be determined that the second size relationship is used to trigger point convolution operations; if the second condition is not met, it can be determined that the second size relationship is not used to trigger point convolution operations.

[0107] Assuming that the third condition is that the time relationship represents the current time being equal to the preset time, if the third condition is met, it can be determined that the time relationship is used to trigger point convolution operation; if the third condition is not met, it can be determined that the time relationship is not used to trigger point convolution operation.

[0108] It should be noted that if at least one of the first, second, and third conditions is met, it can be assumed that the cache contains sufficient depthwise convolution results. If point convolution operations are not performed on these depthwise convolution results in a timely manner, they may be overwritten by subsequent depthwise convolution results cached in the cache, resulting in the loss of some point convolution results. Therefore, in the embodiments of this disclosure, the timing for triggering point convolution operations can be efficiently and quickly determined by referring to the first size relationship, the second size relationship, and the time relationship. This ensures that point convolution operations are performed on these depthwise convolution results in a timely manner when the cache contains sufficient depthwise convolution results, thereby effectively guaranteeing the integrity and accuracy of the output feature map of the final merged convolutional layer.

[0109] exist Figure 4 Based on the illustrated embodiments, as Figure 6 As shown, step 440 includes step 4401.

[0110] Step 4401: Cache at least two depthwise convolution results obtained from the depthwise convolution operation into the first cache sub-area in the cache area in sequence.

[0111] Here, the cache area may include a first cache sub-area and a second cache sub-area. Optionally, the first cache sub-area and the second cache sub-area may have the same or different sizes.

[0112] Step 450 includes steps 4513 and 4515.

[0113] Step 4513: Transfer the depthwise convolution result cached in the first cache sub-region to the second cache sub-region.

[0114] Optionally, if the space occupancy rate of the first cache sub-region is greater than or equal to the preset space occupancy rate, the depth convolution results cached in the first cache sub-region can be transferred to the second cache sub-region; or, if the number of depth convolution results cached in the first cache sub-region is greater than or equal to the first preset number, the depth convolution results cached in the first cache sub-region can be transferred to the second cache sub-region.

[0115] Optionally, when transferring the depthwise convolution result cached in the first cache sub-region to the second cache sub-region, the depthwise convolution result cached in the first cache sub-region can be cleared.

[0116] Step 4515: Based on the point convolution parameters, perform point convolution operation on the depth convolution result of the second cache sub-region.

[0117] Optionally, if at least one of the following three conditions is met: the space occupancy rate of the second cache sub-region is greater than or equal to a preset space occupancy rate, the number of depth convolution results cached in the second cache sub-region is greater than or equal to a first preset number, and the current time is equal to a preset time, then point convolution operation can be performed on the depth convolution results cached in the second cache sub-region based on the point convolution parameters.

[0118] In the embodiments of this disclosure, by dividing the cache area into a first cache sub-area and a second cache sub-area, the first cache sub-area can be used to directly cache the depth convolution result obtained from the depth convolution operation, and the depth convolution result cached in the second cache sub-area can be used for point convolution operation. In this way, it can be considered that the first cache sub-area is dedicated to the depth convolution operation and the second cache sub-area is dedicated to the point convolution operation, thereby enabling the concurrent execution of depth convolution operation and point convolution operation, thereby further improving the operation speed of depth separable convolution.

[0119] Figure 7 This is a flowchart illustrating a method for compiling a neural network model provided in an exemplary embodiment of this disclosure. Figure 7 The method shown includes steps 710, 720, 730, 740 and 750, which are explained below.

[0120] Step 710: Determine the neural network model to be compiled.

[0121] It should be noted that the neural network model to be compiled refers to the neural network model that needs to be compiled. The neural network model to be compiled may include multiple network layers, including but not limited to deep convolutional layers, point convolutional layers, pooling layers, rectified linear unit (ReLU) layers, etc.

[0122] Step 720: Determine the paired first and second convolutional layers from the neural network model to be compiled.

[0123] In step 720, all convolutional layers can be determined from the multiple network layers included in the neural network model to be compiled. Then, a pair of first and second convolutional layers can be selected from all the determined convolutional layers. The pairing of the first and second convolutional layers can mean that the combination of the first and second convolutional layers can achieve depthwise separable convolution. The first convolutional layer can be a depthwise convolutional layer, and the layer parameters of the first convolutional layer can include depthwise convolutional parameters. The second convolutional layer can be a point convolutional layer, and the layer parameters of the second convolutional layer can include point convolutional parameters.

[0124] Step 730: Merge the first convolutional layer and the second convolutional layer to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer.

[0125] In step 730, depth convolution parameters can be extracted from the layer parameters of the first convolutional layer, and point convolution parameters can be extracted from the layer parameters of the second convolutional layer. Then, by merging the first and second convolutional layers, a merged convolutional layer whose layer parameters include depth convolution parameters and point convolution parameters can be obtained.

[0126] Step 740: Allocate a buffer for the merged convolutional layers.

[0127] Optionally, a region can be selected from the SRAM as a buffer and allocated to the merged convolutional layer according to specific rules; or, a region can be randomly selected from the SRAM as a buffer and allocated to the merged convolutional layer.

[0128] Step 750: Based on the merged convolutional layers, buffers, and network layers in the neural network model to be compiled (excluding the first and second convolutional layers), a target neural network model is generated. The target neural network model includes methods for performing feature map processing via an artificial intelligence accelerator (e.g., Figure 4 , Figure 5 or Figure 6 Instructions for the method of processing feature maps via an artificial intelligence accelerator in the illustrated embodiment.

[0129] In step 750, the compiler backend can perform compilation processing based on the merged convolutional layers, buffers, and network layers in the neural network model to be compiled, excluding the first and second convolutional layers, to generate a binary target neural network model. The specific compilation processing method can be any feasible method according to actual needs, and this disclosure will not elaborate on it.

[0130] In the embodiments of this disclosure, during the compilation stage, the paired first and second convolutional layers in the neural network model to be compiled can be merged to obtain a merged convolutional layer. A buffer is allocated to the merged convolutional layer so that the merged convolutional layer, the buffer, and the network layers in the neural network model to be compiled (excluding the first and second convolutional layers) can be used for model compilation to generate the target neural network model. Thus, during the execution stage, based on the depthwise convolutional and pointwise convolutional parameters of the merged convolutional layer, and by introducing a buffer, the fusion of depthwise convolution and pointwise convolution operations is effectively achieved, thereby achieving efficient and fast implementation of depthwise separable convolution. Furthermore, since the embodiments of this disclosure do not use DDR located outside the AI ​​accelerator, but only implement depthwise separable convolution through the buffer inside the AI ​​accelerator, the computational speed of depthwise separable convolution can be effectively improved.

[0131] exist Figure 7 Based on the illustrated embodiments, as Figure 8 As shown, step 720 includes steps 7201, 7203 and 7205.

[0132] Step 7201: Determine the information used for the output of the first convolutional layer in the neural network model to be compiled.

[0133] Optionally, the output usage information of the first convolutional layer can be extracted from the neural network model to be compiled. The output usage information can be used to characterize which network layers in the neural network model to be compiled specifically use the output of the first convolutional layer; the output usage information can also be referred to as the use-def relationship.

[0134] Step 7203: Based on the information used in the output results, determine the number and type of network layers that use the output results of the first convolutional layer in the neural network model to be compiled.

[0135] In one example, the first convolutional layer is denoted as conv1. The output information obtained in step 7201 is in the following form: conv2_output = conv(conv1_output). It can be determined that the output of conv1 is provided to the network layer conv2, and conv2 is a convolutional layer.

[0136] In another example, the first convolutional layer is denoted as conv1. The output information obtained in step 7201 is in the following form: conv2_output = conv(conv1_output), ReLU1_output = conv(conv1_output). It can be determined that the output of conv1 is provided to the two network layers conv2 and ReLU1. Furthermore, conv2 is a convolutional layer and ReLU1 is a modified linear unit layer.

[0137] Step 7205: Based on the number and type of network layers using the output results, determine the second convolutional layer to be paired with the first convolutional layer.

[0138] In one specific embodiment, step 7205 includes:

[0139] In response to the fact that the number of network layers using the output results is a second preset number, and the type of network layer using the output results is a point convolutional layer, the network layer using the output results is determined as a second convolutional layer paired with the first convolutional layer.

[0140] Optionally, the second preset quantity can be 1.

[0141] In step 7205, it can be determined whether the number of network layers using the output of the first convolutional layer is 1, and whether the type of network layer using the output is a point convolutional layer.

[0142] If the number of network layers using the output of the first convolutional layer is 1, and the type of the network layer using the output is a point convolutional layer, then the network layer using the output of the first convolutional layer can be determined as the second convolutional layer paired with the first convolutional layer, and the first and second network layers can be merged subsequently.

[0143] If the number of network layers using the output of the first convolutional layer is not 1, and / or the type of network layer using the output is not a point convolutional layer, it can be determined that there is no second network layer paired with the first network layer in the neural network model to be compiled, and there is no need to merge the first and second network layers in the future.

[0144] In this way, by comparing the number of network layers that use the output of the first convolutional layer with a specific number, and by comparing the type of network layers that use the output with a specific type, the second convolutional layer that is paired with the first convolutional layer can be determined efficiently and quickly.

[0145] In the embodiments of this disclosure, by referring to the output information of the first convolutional layer in the neural network model to be compiled, the number and type of network layers that use the output of the first convolutional layer in the neural network model to be compiled can be determined efficiently and quickly. Based on the determined results, the second convolutional layer paired with the first convolutional layer can be determined efficiently and quickly. Thus, the embodiments of this disclosure can efficiently and quickly identify convolutional layer pairs that can achieve depth-separable convolution through combination.

[0146] In an optional example, during the compilation phase, assuming that there is a conv1 in the neural network model to be compiled, and the type of conv1 (equivalent to the first convolutional layer mentioned above) is a deep convolutional layer, based on the usage relationship of the output of conv1, it is determined that the output of conv1 is only used by one network layer in the neural network model to be compiled, and the type of that network layer is a point convolutional layer. Then, that network layer can be used as a convolutional layer paired with conv1 (equivalent to the second convolutional layer mentioned above).

[0147] Assuming the convolutional layer paired with conv1 is conv2, conv1 and conv2 can be merged to obtain a merged network layer (which can be called DwPw Conv). The layer parameters of the merged network layer can include the depthwise convolution parameters of conv1 (which can be called Depthwise Conv Weight) and the pointwise convolution parameters of conv2 (which can be called Pointwise Conv Weight). In addition, a buffer (which can be called a temporary buffer) can be allocated to the merged network layer. Based on the merged network layer, the buffer, and the network layers other than conv1 and conv2 in the neural network model to be compiled, instructions can be generated through assembly.

[0148] During the execution phase, instructions generated through assembly during the compilation phase can be executed. Specifically, such as... Figure 9 As shown, the depthwise convolution parameters in the layer parameters of the merged network layer can be used to perform depthwise convolution operations on the input feature map (which can be called InputFeature) of the merged convolutional layer, and the resulting depthwise convolution results can be cached in the buffer area in sequence. Then, the pointwise convolution parameters in the layer parameters of the merged network layer can be used to perform pointwise convolution operations on the depthwise convolution results in the buffer area, so that the pointwise convolution results can be used to generate the output feature map of the merged convolutional layer.

[0149] It should be noted that, assuming the input and output of conv1 are represented in the following form: conv1_output = conv(input), and the input and output of conv2 are represented in the following form: conv2_output = conv(conv1_output), then the input and output of the merged convolutional layer can be represented in the following form: conv2_output = DWPW_conv(input).

[0150] In summary, the embodiments of this disclosure fully utilize the fast computation speed of depthwise convolution and the simple computation of pointwise convolution, which only requires relying on a portion of the results of depthwise convolution. By introducing a buffer, pointwise convolution can be performed without waiting for all data in the input feature map to complete depthwise convolution, thereby effectively improving the computation speed of depthwise separable convolution.

[0151] Any of the methods for processing feature maps using an artificial intelligence accelerator provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the methods for processing feature maps using an artificial intelligence accelerator provided in this disclosure can be executed by a processor, such as by a processor executing any of the methods for processing feature maps using an artificial intelligence accelerator mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated upon below.

[0152] The compilation method for any neural network model provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the compilation method for any neural network model provided in this disclosure can be executed by a processor, such as by a processor executing the compilation method for any neural network model mentioned in this disclosure by calling corresponding instructions stored in memory.

[0153] Exemplary device

[0154] Figure 10 This is a schematic diagram of the structure of an apparatus for processing feature maps by an artificial intelligence accelerator, provided in an exemplary embodiment of this disclosure. Figure 10 The device shown includes a first determining module 1010, a second determining module 1020, a first calculation module 1030, a cache module 1040, a second calculation module 1050, and a third determining module 1060.

[0155] The first determining module 1010 is used to determine the merged convolutional layer in the target neural network model. The layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters.

[0156] The second determining module 1020 is used to determine the cache area corresponding to the merged convolutional layer determined by the first determining module 1010.

[0157] The first operation module 1030 is used to perform depth convolution operation on the input feature map of the merged convolutional layer based on the depth convolution parameters of the merged convolutional layer determined by the first determination module 1010.

[0158] The caching module 1040 is used to cache at least two depth convolution results obtained by the depth convolution operation of the first operation module 1030 to the cache area determined by the second determination module 1020 in sequence, wherein any two depth convolution results in the at least two depth convolution results correspond to different local regions in the input feature map.

[0159] The second operation module 1050 is used to perform point convolution operation on the depth convolution result cached in the buffer area determined by the second determination module 1020 based on the point convolution parameters of the merged convolution layer determined by the first determination module 1010.

[0160] The third determining module 1060 is used to determine the output feature map of the merged convolutional layer determined by the first determining module 1010 based on the point convolution result obtained by the point convolution operation of the second operation module 1050.

[0161] In an optional example, such as Figure 11 As shown, the cache module 1040 includes:

[0162] The first determination submodule 10401 is used to determine the first cache sub-area in the cache area;

[0163] The caching submodule 10403 is used to cache at least two depth convolution results obtained by the depth convolution operation of the first operation module 1030 into the first cache sub-area in the cache area determined by the first determining submodule 10401.

[0164] The second arithmetic module 1050 includes:

[0165] The second determination submodule 10501 is used to determine the second cache sub-area in the cache area;

[0166] The transfer submodule 10503 is used to transfer the depthwise convolution result cached in the first cache sub-region determined by the first determining submodule 10401 to the second cache sub-region determined by the second determining submodule 10501.

[0167] The first operation submodule 10505 is used to perform point convolution operation on the depth convolution result of the second cache sub-region cached by the second determination submodule 10501 based on the point convolution parameters of the merged convolution layer determined by the first determination module 1010.

[0168] In an optional example, such as Figure 12 As shown, the second arithmetic module 1050 includes:

[0169] The third determining submodule 10507 is used to determine the first size relationship between the space occupancy rate of the buffer determined by the second determining module 1020 and the preset space occupancy rate.

[0170] The fourth determining submodule 10509 is used to determine a second size relationship between the number of depthwise convolution results cached in the buffer area determined by the second determining module 1020 and the first preset number;

[0171] The fifth determining submodule 10511 is used to determine the time relationship between the current time and the preset time;

[0172] The second operation submodule 10513 is used to trigger point convolution operation in response to at least one of the first size relationship determined by the third determining submodule 10507, the second size relationship determined by the fourth determining submodule 10509, and the time relationship determined by the fifth determining submodule 10511. Based on the point convolution parameters of the merged convolution layer determined by the first determining module 1010, point convolution operation is performed on the depth convolution result cached in the buffer area determined by the second determining module 1020.

[0173] Figure 13 This is a schematic diagram of the structure of a compilation device for a neural network model provided in an exemplary embodiment of this disclosure. Figure 13 The apparatus shown includes a fourth determining module 1310, a fifth determining module 1320, a merging module 1330, an allocation module 1340, and a generating module 1350.

[0174] The fourth determining module 1310 is used to determine the neural network model to be compiled;

[0175] The fifth determining module 1320 is used to determine the paired first and second convolutional layers from the neural network model to be compiled determined by the fourth determining module 1310;

[0176] The merging module 1330 is used to merge the first convolutional layer and the second convolutional layer determined by the fifth determining module 1320 to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer.

[0177] The allocation module 1340 is used to allocate a buffer area for the merged convolutional layer obtained by the merging module 1330;

[0178] The generation module 1350 is used to compile and generate a target neural network model based on the merged convolutional layer obtained by the merging module 1330, the buffer allocated by the allocation module 1340, and the network layers in the neural network model to be compiled determined by the fourth determining module 1310, excluding the first and second convolutional layers determined by the fifth determining module 1320. The target neural network model includes instructions for executing the above-described method of processing feature maps through an artificial intelligence accelerator.

[0179] In an optional example, such as Figure 14 As shown, the fifth determining module 1320 includes:

[0180] The sixth determining submodule 13201 is used to determine the output result information of the first convolutional layer in the neural network model to be compiled, as determined by the fourth determining module 1310.

[0181] The seventh determining submodule 13203 is used to determine the number and type of network layers in the output result of the first convolutional layer in the neural network model to be compiled, based on the output result usage information determined by the sixth determining submodule 13201.

[0182] The eighth determining submodule 13205 is used to determine the second convolutional layer paired with the first convolutional layer based on the number and type of network layers using the output results determined by the seventh determining submodule 13203.

[0183] In an optional example, such as Figure 14 As shown, the eighth determining submodule 13205 includes:

[0184] The judgment unit 132051 is used to determine whether the number of network layers using the output results determined by the seventh determination submodule 13203 is a preset number, and to determine whether the type of network layers using the output results determined by the seventh determination submodule 13203 is a point convolutional layer.

[0185] The determining unit 132053 is configured to determine, in response to the judging unit 132051 determining that the number of network layers using the output result is a second preset number and that the type of network layer using the output result is a point convolutional layer, the network layer using the output result is a second convolutional layer paired with the first convolutional layer.

[0186] Exemplary electronic devices

[0187] Below, for reference Figure 15 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0188] Figure 15 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0189] like Figure 15 As shown, the electronic device 1500 includes one or more processors 1510 and memory 1520.

[0190] The processor 1510 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1500 to perform desired functions.

[0191] The memory 1520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1510 may execute the program instructions to implement the methods for processing feature maps using an artificial intelligence accelerator or the compilation methods for neural network models described in the various embodiments of this disclosure above. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0192] In one example, the electronic device 1500 may also include an input device 1530 and an output device 1540, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0193] For example, when the electronic device is a first device or a second device, the input device 1530 may be a microphone or a microphone array. When the electronic device is a standalone device, the input device 1530 may be a communication network connector for receiving acquired input signals from the first device and the second device.

[0194] In addition, the input device 1530 may also include, for example, a keyboard, a mouse, etc.

[0195] The output device 1540 can output various information to the outside, including determined distance information, direction information, etc. The output device 1540 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0196] Of course, for the sake of simplicity, Figure 15Only some of the components of the electronic device 1500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1500 may include any other suitable components depending on the specific application.

[0197] Exemplary computer program products and computer-readable storage media

[0198] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for processing feature maps by an artificial intelligence accelerator or the methods for compiling neural network models according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0199] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0200] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps in the method of processing feature maps by an artificial intelligence accelerator or in the compilation of a neural network model according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0201] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0202] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0204] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0205] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0206] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0207] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0208] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for processing feature maps using an artificial intelligence accelerator, comprising: Determine the merged convolutional layer in the target neural network model, wherein the layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters; Determine the cache area corresponding to the merged convolutional layer; Based on the depth convolution parameters, a depth convolution operation is performed on the input feature map of the merged convolutional layer; At least two depthwise convolution results obtained from the depthwise convolution operation are sequentially cached in the cache area, and any two depthwise convolution results correspond to different local regions in the input feature map. In response to at least one of the following: the space occupancy rate of the cache is greater than or equal to a preset space occupancy rate, the number of depth convolution results cached in the cache is greater than or equal to a first preset number, and the current time is equal to a preset time, point convolution operation is performed on the depth convolution results cached in the cache based on the point convolution parameters. Based on the point convolution result obtained from the point convolution operation, the output feature map of the merged convolutional layer is determined.

2. The method according to claim 1, wherein, The step of sequentially caching at least two depthwise convolution results obtained from the depthwise convolution operation into the cache area includes: At least two depthwise convolution results obtained from the depthwise convolution operation are sequentially cached into the first cache sub-area of ​​the cache area; The step of performing point convolution operations on the depthwise convolution results cached in the buffer based on the point convolution parameters includes: Transfer the depthwise convolution result cached in the first cache sub-region to the second cache sub-region; Based on the point convolution parameters, point convolution operations are performed on the depth convolution results cached in the second cache sub-region.

3. The method according to claim 1, wherein, In response to at least one of the following: the space occupancy rate of the cache is greater than or equal to a preset space occupancy rate, the number of depthwise convolution results cached in the cache is greater than or equal to a first preset number, and the current time is equal to a preset time, pointwise convolution operations are performed on the depthwise convolution results cached in the cache based on the pointwise convolution parameters, including: Determine a first size relationship between the space occupancy rate of the cache area and the preset space occupancy rate; Determine a second size relationship between the number of depthwise convolution results cached in the cache area and a first preset number; Determine the time relationship between the current time and the preset time; In response to at least one of the first size relationship, the second size relationship, and the time relationship triggering a point convolution operation, a point convolution operation is performed on the depthwise convolution result cached in the buffer based on the point convolution parameters.

4. A method for compiling a neural network model, comprising: Identify the neural network model to be compiled; Determine the paired first and second convolutional layers from the neural network model to be compiled; The first convolutional layer and the second convolutional layer are merged to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer. Allocate a buffer area for the merged convolutional layer; Based on the merged convolutional layer, the buffer, and the network layers in the neural network model to be compiled, excluding the first and second convolutional layers, a target neural network model is compiled and generated. The target neural network model includes instructions for executing the method of processing feature maps by an artificial intelligence accelerator as described in any one of claims 1-3.

5. The method according to claim 4, wherein, The step of determining the paired first and second convolutional layers from the neural network model to be compiled includes: Determine the information used in the output of the first convolutional layer in the neural network model to be compiled; Based on the information used in the output results, determine the number and type of network layers in the neural network model to be compiled that use the output results of the first convolutional layer; Based on the number and type of network layers using the output results, a second convolutional layer is determined to be paired with the first convolutional layer.

6. The method according to claim 5, wherein, The step of determining the second convolutional layer paired with the first convolutional layer based on the number and type of network layers using the output results includes: In response to the fact that the number of network layers using the output result is a second preset number, and the type of the network layer using the output result is a point convolutional layer, the network layer using the output result is determined as a second convolutional layer paired with the first convolutional layer.

7. An apparatus for processing feature maps using an artificial intelligence accelerator, comprising: The first determining module is used to determine the merged convolutional layer in the target neural network model, wherein the layer parameters of the merged convolutional layer include depthwise convolutional parameters and pointwise convolutional parameters. The second determining module is used to determine the cache area corresponding to the merged convolutional layer determined by the first determining module; The first calculation module is used to perform depth convolution operation on the input feature map of the merged convolutional layer based on the depth convolution parameters of the merged convolutional layer determined by the first determining module. A caching module is used to sequentially cache at least two depth convolution results obtained by the depth convolution operation of the first operation module into the cache area determined by the second determination module, wherein any two depth convolution results correspond to different local regions in the input feature map. The second calculation module is used to perform point convolution operation on the depth convolution results cached in the cache area, as determined by the second calculation module, in response to at least one of the following: the space occupancy rate of the cache area is greater than or equal to a preset space occupancy rate, the number of depth convolution results cached in the cache area is greater than or equal to a first preset number, and the current time is equal to a preset time. The third determining module is used to determine the output feature map of the merged convolutional layer determined by the first determining module based on the point convolution result obtained by the point convolution operation of the second operation module.

8. A compilation device for a neural network model, comprising: The fourth determination module is used to determine the neural network model to be compiled; The fifth determining module is used to determine the paired first and second convolutional layers from the neural network model to be compiled determined by the fourth determining module; The merging module is used to merge the first convolutional layer and the second convolutional layer determined by the fifth determining module to obtain a merged convolutional layer. The layer parameters of the merged convolutional layer include the depth convolution parameters of the first convolutional layer and the point convolution parameters of the second convolutional layer. An allocation module is used to allocate a buffer area for the merged convolutional layer obtained by the merging module; A generation module is used to compile and generate a target neural network model based on the merged convolutional layer obtained by the merging module, the buffer allocated by the allocation module, and the network layers in the neural network model to be compiled determined by the fourth determining module, excluding the first convolutional layer and the second convolutional layer determined by the fifth determining module. The target neural network model includes instructions for executing the method of processing feature maps by an artificial intelligence accelerator as described in any one of claims 1-3.

9. A computer-readable storage medium storing a computer program for executing the method of processing feature maps by an artificial intelligence accelerator as described in any one of claims 1-3, or for executing the method of compiling a neural network model as described in any one of claims 4-6.

10. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for processing feature maps by an artificial intelligence accelerator as described in any one of claims 1-3, or to implement the method for compiling a neural network model as described in any one of claims 4-6.

Images