Convolutional neural network mapping method and device, electronic device and storage medium

Abstract

This application discloses a mapping method and apparatus for convolutional neural networks. The mapping method includes: traversing each convolutional layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolutional layer; obtaining the arrangement parameters of the convolutional layer based on the convolution parameters and the parameters of the storage array, wherein the arrangement parameters of the convolutional layer include the number of rows and columns of the weight data of the convolutional layer in the storage, and the parameters of the storage array include the number of storage units and the number of rows and columns of the storage units; sorting multiple convolutional layers according to the convolution type, convolution parameters, and arrangement parameters; and obtaining the arrangement position of each convolutional layer in the storage array according to the sorted order based on the memory loss and / or balance. This application obtains the optional arrangement positions of the convolutional layers in the storage array and optimizes the arrangement positions of the convolutional layers in the storage array based on the memory loss and / or balance of the optional arrangement positions, thereby improving the space utilization of the storage array.

Description

Technical Field

[0001] The present invention relates to the field of in-memory computing technology, and more specifically, to a convolutional neural network mapping method and device, electronic equipment and storage medium. Background Art

[0002] With the rapid growth of data scale, the demand for hardware computing power and data storage capacity in storage-intensive neural network applications has also increased accordingly. In order to break through the memory access bandwidth bottleneck caused by the separation of computing and storage in the traditional von Neumann architecture, more and more research has begun to focus on high-density storage-computing integrated computing architecture, hoping to reduce the additional memory access energy consumption and bandwidth requirements caused by frequent data transfer between storage components and computing components by tightly coupling computing and storage, thereby maximizing the energy efficiency of the hardware architecture.

[0003] The storage-computing integration is data-centric, placing the computing logic in the memory, or directly using the characteristics of the memory itself for computing, thereby greatly reducing the communication overhead between the kernel and the external storage. It is particularly suitable for application scenarios such as deep learning neural networks, which involve large amounts of data and large-scale parallelism. Therefore, storage-computing integrated accelerators that use non-volatile memory (NVM) such as memristors as computing units have emerged. This type of accelerator effectively solves the bottleneck of bandwidth and has the characteristics of low power consumption and high speed.

[0004] Resistive Random Access Memory (ReRAM) is a non-volatile memory with almost zero leakage power consumption. It stores information in a resistance state. This resistive storage principle can provide inherent computing power, so data storage and data processing functions can be integrated at the same physical unit address. ReRAM supports multiple in-memory computing operations and can significantly improve the computing efficiency of neural networks.

[0005] The storage and computing integrated accelerator with ReRAM as the computing unit stores the weight data of the convolution operation in the neural network calculation in advance on the ReRAM array, so that the result of the convolution operation can be obtained immediately after receiving external input data. Due to factors such as the shape of the convolution kernel in the network model, the size of the input and output data, the efficiency and limitations of the hardware row and column switching, and the convolution method, the space utilization rate of ReRAM is insufficient, the overall computing efficiency of the system is low, and the computing overhead is large. Summary of the invention

[0006] In view of the above problems, the object of the present invention is to provide a convolutional neural network mapping method and device, electronic device and storage medium, which can improve the space utilization of the storage array.

[0007] According to a first aspect of the present invention, a mapping method for a convolutional neural network is provided, comprising: traversing each convolutional layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolutional layer, the convolution parameters including the number of input channels, the number of output channels, the convolution filter width and the convolution filter height of the convolution kernel; obtaining the arrangement parameters of the convolutional layer according to the convolution parameters of the convolutional layer and the parameters of a storage array, the arrangement parameters of the convolutional layer including the number of arrangement rows and the number of arrangement columns of the weight data of the convolutional layer in the storage body, the parameters of the storage array including the number of storage bodies and the number of rows and columns of the storage bodies; sorting a plurality of convolutional layers according to the convolution type, convolution parameters and arrangement parameters of the convolutional layer; and obtaining the arrangement position of each convolutional layer in the storage array according to the memory loss degree and / or balance degree in the sorted order.

[0008] Preferably, sorting multiple convolutional layers according to the convolution type, convolution parameters and arrangement parameters of the convolutional layers includes: sorting with the convolution type of the convolutional layer as the first priority; and sorting convolutional layers of the same convolution type according to the convolution parameters and arrangement parameters of the convolutional layers.

[0009] Preferably, the type of the convolution layer includes at least one of a common convolution and a depthwise separable convolution; the common convolution layer is stored before the depthwise separable convolution layer.

[0010] Preferably, the ordering priority of the convolution parameters and arrangement parameters of the convolution layer is: the number of weights of the convolution filter, the number of output channels, the number of arrangement rows and the number of arrangement columns, and the order of priority is adjustable.

[0011] Preferably, obtaining the arrangement position of the convolution layer in the storage array according to the memory loss degree and / or the balance degree includes: obtaining the optional arrangement position of the convolution kernel to be arranged in the convolution layer, the optional arrangement position including an optional starting position and an optional storage body; obtaining the memory loss degree and / or the balance degree of each optional arrangement position according to the optional starting position, the weight arrangement state of the optional storage body and the arrangement parameters of the convolution layer to be arranged; determining the arrangement position of the convolution kernel to be arranged according to the memory loss degree of the optional arrangement position.

[0012] Preferably, the memory loss degree of the optional arrangement position of the ordinary convolution layer is the number of empty rows before the arrangement position of the convolution kernel to be arranged; the balance degree of the optional storage body of the ordinary convolution layer is the difference between the maximum and minimum number of arrangement rows of weight data of each storage body.

[0013] Preferably, the memory loss degree of the optional arrangement position of the depthwise separable convolutional layer is the sum of the number of empty columns in each row to the left of the arrangement position of the convolution kernel to be arranged; the balance degree of the optional storage body of the depthwise separable convolutional layer is the difference between the column where the optional starting position is located and the column where the arrangement starting position of the convolution kernels arranged in each storage body is located.

[0014] Preferably, the weight arrangement states of multiple storage bodies are merged to obtain an optional starting position for the convolution kernel to be arranged.

[0015] Preferably, when the number of output channels is less than the number of storage bodies, the weight data of the convolution filters of different output channels of each convolution layer are arranged in different storage bodies, and the arrangement positions of the weight data of the convolution filters of different output channels in different storage bodies are the same.

[0016] Preferably, the weight data of the convolution filters of the same output channel are arranged in the same storage volume.

[0017] According to a second aspect of the present invention, there is provided a mapping device for a convolutional neural network, comprising: a first acquisition module, used to traverse each convolutional layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolutional layer, wherein the convolution parameters include the number of input channels, the number of output channels, the convolution filter width and the convolution filter height of the convolution kernel; a second acquisition module, used to obtain the arrangement parameters of the convolutional layer according to the convolution parameters of the convolutional layer and the parameters of the storage array, wherein the arrangement parameters of the convolutional layer include the number of arrangement rows and the number of arrangement columns of the weight data of the convolutional layer in the storage body, and the parameters of the storage array include the number of storage bodies and the number of rows and columns of the storage body; a sorting module, used to sort a plurality of convolutional layers according to the convolution type, convolution parameters and arrangement parameters of the convolutional layer; and a mapping module, used to obtain the arrangement position of each convolutional layer in the storage array according to the memory loss degree and / or the balance degree in the sorting order.

[0018] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the mapping method of the convolutional neural network as described above.

[0019] According to a fourth aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the mapping method of the convolutional neural network as described above when executing the computer program.

[0020] The mapping method and device of the convolutional neural network, the electronic device and the storage medium provided by the present invention obtain the optional arrangement position of the convolutional layer in the storage array according to the convolution type, convolution parameters and parameters of the storage array of the convolutional layer, and optimize the arrangement position of the convolutional layer in the storage array according to the memory loss degree and / or balance degree of the optional arrangement position, thereby improving the space utilization rate of the storage array.

[0021] Furthermore, after each convolution kernel is arranged, the weight arrangement state of each storage body is merged to obtain an optional starting position of the convolution kernel to be arranged, so that the weight data of the convolution filters of different output channels have the same arrangement position in different storage bodies, and can be calculated in parallel to improve computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart showing a mapping method of a convolutional neural network provided by an embodiment of the present invention is shown;

[0024] Figure 2 A flowchart of step S140 provided in an embodiment of the present invention is shown;

[0025] Figure 3 A diagram showing a mapping relationship between a common convolutional layer and a storage array provided by an embodiment of the present invention;

[0026] Figure 4 A diagram showing a mapping relationship between another common convolutional layer and a storage array provided by an embodiment of the present invention;

[0027] Figure 5 A schematic diagram showing the relationship between the arrangement position of a common convolutional layer and the memory loss degree provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram showing the relationship between the arrangement position of the depthwise separable convolutional layer and the memory loss provided by an embodiment of the present invention;

[0029] Figure 7 A diagram showing a mapping relationship between a depth-separable convolutional layer and a storage array provided by an embodiment of the present invention;

[0030] Figure 8 A mapping relationship diagram between another depth-separable convolutional layer and a storage array provided by an embodiment of the present invention is shown;

[0031] Fig. 9 A schematic diagram showing the structure of a mapping device for a convolutional neural network provided by an embodiment of the present invention is shown;

[0032] Fig.10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] Various embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by the same or similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale.

[0034] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0035] Figure 1 FIG. 1 is a flow chart showing a convolutional neural network mapping method provided by an embodiment of the present invention. Figure 1 As shown, the mapping method of the convolutional layer includes the following steps.

[0036] In step S110, each convolutional layer of the convolutional neural network is traversed to obtain the convolution type and convolution parameters of the convolutional layer.

[0037] In this embodiment, the convolutional neural network includes multiple convolutional layers, and the convolution type of the convolutional layer includes at least one of ordinary convolution and depth-separable convolution. The convolution parameters include the number of input channels I of the convolution kernel, the number of output channels O, the convolution filter width W of the convolution kernel, and the height H of the convolution filter. The number of output channels O is also the number of convolution filters, and the convolution filter includes multiple channels, and the number of channels is consistent with the number of input channels I. For depth-separable convolution, the number of input channels of the convolution kernel I=1.

[0038] In step S120, the arrangement parameters of the convolution layer are obtained according to the convolution parameters of the convolution layer and the parameters of the storage array, wherein the arrangement parameters of the convolution layer include the number of arrangement rows and columns of the weight data of the convolution layer in the storage body, and the parameters of the storage array include the number of storage bodies (Bank) and the number of rows and columns of the storage bodies (Bank).

[0039] In this embodiment, the storage array includes a plurality of storage banks. In this embodiment, the storage array includes a plurality of row storage banks as an example for description, but it is not limited to this. Specifically, the storage array includes K row storage banks; each row storage bank has the same size, and each row storage bank is a storage cell array of M rows and N columns. The embodiment of the present invention is described by taking K=128, M=256, N=288 as an example, but it is not limited to this.

[0040] According to the number of input channels, the convolution filter width of the convolution kernel and the height of the convolution filter in the convolution parameters, the weight number P = W*H*I of the convolution filter of each output channel is obtained; according to the weight number P of the convolution filter and the number of columns N of the memory bank, the number of rows and columns of the convolution filter in the memory bank is calculated. Specifically, the maximum factor whose weight number P is less than N is selected as the number of columns of each row of the convolution filter, and then the number of rows of the convolution filter is obtained according to the weight number P and the number of columns.

[0041] In step S130, multiple convolutional layers are sorted according to the convolution types, convolution parameters, and arrangement parameters of the convolutional layers.

[0042] In this embodiment, the convolution type of the convolution layer is used as the first priority for sorting; convolution layers of the same convolution type are sorted according to the convolution parameters and arrangement parameters of the convolution layer.

[0043] Specifically, the arrangement of common convolutional layers is performed first according to the type of convolutional layer, and then the arrangement of depthwise convolutional layers is performed. All common convolutional layers are sorted according to the convolutional parameters and arrangement parameters of the convolutional layers; then all depthwise separable convolutional layers are sorted according to the convolutional parameters and arrangement parameters of the convolutional layers. The sorting priority of the convolutional parameters and arrangement parameters of the convolutional layers is: the number of weights of the convolutional filter, the number of output channels, the number of arrangement rows, and the number of arrangement columns. The order of priority is adjustable.

[0044] In step S140, the arrangement position of each convolutional layer in a storage array is obtained in sequence according to the memory loss degree and / or the balance degree in the sorted order, and the storage array includes a plurality of storage banks.

[0045] In this embodiment, see Figure 3 and Figure 7 When the number of output channels is less than the number of memory banks (O<K), the weight data of the convolution filters of different output channels of each convolution layer are arranged in different memory banks, and the weight data of the convolution filters of different output channels are arranged in the same position in different memory banks; the weight data of the convolution filters of the same output channel are arranged in the same memory bank. The weight data of the convolution filters of different output channels are arranged in the same position in different memory banks, which can be used for parallel calculation in the form of broadcasting to improve the calculation efficiency.

[0046] See also Figure 4 and Figure 8 , when the number of output channels is greater than or equal to the number of memory banks (O ≥ K), the weight data of the convolution filter memories of at least two different output channels are arranged in the same memory bank.

[0047] Specifically, step S140 includes the following steps.

[0048] In step S141, an optional arrangement position of a convolution kernel to be arranged in the convolution layer is obtained, where the optional arrangement position includes an optional starting position and an optional storage body.

[0049] In this embodiment, the weight data of the convolution filters of different output channels are arranged in the same position in different storage bodies, and the weight arrangement states of multiple storage bodies need to be merged to obtain the optional starting position of the convolution kernel to be arranged. Figure 6, the weight arrangement states of multiple storage bodies are merged, that is, the maximum number of weight arrangement rows and the maximum number of arrangement columns of the storage body are obtained.

[0050] by Figure 6 Take the example shown for explanation. The weight arrangement of each storage body is four rows, and the number of columns arranged in each row is different. The maximum value Colmax of the number of arrangement columns corresponding to all rows of all storage bodies Bank is obtained. If the maximum value of the number of arrangement columns of the first row row1 is Colmax_1, then the optional starting position of the convolution kernel to be arranged in the first row of any storage body Bank is any position after [1, Colmax_1]; if the maximum value of the number of arrangement columns of the second row row2 is Colmax_2, then the optional starting position of the convolution kernel to be arranged in the second row of any storage body Bank is any position after [2, Colmax_2], and the corresponding maximum value of the number of arrangement columns of the mth row is Colmax_m, then the optional arrangement position of the convolution kernel to be arranged in the first row of any storage body Bank is any position after [m, Colmax_m].

[0051] In step S142, the memory loss degree and / or balance degree of each optional arrangement position is obtained according to the optional starting position, the weight arrangement state of the optional storage body, and the arrangement parameters of the convolutional layer to be arranged.

[0052] In this embodiment, the memory loss degree of the optional arrangement position of the common convolution layer is the number of empty rows before the arrangement position of the convolution kernel to be arranged. The balance degree of the optional storage bank of the common convolution layer is the difference between the maximum and minimum number of arrangement rows of weight data of each storage bank.

[0053] See also Figure 5 , determine the optional starting position of the convolution kernel to be arranged according to the maximum value of the number of weight arrangement rows of multiple storage bodies, for example, the maximum value of the number of weight arrangement rows of all storage bodies is the number of weight arrangement rows of storage body Bank_i, the number of weight arrangement rows of storage body Bank_i is 3 rows, and the optional starting position is, for example, the 4th row. The memory loss degree and / or balance degree of the optional arrangement position can be calculated according to the optional starting position and the weight arrangement state of each storage body. If the number of weight arrangement rows of storage body Bank_j is 2 rows, if a new convolution kernel is arranged in storage body Bank_j, the convolution kernel must also be arranged from the 4th row. In this case, the memory loss degree of the arrangement position of the convolution kernel is one row. After the convolution kernel is arranged in the optional storage body, the smaller the difference between the maximum and minimum values ​​of the weight data arrangement rows of each storage body, the better, and the smaller the difference, the more balanced the weight data arranged in each storage body.

[0054] For the depthwise separable convolutional layer, the optional starting position of the convolution kernel to be arranged is determined according to the maximum value of the weight number of each row of the multiple memory banks; the memory loss degree and / or balance degree of each optional arrangement position can be calculated according to the optional starting position and the weight arrangement state of each memory bank. The memory loss degree of the optional arrangement position of the depthwise separable convolutional layer is the sum of the number of empty columns in each row to the left of the arrangement position of the convolution kernel to be arranged (see Figure 6 ); The balance degree of the optional storage body of the depth-separable convolutional layer is the difference between the column where the optional starting position is located and the column where the arrangement starting position of the convolution kernels of each storage body is arranged.

[0055] In step S143, the arrangement position of the convolution kernel to be arranged is determined according to the memory loss degree and / or balance degree of the optional arrangement position.

[0056] In this embodiment, the memory loss of each optional arrangement position is compared, and the optional arrangement position with the least memory loss and / or good balance is selected as the arrangement position for the convolution kernel to be arranged.

[0057] The mapping method of the convolutional neural network provided in an embodiment of the present invention obtains the optional arrangement position of the convolution layer in the storage array according to the convolution type, convolution parameters and parameters of the storage array of the convolution layer, and optimizes the arrangement position of the convolution layer in the storage array according to the memory loss and / or balance of the optional arrangement position, thereby improving the space utilization of the storage array.

[0058] Furthermore, after each convolution kernel is arranged, the weight arrangement state of each storage body is merged to obtain an optional starting position of the convolution kernel to be arranged, so that the weight data of the convolution filters of different output channels have the same arrangement position in different storage bodies, and can be calculated in parallel to improve computing efficiency.

[0059] Corresponding to the mapping method of the convolutional neural network described in the above embodiment, Fig. 9 FIG. 1 is a schematic diagram showing the structure of a mapping device for a convolutional neural network provided by an embodiment of the present invention. Fig. 9 As shown, the convolutional neural network mapping device 90 includes a first acquisition module 91, a second acquisition module 92, a sorting module 93 and a mapping module 94.

[0060] Among them, the first acquisition module 91 is used to traverse each convolution layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolution layer, and the convolution parameters include the number of input channels of the convolution kernel, the number of output channels, the convolution filter width of the convolution kernel, and the convolution filter height.

[0061] In this embodiment, the convolutional neural network includes a plurality of convolutional layers, and the convolution type of the convolutional layer includes at least one of ordinary convolution and depthwise separable convolution. The convolution parameters include the number of input channels I of the convolution kernel, the number of output channels O, the width W of the convolution filter (Filter) of the convolution kernel, and the height H of the convolution filter. The number of output channels O is also the number of convolution filters, and the convolution filter includes a plurality of channels, and the number of channels is consistent with the number of input channels I. For depthwise separable convolution, the number of input channels I of the convolution kernel is 1.

[0062] The second acquisition module 92 is used to obtain the arrangement parameters of the convolution layer according to the convolution parameters of the convolution layer and the parameters of the storage array, wherein the arrangement parameters of the convolution layer include the number of arrangement rows and columns of the weight data of the convolution layer in the storage body, and the parameters of the storage array include the number of storage bodies and the number of rows and columns of the storage bodies.

[0063] In this embodiment, the storage array includes a plurality of storage banks. In this embodiment, the storage array includes a plurality of row storage banks as an example for description, but it is not limited to this. Specifically, the storage array includes K row storage banks; each row storage bank has the same size, and each row storage bank is a storage cell array of M rows and N columns. The embodiment of the present invention is described by taking K=128, M=256, N=288 as an example, but it is not limited to this.

[0064] According to the number of input channels, the convolution filter width of the convolution kernel and the height of the convolution filter in the convolution parameters, the weight number P = W*H*I of the convolution filter of each output channel is obtained; according to the weight number P of the convolution filter and the number of columns N of the memory bank, the number of rows and columns of the convolution filter in the memory bank is calculated. Specifically, the maximum factor whose weight number P is less than N is selected as the number of columns of each row of the convolution filter, and then the number of rows of the convolution filter is obtained according to the weight number P and the number of columns.

[0065] The sorting module 93 is used to sort the multiple convolutional layers according to the convolution types, convolution parameters and arrangement parameters of the convolutional layers.

[0066] In this embodiment, the convolution type of the convolution layer is used as the first priority for sorting; convolution layers of the same convolution type are sorted according to the convolution parameters and arrangement parameters of the convolution layer.

[0067] Specifically, the arrangement of common convolutional layers is performed first according to the type of convolutional layer, and then the arrangement of depthwise convolutional layers is performed. All common convolutional layers are sorted according to the convolutional parameters and arrangement parameters of the convolutional layers; then all depthwise separable convolutional layers are sorted according to the convolutional parameters and arrangement parameters of the convolutional layers. The sorting priority of the convolutional parameters and arrangement parameters of the convolutional layers is: the number of weights of the convolutional filter, the number of output channels, the number of arrangement rows, and the number of arrangement columns. The order of priority is adjustable.

[0068] The mapping module 94 is used to obtain the arrangement position of each convolution layer in the storage array according to the memory loss degree and / or the balance degree in the sorted order.

[0069] In this embodiment, see Figure 3 and Figure 7 When the number of output channels is less than the number of memory banks (O<K), the weight data of the convolution filters of different output channels of each convolution layer are arranged in different memory banks, and the weight data of the convolution filters of different output channels are arranged in the same position in different memory banks; the weight data of the convolution filters of the same output channel are arranged in the same memory bank. The weight data of the convolution filters of different output channels are arranged in the same position in different memory banks, which can be used for parallel calculation in the form of broadcasting to improve the calculation efficiency.

[0070] See also Figure 4 and Figure 8 , when the number of output channels is greater than or equal to the number of memory banks (O ≥ K), the weight data of the convolution filter memories of at least two different output channels are arranged in the same memory bank.

[0071] Preferably, the mapping module 94 is also used to obtain optional arrangement positions of convolution kernels to be arranged in the convolution layer, and the optional arrangement positions include optional starting positions and optional storage bodies; obtain the memory loss and / or balance of each optional arrangement position according to the optional starting position, the weight arrangement state of the optional storage body and the arrangement parameters of the convolution layer to be arranged; determine the arrangement position of the convolution kernel to be arranged according to the memory loss and / or balance of the optional arrangement position.

[0072] In this embodiment, the weight data of the convolution filters of different output channels are arranged in the same position in different storage bodies, and the weight arrangement states of multiple storage bodies need to be merged to obtain the optional starting position of the convolution kernel to be arranged. Figure 6 , the weight arrangement states of multiple storage bodies are merged, that is, the maximum number of weight arrangement rows and the maximum number of arrangement columns of the storage body are obtained.

[0073] The memory loss degree of the optional arrangement position of the common convolution layer is the number of empty rows before the arrangement position of the convolution kernel to be arranged. The balance degree of the optional storage bank of the common convolution layer is the difference between the maximum and minimum number of arrangement rows of weight data of each storage bank.

[0074] See also Figure 5 , determine the optional starting position of the convolution kernel to be arranged according to the maximum value of the number of weight arrangement rows of multiple storage bodies, for example, the maximum value of the number of weight arrangement rows of all storage bodies is the number of weight arrangement rows of storage body Bank_i, the number of weight arrangement rows of storage body Bank_i is 3 rows, and the optional starting position is, for example, the 4th row. The memory loss degree and / or balance degree of the optional arrangement position can be calculated according to the optional starting position and the weight arrangement state of each storage body. If the number of weight arrangement rows of storage body Bank_j is 2 rows, if a new convolution kernel is arranged in storage body Bank_j, the convolution kernel must also be arranged from the 4th row. In this case, the memory loss degree of the arrangement position of the convolution kernel is one row. After the convolution kernel is arranged in the optional storage body, the smaller the difference between the maximum and minimum values ​​of the weight data arrangement rows of each storage body, the better, and the smaller the difference, the more balanced the weight data arranged in each storage body.

[0075] For the depthwise separable convolutional layer, the optional starting position of the convolution kernel to be arranged is determined according to the maximum value of the weight number of each row of the multiple memory banks; the memory loss degree and / or balance degree of each optional arrangement position can be calculated according to the optional starting position and the weight arrangement state of each memory bank. The memory loss degree of the optional arrangement position of the depthwise separable convolutional layer is the sum of the number of empty columns in each row to the left of the arrangement position of the convolution kernel to be arranged (see Figure 6 ); The balance degree of the optional storage body of the depth-separable convolutional layer is the difference between the column where the optional starting position is located and the column where the arrangement starting position of the convolution kernels of each storage body is arranged.

[0076] In this embodiment, the memory loss of each optional arrangement position is compared, and the optional arrangement position with the least memory loss and / or good balance is selected as the arrangement position for the convolution kernel to be arranged.

[0077] The mapping device of the convolutional neural network provided in an embodiment of the present invention obtains the optional arrangement position of the convolution layer in the storage array according to the convolution type, convolution parameters and parameters of the storage array of the convolution layer, and optimizes the arrangement position of the convolution layer in the storage array according to the memory loss and / or balance of the optional arrangement position, thereby improving the space utilization of the storage array.

[0078] Furthermore, after each convolution kernel is arranged, the weight arrangement state of each storage body is merged to obtain an optional starting position of the convolution kernel to be arranged, so that the weight data of the convolution filters of different output channels have the same arrangement position in different storage bodies, and can be calculated in parallel to improve computing efficiency.

[0079] Fig.10 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10 As shown, the electronic device 100 of this embodiment includes: at least one processor 101 ( Fig.10 Only one is shown in the figure) a processor, a memory 102, and a computer program 103 stored in the memory 102 and executable on the at least one processor 101, and when the processor 101 executes the computer program 103, the steps in any of the above-mentioned storage space allocation method embodiments are implemented.

[0080] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Fig.10 This is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100 . The electronic device 100 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0081] The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0082] In some embodiments, the memory 102 may be an internal storage unit of the electronic device 100, such as a hard disk or memory of the electronic device 100. In other embodiments, the memory 102 may also be an external storage device of the electronic device 100, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 100. Further, the memory 102 may also include both an internal storage unit of the electronic device 100 and an external storage device. The memory 102 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 102 may also be used to temporarily store data that has been output or is to be output.

[0083] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0084] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0086] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0087] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0089] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0090] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A convolutional neural network mapping method, characterized in that: include: Traversing each convolution layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolution layer, wherein the convolution parameters include the number of input channels of the convolution kernel, the number of output channels, the convolution filter width of the convolution kernel, and the convolution filter height; Acquire arrangement parameters of the convolution layer according to the convolution parameters of the convolution layer and the parameters of the storage array, wherein the arrangement parameters of the convolution layer include the number of arrangement rows and the number of arrangement columns of the weight data of the convolution layer in the storage body, and the parameters of the storage array include the number of storage bodies and the number of rows and columns of the storage bodies; Sort multiple convolutional layers according to their convolution types, convolutional parameters, and arrangement parameters; Obtaining the arrangement position of each convolutional layer in the storage array in the sorted order according to the memory loss degree and / or the balance degree; Wherein, obtaining the arrangement position of the convolution layer in the storage array according to the memory loss degree and / or the balance degree includes: Obtaining an optional arrangement position of a convolution kernel to be arranged in a convolution layer, wherein the optional arrangement position includes an optional starting position and an optional storage body; Obtaining a memory loss degree and / or a balance degree of each optional arrangement position according to an optional starting position, a weight arrangement state of an optional storage body, and an arrangement parameter of a convolutional layer to be arranged; The arrangement position of the convolution kernel to be arranged is determined according to the memory loss degree and / or balance degree of the optional arrangement position.

2. The mapping method according to claim 1, characterized in that: Sorting multiple convolutional layers according to the convolution type, convolution parameters, and arrangement parameters of the convolutional layers includes: Sort by the convolution type of the convolution layer as the first priority; Convolutional layers of the same convolution type are sorted according to the convolution parameters and arrangement parameters of the convolutional layers.

3. The mapping method according to claim 2, characterized in that: The type of the convolution layer includes at least one of a common convolution and a depthwise separable convolution; the common convolution layer is stored before the depthwise separable convolution layer.

4. The mapping method according to claim 2, characterized in that: The sorting priority of the convolution parameters and arrangement parameters of the convolution layer is: the number of weights of the convolution filter, the number of output channels, the number of arrangement rows, and the number of arrangement columns. The order of priority is adjustable.

5. The mapping method according to claim 1, characterized in that: The memory loss degree of the optional arrangement position of the ordinary convolution layer is the number of empty rows before the arrangement position of the convolution kernel to be arranged; the balance degree of the optional storage body of the ordinary convolution layer is the difference between the maximum and minimum number of arrangement rows of weight data of each storage body.

6. The mapping method according to claim 1, characterized in that: The memory loss degree of the optional arrangement position of the depthwise separable convolutional layer is the sum of the number of empty columns in each row to the left of the arrangement position of the convolution kernel to be arranged; the balance degree of the optional storage body of the depthwise separable convolutional layer is the difference between the column where the optional starting position is located and the column where the arrangement starting position of the convolution kernels arranged in each storage body is located.

7. The mapping method according to claim 1, characterized in that: The weight arrangement states of multiple storage bodies are merged to obtain an optional starting position for the convolution kernel to be arranged.

8. The mapping method according to claim 1, characterized in that: When the number of output channels is less than the number of storage bodies, the weight data of the convolution filters of different output channels of each convolution layer are arranged in different storage bodies, and the weight data of the convolution filters of different output channels have the same arrangement position in different storage bodies.

9. The mapping method according to claim 1, characterized in that: The weight data of the convolutional filters of the same output channel are arranged in the same memory bank.

10. A mapping device for a convolutional neural network, characterized in that: include: A first acquisition module is used to traverse each convolution layer of the convolutional neural network to obtain the convolution type and convolution parameters of the convolution layer, wherein the convolution parameters include the number of input channels and output channels of the convolution kernel, the convolution filter width and the convolution filter height of the convolution kernel; A second acquisition module is used to acquire arrangement parameters of the convolution layer according to the convolution parameters of the convolution layer and the parameters of the storage array, wherein the arrangement parameters of the convolution layer include the number of arrangement rows and the number of arrangement columns of the weight data of the convolution layer in the storage body, and the parameters of the storage array include the number of storage bodies and the number of rows and columns of the storage bodies; A sorting module, used to sort multiple convolutional layers according to the convolution type, convolution parameters and arrangement parameters of the convolutional layers; A mapping module, used to obtain the arrangement position of each convolution layer in the storage array according to the memory loss degree and / or the balance degree in the sorted order; The mapping module is further used to obtain an optional arrangement position of the convolution kernel to be arranged in the convolution layer, wherein the optional arrangement position includes an optional starting position and an optional storage body; According to the optional starting position, the weight arrangement state of the optional storage body and the arrangement parameters of the convolution layer to be arranged, the memory loss degree and / or the balance degree of each optional arrangement position are obtained; according to the memory loss degree and / or the balance degree of the optional arrangement position, the arrangement position of the convolution kernel to be arranged is determined.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the mapping method of the convolutional neural network as described in any one of claims 1 to 9 is implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the mapping method of the convolutional neural network as described in any one of claims 1 to 10 is implemented.

Images