Bandwidth optimization method and related equipment for convolutional neural network accelerator

By transporting weight parameters from off-chip memory to on-chip memory in a convolutional neural network and transmitting and executing them among multiple computing engines, the problem of low performance of convolutional neural network accelerator is solved, and more efficient computing is achieved Efficiency and bandwidth utilization.

CN114202067BActive Publication Date: 2024-11-08SHANDONG IND RES KUNYUN ARTIFICIAL INTELLIGENCE RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111445760.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-30
Publication Date
2024-11-08
Estimated Expiration
2041-11-30

Smart Images

  • Figure CN114202067B_ABST
    Figure CN114202067B_ABST
Patent Text Reader

Abstract

A bandwidth optimization method for a convolutional neural network accelerator is provided. Before the convolutional neural network runs, a first weight parameter of a convolutional layer to be calculated is transferred from a first off-chip memory to a first on-chip memory, and a second weight parameter of the convolutional layer to be calculated is transferred from a second off-chip memory to a second on-chip memory; when performing convolution calculation of the convolutional layer to be calculated, the first weight parameter is transferred from the first on-chip memory to a first computing engine to perform a first convolution operation, and the second weight parameter is transferred from the second on-chip memory to a second computing engine to perform a second convolution operation; when the first convolution operation and the second convolution operation are completed, the second weight parameter is transferred from the second on-chip memory to the first computing engine to perform a third convolution operation, and the first weight parameter is transferred from the first on-chip memory to the second computing engine to perform a fourth convolution operation; when the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolution layer is performed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural Internet of Things, and in particular to a bandwidth optimization method for a convolutional neural network accelerator and related equipment. Background Art

[0002] With the rapid development of deep learning, convolutional neural networks have been widely used in fields such as computer vision. In order to improve the accuracy of convolutional neural networks, convolutional neural network models have become larger and larger, and the amount of calculation required has also increased. Using CPU to implement convolutional neural network algorithms is slow; using GPU to implement convolutional neural network algorithms consumes high power and has large delays. Summary of the invention

[0003] The embodiments of the present invention provide a bandwidth optimization method and related equipment for a convolutional neural network accelerator, which improves the weight parameter transmission bandwidth, reduces the weight parameter transmission time, and improves the performance of the convolutional neural network accelerator.

[0004] In a first aspect, an embodiment of the present invention provides a bandwidth optimization method for a convolutional neural network accelerator, the method comprising:

[0005] Before the convolutional neural network runs, a first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, and a second weight parameter of the convolutional layer to be calculated is moved from the second off-chip memory to the second on-chip memory, wherein the first weight parameter and the second weight parameter are two parts of weight parameters of the convolutional layer to be calculated in the convolutional neural network;

[0006] When performing the convolution calculation of the convolution layer to be calculated, the first weight parameter is transferred from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is transferred from the second on-chip memory to the second computing engine to perform the second convolution operation;

[0007] When the first convolution operation and the second convolution operation are completed, the second weight parameter is transferred from the second on-chip memory to the first computing engine to perform a third convolution operation, and the first weight parameter is transferred from the first on-chip memory to the second computing engine to perform a fourth convolution operation;

[0008] When the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolution layer is performed.

[0009] Furthermore, before transferring the first weight parameter from the first off-chip memory to the first on-chip memory and transferring the second weight parameter from the second off-chip memory to the second on-chip memory, the method further includes:

[0010] Obtaining weight parameters of each convolutional layer in the convolutional neural network;

[0011] Dividing the weight parameters of each convolutional layer in the convolutional neural network into a first weight parameter and a second weight parameter;

[0012] The first weight parameter is stored in a first off-chip memory, and the second weight parameter is stored in a second off-chip memory.

[0013] Furthermore, after transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the method further includes:

[0014] Determine whether the first on-chip memory has enough space to store first weight data of the next convolutional layer;

[0015] If the first on-chip memory has enough space to store the first weight data of the next convolutional layer, the first weight data of the next convolutional layer is moved from the first off-chip memory to the first on-chip memory in advance.

[0016] Furthermore, after transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the method further includes:

[0017] If the first on-chip memory does not have enough space to store the first weight data of the next convolutional layer, wait for the first on-chip memory to release enough storage space.

[0018] Furthermore, after transferring the second weight parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the method further includes:

[0019] Determining whether the second on-chip memory has enough space to store the second weight data of the next convolutional layer;

[0020] If yes, the second weight data of the next convolutional layer is moved from the second off-chip memory to the second on-chip memory in advance.

[0021] Furthermore, after transferring the second value parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the method further includes:

[0022] If the second on-chip memory does not have enough space to store the second weight data of the next convolutional layer, wait for the second on-chip memory to release enough storage space.

[0023] In a second aspect, an embodiment of the present invention provides a bandwidth optimization device for a convolutional neural network accelerator, the device comprising:

[0024] A first transfer module is used to transfer a first weight parameter of a convolutional layer to be calculated from a first off-chip memory to a first on-chip memory before the convolutional neural network runs, and to transfer a second weight parameter of the convolutional layer to be calculated from a second off-chip memory to a second on-chip memory, wherein the first weight parameter and the second weight parameter are two parts of weight parameters of the convolutional layer to be calculated in the convolutional neural network;

[0025] a second transport module, configured to transport the first weight parameter from the first on-chip memory to the first computing engine to perform a first convolution operation, and to transport the second weight parameter from the second on-chip memory to the second computing engine to perform a second convolution operation when performing the convolution calculation of the convolution layer to be calculated;

[0026] a third transport module, configured to transport the second weight parameter from the second on-chip memory to the first computing engine to perform a third convolution operation, and to transport the first weight parameter from the first on-chip memory to the second computing engine to perform a fourth convolution operation when the first convolution operation and the second convolution operation are completed;

[0027] A computing module is used to perform convolution calculation of a next convolution layer when the third convolution operation and the fourth convolution operation are completed.

[0028] In a third aspect, an embodiment of the present invention provides an AI acceleration chip, comprising the first off-chip memory, the second off-chip memory, the first on-chip memory, the second on-chip memory, an on-chip network, a first computing engine, and a second computing engine. The on-chip network connects the first off-chip memory, the second off-chip memory, the first on-chip memory, the second on-chip memory, the first computing engine, and the second computing engine. The on-chip network is used to execute the steps in the bandwidth optimization method for a convolutional neural network accelerator described in any one of the embodiments of the present invention.

[0029] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the bandwidth optimization method for a convolutional neural network accelerator described in any one of the embodiments of the present invention are implemented.

[0030] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the bandwidth optimization method for a convolutional neural network accelerator described in any one of the embodiments of the present invention are implemented.

[0031] In an embodiment of the present invention, before the convolutional neural network runs, the first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, and the second weight parameter of the convolutional layer to be calculated is moved from the second off-chip memory to the second on-chip memory, the first weight parameter and the second weight parameter are two parts of the weight parameters of the convolutional layer to be calculated in the convolutional neural network; when performing the convolution calculation of the convolutional layer to be calculated, the first weight parameter is moved from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is moved from the second on-chip memory to the second computing engine to perform the second convolution operation; when the first convolution operation and the second convolution operation are completed, the second weight parameter is moved from the second on-chip memory to the first computing engine to perform the third convolution operation, and the first weight parameter is moved from the first on-chip memory to the second computing engine to perform the fourth convolution operation; when the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolution layer is performed. Embodiment In an embodiment of the present invention, multiple computing engines can simultaneously accelerate one or more convolutional neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0033] Figure 1 It is a structural schematic diagram of a bandwidth optimization method for a convolutional neural network accelerator provided by an embodiment of the present invention;

[0034] Figure 2 It is a flow chart of a bandwidth optimization method for a convolutional neural network accelerator provided by an embodiment of the present invention;

[0035] Figure 3 A flowchart of another bandwidth optimization method for a convolutional neural network accelerator provided in this application;

[0036] Figure 4 It is a structural schematic diagram of a bandwidth optimization device for a convolutional neural network accelerator provided in an embodiment of the present invention.

[0037] in:

[0038] DDR: off-chip memory;

[0039] scratchpad memory: on-chip memory;

[0040] noc: network on chip;

[0041] AI engine: convolution computing engine. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but are not to be construed as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without creative work based on the embodiments of the present invention are within the scope of protection of the present invention.

[0043] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “circumferential”, “radial”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0044] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0045] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0047] See also Figure 1 , Figure 1 is a schematic diagram of a bandwidth optimization method for a convolutional neural network accelerator provided by the present application. Figure 1 As shown, it includes two DDRs, two scratchpad memories, one noc (network on chip), and two computing engines. DDR is an off-chip memory with large capacity and slow speed (small bandwidth); scratchpad memory is an on-chip memory with small capacity and fast speed (large bandwidth); AI engine is a convolution computing engine used to accelerate convolutional neural network algorithms. DDR0 is the first off-chip memory, DDR1 is the second off-chip memory, scratchpad memory0 is the first on-chip memory, scratchpad memory1 is the second on-chip memory, engine0 is the first engine, and engine2 is the second engine. noc is an on-chip network that connects DDR, scratchpad memory, and AI engine so that data can be transmitted between the three. The number of DDR, scratchpad memory, and AI engine in the accelerator can be expanded.

[0048] In the embodiment of the present invention, the above-mentioned DDR is a storage space that can be directly addressed by the CPU, and an advanced synchronization circuit is used to enable the main steps of specifying the address, data transmission and output to be executed independently and kept completely synchronized with the CPU. The above-mentioned scratchpad memory is composed of three components: SRAM storage component + address decoding component + data output circuit, and is connected using an on-chip high-speed bus and processor. The above-mentioned engine is a core component for developing programs or systems on an electronic platform. Using the engine, the functions required by the program can be quickly established and laid out, or the program can be operated using a device-assisted program. The above-mentioned noc (Network-on-Chip) is an electronic system based on network communication implemented on a single chip, and its form is an integrated circuit chip.

[0049] In an embodiment of the present invention, a convolutional neural network is accelerated, and by adding a scratchpad memory, the transmission of weight parameters can be reduced by half, thereby reducing bandwidth requirements and improving the performance of the convolutional neural network accelerator.

[0050] See also Figure 2 , Figure 2 A bandwidth optimization method for a convolutional neural network accelerator provided in this application includes:

[0051] 201. Before the convolutional neural network runs, a first weight parameter of the convolutional layer to be calculated is moved from a first off-chip memory to a first on-chip memory, and a second weight parameter of the convolutional layer to be calculated is moved from a second off-chip memory to a second on-chip memory.

[0052] In an embodiment of the present invention, the first weight parameter and the second weight parameter are two weight parameters of the convolutional layer to be calculated in the convolutional neural network. The first weight parameter is the first half weight parameter, and the second weight parameter is the second half weight parameter.

[0053] In an embodiment of the present invention, the above-mentioned convolutional neural network (CNN) includes a convolutional layer and a pooling layer, and the above-mentioned convolutional neural network includes at least two convolutional layers. Convolutional neural network is an efficient recognition method that has been developed in recent years and has attracted widespread attention.

[0054] In the embodiment of the present invention, the above-mentioned convolution layer is composed of a plurality of convolution units, the parameters of each convolution unit are obtained through back propagation algorithm, and the purpose of the convolution operation is to extract different features of the input.

[0055] In the embodiment of the present invention, the storage space of the above-mentioned off-chip memory that can be directly addressed by the CPU uses advanced synchronization circuits so that the main steps of specifying addresses, data transmission and output can be executed independently and maintained in complete synchronization with the CPU.

[0056] In the embodiments of the present invention, CPU is the central processing unit (Central Processing Unit / Processor), which is one of the main devices of electronic computers and the core component of computers. Its main function is to interpret computer instructions and process data in computer software. All operations in the computer are performed by the CPU, which is the core component responsible for reading instructions, decoding instructions and executing instructions.

[0057] In an embodiment of the present invention, the above-mentioned on-chip memory (scratchpad memory) is composed of three components: SRAM storage component + address decoding component + data output circuit, uses on-chip high-speed bus and processing connection, and has a unified non-overlapping address space with the main memory.

[0058] Furthermore, before transferring the first weight parameter from the first off-chip memory to the first on-chip memory, and before transferring the second weight parameter from the second off-chip memory to the second on-chip memory, the present invention also includes: obtaining the weight parameters of each convolutional layer in the convolutional neural network; dividing the weight parameters of each convolutional layer in the convolutional neural network into first weight parameters and second weight parameters; storing the first weight parameters in the first off-chip memory, and storing the second weight parameters in the second off-chip memory.

[0059] In an embodiment of the present invention, before the convolutional neural network is run, the weight parameters can be calculated in advance, and the weight parameters of each layer need to be divided into two parts, the first weight parameters are weight parameters for calculating the first half of the convolutional layer, and the second weight parameters are weight parameters for calculating the second half of the convolutional layer. The present invention stores the first weight parameters in advance in the first off-chip memory, and stores the second weight parameters in the second off-chip memory.

[0060] Furthermore, after the first weight parameter of the convolutional layer to be calculated is transferred from the first off-chip memory to the first on-chip memory, the present invention also includes: judging whether the first on-chip memory has sufficient space to store the first weight data of the next convolutional layer; if the first on-chip memory has sufficient space to store the first weight data of the next convolutional layer, the first weight data of the next convolutional layer is transferred from the first off-chip memory to the first on-chip memory in advance.

[0061] In an embodiment of the present invention, after the first weight parameter of the convolutional layer to be calculated is transferred from the first off-chip memory to the first on-chip memory, it is determined whether the first on-chip memory has sufficient space to store the first weight data of the next convolutional layer; if the first on-chip memory has sufficient space to store the first weight data of the next convolutional layer, the first weight data of the next convolutional layer is transferred from the first off-chip memory to the first on-chip memory in advance.

[0062] Furthermore, after transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the present invention also includes: if the first on-chip memory does not have enough space to store the first weight data of the next convolutional layer, waiting for the first on-chip memory to release enough storage space.

[0063] In an embodiment of the present invention, after the first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, if the first on-chip memory does not have enough space to store the first weight data of the next convolutional layer, the first on-chip memory is waited for to release enough storage space.

[0064] Furthermore, after transferring the second weight parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the present invention also includes: determining whether the second on-chip memory has sufficient space to store the second weight data of the next convolutional layer; if so, transferring the second weight data of the next convolutional layer from the second off-chip memory to the second on-chip memory in advance.

[0065] In an embodiment of the present invention, it is determined whether the second on-chip memory has sufficient space to store the second weight data of the next convolutional layer; if so, the second weight data of the next convolutional layer is moved from the second off-chip memory to the second on-chip memory in advance.

[0066] Furthermore, after transferring the second value parameter of the convolution layer to be calculated from the second off-chip memory to the second on-chip memory, the present invention also includes: if the second on-chip memory does not have enough space to store the second weight data of the next convolution layer, waiting for the second on-chip memory to release enough storage space.

[0067] In the embodiment of the present invention, after the second value parameter of the convolution layer to be calculated is transferred from the second off-chip memory to the second on-chip memory, if the second on-chip memory does not have enough space to store the second weight data of the next convolution layer, then wait for the second on-chip memory to release enough storage space.

[0068] 202. When performing a convolution calculation of the convolution layer to be calculated, a first weight parameter is transferred from a first on-chip memory to a first computing engine to perform a first convolution operation, and a second weight parameter is transferred from a second on-chip memory to a second computing engine to perform a second convolution operation.

[0069] In an embodiment of the present invention, the above-mentioned convolution layer is composed of a number of convolution units, and the parameters of each convolution unit are obtained through the back propagation algorithm; the purpose of the convolution operation is to extract different features of the input. The first convolution layer may be some low-level features of the intelligent body such as edges, lines, and corners. More layers of the network can iteratively extract more complex features from low-level features.

[0070] In the embodiment of the present invention, the above-mentioned back propagation algorithm is suitable for a learning algorithm of a multi-layer neural network, which is based on the gradient descent method.

[0071] In the embodiment of the present invention, when executing the calculation of the convolution layer to be calculated, the first weight parameter is transferred from the first on-chip memory to the first computing engine to perform the first convolution operation, and firstly, it is determined whether the first convolution operation is completed; if the first convolution operation is completed, the next step is continued; if the first convolution operation is not completed, the first weight parameter is again transferred from the first on-chip memory to the first computing engine to perform the first convolution operation, and the convolution operation of the next convolution layer is entered after the operation is completed.

[0072] In the embodiment of the present invention, when executing the convolution calculation of the convolution layer, the second weight parameter is transferred from the second on-chip memory to the second computing engine to perform the second convolution operation, and firstly, it is determined whether the second convolution operation is completed; if the second convolution operation is completed, the next step is continued; if the second convolution operation is not completed, the second weight parameter is again transferred from the second on-chip memory to the second computing engine to perform the second convolution operation, until the operation is completed and the convolution operation of the next convolution layer is entered.

[0073] In an embodiment of the present invention, for the same layer of convolutional network, when the first computing engine reads the upper half of the weight parameters for calculation, the second computing engine can read the lower half of the weight parameters for calculation, and each computing engine reduces the transmission of the weight parameters by half. For example, if the first computing engine originally wants to read the complete weight parameters once, and reads them from the same off-chip memory, the second engine needs to wait for the first engine to read before it can read. Assuming that the time to read the complete weight parameters is 1 second, the two engines need 2 seconds to get the same complete weight parameters. If the complete weight parameters are stored in both off-chip memories, the first engine can read from the first off-chip memory, and at the same time, the second engine can read from the second off-chip memory. The two engines only need 1 second to get the same complete weight parameters. In the present invention, the complete weight is split into two weight parameters, and the reading time of each weight parameter is 0.5 seconds. The first engine can get one weight parameter from the first off-chip memory, and the second engine can get another from the second off-chip memory.

[0074] In the embodiment of the present invention, while the current layer of convolutional neural network is being calculated, the weight parameters of the next layer of convolutional neural network in the off-chip memory are transferred to the on-chip memory in advance. When the next layer is calculated, the weight parameters are directly taken from the on-chip memory. The bandwidth of the on-chip memory is greater than the bandwidth of the off-chip memory, thereby increasing the bandwidth for weight parameter transmission, reducing the transmission time of weight parameters, and improving the performance of the convolutional neural network accelerator.

[0075] 203. When the first convolution operation and the second convolution operation are completed, the second weight parameter is transferred from the second on-chip memory to the first computing engine to perform a third convolution operation, and the first weight parameter is transferred from the first on-chip memory to the second computing engine to perform a fourth convolution operation.

[0076] In the embodiment of the present invention, when the first convolution operation and the second convolution operation are completed, the second weight parameter is transferred from the second on-chip memory to the first computing engine to perform the third convolution operation; first, it is determined whether the third convolution operation is completed; if the third convolution operation is completed, the next step is continued; if the third convolution operation is not completed, the second weight parameter is again transferred from the second on-chip memory to the first computing engine to perform the third convolution operation, until the operation is completed and the convolution operation of the next convolution layer is entered.

[0077] In the embodiment of the present invention, when the first convolution operation and the second convolution operation are completed, the first weight parameter is transferred from the first on-chip memory to the second computing engine to perform the fourth convolution operation; first, it is determined whether the fourth convolution operation is completed; if the fourth convolution operation is completed, the next step is continued; if the fourth convolution operation is not completed, the first weight parameter is again transferred from the first on-chip memory to the second computing engine to perform the fourth convolution operation, until the operation is completed and the convolution operation of the next convolution layer is entered.

[0078] In an embodiment of the present invention, for the same layer of convolutional network, when the first computing engine reads the upper half of the weight parameters for calculation, the second computing engine can read the lower half of the weight parameters for calculation. The transmission of the weight parameters for each computing engine is reduced by half. For example, if the first computing engine originally wants to read the complete weight parameters once, and reads them from the same off-chip memory, the second engine needs to wait for the first engine to read before it can read. Assuming that the time to read the complete weight parameters is 1 second, the two engines need 2 seconds to get the same complete weight parameters. If the complete weight parameters are stored in both off-chip memories, the first engine can read from the first off-chip memory, and at the same time, the second engine can read from the second off-chip memory. The two engines only need 1 second to get the same complete weight parameters. In the present invention, the complete weight is split into two weight parameters, and the reading time of each weight parameter is 0.5 seconds. The first engine can get one weight parameter from the first off-chip memory, and the second engine can get another from the second off-chip memory.

[0079] In the embodiment of the present invention, while the current layer of convolutional neural network is being calculated, the weight parameters of the next layer of convolutional neural network in the off-chip memory are transferred to the on-chip memory in advance. When the next layer is calculated, the weight parameters are directly taken from the on-chip memory. The bandwidth of the on-chip memory is greater than the bandwidth of the off-chip memory, thereby increasing the bandwidth for weight parameter transmission, reducing the transmission time of weight parameters, and improving the performance of the convolutional neural network accelerator.

[0080] 204. When the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolution layer is performed.

[0081] In the embodiment of the present invention, when the third convolution operation and the fourth convolution operation are completed, the calculation of the next convolution layer is performed.

[0082] In an embodiment of the present invention, the third convolution operation is a convolutional neural network operation that transfers the second weight parameter from the second on-chip memory to the first computing engine for execution; after completing the third convolution operation, the convolutional neural network operation of the next convolutional layer is entered. The fourth convolution operation is a convolutional neural network operation that transfers the first weight parameter from the first on-chip memory to the second computing engine for execution; after completing the fourth convolution operation, the convolutional neural network operation of the next convolutional layer is entered.

[0083] In the embodiment of the present invention, while the current layer of convolutional neural network is being calculated, the weight parameters of the next layer of convolutional neural network in the off-chip memory are transferred to the on-chip memory in advance. When the next layer is calculated, the weight parameters are directly taken from the on-chip memory. The bandwidth of the on-chip memory is greater than the bandwidth of the off-chip memory, thereby increasing the bandwidth for weight parameter transmission, reducing the transmission time of weight parameters, and improving the performance of the convolutional neural network accelerator.

[0084] In an embodiment of the present invention, before the convolutional neural network is run, the first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, and the second weight parameter of the convolutional layer to be calculated is moved from the second off-chip memory to the second on-chip memory, the first weight parameter and the second weight parameter are two weight parameters of the convolutional layer to be calculated in the convolutional neural network; when performing the convolution calculation of the convolutional layer to be calculated, the first weight parameter is moved from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is moved from the second on-chip memory to the second computing engine to perform the second convolution operation; when the first convolution operation and the second convolution operation are completed, the second weight parameter is moved from the second on-chip memory to the first computing engine to perform the third convolution operation, and the first weight parameter is moved from the first on-chip memory to the second computing engine to perform the fourth convolution operation; when the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolutional layer is performed. In an embodiment of the present invention, multiple computing engines can simultaneously accelerate one or more convolutional neural networks.

[0085] See also Figure 3 , Figure 3 A flowchart of another bandwidth optimization method for a convolutional neural network accelerator provided in this application, the method comprising:

[0086] Step S0, start;

[0087] Step S1: Before the convolutional neural network runs, the weight parameters can be calculated in advance. The weight parameters of each layer need to be divided into two parts, one half is stored in DDR0 and the other half is stored in DDR1. The accelerated neural network model here has only two convolutions, so L0_coef0 and L1_coef0 are stored in DDR0 in advance.

[0088] In the embodiment of the present invention, the DDR0 is the first off-chip memory, the DDR1 is the second off-chip memory; L0_coef0 represents the first weight parameter of layer0, and L0_coef1 represents the second weight parameter of layer0. The layer0 is the first convolution layer, and layer1 is the second convolution layer.

[0089] Step S2: Store L0_coef1 and L1_coef1 into DDR1 in advance.

[0090] In the embodiment of the present invention, the above L0_coef1 is the second weight parameter of the first convolutional layer, and the above L1_coef1 is the second weight parameter of the second convolutional layer.

[0091] Step S3: Move L0_coef0 from DDR0 to scratchpad memory0 (scpd0);

[0092] In the embodiment of the present invention, the scratchpad memory0 (scpd0) is the first on-chip memory.

[0093] Step S4: Move L0_coef1 from DDR1 to scratchpad memory1 (scpd1)

[0094] In the embodiment of the present invention, the scratchpad memory 1 (scpd1) is a second on-chip memory.

[0095] Step S5: Determine whether all L0_coef0 are transferred to scpd0. If yes, proceed to step S7 and step S10, otherwise proceed to step S3 and continue to transfer L0_coef0 data.

[0096] Step S6: Determine whether all L0_coef1 are transferred to scpd1. If yes, proceed to step S8 and step S11, otherwise proceed to step S4 and continue to transfer L0_coef1 data.

[0097] Step S7: Determine whether scpd0 has enough space to store L1_coef0. If yes, proceed to step S9, otherwise wait for scpd0 to release enough storage space.

[0098] Step S8: Determine whether scpd1 has enough space to store L1_coef1. If yes, proceed to step S12, otherwise wait for scpd1 to release enough storage space.

[0099] Step S9: Move L1_coef0 from DDR0 to scpd0

[0100] Step S10: engine0 takes out L0_coef0 from scpd0 and performs a convolution operation.

[0101] In the embodiment of the present invention, engine0 is the first engine.

[0102] Step S11: engine1 takes L0_coef1 from scpd1 and performs convolution operation;

[0103] In the embodiment of the present invention, engine1 is the second engine.

[0104] Step S12: Move L1_coef1 from DDR1 to scpd1

[0105] Step S13: Determine whether L1_coef0 has been transported. If so, check whether step S19 has been completed. If step S19 has been completed, proceed to step S21. Otherwise, wait for the transport to be completed.

[0106] Step S14: Determine whether engine0 has completed the calculation of L0_coef0 and engine1 has completed the calculation of L0_coef1. If satisfied, proceed to step S17, otherwise continue to wait for the calculation to be completed.

[0107] Step S15: Determine whether engine0 has completed the calculation of L0_coef0 and engine1 has completed the calculation of L0_coef1. If so, proceed to step S18, otherwise, continue to wait for the calculation to be completed.

[0108] Step S16: Determine whether L1_coef0 has completed the transfer. If so, check whether S20 has been completed. If step S20 has been completed, proceed to step S22. Otherwise, wait for the transfer to be completed.

[0109] Step S17: engine0 takes L0_coef1 from scpd1 and performs convolution operation

[0110] Step S18: engine1 takes L0_coef0 from scpd0 and performs convolution operation

[0111] Step S19: Determine whether engine0 has completed the L0_coef1 calculation, and whether engine1 has completed the L0_coef0 calculation. If satisfied, check whether S13 is satisfied. If S13 is also satisfied, proceed to step S21

[0112] Step S20: Determine whether engine0 has completed the L0_coef1 calculation, and whether engine1 has completed the L0_coef1 calculation. If satisfied, check whether S16 is satisfied. If S16 is satisfied, proceed to step S22

[0113] Step S21: engine0 starts the second layer of convolutional neural network operation

[0114] Step S22: engine1 starts the second layer of convolutional neural network operation

[0115] In the embodiment of the present invention, while the current layer of convolutional neural network is being calculated, the weight parameters of the next layer of convolutional neural network in the off-chip memory are transferred to the on-chip memory in advance. When the next layer is calculated, the weight parameters are directly taken from the on-chip memory. The bandwidth of the on-chip memory is greater than the bandwidth of the off-chip memory, thereby increasing the bandwidth for weight parameter transmission, reducing the transmission time of weight parameters, and improving the performance of the convolutional neural network accelerator.

[0116] In an embodiment of the present invention, before the convolutional neural network is run, the first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, and the second weight parameter of the convolutional layer to be calculated is moved from the second off-chip memory to the second on-chip memory, the first weight parameter and the second weight parameter are two weight parameters of the convolutional layer to be calculated in the convolutional neural network; when performing the convolution calculation of the convolutional layer to be calculated, the first weight parameter is moved from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is moved from the second on-chip memory to the second computing engine to perform the second convolution operation; when the first convolution operation and the second convolution operation are completed, the second weight parameter is moved from the second on-chip memory to the first computing engine to perform the third convolution operation, and the first weight parameter is moved from the first on-chip memory to the second computing engine to perform the fourth convolution operation; when the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolutional layer is performed. In an embodiment of the present invention, multiple computing engines can simultaneously accelerate one or more convolutional neural networks.

[0117] See also Figure 4 , Figure 4 A bandwidth optimization device for a convolutional neural network accelerator provided in this application includes:

[0118] The first transfer module 401 is used to transfer the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory before the convolutional neural network runs, and to transfer the second weight parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, wherein the first weight parameter and the second weight parameter are two parts of the weight parameters of the convolutional layer to be calculated in the convolutional neural network;

[0119] A second transfer module 402 is used to transfer the first weight parameter from the first on-chip memory to the first computing engine to perform a first convolution operation, and to transfer the second weight parameter from the second on-chip memory to the second computing engine to perform a second convolution operation when performing the convolution calculation of the convolution layer to be calculated;

[0120] A third transfer module 403 is used for transferring the second weight parameter from the second on-chip memory to the first computing engine to perform a third convolution operation, and transferring the first weight parameter from the first on-chip memory to the second computing engine to perform a fourth convolution operation when the first convolution operation and the second convolution operation are completed.

[0121] The calculation module 404 is used to perform convolution calculation of the next convolution layer after the third convolution operation and the fourth convolution operation are completed.

[0122] Further, before the first weight parameter is transferred from the first off-chip memory to the first on-chip memory, and the second weight parameter is transferred from the second off-chip memory to the second on-chip memory, the device further includes:

[0123] An acquisition module, used to obtain weight parameters of each convolutional layer in the convolutional neural network;

[0124] A division module, used for dividing the weight parameters of each convolutional layer in the convolutional neural network into a first weight parameter and a second weight parameter;

[0125] The storage module is used to store the first weight parameter in a first off-chip memory, and store the second weight parameter in a second off-chip memory.

[0126] Further, after transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the device further includes:

[0127] A first judgment module, used to judge whether the first on-chip memory has enough space to store the first weight data of the next convolutional layer;

[0128] The fourth transfer module is used to transfer the first weight data of the next convolutional layer from the first off-chip memory to the first on-chip memory in advance if the first on-chip memory has enough space to store the first weight data of the next convolutional layer.

[0129] Further, after transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the device further includes:

[0130] The first releasing module waits for the first on-chip memory to release sufficient storage space if the first on-chip memory does not have enough space to store the first weight data of the next convolutional layer.

[0131] Further, after transferring the second weight parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the device further includes:

[0132] A second judgment module, used to judge whether the second on-chip memory has enough space to store the second weight data of the next convolutional layer;

[0133] The fifth transfer module is used to transfer the second weight data of the next convolutional layer from the second off-chip memory to the second on-chip memory in advance, if any.

[0134] Further, after transferring the second value parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the device further includes:

[0135] The second releasing module is used to wait for the second on-chip memory to release sufficient storage space if the second on-chip memory does not have enough space to store the second weight data of the next convolutional layer.

[0136] In an embodiment of the present invention, before the convolutional neural network is run, the first weight parameter of the convolutional layer to be calculated is moved from the first off-chip memory to the first on-chip memory, and the second weight parameter of the convolutional layer to be calculated is moved from the second off-chip memory to the second on-chip memory, the first weight parameter and the second weight parameter are two weight parameters of the convolutional layer to be calculated in the convolutional neural network; when performing the convolution calculation of the convolutional layer to be calculated, the first weight parameter is moved from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is moved from the second on-chip memory to the second computing engine to perform the second convolution operation; when the first convolution operation and the second convolution operation are completed, the second weight parameter is moved from the second on-chip memory to the first computing engine to perform the third convolution operation, and the first weight parameter is moved from the first on-chip memory to the second computing engine to perform the fourth convolution operation; when the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolutional layer is performed. In an embodiment of the present invention, multiple computing engines can simultaneously accelerate one or more convolutional neural networks.

[0137] An embodiment of the present invention also provides an AI acceleration chip, comprising a first off-chip memory, a second off-chip memory, a first on-chip memory, a second on-chip memory, an on-chip network, a first computing engine, and a second computing engine. The on-chip network connects the first off-chip memory, the second off-chip memory, the first on-chip memory, the second on-chip memory, the first computing engine, and the second computing engine. The on-chip network is used to execute the steps in the bandwidth optimization method for a convolutional neural network accelerator described in any one of the embodiments of the present invention.

[0138] An embodiment of the present invention also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any bandwidth optimization method for a convolutional neural network accelerator as recorded in the above method embodiments.

[0139] An embodiment of the present invention also provides an electronic device, comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps of any one of the bandwidth optimization methods for a convolutional neural network accelerator as recorded in the above method embodiments.

[0140] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0141] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software program modules.

[0145] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0146] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0147] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A bandwidth optimization method for a convolutional neural network accelerator, characterized in that: The method comprises: Before the convolutional neural network is run, the complete weight parameter of each convolutional layer to be calculated is split into two parts, and the first weight parameter of the convolutional layer to be calculated is pre-transferred from the first off-chip memory to the first on-chip memory, and the second weight parameter of the convolutional layer to be calculated is pre-transferred from the second off-chip memory to the second on-chip memory, wherein the first weight parameter and the second weight parameter are two parts of the weight parameter of the convolutional layer to be calculated in the convolutional neural network, wherein the first weight parameter is the weight parameter of the first half of the convolutional layer to be calculated, and the second weight parameter is the weight parameter of the second half of the convolutional layer to be calculated; When performing the convolution calculation of the convolution layer to be calculated, the first weight parameter is transferred from the first on-chip memory to the first computing engine to perform the first convolution operation, and the second weight parameter is transferred from the second on-chip memory to the second computing engine to perform the second convolution operation; When the first convolution operation and the second convolution operation are completed, the second weight parameter is transferred from the second on-chip memory to the first computing engine to perform a third convolution operation, and the first weight parameter is transferred from the first on-chip memory to the second computing engine to perform a fourth convolution operation; When the third convolution operation and the fourth convolution operation are completed, the convolution calculation of the next convolution layer is performed.

2. The bandwidth optimization method for a convolutional neural network accelerator according to claim 1, characterized in that: Before the first weight parameter is transferred from the first off-chip memory to the first on-chip memory, and before the second weight parameter is transferred from the second off-chip memory to the second on-chip memory, the method further includes: Obtaining weight parameters of each convolutional layer in the convolutional neural network; Dividing the weight parameters of each convolutional layer in the convolutional neural network into a first weight parameter and a second weight parameter; The first weight parameter is stored in a first off-chip memory, and the second weight parameter is stored in a second off-chip memory.

3. The bandwidth optimization method for a convolutional neural network accelerator according to claim 1, characterized in that: After transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the method further includes: Determine whether the first on-chip memory has enough space to store first weight data of the next convolutional layer; If the first on-chip memory has enough space to store the first weight data of the next convolutional layer, the first weight data of the next convolutional layer is moved from the first off-chip memory to the first on-chip memory in advance.

4. The bandwidth optimization method for a convolutional neural network accelerator according to claim 3, characterized in that: After transferring the first weight parameter of the convolutional layer to be calculated from the first off-chip memory to the first on-chip memory, the method further includes: If the first on-chip memory does not have enough space to store the first weight data of the next convolutional layer, wait for the first on-chip memory to release enough storage space.

5. The bandwidth optimization method for a convolutional neural network accelerator according to claim 1, characterized in that: After transferring the second weight parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the method further includes: Determining whether the second on-chip memory has enough space to store the second weight data of the next convolutional layer; If yes, the second weight data of the next convolutional layer is moved from the second off-chip memory to the second on-chip memory in advance.

6. The bandwidth optimization method for a convolutional neural network accelerator according to claim 5, characterized in that: After transferring the second value parameter of the convolutional layer to be calculated from the second off-chip memory to the second on-chip memory, the method further includes: If the second on-chip memory does not have enough space to store the second weight data of the next convolutional layer, wait for the second on-chip memory to release enough storage space.

7. A bandwidth optimization device for a convolutional neural network accelerator, characterized in that: The device comprises: A first transport module is used to split the complete weight parameter of each convolution layer to be calculated into two parts before the convolution neural network runs, and to transport the first weight parameter of the convolution layer to be calculated from the first off-chip memory to the first on-chip memory in advance, and to transport the second weight parameter of the convolution layer to be calculated from the second off-chip memory to the second on-chip memory, wherein the first weight parameter and the second weight parameter are two parts of the weight parameter of the convolution layer to be calculated in the convolution neural network, the first weight parameter is the weight parameter of the first half of the convolution layer to be calculated, and the second weight parameter is the weight parameter of the second half of the convolution layer to be calculated; a second transport module, configured to transport the first weight parameter from the first on-chip memory to the first computing engine to perform a first convolution operation, and to transport the second weight parameter from the second on-chip memory to the second computing engine to perform a second convolution operation when performing the convolution calculation of the convolution layer to be calculated; a third transport module, configured to transport the second weight parameter from the second on-chip memory to the first computing engine to perform a third convolution operation, and to transport the first weight parameter from the first on-chip memory to the second computing engine to perform a fourth convolution operation when the first convolution operation and the second convolution operation are completed; A computing module is used to perform convolution calculation of a next convolution layer when the third convolution operation and the fourth convolution operation are completed.

8. An AI acceleration chip, characterized in that: The first off-chip memory, the second off-chip memory, the first on-chip memory, the second on-chip memory, the on-chip network, the first computing engine and the second computing engine, the on-chip network connects the first off-chip memory, the second off-chip memory, the first on-chip memory, the second on-chip memory, the first computing engine and the second computing engine, and the on-chip network is used to execute the steps in the bandwidth optimization method for a convolutional neural network accelerator as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the bandwidth optimization method for a convolutional neural network accelerator as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the bandwidth optimization method for a convolutional neural network accelerator as described in any one of claims 1 to 6.