Neural network batch normalization layer hardware implementation method, device, equipment and medium

Abstract

The present application relates to the field of neural network computing technology, and in particular to a method, device, equipment and medium for hardware implementation of a neural network batch normalization layer, wherein the method includes: storing the weight parameters of the neural network in the form of conductance in a memristor array; based on the convolution result of the previous convolution layer, obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array; sending the quantization result to the next convolution layer for convolution layer calculation. Thus, the ADC module commonly used in storage and computing tasks based on the memristor array is adopted to realize the BN layer calculation and the activation function module, saving the extra overhead of the processor for BN layer calculation and improving the system energy efficiency.

Description

Technical Field

[0001] The present application relates to the field of neural network computing technology, and in particular to a method, device, equipment and medium for hardware implementation of a neural network batch normalization layer. Background Art

[0002] Batch Normalization (BN) is a commonly used module in deep neural network training. It concentrates the output results with relatively discrete distribution within a certain range, avoids the gradient vanishing problem, and speeds up network training. Among the hardware implementation methods of the BN layer, the most common one is to compare with the improved threshold.

[0003] In related technologies, such as Figure 1 As shown in Figure 1, in a binary neural network, the output value of each layer is 1 or -1. N , Y N , Z N They are the results of the Nth convolutional layer, BN layer, and sign bit judgment module. After the output vector of the convolutional layer passes through the BN layer, its sign bit needs to be judged and sent to the next convolutional layer for calculation, as shown in the following formula:

[0004]

[0005] The improved threshold solution combines the BN layer calculation with the subsequent sign bit judgment module. The specific calculation formula is as follows:

[0006]

[0007] Assuming k>0, k<0 is the same. This method avoids the tedious calculation steps of the BN layer and greatly reduces the computational overhead.

[0008] However, the BN layer implementation scheme of the improved threshold is designed for binary neural networks, and the output value of each layer has only two values, and the calculation steps are relatively simple. However, this method is not applicable to the hardware implementation of neural networks with higher accuracy, and it needs to be solved urgently.

[0009] Application Contents

[0010] The present application provides a method, apparatus, device and medium for hardware implementation of a neural network batch normalization layer, which uses an ADC (Analog-to-digital converter) module commonly used in storage-computation tasks based on a memristor array to implement BN layer calculations and an activation function module, thereby saving the processor's additional overhead for BN layer calculations and improving system energy efficiency.

[0011] The first aspect of the present application provides a method for hardware implementation of a neural network batch normalization layer, comprising the following steps:

[0012] The weight parameters of the neural network are stored in the memristor array in the form of conductance;

[0013] Based on the convolution result of the previous convolution layer, a corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array; and

[0014] The quantization result is sent to the next convolution layer for convolution layer calculation.

[0015] Optionally, obtaining a corresponding quantization result according to an actual current flowing through each source line of the memristor array includes:

[0016] The preset range is quantized by 8 bits to obtain the quantization result.

[0017] Optionally, the preset range is:

[0018]

[0019] in, Z max is the upper limit of the calculation result of the batch normalization layer, β, γ, σ, μ are all parameters of the batch normalization layer, and ε is the minimum value.

[0020] Optionally, before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, the method further includes:

[0021] The capacitor in the integrator is charged and discharged according to a preset charging and discharging strategy to integrate the convolution result of the previous convolution layer.

[0022] Optionally, performing 8-bit quantization on the preset range to obtain the quantization result includes:

[0023] The output voltage of the integrator is sent to an 8-bit ADC, and the convolution result is quantized to multiple levels within a preset voltage.

[0024] The second aspect of the present application provides a hardware implementation device of a neural network batch normalization layer, including:

[0025] A storage module, used for storing the weight parameters of the neural network in the form of conductance in the memristor array;

[0026] An acquisition module, configured to obtain a corresponding quantization result based on the convolution result of the previous convolution layer and the actual current flowing through each source line of the memristor array; and

[0027] The quantization module is used to send the quantization result to the next convolution layer for convolution layer calculation.

[0028] Optionally, the acquisition module is specifically used to:

[0029] The preset range is quantized by 8 bits to obtain the quantization result.

[0030] Optionally, the preset range is:

[0031]

[0032] in, Z max is the upper limit of the batch normalization layer calculation result, β, γ, σ, μ are all parameters of the batch normalization layer, and ε is the minimum value. Optionally, before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, the acquisition module is further used to:

[0033] The capacitor in the integrator is charged and discharged according to a preset charging and discharging strategy to integrate the convolution result of the previous convolution layer.

[0034] Optionally, the acquisition module is specifically used to:

[0035] The output voltage of the integrator is sent to an 8-bit ADC, and the convolution result is quantized to multiple levels within a preset voltage.

[0036] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hardware implementation method of the neural network batch normalization layer as described in the above embodiment.

[0037] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the hardware implementation method of a neural network batch normalization layer as described in any one of claims 1 to 5.

[0038] Therefore, the weight parameters of the neural network can be stored in the memristor array in the form of conductance, and based on the convolution result of the previous convolution layer, the corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array, and the quantization result is sent to the next convolution layer for convolution layer calculation. Therefore, the ADC module commonly used in storage and computing tasks based on the memristor array is adopted to realize the BN layer calculation and the activation function module, saving the extra overhead of the processor for BN layer calculation and improving the system energy efficiency.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A schematic diagram of a convolutional layer with a BN layer is a flowchart of a method for hardware implementation of a neural network batch normalization layer according to an embodiment of the present application;

[0042] Figure 2 This is a comparison diagram of two ways to implement BN layer calculation and ReLU function;

[0043] Figure 3 This is a schematic diagram of the result after the Nth layer of convolution and the BN calculation result;

[0044] Figure 4 A flowchart of a method for hardware implementation of a neural network batch normalization layer provided according to an embodiment of the present application;

[0045] Figure 5 A schematic diagram of a method for implementing a convolutional layer, a BN layer, and a ReLU activation function according to an embodiment of the present application;

[0046] Figure 6 This is an example diagram of using the quantization module to implement the BN layer and the ReLU activation function calculation;

[0047] Figure 7 An exemplary diagram of a hardware implementation device for a neural network batch normalization layer according to an embodiment of the present application;

[0048] Figure 8 is an exemplary diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0050] The following describes the hardware implementation method, device, equipment and medium of the neural network batch normalization layer in the embodiments of the present application with reference to the accompanying drawings.

[0051] Before introducing the hardware implementation method of the neural network batch normalization layer in the embodiment of the present application, Figures 2 to 4 Explain that this application The range is quantized to 8 bits, skipping the original BN layer calculation and activation function steps.

[0052] like Figure 2 As shown, Figure 2 (a) is the flow chart of the computational process of the Nth layer of the neural network. Figure 2 (b) Flowchart of the improved scheme for implementing BN layer calculation and ReLU activation function.

[0053] Specifically, take the three modules of the Nth convolutional layer, BN layer, and ReLU activation function in the neural network as examples.

[0054] Y N =W N ·X N ;

[0055] Among them, Y N is the convolution result of the Nth layer, W N is the weight matrix obtained by expanding and concatenating n convolution kernels, X N is the input vector of the Nth layer; Y N Quantization is performed using an 8-bit ADC.

[0056] The BN layer calculation formula is as follows:

[0057]

[0058] The calculation step of the BN layer can be regarded as a linear transformation function: Z N = k·Y N +b, where The calculation results of the BN layer need to be sent to the ReLU activation function module.

[0059]

[0060] Assume that Z of the BN layer N The range is (Z min , Z max ) min <0), then X N+1 The range is (0, Z max ) range; after BN layer calculation, in (0, Z max ) in the range of Z N It needs to be quantized to 8 bits and sent to the ReLU activation function module, such as Z N =0, the quantization result corresponds to 0, Z max Corresponds to 255.

[0061] According to the above Z N and Y N The conversion relationship between them is known. Depend on Figure 3 It can be seen that Figure 3(a) is a schematic diagram showing the quantization of the convolutional layer output to 8 bits after the improvement (i.e. the result after the Nth convolution layer). Figure 3 (b) is a schematic diagram of the need to quantize the output value of the BN layer to 8 bits in the unimproved network (i.e., the calculation result of the Nth BN layer). Therefore, if we want to obtain Figure 3 (b) Same quantization result, Y N The quantization range needs to be corrected. Here, we assume that k is greater than 0, (0, Z max ) range corresponds to

[0062] Therefore, the embodiments of the present application need to The range is quantized to 8 bits, the original BN layer calculation and activation function steps are skipped, and the hardware implementation of BN layer calculation and ReLU function is realized through the improved output result quantization scheme.

[0063] Specifically, Figure 4 A flowchart of a method for hardware implementation of a neural network batch normalization layer provided in an embodiment of the present application.

[0064] like Figure 4 As shown, the hardware implementation method of the neural network batch normalization layer includes the following steps:

[0065] In step S401, the weight parameters of the neural network are stored in the memristor array in the form of conductance.

[0066] In step S402, based on the convolution result of the previous convolution layer, a corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array.

[0067] Optionally, in some embodiments, obtaining a corresponding quantization result according to an actual current flowing through each source line of the memristor array includes: performing 8-bit quantization on a preset range to obtain a quantization result.

[0068] Optionally, in some embodiments, the preset range is:

[0069]

[0070] Where b is k is Z max is the upper limit of the batch normalization layer calculation results. β, γ, σ, μ are all parameters of the batch normalization layer, where β and γ are trainable parameters that gradually converge during the training process, and σ and μ represent the standard deviation and mean of the output values, respectively, and are related to the training samples. ε is a negligible minimum value introduced to prevent the denominator from being zero.

[0071] In some embodiments, 8-bit quantization is performed on a preset range to obtain a quantization result, including: sending the output voltage of the integrator to an 8-bit ADC, and quantizing the convolution result to multiple levels within the preset voltage.

[0072] Optionally, in some embodiments, before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, it also includes: charging and discharging the capacitor in the integrator according to a preset charging and discharging strategy to integrate the convolution result of the previous convolution layer.

[0073] Specifically, the embodiment of the present application can store the weight parameters of the neural network in the form of conductance in the 2T2R memristor array, and the current value flowing through each source line needs to be converted into a voltage value and quantized. Figure 5 Taking the output quantization module (including integrator and 8-bit ADC) as an example, the current-to-voltage module can convert the source line current into voltage through a transimpedance amplifier or an integrator. The reference voltage of the integrator is 2.5V. Integrate the convolution layer result, that is, the capacitor C in the integrator integ Charge and discharge. Assume When the voltage on the capacitor drops to 2V, The output voltage of the integrator is then sent to the 8-bit ADC, which quantizes the result to 256 levels within 2-5V. Figure 6 For the implementation method shown, the subsequent BN layer and ReLU activation function calculation steps are no longer required, which saves the processor's additional overhead for BN layer calculation and improves system energy efficiency.

[0074] It should be noted that for each convolution kernel in the convolution layer, the parameters k1, b1, k2, b2, ..., kn, bn of the BN layer corresponding to the output neuron are different, so the details of each integral capacitor need to be carefully designed to ensure that the subsequent voltage range is between 2V and 5V to achieve the reuse of 8-bit ADC.

[0075] In step S403, the quantization result is sent to the next convolution layer for convolution layer calculation.

[0076] Therefore, combined with the BN layer implementation of ADC, the quantization result can be directly sent to the next convolution layer for calculation without going through the BN layer calculation and ReLU activation function steps.

[0077] According to the hardware implementation method of the neural network batch normalization layer proposed in the embodiment of the present application, the weight parameters of the neural network can be stored in the memristor array in the form of conductance, and based on the convolution result of the previous convolution layer, the corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array, and the quantization result is sent to the next convolution layer for convolution layer calculation. Thus, the ADC module commonly used in the storage and calculation tasks based on the memristor array is adopted to realize the BN layer calculation and the activation function module, saving the extra overhead of the processor for BN layer calculation and improving the system energy efficiency.

[0078] Next, a hardware implementation device for a neural network batch normalization layer proposed in an embodiment of the present application will be described with reference to the accompanying drawings.

[0079] Figure 7 It is a block diagram of a hardware implementation device of a neural network batch normalization layer according to an embodiment of the present application.

[0080] like Figure 7 As shown, the neural network batch normalization layer hardware implementation device 10 includes: a storage module 100, an acquisition module 200 and a quantization module 300.

[0081] Wherein, the storage module 100 is used to store the weight parameters of the neural network in the form of conductance in the memristor array;

[0082] The acquisition module 200 is used to obtain a corresponding quantization result based on the convolution result of the previous convolution layer and the actual current flowing through each source line of the memristor array; and

[0083] The quantization module 300 is used to send the quantization result to the next convolution layer for convolution layer calculation.

[0084] Optionally, the acquisition module 200 is specifically used for:

[0085] Perform 8-bit quantization on the preset range to obtain the quantization result.

[0086] Optionally, the preset range is:

[0087]

[0088] in, Z max is the upper limit of the batch normalization layer calculation result, β, γ, σ, μ are all parameters of the batch normalization layer, and ε is the minimum value. Optionally, before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, the acquisition module 200 is further used to:

[0089] The capacitor in the integrator is charged and discharged according to the preset charge and discharge strategy to integrate the convolution result of the previous convolution layer.

[0090] Optionally, the acquisition module 200 is specifically used for:

[0091] The output voltage of the integrator is sent to the 8-bit ADC, and the convolution result is quantized to multiple levels within the preset voltage.

[0092] It should be noted that the aforementioned explanation of the embodiment of the method for hardware implementation of the neural network batch normalization layer is also applicable to the hardware implementation device of the neural network batch normalization layer of this embodiment, and will not be repeated here.

[0093] According to the hardware implementation device of the neural network batch normalization layer proposed in the embodiment of the present application, the weight parameters of the neural network can be stored in the memristor array in the form of conductance, and based on the convolution result of the previous convolution layer, the corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array, and the quantization result is sent to the next convolution layer for convolution layer calculation. Thus, the ADC module commonly used in the storage and calculation tasks based on the memristor array is adopted to realize the BN layer calculation and the activation function module, saving the extra overhead of the processor for BN layer calculation and improving the system energy efficiency.

[0094] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0095] A memory 801 , a processor 802 , and a computer program stored in the memory 801 and executable on the processor 802 .

[0096] When the processor 802 executes the program, the hardware implementation method of the neural network batch normalization layer provided in the above embodiment is implemented.

[0097] Furthermore, the electronic device further comprises:

[0098] The communication interface 803 is used for communication between the memory 801 and the processor 802 .

[0099] The memory 801 is used to store computer programs that can be executed on the processor 802 .

[0100] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0101] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the communication interface 803, the memory 801 and the processor 802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0102] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0103] The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0104] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned hardware implementation method for a neural network batch normalization layer.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0106] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0107] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0109] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0110] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for hardware implementation of a neural network batch normalization layer, characterized in that: The following steps are involved: The weight parameters of the neural network are stored in the memristor array in the form of conductance; Based on the convolution result of the previous convolution layer, a corresponding quantization result is obtained according to the actual current flowing through each source line of the memristor array; as well as Sending the quantization result to the next convolution layer for convolution layer calculation; The step of obtaining a corresponding quantization result according to the actual current flowing through each source line of the memristor array includes: performing 8-bit quantization on a preset range to obtain the quantization result, The preset range is: in, Z max is the upper limit of the calculation result of the batch normalization layer, β, γ, σ, μ are all parameters of the batch normalization layer, and ε is the minimum value.

2. The method according to claim 1, characterized in that Before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, the method further includes: The capacitor in the integrator is charged and discharged according to a preset charge and discharge strategy to integrate the convolution result of the previous convolution layer.

3. The method according to claim 2, characterized in that The step of performing 8-bit quantization on the preset range to obtain the quantization result includes: The output voltage of the integrator is sent to an 8-bit ADC, and the convolution result is quantized to multiple levels within a preset voltage.

4. A hardware implementation device for a neural network batch normalization layer, characterized in that: include: A storage module, used for storing the weight parameters of the neural network in the form of conductance in the memristor array; An acquisition module, configured to obtain a corresponding quantization result based on a convolution result of a previous convolution layer and an actual current flowing through each source line of the memristor array; as well as A quantization module, used for sending the quantization result to the next convolution layer for convolution layer calculation; The acquisition module is specifically used to: perform 8-bit quantization on the preset range to obtain the quantization result, The preset range is: Where b is k is Z max is the upper limit of the calculation result of the batch normalization layer, β, γ, σ, μ are all parameters of the batch normalization layer, and ε is the minimum value.

5. The device according to claim 4, characterized in that Before obtaining the corresponding quantization result according to the actual current flowing through each source line of the memristor array, the acquisition module is further used to: The capacitor in the integrator is charged and discharged according to a preset charge and discharge strategy to integrate the convolution result of the previous convolution layer.

6. The device according to claim 5, characterized in that The acquisition module is specifically used for: The output voltage of the integrator is sent to an 8-bit ADC, and the convolution result is quantized to multiple levels within a preset voltage.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hardware implementation method of a neural network batch normalization layer as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the hardware implementation method of a neural network batch normalization layer as described in any one of claims 1 to 3.

Images